I have been asked this question many times over the years, and the best answer I can give is that, it is only as safe as you make it.
Having flown ultralights for around 20 years, and hearing of the many ultralight accidents over that time, I can only deduce the following:
If you follow these 5 points, you should have many good flying years ahead of you.
1. Do a serious walk-around and visual check of your aircraft, checking for loose or leaking components (your flight manual should have a check list).
2. Make sure you have the right amount of fuel (with reserve) for your flight, plus clean fuel filters.
3. Do a proper warm up and runup of the engine to the engine manual specs.
4. Check your weather from multiple sources, and don't rely on "oh, the weather is good now, maybe it will stay that way".
5. And this is the most important of the lot. Put your common sense hat on. If things don't seem right, within yourself, the weather, your aircraft, or your inner sixth sense, then DON'T FLY. There will be other days or other times to follow your passion flying ultralight aircraft. Too many have come undone by travelling an hour or more to the airfield, then convincing themselves that they have come all this way and deserve to get a fly in.
On the subject of safety, GA pilots fly more complicated aircraft, with many more "things to keep an eye on", and some (not all) feel that if they can fly a GA they can fly anything. Unfortunately some GA pilots have come to grief by trying to fly an ultralight like a GA (the main issue being trying to land a draggy ultralight like a Cessna). Thankfully, RA-Aus have introduced conversion training so that the GA fraternity can enjoy what us ultralighters have been enjoying for many years. For GA conversion training CLICK HERE for an example.
Bert Moonen - Quicksilver Aircraft Australia