Long story short, you SHOULD wear them and I would never advise someone not to wear them. But it looks like we have a bunch of Coral Whisperers on this site. Myself included though, I NEVER wear them and have been stung several times. I also got 'swimmers itch' I think or some other rash in the past. There are horror stories all over the net about dangers of it, and some stories on this site about members/family members, including ricklalonde who have suffered ailments from toxins.
There are a ton of risks though, bristle worm and other pokes, too many to list. There are poisons, Palytoxin and others. Symptoms include a wound that heals poorly or continues to drain pus or cloudy fluid, swelling around the cut, swollen lymph glands, fever, chills and fatigue. IF they persist and untreated, they could be fatal. If these symptoms are present, you should see a physician, who may treat with an antibiotic or corticosteroid medication.
There is also 'swimmers itch' which are the immature larval forms of flatworms. The flatworms penetrate the outer layer of the skin, where itching is noted within minutes. Shortly afterwards, the skin becomes reddened and swollen, with an intense rash and, occasionally, hives. Blisters may develop over the next 24 to 48 hours. Untreated, the affliction is limited to 1 to 2 weeks.
