I have been using lipos in feathers for over 3 years now. A similar rated pack to what turbulence used to run in Nicads.
equivilent of 24v @ 5000mA and never had a problem. They are perfectly safe as long as your not stupid with them.
As regards to charging them, do it properly and you will never have a problem. Its not actually very likely a lipo will catch fire either. They need to be almost fully charged and punctured to have a good chance of catching fire (i know, i've tried it myself ).

If they go up in the arena, well who cares? The fumes are less toxic than nicad fires, and the robot needs to have a metal base to run lipos anyway. So it cant harm anyone.

They need to be charged in a metal box of some form aswell (personally prefer in the robot myself)
so it shouldnt matter if they go up when out of the arena either due to bad charging.