I think you're supposed to be able to pull on the wire. If it's coming undone, either find a better connector (like the red ones which require you to push the contact into the connector end) or check your solder connections. I suppose you could also try reinforcing it with some tape, but I'd only do that as a last resort.