When you drilled the holes, did you use a centrepunch? That should make an indent to stop your drillbit walking off.