Same happened to mine and I couldn't fix due to the plastic bit that held it in place had snapped.
If you look at the memory card covers I think there held on in the same way so you can then see how the USB one is held in place. Hopefully yours isn't broken and you can just get the spring back in place.
As luck would have it I have a spare one lying on my desk, so took a piccy for you.
If you call MS they'll send you a new one btw, or you can have this one if you want. Been sitting on my desk for months (MS sent it to me by mistake).