I can't condone the bikers actions, but I can't say I've never done it myself.
Last time I did it was when a car forced me in to the gravel area of the central reservation and then gave me the finger when I sounded my horn at him (after regaining control of my now flailing bike). In a fit of rage, I punched his mirror downwards to remove it from his car and returned the finger gesture.
I would imagine he punched your mirror rather than kicked it off. I have a sports bike and I can tell you. It would be almost impossible to kick a mirror off at the angle one has to sit on a bike. Well, unless you drive something really low like an Elise or a Westfield or something.