I don't see why not?
The driver's door is on that side, so the OP would have been getting in and out of that side regardless - the guy could be claiming he hit his car with the driver's door - the OP doesn't specify
Regardless, it stinks of a scam - the guy was probably hoping that instead of giving your insurance details you'd just give him the £50 - it's a low enough value that most people would rather pay it than lose their NCB and suffer increased premiums for 3-5 years.
Chances are nothing will come of it; if he does try to go through insurance then he's an idiot, because a) he'll have to declare it and the increase in his premiums will likely be more than the £50, b) there's no proof you did it, so you could just deny it and if asked why you gave your details then tell them he was intimidating you and your daughter and you were trying to defuse the situation.