Practically every cycle path and road side in the entire UK is fine for cycling, even going over drain covers is absolutely fine - if you ride a mountain bike or something with some decent width tyres.
The problem is you've got guys on bikes that are the equivalent of delicate sports cars not wanting to go over the slightest bump, so they take the prime part of the road, at least two feet from the kerb, causing even more dangerous overtakes by cars.
I was on a cycling holiday recently and saw numerous racer bike riders ignoring massive width cycle paths to go on the road instead as the cycle path had a slight bit of gravel on it. This was on thin country roads with low visibility and plenty of hedge row turns. Maybe doing 22-25mph, but still holding up traffic and clearly infuriating drivers putting themselves and everyone else on the road at risk.
Unfortunately the guy who kicked your wing mirror is one of those selfish types of rider. I'm on a mountain bike every day riding London roads, never wear lycra, never doing anything unsafe, try show as much respect to motorists and I ride close to the kerb or a cycle path whenever possible.
I'm sorry you encountered this kind of rider that gives us all a bad name, believe me I see bad cycling every day, usually on guys in lycra riding expensive and delicate sports bikes on roads entirely not suited to them.