Does he live alone? If so you know what to do! Joking aside probably report it to the police, just so they have record of it, with the CCTV if he says he didn't speak to you then he's already talked himself into a hole.
Could be rental, or maybe he has no intention of selling and has lived their for years, you're right though that is something to think about otherwise.Difficult and horrible situation. Normally I'd agree with everyone saying to report him, but I believe it was someone here who raised the point about any official neighbourly fuedes etc popping up on future solicitor searches and potentially effecting a future sale.
Would you want to move into a house where the current occupier has filed a police report on a neighbour?
Obviously if he actually takes a swing then get straight down to the station, but I'd probably take a minute to just laugh at how pathetic he is, but otherwise ignore him/forget about him and not let him effect my life.