These people tend to crawl back into their hole after a while. Might have been having a bad day. If he mentions it again, do this:
Be neighbourly and invite him round to give you a hand