Sounds like a douche. I sold my car last year when I became too ill to work. Getting better now and hope to get back into work soon. However, during my time without a car, I've noticed both my neighbours using my private parking bay. I don't have a problem with this, even though it would have been nice if they asked first. Anyway, OP, best thing to do is confront him and tell him what you have told us. At least that way you'll be getting this off your chest.
Your neighbours are making you beta by parking in your private space. Set boundaries and show them how Alpha you are.