Turns out the neighbours were actually "giving in" to our cat and feeding him to shut him up so he would leave and throwing treats so he would move away from the door/out from under car when they were leaving. Well, I'm not sure what they thought rewarding that sort of behaviour would turn out for them and our Roary...
Either way we have some dialogue going now and are fully understanding each others situation. They have blocked up their dog flap (it's no longer summer, so hopefully we can get this sorted over the colder months), they have said they won't feed them any more, let them in or reward them with any affection at all - this I hope will deter our guys from going around there terrorising them. We are now keeping them inside which isn't going down too well with the 3 year old, and letting them out only while we are home so that we can then keep them in when they return later in the day - they refuse to use the cat flap, so work in progress here. Also requested the neighbours let us know if they keep bothering them or bring them over if they get into theirs too often.
It's a pain, but a relief to know they aren't actually being "stolen" more like inadvertently befriended through ignorance and lack of communication on both parties.
Here's another picture of Roary in his favourite dirty spot in the shed...