No, tbh, it would annoy me too. I was just checking thaat you hadn't just moved into an area where street play was already established.
As another poster said, contact the council and get them to put "no ball games" signs up.
If they are climbing into your garden, that is trespass. Speak to the council again to see if they have any "community patrols" who may be able to have a word.
I any case, try your best not to cause a row between you and your neighbours - things like this can blow up very easily and make living in your home hell.