The kid made a beeline straight to the front door. He didn’t spend anytime ‘playing’ on the gravel other than the time it took him to walk over it.
If you look where they put him down, you can see they were expecting him to just have a little walk on the gravel near them, they weren't expecting him to go over to the door. Kids sometimes don't do what you expect you know!
Anyway, I won't spend too much time analysing this. The key thing about this is they crossed the street and made a specific decision to let their kid play on someone's drive way, that is the slightly disrespectful/impolite part, and I would not do it. If they had just been passing on the pavement next to the house, it wouldn't have been odd. It's the fact they decided to take the kid over and play. As the driveway is right next to the pavement, you can see why they would have thought this would not cause offence, they probably weren't expecting their kid to go over to the door.
Personally I think the OP has nothing to be concerned about and I hope he can now sleep safely at night!! If he is that precious about his driveway, then he needs to get some gates.