The only weird part is they took a 90 degree turn from their walk and went to your driveway. As for where the toddler started to go, that seems to me like the parents letting the kid go and explore a bit. The reason they are not talking is because they are keeping an eye on him. There was no direction from the parent to tell him where to go and from the body language they are not checking the neighbours to see if people are looking at them or scouting the surroundings. Their attention is on the kid. Soon as the kid bent down he rushed over and stopped him.
I don’t see any malice here. The only concern would be your property, but some people sees open space right next to the main road to be less private to the back garden. If you take this to the police they will tell you to go away, it’s a civil matter.
I wouldn’t lose sleep over it.