Correct. Of course if you were questioned as to why you were doing it and you had no real reason, and/or the subject wanted to prosecute for harassment or restraining etc, then that would be valid but it's irrelevant to the photography element. You could equally just be watching them or using binoculars etc.
That's an extreme example remember. A shopping centre is private property, so technically a security guard could prevent you from taking photos inside but they can't stop you from taking photos of it or into it.
Most of the times this is done it's completely innocent and you wouldn't think twice. Press photographers probably take photos every day of people on private property; leaving courts, celeb weddings, etc etc.
That's why the Kate Middleton topless photo thing was interesting, they were in France and on private property so it was illegal aside from any moral/harassment etc issues.