I don't think you understand the concept of an invasion of privacy, it's nothing to do with the law but how the person who's privacy you're invading perceives it.
e.g. If I'm in the house and see some neckbeard pointing a camera at me, I regard that as an invasion of my privacy.
But that is where your perception is wrong with the majority of society, which is why the law is they way it is. When people photograph the house and not specificity you within it then the house is the subject of the photo and the house has no concept of privacy. Whether you are in the house at the time or not shouldn't matter. It is just irrational paranoia to think something untoward is happening and it is simply not healthy to think that way.