You are assuming it'll get physical so you bolted, well that just re-enforced to the householder he had something to be concerned about and makes the life of the next photographer even harder. Well done.
Grow a pair, talk to the guy as civil as possible and explain in clam and rational manner what you're doing and engage him. If he still doesn't like it while not 'right' delete the image and move on with your life.
Ideally a civil conversation would be best but depending on how the other guy reacts initially, I can definitely understand why he'd walk away before anything stats.
I'm not a small guy but I wouldn't be keen to try to start a conversation after some guy shouted at me out of a window. Some people can react unreasonably to this kind of thing and having a pair of balls won't help you if this guy decides to come out and want a fight... Much less if he's definitely some kind of nutter who comes out with a knife or something.
Personally I've never taken pictures of people's houses for this precise reason. Sadly, this is the world we live in and whilst we may have the right to take the photos, we also have the right to deal with the crazies that go along with it :/
It's not something that's likely to happen but I weigh that up with the fact that there isn't really a house out there that I want a picture of that badly! If there was a house like that then I guess I'd get permission first.