Don't take this the wrong way, but I haven't seen it already mentioned. But make sure they're not suffering from fleas or worms. The frustration and annoyance of fleas can cause a cat to go right out of character.
I completely support the neutering route, until that's done you don't know the real character of your cat.
But learn to read your cat, my cat's a bit bitey, which has been slowly becoming less and less as time goes by. But what's telling is that she's so much less bitey with me, but then I can read when she needs to be backed off from. I watch my son and wife and I always seem to have to tell them when they should stop whatever it is they're doing with her as I can see her demeanour change. It's not as simple as ears back and tail flicking = leave alone, it's how they react to everyday contact.
Sounds like you've got a lot of love for him and I really hope he learns to cope with whatever has made him like this. You've got a bit of a problem cat by the sounds of it, who's either going to have the loving owners with the patience to nurture social behaviour, or owners without the patience who end up inadvertantly increasing the bad behaviour with their reactions to it. You sound like the former to me, best of luck to you.
.
