Erm, I think the OP has stated that the cat is neutered.
OP, just make sure you attempt to correct any bad behaviour. And this doesnt mean shouting and projecting crazy behaviour. Let the cat know when it is misbehaving and eventually it will learn. Also, if you project calmness when your guests are over, the cat should be re-assured by your calm behaviour. And dont be scared of holding the cat by the skin behind its neck, while it calms down, when it is seriously misbehaving.
The hissing and growling will eventually subside when the person/animal it is hissing at, is calmly in the cat's presence, over a period of time. Eventually, the cat will run out of steam and stop growling/hissing.
The key is socialising.