Noise can sometimes be an issue, sure, but I wouldn't say it's any worse than when I've lived in terrace/semi detached houses in the past. It's all in the quality of the construction ultimately.
And the neighbours. You can have the kindest, quietest people living next door for years. They move out or decide to rent the property out and you get inconsiderate neighbours playing stupidly loud music like they think they're the next Glastonbury!