You really need to know how loud it is for them when you think it's a reasonable volume for you ... otherwise there is now way of knowing how justified their complaints are.
Yes if you live in a flat, or a semi/terraced house, then you should be mindful of those around you but I don't think you should expect to hear zero noise from the people around you and they shouldn't expect that from you. I live in a semi and at the moment I can just hear a TV next door and the foot steps of the kids running around ... do I have an issue with that ... not really.
My previous neighbour here was a nutcase ... I am a very quiet person, (I go to work very early so am normally in bed asleep by 10pm and have probably been watching TV collapsed on it using headphones for quite some time before then. Even during the weekend I don't tend to have music on much). It got to the stage where if she heard
any noise at all she would start shouting and banging on the walls ... and I mean any noise (in the summer, windows open, I even had a complaint about opening and closing a DVD players drawer!). I saw her charging around one Saturday morning ready to have a right go ... only when she got 3/4 of the way she realised the noise was the neighbour on the other side of me mowing his lawn.
In the end I flipped the master power off on the fuse box, went around and asked about the current noise level. She said it was still to loud ... so I wrote to her explaining what I had done (and the various other issues up to that point) and if she continued as she was then I would be reporting her for harassment. She left the following month (the night before she moved out she put radios on full blast in every room all night and went and stayed at her Mum's ... I could barely hear them and slept soundly

). Then the current lot moved in and no problems since.