I had a big argument and lost my temper with a lady pushing someone in a wheelchair over this. I was at a motorway cafe with my daughter who was around 5yrs old. I didn't want to take her into the mens with me because she was getting older and I didn't feel she was old enough to go into the ladies on her own. So I took her into the disabled/accessible toilet.
When we came out I got a barrage of abuse from a lady waiting outside with an elderly lady in a wheelchair. I wouldn't have minded her politely questioning it but the attitude from her and her sense of entitlement was terrible. She was very rude and very aggressive. It's one of the few times I've lost my temper with anyone and shouted them down. I simply wasn't going to take that kind of abuse from anyone. The wheelchair bound lady in her care seemed quite ebarrassed by it. My daughter had just as much right as anyone to use that toilet.
I've never actually had any issues with people moaning about it.
However, she was completely on the wrong there, they aren't to be called disabled, or restricted to disabled people only because it's discrimination.