You don't need to be advocating mistreatment of disabled people to lack understanding or positivity.
This.
I've seen people wonder why the disabled complain about certain building features, or fully able bodied people complaining about why things like new builds have certain features that look odd (mainly because they're used to the "traditional" way which often became common because it was the fast/easy way at the time to do something and not necessarily the best way even then).
About the only time I've seen disabled people complain about "ablism" is when they've been commenting on designs that take no account of the statistically fairly large number of people who will have trouble using them, and the replies they've received from able bodied people who can't understand why it's an issue, as many many design teams have no experience with disabilities and it's not common to say have products tested to make sure they're reasonably user friendly for a wide range of people.
It's similar to the way you get complaints about UI's when a group of programmers make something and release it without thinking about how people who haven't been in on the thinking behind it and the design process will deal with it on their first encounter, or things like hand dryers that don't work reliably if your skin is anything other than white because the design team used a sensor that wasn't good enough for a wide range of lighting and skin colours, or you get a new building where it technically meets the legal requirements for disabled access but no one has thought about things like the room needed to easily turn a self propelled wheelchair (let alone a pushed one or a motorised one), or how many car parking spaces you see that are disabled but only have room to open the car door wide on one side (thus meaning that if you need to open the door wide the number of options is halved).
In an ideal world every product testing team should have a mix of people from widely different backgrounds, as so many otherwise good products fail due to things like them being designed with a narrow or outdated data set and not enough testing by the sorts of people that are expected to use it so what become blindingly obvious problems in the real world are completely missed.
Going back a few years there was a thread on here (I think it was) where people were complaining about how new builds had to have "a load of silly features" and why couldn't they just put them in for the houses the old/disabled were going to use, or fit them afterwards, where a number of users couldn't grasp the idea that it's far more expensive and time consuming to retrofit basic disabled friendly features than to put them in from the word go*.
I try to be aware if accessibility/ease of use for the disabled because my mother was on crutches and a wheelchair/motorised scooter for decades and we had to learn the hard way what worked (things as simple as a gate latch that could be opened from either side allowing the gate to be pushed open was a struggle to find), and made adaptions to the house that surprised the experts when they did assessments, as we'd already raised sockets so they were easy to reach, changed the light switches to larger surface areas, put in ramps front and back, easy use "lever" taps etc, My father is now going blind and we're starting to see (if you'll pardon the expression) a whole new load of issues we need to sort out, some of which we've already worked on (improved lighting) others we're going to have to deal with.
Basically you as you say, you don't have to be actively advocating against disabled people to have a very narrow view that doesn't consider them, or their needs.
*Not to mention many of the changes are actually useful for able bodied people, including parents with small children (things like low/no steps, slightly wider doors), or when moving white goods in and out.
