If that was a two story house, you had 1 wc on the ground and 4 ensuites upstairs along with a main bathroom, that is one thing, its overkill because one ensuite and one main bathroom would do(though some would have a second ensuite for a guest room). But once you start talking about different floors. You don't want to do several flights of stairs just to take a pee, nor go downstairs to take a pee or have a morning shower and find someone is in the bathroom, so you go back upstairs to wait for them to finish and then do the stairs again, and again and again.
The only way I'd rip out ensuites would be removing them all and putting one bathroom on each floor, but depending on how the house is setup, you'd be losing the space of a bathroom on each floor and would cost money to get basically the same thing.
This is where house set up comes into play, if its a house with stairs in the middle and a big room on each side on 4 floors, there is no real natural place to put a bathroom on each floor so I can understand the en-suites. If its a house with 3-4 rooms on each floor, one bathroom makes sense, but at least one bathroom per floor makes absolute sense.
At some stage in life, stairs become a problem be it having your parents or even grandparents come and stay for a night, friends, you break a leg, twist an ankle, have kids and they will get injuries, stairs become a pain in the arse. You have noise at night, someone gets up to pee, the less rooms they walk past, the less stairs used, the further a bathroom is from everyone else, the better. If you had 3 kids, one ensuite per floor and a bathroom in the basement, then unlucky kid who gets a bedroom on the second floor has to go down 3 floors just to take a leak in the middle of the night.
As for moving, never say you won't move and certainly not for reasons of stamp duty. If you both(assuming you both work) lost your jobs and ran out of cash, and have to sell and move to somewhere smaller, thats life, you don't refuse to do it because of stamp duty. You might have kids, they could be sick and you might need to move near a certain specialised hospital, you might get sick and need the same, you might be unable to do stairs in 5 years due to illness, you might get a job that moves you to the other side of the country, or world.
I'd never change a house based on the absolute certainty that I'd not move for years and years, because anything can change.
One bathroom per floor, is something I would stick with, for certain, unless you can get rid of both en-suites and put in a single bathroom, or make one of the en-suites accessible through another door, I wouldn't be taking any of them out.
One of the bedrooms will get converted in a study and I'm calling dibs on a games room until our family expands. But even with all four bedrooms occupied, ripping out two bathrooms would leave us with three bathrooms + one WC.