I guess there are three camps with skin in this game.
One being the people who don't want to see statues of slavers in their home town, or places they might like to visit. They might also think that these statues normalise what is now considered a crime against humanity. They find the statue horrifying or at least unpleasant. Then there are people who consider it so important to have memorials to history that having a statue of a slaver in the middle of a modern British city means they'd rather see it standing, regardless of any affront it gives to group 1 (even though there's been no attempt to make the statue into a memorial). Then there is the last group who want a statue of a slaver in the middle of a modern British city because it somehow validates their identity and they're at least a little bit racist/supremacist/rule Britannia.
Lastly, there is another group who don't really care either way. I suppose it's up to us as individuals to decide which group they feel at home in.