I see no reason to keep a domain separate from hosting. If you don't trust a host to look after your domain, why on Earth trust them with data which is important to yourself (and possibly your business)? The same applies vice versa, too.
It's true there are companies who make life difficult but these are becoming rarer as their clients migrate away and post negative reviews, whereas the decent hosts state their policies
clearly (disclaimer, that's my company but just posting here as an example).
We've had all sorts of issues trying to gain control of our work domain names - unwravelling a web of companies to get it (between marketing/designers, hosts, ISPs and the like).
Because you've unfortunately chosen dodgy hosts or not read their T&Cs sadly. That doesn't make it right though but then business is business perhaps.
I'm pretty sure all hosts recommended here makes zero charge whatsoever for moving away a domain as any other good host would. But on the other hand if a host is selling something uber-cheap, the transfer away charge is there to of course produce profit, but also ensure they aren't taken advantage of, so swings and roundabouts perhaps.
It's more than just changing the nameservers, if you cancel the hosting account the domain would go with it, you'd have to change the IPS tag too.
I can think of one large well known host in particular who do that but there's no more to it with a decent company who don't tie you into long, complex contracts at silly prices.
We had a large corporation migrate hundreds of domains to us for precisely this reason. Since that time it's taken about 4 weeks to get all the domains transferred over!
Whatever you choose - read the T&Cs carefully as with any online purchase.