It would SPEED UP the game, not slow it down. Why, because currently people grab shirts at every corner and ref's constantly pull up games to give a pair of players a talking to, after a few weeks of giving away cheap penalties and red cards, this would simply stop happening.
Make throw in's non challengable, not exactly difficult, but more importantly like just about every other sport out there have 2 or 3 challenges, in which case there is little to no benefit in time wasting or using them on pointless decisions. I would guess that if we had 3 challenges per manager(where you lose one if you get it wrong) then we wouldn't have one made over a throw in , in the next decade of football.
After a few weeks we'd have, less arguing with the ref's, less freekicks, less diving, less taking ages arguing if something was a foul, less complaining after goals, less mouthing off at ref's, faster corners with less cheating, etc, etc. A huge amount of time is wasted every game purely through cheating, dives and long freekicks, corners, etc, etc. Though most of this isn't added on, its just less time in the 90mins actually spent playing football.
They also aren't interested because if you say video replays, 95% of fans INCORRECTLY say it will slow down the game. Ref, 30 second replay from multiple angles, it will either happen a few times some games, loads in some games, none in others. It will improve referee's by giving them real time feedback in their decisions rather than reviewing it 2 hours after the fact with no chance of it being fresh in their mind. They'll start getting decisions right more often which will negate the need for challenges.
Fact is ref's suck and players constantly cheat and its getting worse every year and video replay will fix 99% of cheating almost instantly and 99% of incorrect decisions. Every time anyone mentions it in a pub, tell them why it will speed up the game, when the majority of fans want video replays, Fifa will get it going, most fans have a whole raft of reasons why it will somehow ruin football and they are frankly all wrong.