For me, although Top Gear became more a car based entertainment show, the fact that the trio were car savvy journalists still came through.
Matt and Chris just sound like blokes who happen to be rich and like, and are able, to buy expensive cars, not true petrol heads.
Someone called it when they talked about the lap commentary, it sounded like your average woman talking about football (if that's not too sexist a thing to say) when they've never ever kicked a ball in their life.
I also like the description above likening it to a cover band.
If they were going to change the celebrity bit, rather than the rally cross, maybe use Sabine more for that segment. Maybe have a selection of cars to account for the different ability of the celebs and they have to try and get within 125% of her time on the day (or whatever is appropriately challenging).