If you find it adding more, there's a simple solution. Keep the charging cable in the car and just plug the FitBit in when you're driving as it won't register steps when charging.I was under the impression that it added much more than that. An extra 50 or so makes very little difference as you say.
I did that to start with but soon realised that the number of random steps added is minuscule and I kept forgetting to unplug it and stick it back in my pocket!