No offence, but I don't really get why you wouldn't want to install the drivers. This is especially true if you happen to have the disks. If it is something like a 56K modem I usually don't bother to install drivers (assuming the person isn't using it) and either disable it in device manager or remove the card.
Its basically improving system performance, may as well have it!
That could be the IDE drivers. On my Nvidia mobo I get a choice between MS' and Nvidia drivers.
Are we talking a couple of seconds here though? Surely it wont make THAT much of a difference...