Its fine. Theres no where near enough to wreck it. Hell, even a small magnet would be fine. Honestly, dont worry.
I think it's been established now that the magnets used in the head of a hard drive are far more powerful and operate far closer to the magnetic platters than you're going to get with a screwdriver.
So as long as the PC's not on when you're doing all of this, you should be fine