How annoying! I used to have a mic plugged into a Razer keyboards mic slot, and that caused a few complaints in TS 'Do you have a vibrator running, dow?!' they used to say to me too. Anyway, a few ideas... even though you have found that removing the headphone amp cures it, there might be a way to still use that amp and get rid of the noise.
Could be caused by poor quality cable between the amp and the soundcard perhaps? The ones from my Xonar D2 and later D2X haven't picked up any interference mind. Try to keep some distance between the analogue cable from the sound card to the amp, and any digital cables round the back of the computer. Ie, don't have them all cable tidied together and running parallel.
Also, how far is the amp positioned from the rest of the system? Try to see if putting it as far away as you can manage makes the noise any lower. This also means keep it away from things like the monitor and any USB devices/hubs, routers, etc.
There could also be a poor earth in the chassis of the computer. The soundcard is screwed into the metal of the chassis with contact between the metals? If the chassis is painted, the threads of the screws will be providing an earth, but if you've moved the card slot recently this should be okay.
Good luck.