It does matter.
Performance is only there, if the fans are sounding like windtunnels and still failing to prevent the card from throttling.
Before I discovered 'manual fan control', I used to run an ATI 1800XT which would hit +80C° temps. That card stopped working within two years and gave me plenty errors and artifacts on its way out. With my current HD 7970 OC, I put the fan up to 50% to keep temps down where I like them. This gets me a further 300Mhz on base clock speeds, but damn, is it loud (although not as loud as GTX 980 ti fans it seems).
I refuse to have either another ridiculously loud or boiling hot GPU in my desktop ever again.
Thousands of people buying the 980ti don't seem to care