Well, if you still want the performance, but want less heat your ONLY option is a 5770 or above.
I do know what you mean about heat, I've got a room that is basically a radiator in summer, south facing, flat roof above, so ceiling/south wall in sun ALL day and left wall for half a day, and I have the hot water tank, the computer is just the icing on the cake. I had a 4870x2, was great, but replaced witha 4890, less performance but less heat output, replaced screen with a 35W LED LCB over a power hungry and hot running but much better quality screen(130w vs 35w).
40nm process, same performance less power, or more performance same power. So the only option is to move to the new gen. The AMD and Nvidia series both use significantly less power at idle than they did last generation which is a nice addition, down to 20W vs about 60-80W last year on the top end single gpu cards. a 5770 offers probably mostly better performance than a 4870 but at not far off half the power and lower idle power usage by a long way.
Nvidia wise, well the only option is 470gtx and up, and they use ridiculous power this gen so you won't get a lower power/quieter card on the Nvidia side frankly.
As for being passive, my 5850's(i'm a sucker) run very quietly, the original one I got was quiet except when overclocked(and I expected as we hit summer weather the fan would need to spin up further). So I stuck a prolimatech Mk13 on it, expensive but should fit new cards also, if it fits my next 2 cards aswell, it will end up as £15 per card, which is great value. Zalman at £32 is also a fantastic cooler, mine runs literally silent and right now in a ridiculously hot room its running 28C idle, it runs at 45C load in a closed case with another 5850 and overclocked quad core.
Passive just isn't going to happen, you can't get even close to 4870 performance on a passive card, silent with fans, yes, passive, no, I think something like a passive gt240 type performance is about as close as you'll get and I'm not sure even they come in passive.
I'd recommend a 5770/5750 for now, see how you get on with the fan noise(I'd imagine you wouldn't notice it in normal use at all), if you have to, get a Zalman/Prolimatech in the future and keep in mind it will probably fit any card you have a for a few years so will be decent value long term.