It's because we went from 1000 to 100,000 polygons per car / player character etc over a period of 5 years or so, and that is a huge jump in fidelity, a 100x increase in the number of polygons. Around this time a wide range of effects / tricks were introduced and wowing us (who remembers seeing the Quake 2 lighting for the first time).
In the last five years we may have gone from, I don't know, 2,000,000 to 2,500,000 polygons, so whilst 50 times number of polygones increased compared to 20 years ago, (I am just making these numbers up by the way!), this is only a 0.25x increase - of course you're not going to ntoice it compared to previous increases. Not to mention the effects we managed to implement into games have, well, already been implemented - yes they look much better and are rendered in a different way now but we can't be wowed by the real time lighting because it's alreay been done if you know what I mean.
Finally resolutions have increased from 0.3mp (640x480) to 0.7mp (1024x768) to ~1mp (720p) but then 2mp (1080p) and now 8mp (4k) so the number of pixels that all these polygons need to be rendered in(?) is increasing exponentially too so its exponentially, exponentially harder to make a game look much much better.
The other way of looking at it is that the orginal 3D acellerators on the PCI bus would use 25 watts max and not even need a heatsink. AGP came along and raised that to ??? and heatsinks and sometimes fans were needed. Then PCI-E raised that to 75 Watts and we needed good heatsink and fans. Then we got the 6 pin connection (another 75 watts) and we needed dual slot coolers... then 2x 6 pin, and then 8 pin (150 watts),now we're at, watt, 300+ watts and a triple slot cooling for a high end graphics card? I feel this increase has been pretty linear over the years so the graphics tech has definitely been ramping up, we just struggle to see the increased quality now its got so good