The thing is that even an increase to frequencies of 10-15 GHz, would still be a pretty significant improvement!
I think Dave was getting at the issue of the path delays being a dominant factor in the frequency of the ICs. So, great, you've got a flip-flop (single bit store element) that works at 10GHz, but can the signals (electrons or lack thereof) wobble their way down the routing between flip-flops between clock edges?
At 10GHz, your time between clock edges is 100ps. The speed of light is 300,000,000m/s in a vacuum, where the wavefront of an electronic signal in silicon propagates at about half the velocity of a vacuum [requires citation]. So that's 150,000,000m/s * 100ps, giving 15mm!
The problem is, this is one of many factors that add up to your propagation delay. You've got the delay of your combination logic (the stuff doing the actual "maths" between flip-flops), and the parasitic capacitance that need to be charged before your signal reaches your gate thresholds. Also, you have a setup and hold delay to meet, which is the time before the clock edge that the signal has to be stable.
So lets assume you can (if you're lucky) get your signal down a 5mm path before the next valid clock edge. 5mm is quite difficult to glue cores, caches, IMCs, etc together. Also keep in mind the routing is never as the crow flies, and can be quite convoluted.
Anyways, really dodgy maths, but it is the rough idea.