There is potential to increase single threaded performance significantly but ultimately its not... its not correct to say its not needed, its just inefficient.
2 cores running at 4Ghz will use exponentially less power than 1 core running at 8 Ghz, and frankly 2 cores at 4Ghz, the die size cost is expensive vs 1 core, but massively cheaper than creating incredibly high speed capable processes.
Ultimately so many devices DON'T need single threaded performance or high clock speeds and having a process and foundry that can make a whole range of chips for various devices needs to be a relatively one size fits all process. We're talking about billions, realistically 10's of billions now in R&D for future processes(much spread across many companies) and similar amounts in equipping a fab, the more chips and the more types of chips that fab can produce the better the cost gets spread.
To be honest I think hardware always came before software, and Intel decided to stop adding cores quite a few years ago in mainstream. They are making very small chips but creating a lack of upgrading urgency. By not pushing 8 cores into mainstream pricing, they aren't pushing any game makers or any other high performance software makers to come up with ways to use 8 cores of performance. But because of the very small upgrade in performance they've moved the market more from "upgrade because I need more performance as my computer is slow" to "upgrade when my old computer dies".... which is FAR less frequently.
5 or 10 years ago I never saw an enthusiast forum full of people saying "meh the new Intel isn't worth the upgrade, wait for something in two gens when maybe Intel add some more cores"
Whats quite funny is with Intel that, they've reduced sales because people are upgrading less, and they've reduced die size, so they are currently running WAY below capacity at their fabs which cost a crapload to kit out. If they were making 8 core chips they would have more demand, and because the chips would be bigger, the fabs would be closer to capacity
CPU's have been boring for ages. A 2500k would last you freaking ages.