I think it simply comes down to demand for Milan-X taking all the chiplets.
AMD will have known from the outset that big chunks o cache will benefit some workloads better than others, it's not like they're slapping things on blindly. Since a chiplet is a chiplet is a chiplet, I do believe that AMD's intention was a full 5000 series refresh with 3D cache which would close the gap on Alder Lake for 6-9 months before Zen 4 comes along.
But then everything got in the way so AMD changed their plans.
I recall AMD saying that demand for Milan-X was higher than anticipated, so straight away that means Ryzen X3D is getting cut back. Then I'm sure something else Covid-y got in the way which pushed Ryzen X3D release back to Q2. So now AMD are in the situation where they don't have as many chiplets as they planned for Ryzen and the delayed release puts it within 6 months of Zen 4.
So, spin the marketing narrative that 3D cache is part of AMD exploring diversification into custom SKUs, and the best desktop use for 3D cache is for gaming, so they release the definitive gaming CPU: 5800X3D.
I think because its limited to an 8 core SKU,the 16 core models would beat it overall because of sheer core count.
But would it? How much does a 5950X beat a 5800X in gaming? That's the marketing narrative for the 5800X3D: it's a dedicated gaming CPU. So would a 5950X3D beat a 5800X3D in gaming?
The "limited to 8 core SKU" is simply because they don't have the chiplets to make any other SKUs. Had Milan-X not been as popular then AMD will have more 3D chiplets to go around, including the 6 core chiplets for 6 and 12 core CPU options.
Even if AMD were genuinely surprised in final tests that the cache boost only benefited gaming on desktop, the fact we don't
also have a 5600X3D to use those 6 core chiplets pretty much confirms Milan-X is gobbling everything up.