HT was in the P4 in 2002, it's been 17+ years, almost twice what you state and nothing to do with the i series of chips, intel removed it from lower end variants, but that's not the issue here. Also intel weren't caught off guard, they gambled on being able to transition to 10nm and lost, that's why refresh after refresh has taken place recently. It just so happened that AMD was able to be competitive at the same time, not only on price, but on performance and supply. Zen2 looks set to move the goalposts completely if it moves to the rumoured 16c/32t top end as that's well into Xeon/TR territory. That's not ground intel wants to compete on, ever, as that's where the money is. Intel have history (x58/79/99) for dropping Xeon chipsets in consumer boards for HEDT and charging a premium, it didn't need to improve on 4c/8t for the masses - most of them have little to gain and those that do were a tiny percentage of the market - they were supposed to buy HEDT.
At face value your Xeon cost point is correct, but this is OCUK, so look a little deeper, the critical part of Xeon ownership for enthusiasts has always come down to 2 letters, the last was always an S, the first was either an E or a Q. With a little knowledge, 8C/16t CPU's were inexpensive for x99, in some cases you could buy the retail stepping for peanuts vs. the retail price, 24c/48t was possible. My 2630 v3 set-up (board, CPU, RAM) cost less than my Ryzen set-up, even with the latter enjoying some very significant discounts. Also remember OCUK sold x99 boards for as little as £99 new (Gigabyte EATX from memory).