The funny thing is many early personal computers were often synonymous with controllers/joysticks, they'd often come as part of a package deal with the computer, as a young kid I played many games on Spectrum and C64 using joysticks. This is a time when you wouldn't get a mouse as standard and mouse use was far less common than controller use for home entertainment users.
It was only the IBM PC compatibles mostly intended for business and similar that didn't really much focus on controllers, I think that's where a lot of PC controller reticence stems from.
There were also no really good controller standards on PC except Sidewinder that wasn't great until the Xbox 360 came out and gave PC gaming a great and more importantly standardised controller format.
Now the home of the greatest and most varied controller games library is on PC once again, with 3 console generations worth of games that can be played with controller on Steam, plus many earlier types of games with it as standard as well, there's more controller based game currently playable on PC than any console has ever had.