This sort of statement sounds silly but in reality is typical of the attitude of most people, even most modders I'd wager, buying someone elses car.
Someone who is selling a modded "normal" car (I'd exclude some Jap hypercars - e.g. Skylines, Supras, Evos, etc from this) will appear to your average layperson to be the sort of people who "drives spiritedly around" (aka boy racer). That may not be the case but that's going to be the automatic first impression. no one wants to think that the car they're potentially going to be buying has been ragged about, even if that's exactly what they intend to do when they buy it themselves!
For that reason I too would have reservations about buying a modded car, or one that has been "returned to standard". It makes no sense, but it is what it is.