Potentially cutting your life short.
People do all manner of dangerous things, they do them because it provides them with pleasure. I'd much rather have a short life that is full of pleasure than a long, boring life that I abstain from things that I know that I'll enjoy.
Yeah there's a great many drugs that'll give you pleasure. Herion being the notable one. But I don't see many people taking the same view as yours on that.