Not wanting to pay for something i do not want or need is the dumb?
Large scale... yep.
My taxes pay for the fire service. I have used them zero times.
My taxes pay for roads, I don't drive.
My taxes pay to educate kids, I don't have any nor do I intend to.
All of these things benefit society as a whole, and therefor benefit me.