My view is that because these people live in the same society as you and I, then yes we should care about them because the consequences of their actions can affect the quality of life for lots of other people. These troubled and troublesome people are the way they are for a reason, they often need help and support to break the cycle of depravation. Who should deliver this support? A combination of the state, charities and individuals or as I like to think of it - society.
It's easier to just label people scumbags and segregate from them though. Live in a more affluent area, a gated community if you can afford that, then pretend poor people don't exist.
I sometimes wish I did live in an era where community was more emphasised upon. I think in those times though there was much less disparity between poor>comfortable>rich. The rich were not so, and the poor were more (numerous) so. The need to rely on one another was more apparent, now it's every man for himself in a dog eat dog world. Thing is, I'm all for competitive markets, the free economy ethos, and a bit of dog-eat-dog. That framework for our 'Western' way should not be to the detriment of decent human nature though, hence why a decent government who can regulate accordingly should be in place.... which party is that though??