I grew up in the area and still live close by. I've never had any trouble there but you have to remember it's still got crime problems and you can't own anything nice there. It will get stolen, your car (if you have one) will get knocked into and scratched and as a borough I find Lambeth is VERY unpleasant. The problem is all the scum that used to live there have just been pushed into the surrounding areas like Streatham and Stockwell. They are effectively still there and you still have to pay for their problems through your council tax.
Don't get unwell there either, Kings College has to be one of the most unpleasant, dirty hospitals in the country and the ward staff are awful.