It depends on the amount of inbound links from these 'bad neighbourhoods'. Only if it trips a certain percentage of your inbound links will google penalise you. If the link is reciprocal google will also try to work out if your part of a link farm, which if they think you are you'll also be penalised.
Have a read of seomoz - brilliant site.
Also, post a link to your website here so we can see how much work needs to be done to give it decent SEO - it'll also give you a free inbound link!