I'm very light brown (mixed race). I'm not really bothered as such by it, more perplexed. I'm extremely interested in the psychology behind it as well. The most amusing thing about the whole thing though is that the most vocal anti-chocolate people brigade, especially the ones who go on about race preservation, will have quite a varied make up, just like the rest of us.
Most of these people focus on skin colour because it's really the only way they can rationalise their broken mentality, but the reality is there's a lot of "brown" people here that have a stronger ancestral route within the UK than some "white" families. Sensible people know skin colour means very little outside of what shade make up to buy if you're a woman.