I think it might be more to do with what you deem unusual. If someone had a massive scar on their face you might not notice they had a slight squint because you focus on the scar. Similarly it sounds like you don't see a huge amount of people from different races and, as such, when you do see one your mind concentrates on their skin colour rather than something which would help you recognise them from someone else of their race, like the shape of their face or other defining feature. When you see someone of your own race, because their skin colour is not unusual to you, it fades into the background to be overtaken by something that helps you more easily delineate them from others. The brain is very good at throwing away details that it doesn't deem important and concentrating on those that it thinks are. This, I think, may be one of those situations.