That's only true for some times/places of the world. If, for example, you were to go back to the Roman empire it wouldn't matter. If you went back to, for example, the Mali empire then being "white" would be a definite disadvantage. In Norman England, a place famed for keeping records of everything, they cared so little about "race" that they didn't record it. We know that some people migrated from Africa to England in those days, but we don't know how many because nobody cared. There are references to people coming from various African ports and that's it. If you were a Christian and you weren't French, that was fine.