It's the same as the use of the phrase "blackboard". It's called a blackboard because it's a board that is coloured black. There was a time where some people were getting equally as upset about that insisting we should call them chalkboards to avoid the racist links to the word. It's the same crowd that want to ban the use of the word "christmas" and instead call it the "holidays" so as to avoid offending non Christians.
It's all absolutely nonsense, have plenty of friends and colleagues across numerous different nationalities and faiths, none of these people find it offensive when you use the word Christmas, or use the word field. If you start saying a black person must know how to farm a cotton field because they're black - now we're being racist. But the word isn't racist in itself. I'm aware the right likes to use this stuff as evidence that anybody who doesn't agree with them is the "radical left" so i'm not going there. Just talking about common sense here.
Show me the massive petition from African American's wanting the use of the word field banned ... go on i'll wait.