EDIT / Disclaimer: None of this post is in any way meant as inflammatory, it's just a part of a discussion which I think we as a species need to have more often to better understand what is or isn't offensive to others so as to avoid an unnecessary / unintentional conflict.
Racism can be very subjective, depending on the context and the language.
A good example would be Holland's
"Negerzoenen" which isn't derogatory at all, yet can still (understandably) be perceived as racist.
Another good example is brought to you by the shining light of racism, South Africa:
N***erballs, which here in the UK we simply call Gobstoppers, except they're black. This is of course very heavily racist.
An interesting one is Zwarte Piet, translated to "black pete" which are meant to be Sinte Klaas' (St Nicholas) sidekicks. This role is traditionally performed by white people in black face, and these fellas are jolly, happy folks however it was determined by people many years ago that it stems from slavery, and has since been banned.
It's interesting to see how these things change, when I was a kid in Saffrica I used to ask for N***erballs all the time, didn't even cross my mind that it was a derogatory term. It's just what the sweets were called and back in those days I don't think I even knew what N***er was as it's not a South African term at all. I, like many others saffricans, only discovered the word when the internet started spreading, as well as music videos on MTV i the 90s.
I'm rather intrigued by the word Kaffir. This, in south africa, is probably the most insulting thing to call a black person. Quite where it came from is anyone's guess, but here in the UK, it's a type of lime with no racist connotations at all. Call a black person a Kaffir in the UK and odds are you'll get a puzzled look. Do it in South Africa and you'll likely be staring down the barrel of a gun.