You're telling me that if no one cares about the racism then it is no longer considered racist?! Whether people are offended or not does not change the definition of racism.
I'm saying that just because someone may take offence, and try to claim it as racism, that doesn't mean it WAS racism. It all depends on the intent of the person who said it.