I disagree. There's no logical reason for the offence taken to differ.
Call someone you know at work an idiot, then call someone else a black idiot. Let me know how that works out for you.
You can argue the semantics or 'logic' behind it all you want, fact is that by adding someone's skin colour into the mix creates a totally different situation. If Terry said what everyone thinks (and saw) then he commited a criminal act, it would not have been a criminal act had he not brought colour into it however logical any of us think that is.