That assumes that 'offence' is going to be the only thing taken into account, which is unrealistic. Ross is a AAA star of the BBC. Also, he apologised immediately at the time of his comments.
A gollywog has negative connotations since they were the villains of the Noddy books.
In other words, he looks like a thieving villain, based on his colour. Not a pleasant thing to say, even if it wasn't intentional to have that meaning.
I really don't care what the golliwogs were in Noddy. the fact of the matter is that the bloke looks like a golliwog, that is what she said and its quite obvious what she meant. Im quite sure that she was not implying that he is a villain.