Whether it should happen or not, you will find groups of friends that will make offensive remarks to each other and no actual offence is caused. There's an interview with the boxer Billy Joe Saunders on YouTube where he, the interviewer (a guy of Sri Lankan descent), a black friend and a mixed race friend of his all discuss racism in which they all openly make offensive jokes to each other but it's clear that no genuine offence was meant or taken by anybody.
This situation is different though. There's no two way interaction with the black players involved, it's a generic term being used to describe 3 people purely because of their skin colour so by default it's fair to assume the person saying this thinks the term applies to all black people and it's been said to the world! Given the scrutiny your club and country are under for issues around racism you'd have to be the thickest person on the planet to use terms like this publicly, which makes you assume that he thought it was a perfectly fair term to use.
You're going to play stupid on this too? Come on Shami, you know what he means.