The problem really is that the comment has come from a popular politician knowing very well it's going to hit the press and stir up opinions. I don't see the comment as being racist/anti-Islam etc necessarily if it came from someone else, but when you consider his position, his association to Brexit and the other idiot involved in Brexit then it does come across as possibly showing racism and trying to stir that up in others, IMO
It is only doing that because people have found out about it, republished snippets of it and then the screeching started... it could have just been an opinion piece that various telegraph readers just read and then mostly nod in agreement with, it was the people looking to be offended by things that gave it all this publicity and started this whole story.
The sort of person who might abuse women in the street etc.. likely wouldn't have even read the telegraph column, they're only learning about the comments because of the screeching. If you look at the interviews in the aftermath there were a bunch of people saying "I agree with Boris, the burka should be banned" making it clear they didn't read the original article at all but are reacting to the reporting of it/snippets of what he said as they've got the completely opposite impression of what his actual views are.
I'm glad he's not apologised and frankly I'd like to see more politicians speaking freely like that, the more it happens the less people who try to shut down speech by claiming to be "offended" can have an impact. In fact it would be much better if the article was just written and simply read by people who chose to read the publication it was written for because people finally accepted that not everyone is going to have the same opinion as them.