Well, I think it would be best to move it a bit further away just to keep folks happy, its not like the Muslim's are going to be put out by it being in a slightly different place.
you are right they wont but calling it 'the ground zero mosque' when its not is just feeding the backlash