A quick search on Wikiquote reveals that it's J. S. Mill, and that the full quotation is:
"If all mankind minus one, were of one opinion, and only one person were of the contrary opinion, mankind would be no more justified in silencing that one person, than he, if he had the power, would be justified in silencing mankind."
...which was later rehashed by Noam Chomsky as:
"If you believe in freedom of speech, you believe in freedom of speech for views you don't like. Goebbels was in favor of freedom of speech for views he liked. So was Stalin. If you're in favor of freedom of speech, that means you're in favor of freedom of speech precisely for views you despise."