I can gurantee you would care what people said about you if it grew out of control, starting trending or became the cool thing to jump on.
It would definitely affect you one way or another. How about if someone starting randomly tweeting awful, unjustied words about any of your loved ones, are you suggesting you could just move on and forget about it?
I'm guessing you've never been bullied?