Button didn't walk away from Brawn, he was pushed away from Mercedes.
I think Fry was just a bit bitter about the whole thing. Brawn/Button win, and then Merc muscle in, Button goes elsewhere, and hes left a lot less important in the whole scheme of things.