It's not about being brave. It's scary as hell. As you said in your OP, the fear hits. It's just about getting yourself out of the situation as quickly as possible. You could spend a minute puffing up your chest along with the aggressor and trading insults... but all that is, is both parties squaring off to see who is going to throw the first punch. And getting punched isn't nice. So don't get punched first.
Do you know any martial arts? Or do you just hit and hope for the best? Could be a bad ending for you if you go up against someone who knows how to defend themselves.