90% of users calling OP a coward would have probably done the same under the circumstances, it's just too easy to have a go at him and look the man in front of everyone else on the forum.
I think the trouble is that authorities are so overzealous and lacking in common sense nowadays that people are scared to get involved in any disputes, you only have to offend/scare someone on Twitter and the police get involved.
Exactly this.
Internet is full of keyboard warriors who can lead a army of pixels, but physical confrontation is something entirely different for us all.
I've been involved in fights throughout school and college, even been attacked by suspect with a knife, but I tell you this, escalation is so insanely easy to achieve especially with adrenalin and testosterone that it just isn't worth the risk.
Who was to say the woman's bf wasn't nearby and was tooled up, or her brother or some friends?
It is more likely that the big guy gets targeted and that elderly generally do not get anything but a mouthful of chav-abuse.
In this situation (after assessing it carefully - which nobody here apart from the OP has done) I'd have observed an acted as a witness initially. If she got physical, then intervene. Otherwise if no escalation, when she's gone ask if the old folks are ok.
For the record, my mate defended his gf being attacked by 15 drunken teenagers. He was charged as was his gf for hitting 'innocent defenseless children'. The appeal and charges were dropped - the drunk teenagers got some big compensation.