Since the symptoms of a cold are caused by the person's immune system over-reacting to a harmless virus, a weaker immune system would result in a cold being less severe, not more severe.
I have only ever heard "man flu" used as a way to promote sexism, usually to rationalise dismissing any consideration of men who are ill but sometimes by a man trying to claim a more severe illness than he actually has.
Calling a cold "flu" also gets on my nerves. They're very different things.
When I do have a cold, I'd love to have a week by myself. Not because I'm very ill - I'm talking about having a cold, not flu - but because it would save me having to deal with far too many people who are pointlessly annoying in one or more of these ways in response to me having a cold:
1) Saying I have manflu, so they can contemptuously dismiss me and pretend it's a reasonable thing to do. I say I have a cold, they say I have manflu and then pretend that I said I had flu and are hugely exaggerating how ill I am. Their lying and their sexism is annoying.
2) Saying that I have flu, that I'm "all flu'd up". When I've said that I have a cold. Not that I have flu. If I said that I had acid reflux, would they say that I'd had a heart attack? Annoying.
3) Saying "keep it to yourself" when they can see that whenever I sneeze I turn away and cover my noise and mouth with my elbow so I not only block the spray but also ensure it doesn't get on my hands so I won't then transfer it to anything or anyone else. Bloody annoying. If they saw me going into a toilet, would they command me to not urinate on their sofa? What they're saying is either mindless nonsense or a false accusation and both are annoying. Some of them up the annoyance factor some more by coughing, open mouthed of course and without any attempt to block it because they don't care.