Average people think selfishness is a vice. Rich people think selfishness is a virtue. - so true
funny how many rich people are philanthropists then.
Broadly speaking, imo you'll usually accumulate enough money to be able to give lots of it away by being selfish first.

When you are brimming over with money, you would probably change and be happy to give millions away whilst still keeping more than enough for yourself. When you're first struggling to accumulate money however, you would be ruthless (in many cases anyway), hoarding, saving and investing your hard fought for money until it gets to the point that you CAN afford to become a philanthropist.
So if being selfish means that ultimately you can give money away and help others, then more power to being selfish.