Mix of many factors I think:
-Increased use of voice comms and focus on teamplay (as a game mode, not the actual mechanics of teamplay) in modern games meaning that having huge numbers of players per team could get pretty confusing/annoying compared to smaller teams
-Need for maps designed to work well with that number of players (and by work well, I don't just mean bigger in scale, depending on the game mechanics you need to factor in spawn points, choke points, potential exploits, need for balancing etc etc)
-General resource constraints both on client and server side - increasing player counts can vastly increase the amount of entities that need to be communicated/processed
-Lack of demand i.e. are people really clamouring for 100+ player FPS games?
-Potential risks from under-populated servers e.g. if you have a server running 100 player maps but there are only 20 odd people joining to play does that cause an issue?
In general I think it must be pretty hard to get a high quality, massive scale team game as most of the time it will likely descend into pseudo-FFA with some very basic collaboration and small groups working together. My background is in Quakeworld and I can say for certain that the larger games (16v16) whilst fun were not in remotely the same league as 4v4 in terms of maturity of gameplay/TP.