I think it should be limited to sports that showcase some form of athletic ability, be it strength, speed or natural technique - and only sports with the use of an instrument that uses kinetic energy at most (such as bows for archery). I disagree with the use of rifles in shooting and horses in equestrian. In shooting, the rifle is making the projectile move, there's not a lot of athleticism in pulling a trigger, I'm sorry. I like to shoot myself but I wouldn't call myself practicing an Olympic sport. This differs to archery in the sense that archers have to use arm and back muscles to continually pull back and release the bow. That actually gets tiresome after a bit, it's certainly more physically demanding than holding an air rifle on a stand.
As for equestrian, especially dressage. 75% of that sport is the horse's physique and intelligence. It's not a showcase of human performance (which is what I thought the Olympics was).
Football and tennis are just that, so in my opinion have every right to be in the Olympics, regardless of whether or not there are bigger and better tournaments.