My first thought would be that Gentoo would be the optimum system for this - everything compiled in place with perfectly tweaked compile options. Then it occurred to me just how long it would take to build Gentoo on one of these machines - I was doing some rough estimates under QEMU with ARM1176 and it wasn't exactly quick. I'm not so sure just how much of that is QEMU though.
As the Raspberry Pi hardware is common then there should not be any issues with starting from a Stage 3 Gentoo image, as it will already be optimised for your hardware. The problem comes with getting to the point where a State 3 image is availale