Ooooh.
Right, first of all, you need to decide how small you really want this SFF system of yours to be. You have three choices - Shuttle, mITX, mATX.
In terms of Shuttle, well, they're fairly obvious. You have a range of SFF barebones systems supporting a number of Socket 775 chipsets - you can buy new and go for something like an X48/P45 model, or look on second hand sites for a P35/X38 model. I'm not sure about how well they overclock, but the PSUs are often strong, and the cooling good, so there's no danger of things not working out.
Next up would be mITX. There's only one Socket 775 mITX board with good features, and that's Zotac's 9300-ITX WiFi. It takes just about any Socket 775 CPU you can find, DDR2, and also packs in a full PCIe x16 lane for a good graphics card.
The problem with this board comes when you begin to consider power and cooling.
There's only really a couple of cases that you can use with this board if you're looking to build something powerful and overclockable - the Silverstone Sugo SG05 and SG06. I understand there's also a Lian-Li mITX case that can hold a full ATX PSU, but at that point I believe you're getting so close in terms of overall size to mATX that limiting yourself to a smaller board etc. seems pointless, you may as well make the step up.
The Sugos both come with their own PSUs - 80Plus certified 300W units. Powerful enough for a single 4850/GTX250, even at a stretch a 4870/4890/GTX260/GTX275. Anything bigger and I believe you'll hit problems, fast.
Also don't forget you get 78mm worth of height in which to get a CPU cooler in. It's not a lot at all, and limits you to things like the Zalman flower shaped coolers, the Intel stock coolers, or the NT06. And that'll limit overclocking too.
The biggest form factor of the three is microATX. Here's where you start to get a lot of choice. The good motherboards for Socket 775 begin with the G33/G35-based Gigabyte GA-G33M-DS2R and the Asus P5E-VM. However if you want the very best, you're looking at the DFI Lanparty JR P45-T2R. It's got two PCIe x16 slots so it can do Crossfire in 8x8x configuration.
I believe however, that you may as well just get one of the lesser featured two unless you really want Crossfire, the DFI is an expensive board compared.
Then you move onto the case. There's a lot of choice here too. The best (IMO) would be the Sugo range - SG01, SG02, SG03, and the Lian-Li V350 and V351. The Sugos have the same drawback as their mITX brothers with regards to CPU cooler - you've got 78mm space before it won't fit with the PSU. The Lian-Li cases have much more room, but don't get carried away, you're not going to get a TRUE-120 in!
All of the mATX cases i've mentioned take a full sized ATX PSU, so power isn't a problem. Just make sure to get something modular as it's a nightmare working in a cramped case otherwise.
I think that's it, any more questions just ask.
[Edit] Just thought i'd add, there are more mATX cases than that, but IMO, those are the best ones. The Mini-P180 is a lovely case and you can get a full sized cooler like the TRUE-120 in, but it's hardly smaller than the P180 proper so you're not really saving space, you might as well just get the full ATX version.
There's also a Thermaltake SFF case, the Lanbox. Nobody seems to have one, and from my experience with TT cases they're style over substance.
