It depends what OS you want to use. I've gone the Windows Home Server route so I have an Atom powered Mini-ITX box with room for four drives that is running WHS and all my media is stored on there. We then have a second PC that is used for gaming etc and finally an HTPC connected to the TV and AV Receiver. WHS makes the sharing easy and also adds a nice backup system that will even do bare metal restores but ideally it shouldn't be used for running Office apps.
That said with the sort of horsepower you're talking it would easily run Office/Visio whilst still acting as a media server. If you want to use WHS as the OS then you can install Office, you just have to make sure that anyone who logs into the box directly doesn't play with any of the Windows server settings or you could even go as far as installing something like VirtualBox and running a separate desktop OS like Windows 7.
What swung things towards a separate media server for me was the fact that I wanted to leave it on 24/7 and an Atom system uses so much less power.