Personal VM Server Build

grindiron · 16 Feb 2013 at 17:22

Hi Overclockers UK members,

I'd like your advice on a few aspects of a system I am planning to put together if you wouldn't mind offering your support

Note I have asked this same question on "Tom's hardware" but I wanted to try and reach out to another set of very knowledgeable users on Overclockers UK - I hope this doesn't come over as cheap

Approximate Purchase Date (dd/mm/yyyy): 01/04/2013 - 01/10/2013

Budget Range: £1000 (This is for the initial foundation, I will continually add to it over time)

System Usage: Running multiple VM's (Initially around 5 - 10 eventually 30 - 40 ~ All types of OS's), Production Server (Programming & various services (FTP, Samba, SSH, etc)), Kernel Compilation and possibly some intensive CPU tasks like Video/Music Conversion.

Parts NOT required: Mouse, Keyboard, Monitor, Speakers, OS

Preferred Website(s) for parts: Big names preferably (Crucial, ******, Overclockers, Amazon), however I have seen some parts for cheap on less popular sites so as long as I am protected through paypal or some other service I am happy taking the risk.

Country: UK

Overclocking: Nope (i'm hoping my system will be good enough!)

SLI or Crossfire: Nope (the motherboards I'm looking at don't do either) However I may need to consider a graphics card for VGA pass through eventually!

Parts Preference: I have done a few week's of research and looking around for making a personal home server that is solid for my needs but doesn't require the funding of a small company to purchase! If you know of comparable parts that would be better for my build and are around the same price point please provide a link so I can tweak my build idea.
Below are my initial searches over the few weeks I have had time to look into this:

- Motherboard (required to be at minimum dual socket): C32 Socket - KCMA-D8

- Processor (For the C32 motherboard type - start with just the 1 then eventually buy another down the line): Opteron 8-Core 4386

- RAM (Max the motherboard (above) can take is 128GB (RDIMM) so require 16GBsticks of RAM to fill it over time - I have a few questions about RAM see further down this post): 16GB Crucial DDR3 PC3-10600

- Hard Disk(s) (Again starting with just the one and building over time - I realise lots of VM's on a single HDD will make the system crawl - I have a few small HDD's lying around the house I can use in the mean time): WD Red - 3TB

- Case: (Need something pretty big, I'd like to fill it with at minimum 10 HDDs and possibly a Blu Ray Player - this was only a quick look): Lian Li PC-A76X Full Tower Case

- PSU: I did an online PSU calculator check from here and when i maxed out parts (10 HDD's, blu ray drive, loads of fans) and added things such as a single high performance graphics card (GTX 680) I got around 800W recommended PSU. Would I need to go after a 1000w PSU in such a case to ensure it stays stable even after it ages.

I have a few more questions as well regarding this build, i'll start with the RAM:

General RAM Q's:

- What is the difference between quad ranked and dual ranked memory? - I will have 4 slots of DDR3 RAM for each CPU allowing a total of 64GB per CPU which type will be the most beneficial in this scenario?

- Is it best to go for lower Column latencies (CL) or higher HZ's for the sort of workload I expect the system to do?

Looking at the motherboard it offers an ASUS specific PIKE Slot for hardware RAID - this is something I am very interested in obtaining (again once I have a large amount of HDDs). Now I am thinking of going with RAID 6 with the max amount of HDDs it can take of which in this case is 8. PIKE 2108 .
I am slightly confused by this part however. All the other standard enthusiast RAID cardshave 2 SAS ports capable, each capable of connecting to 4 Sata HDD's but the diagram the PIKE 2108 does NOT show where you would plug such a cable in to.

Looking at the KCMA-D8 motherboard it has a set of 8 Sata ports on the left hand side so i am not sure if you would plug the HDD's into that then the card would interface through the motherboard to those Sata ports and ultimately to the 8 HDD's.
I've never dealt with hardware RAID but have played around with Linux software raid (mdadm) and I know its really easy to add a HDD to an exisiting RAID array. Would it be easy to do the same through a standard RAID card - do you do it normally through BIOS?

grindiron · 16 Feb 2013 at 22:15

Thanks for the replies Uhtred and [Darkend]Viper.

UHtred: I am planning on using either Xen or KVM for my virtualisation platform so there shouldn't be any limits with regards to CPU or RAM.

[Darkend]Viper: I had a quick Google search of the r710 server but found it was well over £1000 unfortunately so I won't be able to afford that i'm afraid. My aim is just to have something to learn from whilst making mini environments - really its for my own interest and understanding.
Out of curiosity what OS's have you been running in your VM's?

grindiron · 18 Feb 2013 at 23:40

Thanks for the response rotor,

Processor is almost never the bottleneck in a virtual environment

This is understandable, most VM's will act as clients in the environment I plan to play around with and when i'm not using them directly they are very likely to be idle. Please bare in mind though that I may want to do some computationally expensive tasks as stated in my (verl long I admit...) original post - again I know this sounds odd but it's just me.
I realise the dual socket mobo is expensive but I feel I will future proof myself by investing now.

Maxing out a server on memory is INCREDIBLY expensive. Going for the highest capacity DIMMs usually costs 3 - 4 more than buying the equivalent of the next size down

It certainly is. The price of RAM falls (albeit slower for these larger module sizes) over time and I don't want to be in the situation months down the line where I have filled all the memory banks with 8GB sticks and am now suffering from not having enough RAM (I know 64GB is more than enough but I want to MAX out this puppy!) for the ton of VM's i want to mess with.
Again linking back to my original post I will initially just buy the one stick of 16GB DDR3 RAM which is around £165 (painfully more than twice as much as 2 sticks of 8GB - I get this) and then over time buying another stick and so on.

After RAM, disk is the biggest bottleneck

Of course. I plan to use the tactic similar to that of the RAM where I buy it over time and not in one go (unless I want to take a mortgage out for it...). Once I hit 4 hard disks I plan to obtain some RAID controller be it the proprietary ASUS PIKE card or another RAID controller brand.
I need to ask for recommendations for a RAID 6 capable hardware RAID controller on another thread methinks (again I know these will cost in the vicinity of £300)
Understandably at the beginning with just the 1 HDD i'm going to be grinding slowly if I try and load a fair few VM's, fortunately I have a few small sized HDD's lying around so I can move VM's onto those until I can purchase more of my specific HDD.
As great as SSD's are, I still prefer a larger size of storage as I am considering making it a hub for my family and friends to put their photos and videos as a backup point. (I know another random thing for my server - sorry!)

I would consider 2 servers but I feel I would hit a lower maximum in terms of CPU, RAM and HDD space even when they are both combined (like your set up for example). It's something I want to continue adding to over time, like a little project really

grindiron · 19 Feb 2013 at 10:06

Hey rotor, I hope I haven't come across too aggressively in my stance on the build as this is by no means my intention.
This build is still very much up in the air and I am considering using a combination of SSDs and HDDs after rethinking the size necessary for my VM's whilst still having the storage for things like photos and videos of family and friends.

RAM does *not* get cheaper over time. As the standards change (DDR4 replaces DDR3, and so on), the manufacturers stop making the older type DIMMs

Very true, however I don't think I was very clear on my initial point regarding the price of RAM. I believe that the drop in price will be very marginal (still worth a few beers though!) as I feel that the the price of DDR3 RAM has fallen to a near "sweet spot" before the mainstream introduction of DDR4.
A quick search of "DDR3 DIMM" on http://uk.camelcamelcamel.com/ shows how DDR3 DIMM RAM has (on the whole) dropped in price over the years.

but also because in the future, hardware will be faster, cheaper and consume less power, so buy it then, if and when you need it.

I'm 100% with you that a 1 x 16GB DDR4 DIMM stick will be cheaper, the questions is how much cheaper will it be and how long will it take for DDR4 to become the widely adopted standard? Once it hits that point wouldn't it be worth waiting for DDR5 and then the cycle could just keep going for waiting on the next revision of hardware?

If you want large amounts of disk space, use SSD for the system disks of the VMs, and present spinning disks as RDM (Raw Device Mapped) devices (or whatever KVM calls their equivalent, assuming they have one).

My plan is to have 8 x 3TB HDD's for storage (and initially I thought VM's) and some other disks for my OS. Having 3/4 256GB SSD's would be more than sufficient for my VM's in terms of performance (will be able to run more VM's as a result VS my initial idea of the standard HDDs) and storage so it's definitely something to consider.

In terms of one dual-processor machine being more powerful than two single-processor machines... how do you figure?

I may of made a too sweeping statement that has been interpreted not how I intended - sorry!
My comparison was to your system setup in particular. What I don't want to do is cobble together servers that have some brunt but one having more power than another (not that there is anything wrong with it - each to their own) and having to shuffle VM's between the two to get optimal performance. Dealing with two physical boxes is also just more hassle for me.
I want to make clear (after re-reading my post I made it awfully difficult to interpret) that I realise you can make a setup where the two single-processor machines could house the same combined hardware as the dual processor machine and thus have the same performance. However a separate case and PSU for each one can quickly mount up and cost the same if not more than the as the dual processor set up. Also remember if I want to utilise both processors simultaneously for doing one job, i.e. kernel compilation, I would not be able to do so with two single processor machines, same goes for the RAM if I ever needed to use over 64GB (E.g. lots of VM's running and only have 4GB renaming on both boxes but I need 8GB for the VM - I know it's picky but unless the price is considerably less I don't feel the need to move).
Ignore where I have said HDD space is less when combined, 2 servers would give me far more room for expanding storage - I clearly didn't think through the scenario well enough! Although Hardware RAID for both could be quite an expensive solution VS the single dual processor set up.