Home server for multiple VM's

Boxfreshchris · 22 Mar 2019 at 07:32

Quick question chaps.

Say I wanted to build a home server for running multiple Virtual Machines using VMware, the plan would be to have this running 10 simultaneous VM's and I would remote connect from my main machine. With the VMs I'd need lots of threads/cores, big ram and a couple of SSD's, with a pov spec GPU. Initially I was looking to build a Threadripper rig, however, would something like a 2nd hand Dell workstation work just as well?

*** no ebay computer equipment links please ***

A custom threadripper build would set me back £2k-£3k, it seems I could pick up something like the above for much less, but would it suit my needs just as well?

Cheers for any help.

Throrik · 22 Mar 2019 at 07:49

These work fine yeah - a lot of homelabbers purchase ex-enterprise gear such as this or rack mounted servers to run their VMWare environment at home, or their HV environment at home. The one downside is that it will consume a fair amount of power and you might see a significant increase on your monthly energy bill in comparison to a thread ripper build. So you have to weigh up the pros and cons of short-term versus long term. They're also noisy as hell.

Boxfreshchris · 22 Mar 2019 at 09:01

The increased power consumption will be fine, and if its too noisy/runs too hot it can be relegated to the garage.

I guess my main concern with a system like this would be maintenance/diagnostics/upgrades. A custom build with consumer level parts I'd be happy with diving into, but I wouldn't be comfortable doing the same with this system. Simple things like installing a few SSD's looks a lot more complicated than with consumer grade kit.

jaybee · 22 Mar 2019 at 09:15

You can find some real bargains on ebay where company's literally throw stuff out more to do with space, than trying to actually get the actual worth of the goods. It's lost space and hence money keeping all their old kit which has fallen out of support so they often bung it all on ebay with a low starting bid. The rack mount kit is especially bargain worthy as not many people want to re-purpose rack mountable kit at home or small businesses. They tend to want the large tower case based stuff like the Dells above. If you can live with a small rack or space to house a long rack mountable server just on a bench/shelf/cupboard somewhere, then you can get monsters for next to nothing.

As said though the drawback can be power consumption and noise. Depends on the generation of server. Some of them were never built for low power consumption and were more for performance. A lot will be extremely noisy if not in a room at a constant 16c as the fans will have to ramp up to compensate. Just do a local search for "IBM Server", "Dell Server", "HP Server" etc and see what you find.

jaybee · 22 Mar 2019 at 09:17

Boxfreshchris said:
The increased power consumption will be fine, and if its too noisy/runs too hot it can be relegated to the garage.

I guess my main concern with a system like this would be maintenance/diagnostics/upgrades. A custom build with consumer level parts I'd be happy with diving into, but I wouldn't be comfortable doing the same with this system. Simple things like installing a few SSD's looks a lot more complicated than with consumer grade kit.

How so? They either plug in to the mobo or they don't? Just check the underlying mobo for connections and input/output.

edscdk · 22 Mar 2019 at 09:35

Don't forget that machine you listed is 1st gen xeon Westmere.. Low ipc possibly no faster than a 8 Core modern cpu.. I had a dual Oct sandy bridge xeon system eventually swapped for a i7 Oct oc to 4.5 GHZ.. Westmere combined GHZ 36ghz new system combined still 36 (and better ipc) sata3 faster ram, usb3 lower power

Boxfreshchris · 22 Mar 2019 at 09:44

Just stumbled across *** don't link competitors ***

I can get a T7500 running 2x Xeon X5675 (12 cores, 24 threads), 128GB ram and 2x 256GB SSD'd for £650, I'll also have a look on ebay for rack style systems, I have no issues sitting it in the garage on a shelf

Avalon · 22 Mar 2019 at 10:11

Stop, buying that for £600 is a horrible/bad idea. ESXi support for certain x56xx CPU’s was pulled in 6.7 - you can get a R710 for a lot less with better spec.

How about a cheap X99 LGA 2011-3 based build? A nice Xeon ES (10-14 cores for not a lot), a load of DDR4, bootable NVMe support for running multiple VM’s easily will set you back way less and be much better and much more power efficient and have long term support.

R720 for £2-400 if you have a rack, decent pair of CPU’s for £180-200 depending what you buy, easy/cheap to upgrade and again much more power efficient/future proof.

You could also grab a Ryzen cheaply and do the same with current hen hardware and the option of a 3000 series upgrade in the future. Decent IPC, clock speed and core count now, with potentially more later. ESXi support is a lot better than launch.

Either way the money you are looking to spend can be spent better.

Boxfreshchris · 22 Mar 2019 at 10:27

Thankyou. See this is what I meant when I said I wasn't comfortable with enterprise grade equipment, I've never used it before, but I'm comfortable with consumer grade kit. I was originally recommended a Threadripper based build, but consumer level CPU's like this, and the motherboards & Ram that go with them, seem overly expensive when compared to their enterprise counter parts. However, there's obviously more to it than that, which I'm unaware of due to my very limited experience.

Perhaps I should have tried a different approach. Can I build a home server, capable of running 10+ VMware machines simultaneously, for around (or less than) £1000?

jaybee · 22 Mar 2019 at 12:24

Yes you can. Also consider ECC.

BongoHunter · 22 Mar 2019 at 13:25

Boxfreshchris said:
Thankyou. See this is what I meant when I said I wasn't comfortable with enterprise grade equipment, I've never used it before, but I'm comfortable with consumer grade kit. I was originally recommended a Threadripper based build, but consumer level CPU's like this, and the motherboards & Ram that go with them, seem overly expensive when compared to their enterprise counter parts. However, there's obviously more to it than that, which I'm unaware of due to my very limited experience.

Perhaps I should have tried a different approach. Can I build a home server, capable of running 10+ VMware machines simultaneously, for around (or less than) £1000?

I would think so

An 8 core 16 thread Ryzen 1700 is £140
64GB of DDR4 is about £360
Motherboard for say £80
Cheap as chips £30 GPU
Decent PSU for £70

Spend the rest on a cheap case and SSDs

Edit: RAM will be the limiting factor, will the VMs use 64GB simultaneously?

Avalon · 22 Mar 2019 at 13:47

What are the VM workloads anticipated to be? Any specific usage/requirements we need to know about?

Boxfreshchris · 22 Mar 2019 at 22:07

Avalon said:
What are the VM workloads anticipated to be? Any specific usage/requirements we need to know about?

I thought I posted this reply early, but forgot to actual click 'post reply'.

Flash and HTML5, they'll be running webpage bots. Its probably unlikely all 10 will be going balls out at once, and I'll only be running 5 for a while yet, but I want the ability to run all 10 if needs be.

BongoHunter · 22 Mar 2019 at 22:24

Boxfreshchris said:
I thought I posted this reply early, but forgot to actual click 'post reply'.

Flash and HTML5, they'll be running webpage bots. Its probably unlikely all 10 will be going balls out at once, and I'll only be running 5 for a while yet, but I want the ability to run all 10 if needs be.

You can probably get away with VMs with around 4GB of RAM for that assuming they are windows, so any box with 64GB of ram total should be more than enough.

Armageus · 22 Mar 2019 at 22:31

Boxfreshchris said:
Say I wanted to build a home server for running multiple Virtual Machines using VMware

Does it have to be VMWare/ESXi?

Hyper-V is another option (and works quite well with Windows Admin Centre installed so you can manage it via a webpage), or worth looking at Proxmox (particularly if you end up with older enterprise kit - it's based on a Linux distro so has much wider hardware support)

agw_01 · 22 Mar 2019 at 22:33

Not all rack servers are some noisy energy guzzling behemoth.

The HP DL380 G7 is quiet and quite power efficient. I used to run one 24/7 with dual X5650s, 128GB RAM and 4x 300GB 10K SAS discs. Configured for lower power mode it would use about 150w. Very quiet unless running non HP PCI-E adapters which would ramp up the fan speed a bit. It did get a bit warm though with the fans at low speed so make sure you have adequate ventitation.

I used to run 17 VMs without too much of an issue. Disk performance was the main bottleneck so perhaps replacing the discs with some SSDs would be beneficial.

Avalon · 23 Mar 2019 at 00:50

agw_01 said:
Not all rack servers are some noisy energy guzzling behemoth.

The HP DL380 G7 is quiet and quite power efficient. I used to run one 24/7 with dual X5650s, 128GB RAM and 4x 300GB 10K SAS discs. Configured for lower power mode it would use about 150w. Very quiet unless running non HP PCI-E adapters which would ramp up the fan speed a bit. It did get a bit warm though with the fans at low speed so make sure you have adequate ventitation.

I used to run 17 VMs without too much of an issue. Disk performance was the main bottleneck so perhaps replacing the discs with some SSDs would be beneficial.

G7... Surely that's completely inappropriate.
1. HP = Paywalled drivers/BIOS updates - yes I am aware (through necessity) of workarounds, but why should you have to work around anything to get a BIOS update for a server you've paid for? Dell/Fujitsu/Supermicro/Intel don't.
2. Hardware support sucks, even moving certified hardware between servers of the same generation can result in maximum fan RPM just because HP. Putting anything not specifically certified for your server model/bios version just results in a horrible situation - not so much of a problem in a DC environment, but pretty annoying in a home environment unless you are deaf or like the sound of a hairdryer.
3. As already pointed out, much of the the Xeon 5 series is not supported in current ESXi which the op specified, it can be bodged, but again probably not a good idea.

Op, one of the points raised by Armageus is quite relevant, ESXi is basically the industry standard, for commercial experience/homelab usage it's where your time (and likely money) may be best spent. If it's literally for you to screw about with then Proxmox is great for free usage, but has some frustrating and long standing issues (Intel Pro 1000 driver anyone? Yes I know that's a host OS issue, not a Proxmox specific issue, but OOTB support for PFSense?), Hyper-V is OK, but the only time you're likely to find it in business is when Windows and 'free' are a priority, that's not a bad thing, but it's quite rare to find in larger commercial environments.

agw_01 · 23 Mar 2019 at 01:05

OP was asking for a home server with lots of cores, loads of RAM and wasn't concerned by power usage or heat output.

I read the first few posts where people were moaning about how enterprise kit is loud or power hungry, I just posted a couple of comments about my experience with a cheap enterprise grade server where it was neither loud or power hungry. What more do you expect for a £150 rack server. I'd say it offered very good performance for the price.

I didn't suggest the OP go out and buy a G7, I was simply offering my experience. I'm well aware of the shortfalls of the G7 and old Xeon CPUs. Hell, I'd love to build a nice new modern lower power server with lots of cores.

Hardly 'completely inappropriate' given the OP.

Sparky__H · 23 Mar 2019 at 09:01

Depends if you want computing power, efficiency, price or cool and quiet.

Supermicro X10SDV-6C+-TLN4F ([email protected]) - £650
M393A4K40BB1-CRC (32GB ECC) x 4 - £800 (128GB)

For value its hard to out price a ryzen system but they are capped at 64GB ram where the above is designed for dozens of VMs while not been massive/power hungry at 40w!!!!.

At the moment I'm using the Dell T20 which IMHO is the pinnacle of size,noise,power and cost.

Boxfreshchris · 23 Mar 2019 at 10:29

At the moment I'm leaning towards a Ryzen based build. It might not offer the best value for money, but i'm just a lot more comfortable with consumer grade equipment, I know I'll have hardware/software compatibility, and it'll be a lot easier for me to build/diagnose/upgrade/sell on.

I can put together a 1700x build for around £1000, or a Threadripper 2920x build for £1600, both with 64GB ram and SSD's running the show. No ECC ram though, unfortunately its expensive and limits motherboard choice a little too much.