Home Server for 20-40 VMs

JakeyyB · 30 Mar 2021 at 13:20

Hi folks.

I'm looking to build a new PC that will run 24/7 for 20-40 VMware VMs.

The VMs themselves will only be used for HTML5 stuff on web browsers, so nothing too intensive. Will be 3gb RAM per VM.

The idea is that I'll remote into the server from my main PC so I can access the different VM's quickly.

I've built myself a parts list from some research but I'm fairly new to this stuff so want to run it by some more experienced people first.

I'm pretty sure this is already a bit overkill, but allows room for more VM's in the future if needed.

My main concern is the RAM. Someone else told me I'd be better off getting an ECC RAM kit rather than purchasing RAM separately, but I'm not sure if this is actually required for my needs? Was also told to get either 4 or 8 RAM sticks rather than 6.

The GPU is from a previous build as there's no others available for obvious reasons, but as I won't be gaming it should be fine anyway.

Appreciate any advice.

Journey · 30 Mar 2021 at 13:27

It depends how much you care about stability and data loss.

You can look at the Micron MTA18ASF4G72AZ-3G2B1 modules which are 32GB each ECC Unbuffered DIMMs and run at 3200MHz, the stock timings are pretty poor but they can be tightened easily. I've used it on two different builds and it was great, it's also pretty darn cheap so that is also a bonus. You want multiples of 4x RAM modules for Quad channel configuration

Rest of your parts look fine, although I'd probably go for PCI-E 4.0 M.2 drives if you are spending all that money, more IOPS = more better, and the extra speed can't be a bad thing.

JakeyyB · 30 Mar 2021 at 15:40

Journey said:
It depends how much you care about stability and data loss.

You can look at the Micron MTA18ASF4G72AZ-3G2B1 modules which are 32GB each ECC Unbuffered DIMMs and run at 3200MHz, the stock timings are pretty poor but they can be tightened easily. I've used it on two different builds and it was great, it's also pretty darn cheap so that is also a bonus. You want multiples of 4x RAM modules for Quad channel configuration

Rest of your parts look fine, although I'd probably go for PCI-E 4.0 M.2 drives if you are spending all that money, more IOPS = more better, and the extra speed can't be a bad thing.

Thanks for the reply.

That RAM looks ideal, I guess it's probably worth getting to increase stability, especially as I want this thing running 24/7 without many hiccups.

I was going to get 196gb RAM purely because I think I'm going to be using around 120gb to start with and that would have left me some wiggle room. If I get 8x of the Micron modules then it's hugely overkill, but I guess isn't much more expensive than what I was spending anyway.

Yeah good point on the M.2 drives, makes sense to upgrade those to future proof the machine.

The Micron RAM is fine with the threadripper I take it?

Thanks again

Journey · 30 Mar 2021 at 15:47

JakeyyB said:
The Micron RAM is fine with the threadripper I take it?

As I said, I have had zero issues on the two builds I used it with. You can always check the QVL list for the motherboard, or e-mail them and ask about compatibility, I used them in ASRock and ASUS boards.

Armageus · 30 Mar 2021 at 15:48

JakeyyB said:
I was going to get 196gb RAM purely because I think I'm going to be using around 120gb to start with and that would have left me some wiggle room. If I get 8x of the Micron modules then it's hugely overkill, but I guess isn't much more expensive than what I was spending anyway.

You probably won't even need anywhere near that much RAM - if all the VMs are running similar images/workload, Memory Sharing could be used to reduce that amount massively

https://4sysops.com/archives/vmware-memory-optimization/

Edit:
Depending on your noise/space/power usage requirements, then you may even be better looking at some used enterprise servers that are on the ESXi compatibility lists etc - you could get several rackmount HP DL360p Gen 8 servers (or Dell equivalent) for the cost of that build, giving you more total RAM/Cores etc

JakeyyB · 30 Mar 2021 at 16:36

Armageus said:
You probably won't even need anywhere near that much RAM - if all the VMs are running similar images/workload, Memory Sharing could be used to reduce that amount massively

https://4sysops.com/archives/vmware-memory-optimization/

Edit:
Depending on your noise/space/power usage requirements, then you may even be better looking at some used enterprise servers that are on the ESXi compatibility lists etc - you could get several rackmount HP DL360p Gen 8 servers (or Dell equivalent) for the cost of that build, giving you more total RAM/Cores etc

I have looked at some older enterprise workstations that would probably be ok, but unfortunately noise/space is an issue atm hence going down this route.

BongoHunter · 30 Mar 2021 at 18:00

What would the VMs actually be doing - just sitting there with a single browser tab open?
If you can run Linux VMs then you might find you could get away with a lot less memory per machine - maybe as little as 1GB per machine.

JakeyyB · 30 Mar 2021 at 18:38

BongoHunter said:
What would the VMs actually be doing - just sitting there with a single browser tab open?
If you can run Linux VMs then you might find you could get away with a lot less memory per machine - maybe as little as 1GB per machine.

Yeah will just be that primarily. So Linux certainly an option. Would allow me to put half the RAM in.

wesley · 30 Mar 2021 at 21:30

How will the hard drives setup will be? Running 20+ VMs at same time will require very high I/O speed.

Armageus · 30 Mar 2021 at 21:37

wesley said:
How will the hard drives setup will be? Running 20+ VMs at same time will require very high I/O speed.

Not if they are all idle?

jaybee · 30 Mar 2021 at 22:11

Maybe spec out a single VM first to assist with calculating what you will need? Does it have to be an entire VM for a single web browser? I guess the answer is yes to make it isolated for whatever your requirement is. Will they def run win 10?

I.e. 50gb disk
4gb ram
2 threads each

Multiply all by 40
2tb M2 disk minimum divided up. Probably best to split these up into as many as the board will take for I/o spread.

160gb ram required so yes 64gb X 3

CPU choice seems fine. That's 48 threads ? Giving 2 per vm. And leaving enough to run the main host.

Journey · 30 Mar 2021 at 22:18

jaybee said:
Multiply all by 40
2tb M2 disk minimum divided up. Probably best to split these up into as many as the board will take for I/o spread.

I usually use a decent 16x PCI-E to 4x M.2 adapter in these circumstances, means you can keep individual drives and keep them cool.

jaybee · 30 Mar 2021 at 22:29

Journey said:
I usually use a decent 16x PCI-E to 4x M.2 adapter in these circumstances, means you can keep individual drives and keep them cool.

Yeah or just run some sata el cheap 512 GB ssds. I doubt you would notice the difference in speed using some tabs in chrome.

Journey · 30 Mar 2021 at 22:32

jaybee said:
Yeah or just run some sata el cheap 512 GB ssds. I doubt you would notice the difference in speed using some tabs in chrome.

SATA SSD is the same price as NVME storage these days.

Armageus · 30 Mar 2021 at 22:37

Journey said:
SATA SSD is the same price as NVME storage these days.

But much easier to run large number of drives

JakeyyB · 30 Mar 2021 at 23:17

Yeah the reason I went 2 x 1TB M.2 is to split the VM's over them as I've read about people having issues if you have all of them running off of 1 SSD.

Went M.2 as like Journey said they're not far off sata in terms of price these days.

Could always add more SATA later down the line if needed?

So would I be okay getting 32gb x6 DIMMs, or is the added benefit of quad channel configuration worth me getting 8x DIMMs? Obviously I only really need 6 but not sure if that's going to have a big impact?

Quartz · 30 Mar 2021 at 23:19

JakeyyB said:
The VMs themselves will only be used for HTML5 stuff on web browsers, so nothing too intensive.

Exactly what are you trying to achieve here? Unix is inherently multi-user and can keep users apart so you might not require VMs at all. Microsoft Terminal Server can dynamically spawn sessions so you don't need to worry if you have enough VMs - or too many.

Deleted member 77746 · 30 Mar 2021 at 23:35

64GB ram with as many private browser windows as you need? Far cheaper.

Spleen Sauce · 31 Mar 2021 at 09:42

Dumb question, but could you not achieve this with AWS or Azure (or another public cloud offering)? Would allow you to easily scale, avoid the upfront costs, etc.

Semple · 1 Apr 2021 at 17:45

Bouton Aide said:
64GB ram with as many private browser windows as you need? Far cheaper.

Would presume he needs a separate IP address per browser instance.

Spleen Sauce said:
Dumb question, but could you not achieve this with AWS or Azure (or another public cloud offering)? Would allow you to easily scale, avoid the upfront costs, etc.

If it's short term, then this would certainly be the cheaper option. But if someones splashing 3k on a system, i'm suspecting it to be more of a long use project.