Hyper-threading on ESXi - include in CPU capacity calcs?

#Chri5# · 2 Aug 2019 at 10:03

I'm putting together a design for a small vSphere / ESXi cluster which will need to run 6 or 9 VMs. The spec' required for each VM is 16 vCPUs and 32GB RAM.

The Xeon Silver 4216 is a 16C/32T CPU, so a dual socket 2U box would give me 32C and 64T. Three of the VMs running on a single host would be 48 vCPU. Assuming a heavy load, three VMs would exceed the physical core count and I'm wondering if HT would be good or not? I've seen a figure somewhere that HT can add 30% performance.

Vince · 2 Aug 2019 at 10:45

#Chri5# said:
I'm putting together a design for a small vSphere / ESXi cluster which will need to run 6 or 9 VMs. The spec' required for each VM is 16 vCPUs and 32GB RAM.

The Xeon Silver 4216 is a 16C/32T CPU, so a dual socket 2U box would give me 32C and 64T. Three of the VMs running on a single host would be 48 vCPU. Assuming a heavy load, three VMs would exceed the physical core count and I'm wondering if HT would be good or not? I've seen a figure somewhere that HT can add 30% performance.

Over subbing cores isn't the end of the world, are you likely to have load across all servers pushing the cpu constantly? Also have you considered EPYC? I know hp are currently loading Rome based epyc cpu's into their systems for sale so a single socket rome might be a considerably better option.

#Chri5# · 2 Aug 2019 at 10:59

Yes, there should be data going into the VMs constantly (if not, something else is broken!). Not sure how the traffic coming in is load balanced across the cluster of VMs at the moment though.

Will have a look at the Rome Epyc chips, availability could be an issue.

Avalon · 2 Aug 2019 at 13:31

vCPU includes HT, so a dual socket 16c/32t aka 64vCPU, which means up to 4 VM's per box, not 3. Given you're only counting physical cores here, you're 25% under capacity per box, not 50% over as you suggest. Given the budgets involved here and you not seeming overly familiar with the terminology, is this something you could get some advice from the vendor on? If you supply them with the requirements, they'll be happy to quote you.

#Chri5# · 2 Aug 2019 at 14:17

I've done small clusters before but run out of disk / RAM before becoming CPU constrained.

Reading around I've found articles like this one:

http://www.vmwarebits.com/content/vcpu-and-logical-cpu-sizing-hyper-threading-explained (my bold)

This entire article can be summarized in one sentence: never assign more vCPU's than the number of physical cores your host has. You could also argue that you should leave some headroom in not maximizing the number of physical cores in your VM's but that is beyond the scope of this article

Dated 2015 so old but illustrates my question / concern. I have pinged an email over to a vendor SE to double check their stance on physical vs logical cores.

Vince · 2 Aug 2019 at 15:17

#Chri5# said:
Yes, there should be data going into the VMs constantly (if not, something else is broken!). Not sure how the traffic coming in is load balanced across the cluster of VMs at the moment though.

Will have a look at the Rome Epyc chips, availability could be an issue.

See my new thread under this section - I have pricing and good availability on Rome.

Avalon · 2 Aug 2019 at 16:34

#Chri5# said:
I've done small clusters before but run out of disk / RAM before becoming CPU constrained.

Reading around I've found articles like this one:

http://www.vmwarebits.com/content/vcpu-and-logical-cpu-sizing-hyper-threading-explained (my bold)

Dated 2015 so old but illustrates my question / concern. I have pinged an email over to a vendor SE to double check their stance on physical vs logical cores.

While I agree with having more being a much better option, it's a moot point when person specifying requirements has stated vCPU and not physical cores.

khrall · 3 Aug 2019 at 00:04

A couple of things to look out for when speccing up this cluster:

Memory population performance with Intel Xeon Scalable systems, if your not correctly populating the channels, you can have a huge hit on performance:
https://www.thomas-krenn.com/en/wiki/Optimize_memory_performance_of_Intel_Xeon_Scalable_systems

Enable hyperthreading, but think of it as a bonus, don't count on it in your calculations, exclude it from your ratios.
https://www.vmware.com/content/dam/...nter-server-67-performance-best-practices.pdf

Read up on NUMA, local memory is faster then memory being accessed from the other CPU:
https://itnext.io/vmware-vsphere-why-checking-numa-configuration-is-so-important-9764c16a7e73

And if your going EPYC:
https://frankdenneman.nl/2019/02/19/amd-epyc-and-vsphere-vnuma/

#Chri5# · 5 Aug 2019 at 12:00

khrall said:
A couple of things to look out for when speccing up this cluster:

Memory population performance with Intel Xeon Scalable systems, if your not correctly populating the channels, you can have a huge hit on performance:
https://www.thomas-krenn.com/en/wiki/Optimize_memory_performance_of_Intel_Xeon_Scalable_systems

Enable hyperthreading, but think of it as a bonus, don't count on it in your calculations, exclude it from your ratios.
https://www.vmware.com/content/dam/...nter-server-67-performance-best-practices.pdf

Read up on NUMA, local memory is faster then memory being accessed from the other CPU:
https://itnext.io/vmware-vsphere-why-checking-numa-configuration-is-so-important-9764c16a7e73

And if your going EPYC:
https://frankdenneman.nl/2019/02/19/amd-epyc-and-vsphere-vnuma/

Thanks, especially the DIMM population article which elaborates on a note in the Dell 740 Technical Guide (below). Will see if 192GB can be had in budget, even if it's over spec'd.

Populate six memory modules per processor (one DIMM per channel) at a time to maximize performance.

DustyMiller · 9 Aug 2019 at 23:48

You also have to factor in that when vCPU is assigned to a machine it has to have all of the cores available to assign, so running that many machines with that many vCPU's to pCPU's is not very good as you will be sat in CPU_WAIT issues.

khrall · 12 Aug 2019 at 23:24

Whilst the below is based on the VMC on AWS, the information is relevant to your sizing.
https://docs.vmware.com/en/VMware-C...98/GUID-F3C4C0FA5C36FC67FBF918030728DD22.html

You may not be running NSX & VSAN, so your overheads will be a little lower, but it does show you the effects of sizing your VMs vs physical cores available vs quantity of servers you can run of that size, based on a similar sized workloads.

Are you also sizing for HA, supplying sufficient available resources for a host failure/maintenance?

There are many considerations when sizing, depending on the business requirements.

#Chri5# · 15 Aug 2019 at 09:55

khrall said:
Whilst the below is based on the VMC on AWS, the information is relevant to your sizing.
https://docs.vmware.com/en/VMware-C...98/GUID-F3C4C0FA5C36FC67FBF918030728DD22.html

You may not be running NSX & VSAN, so your overheads will be a little lower, but it does show you the effects of sizing your VMs vs physical cores available vs quantity of servers you can run of that size, based on a similar sized workloads.

Are you also sizing for HA, supplying sufficient available resources for a host failure/maintenance?

There are many considerations when sizing, depending on the business requirements.

Another good link, thanks again.

No NSX, VSAN or HA. Budget at present will cover 3 hosts, so the normal workload will be two VMs per host, plus a small VCSA and one other VM. Will see where the pricing sits when it comes to placing an order, but we could trade memory bandwidth (128GB instead of 192GB) for additional CPU cores (18C Gold instead of 16C Silver ).