ESXi 6 Memory Woes

Misan · 22 Jun 2016 at 00:36

Hi guys,

I recently upgraded a single ESXi host from 64GB to 128GB RAM.

ESXi detects the RAM, however even with all VM's powered off after a reboot, 69GB RAM is being used.. what on earth is using this memory?

Cheers

Little_Crow · 22 Jun 2016 at 08:37

Under configuration check 'system resource allocation', someone could have got a bit carried away with how much memory ESXi reserves....
(For reference our 128GB hosts have got a reservation of 4206MB)

Alternatively, have you got any memory reservations set for VM's?
I guess ESXi might reserve the memory for a machine even if it's switched off - we don't have memory constraints so haven't got this set for me to check for you.

traveyb · 22 Jun 2016 at 09:24

Might be a but in the metrics or something, give the host or vCenter a reboot and see if it clears up.

swain90 · 22 Jun 2016 at 10:07

It's not stuck in a "pool" of resources is it?

As in you assigned it to a pool and it's reserved in there for when the VM's come online?

ASE001 · 22 Jun 2016 at 14:45

Not sure, but I think you need 1.5 times memory size space available on the ESX memory page disk. So if the Datastore configured as the default page memory location is full this may restrict the memory you can use?

Misan · 22 Jun 2016 at 15:22

Little_Crow said:
Under configuration check 'system resource allocation', someone could have got a bit carried away with how much memory ESXi reserves....
(For reference our 128GB hosts have got a reservation of 4206MB)

Alternatively, have you got any memory reservations set for VM's?
I guess ESXi might reserve the memory for a machine even if it's switched off - we don't have memory constraints so haven't got this set for me to check for you.

System resource allocation shows memory as 4GB and CPU at 100Mhz.

Checked through the VM's and I have no reservations set. Should there be limits set for each VM though? - most have the 'unlimited' check box.

I wondered if that might be the case regarding powered off VM's - anyone able to confirm if ESXi sets aside RAM even if the machines are powered off?

Misan · 22 Jun 2016 at 15:24

traveyb said:
Might be a but in the metrics or something, give the host or vCenter a reboot and see if it clears up.

I've rebooted the host twice now. Same issue, 69GB RAM being used straight off the bat without any VM's powered on. The VCSA is powered off for the time being, so don't think it's related to vCenter.

Misan · 22 Jun 2016 at 15:26

swain90 said:
It's not stuck in a "pool" of resources is it?

As in you assigned it to a pool and it's reserved in there for when the VM's come online?

There are no resource pools setup, as far as I can tell.

Misan · 22 Jun 2016 at 15:32

ASE001 said:
Not sure, but I think you need 1.5 times memory size space available on the ESX memory page disk. So if the Datastore configured as the default page memory location is full this may restrict the memory you can use?

I have two datastores, the first has 447GB free the second has 270GB free.

I noticed whilst looking at Configuration > Memory it shows

Total: 131070.5MB
System: 66622.5MB
Virtual Machines: 64448.0MB

blueboy2001 · 22 Jun 2016 at 20:33

I've never seen this in ESXi. It looks like the system has allocated it rather than ESXi.

I take it this isn't a run of the mill server? Has it been allocated for video memory or some sort of cache maybe?

Semple · 22 Jun 2016 at 20:46

Do you have CLI access to the host?

Post the output from esxtop - (press m to show memory usage)

Misan · 22 Jun 2016 at 21:00

Pretty standard server spec's, nothing special.

Supermicro board, LSI-9260-8i card, 128GB RAM

esxtop with M option.

There are a few VM's running, but the issue occurs with all VM's powered off too.

blueboy2001 · 22 Jun 2016 at 21:16

Not got memory mirroring enabled in the BIOS have you?

Misan · 22 Jun 2016 at 21:23

Just having a look at the manual for the board now (Supermicro X7DWN+)

This section looks applicable:-

Memory Branch Mode
This option determines how the two memory branches operate. System address
space can either be interleaved between the two branches or Sequential from
one branch to another. Mirror mode allows data correction by maintaining two
copies of data in two branches. Single Channel 0 allows a single DIMM population
during system manufacturing. The options are Interleave, Sequential, Mirroring,
and Single Channel 0.

Gonna have a look at the BIOS settings and see if this is the case

Misan · 22 Jun 2016 at 23:23

Memory Branch Mode was already set to interleaved, so it doesn't appear to be that.

Would the VMWare client still report 128GB if this was the case though?

Gonna try a BIOS flash next..

Gzero · 22 Jun 2016 at 23:56

Misan said:
Pretty standard server spec's, nothing special.

Supermicro board, LSI-9260-8i card, 128GB RAM

esxtop with M option.

There are a few VM's running, but the issue occurs with all VM's powered off too.

https://communities.vmware.com/docs/DOC-9279

Q: Why is total not the same as RAM size plugged in my memory slots?
A: This is because some memory range is not available for use. It is fine, if the difference is small. If the difference is big, there might be some hardware issue. Check your BIOS.

Sorry haven't seen this happen before so apart from reseating sticks, not much else I can suggest.

Misan · 23 Jun 2016 at 16:32

Flashing the BIOS fixed the problem. A heads up for anyone running a Supermicro X7DWN+ with 128GB RAM.

Many thanks for all the help guys, much appreciated