ESXi 6 Memory Woes

Hi guys,

I recently upgraded a single ESXi host from 64GB to 128GB RAM.

ESXi detects the RAM, however even with all VM's powered off after a reboot, 69GB RAM is being used.. what on earth is using this memory?



Cheers

Under configuration check 'system resource allocation', someone could have got a bit carried away with how much memory ESXi reserves....
(For reference our 128GB hosts have got a reservation of 4206MB)

Alternatively, have you got any memory reservations set for VM's?
I guess ESXi might reserve the memory for a machine even if it's switched off - we don't have memory constraints so haven't got this set for me to check for you.

Might be a but in the metrics or something, give the host or vCenter a reboot and see if it clears up.

It's not stuck in a "pool" of resources is it?

As in you assigned it to a pool and it's reserved in there for when the VM's come online?

Not sure, but I think you need 1.5 times memory size space available on the ESX memory page disk. So if the Datastore configured as the default page memory location is full this may restrict the memory you can use?

Little_Crow said:

Under configuration check 'system resource allocation', someone could have got a bit carried away with how much memory ESXi reserves....
(For reference our 128GB hosts have got a reservation of 4206MB)

Alternatively, have you got any memory reservations set for VM's?
I guess ESXi might reserve the memory for a machine even if it's switched off - we don't have memory constraints so haven't got this set for me to check for you.

Click to expand...

System resource allocation shows memory as 4GB and CPU at 100Mhz.

Checked through the VM's and I have no reservations set. Should there be limits set for each VM though? - most have the 'unlimited' check box.

I wondered if that might be the case regarding powered off VM's - anyone able to confirm if ESXi sets aside RAM even if the machines are powered off?

traveyb said:

Might be a but in the metrics or something, give the host or vCenter a reboot and see if it clears up.

Click to expand...

I've rebooted the host twice now. Same issue, 69GB RAM being used straight off the bat without any VM's powered on. The VCSA is powered off for the time being, so don't think it's related to vCenter.

swain90 said:

It's not stuck in a "pool" of resources is it?

As in you assigned it to a pool and it's reserved in there for when the VM's come online?

Click to expand...

There are no resource pools setup, as far as I can tell.

ASE001 said:

Not sure, but I think you need 1.5 times memory size space available on the ESX memory page disk. So if the Datastore configured as the default page memory location is full this may restrict the memory you can use?

Click to expand...

I have two datastores, the first has 447GB free the second has 270GB free.

I noticed whilst looking at Configuration > Memory it shows

Total: 131070.5MB
System: 66622.5MB
Virtual Machines: 64448.0MB

I've never seen this in ESXi. It looks like the system has allocated it rather than ESXi.

I take it this isn't a run of the mill server? Has it been allocated for video memory or some sort of cache maybe?

Do you have CLI access to the host?

Post the output from esxtop - (press m to show memory usage)

Pretty standard server spec's, nothing special.

Supermicro board, LSI-9260-8i card, 128GB RAM

esxtop with M option.

There are a few VM's running, but the issue occurs with all VM's powered off too.

Not got memory mirroring enabled in the BIOS have you?

Just having a look at the manual for the board now (Supermicro X7DWN+)

This section looks applicable:-

Memory Branch Mode
This option determines how the two memory branches operate. System address
space can either be interleaved between the two branches or Sequential from
one branch to another. Mirror mode allows data correction by maintaining two
copies of data in two branches. Single Channel 0 allows a single DIMM population
during system manufacturing. The options are Interleave, Sequential, Mirroring,
and Single Channel 0.

Gonna have a look at the BIOS settings and see if this is the case

Memory Branch Mode was already set to interleaved, so it doesn't appear to be that.

Would the VMWare client still report 128GB if this was the case though?

Gonna try a BIOS flash next..

Misan said:

Pretty standard server spec's, nothing special.

Supermicro board, LSI-9260-8i card, 128GB RAM

esxtop with M option.

There are a few VM's running, but the issue occurs with all VM's powered off too.

Click to expand...

https://communities.vmware.com/docs/DOC-9279

Q: Why is total not the same as RAM size plugged in my memory slots?
A: This is because some memory range is not available for use. It is fine, if the difference is small. If the difference is big, there might be some hardware issue. Check your BIOS.

Click to expand...

Sorry haven't seen this happen before so apart from reseating sticks, not much else I can suggest.

Flashing the BIOS fixed the problem. A heads up for anyone running a Supermicro X7DWN+ with 128GB RAM.

Many thanks for all the help guys, much appreciated