VMware ESXi is a hypervisor that efficiently manages the use of your system resources (RAM, CPU, ...).
VMware ESXi has several mechanisms to manage memory efficiently and to reclaim memory when needed.
In the rest of this article, you will be able to understand different notions and the different mechanisms that allow VMware ESXi to make the best use of your physical RAM and to recover RAM when necessary :
Note that the mechanisms mentioned above are used one after the other to make the best use of the RAM of your VMware ESXi host. Then, to be able to recover more and more RAM on it when necessary.
VMware ESXi is a hypervisor that allows excessive memory commitment.
Which means you can allocate more RAM to your virtual machines than the amount of RAM available on your VMware ESXi host.
In the 1st diagram (A), the memory is undercommitted (memory-undercommitted).
Indeed, in the example below :
If you start the 2 VMs with 4 GB of RAM allocated each, the allocated memory will be 8 GB.
Which is less than the amount of RAM (16 GB) available on this host.
The memory is therefore undercommitted (memory-undercommitted) and this therefore doesn't pose any problem.
In the 2nd scheme (B), it can happen that the memory is excessively committed (memory-overcommitted).
Indeed, in the example below :
If you start the 3 virtual machines and they are therefore powered on simultaneously, you will therefore end up with a RAM allocation of 20 GB, while your VMware ESXi host really only has 20 GB.
If your virtual machines are not currently using a lot of RAM, it's quite possible that the memory is still undercommitted (memory-undercommitted). Which is no problem.
In fact, memory is overcommitted when the RAM consumed (actually used) by your virtual machines exceeds the amount of RAM physically available on your VMware ESXi host.
Memory overcommitment is a very nice feature of VMware ESXi because when you allocate RAM to virtual machines, the RAM is very often underutilized.
In effect :
In short, if memory becomes excessively committed, VMware ESXi will reclaim unused RAM from your virtual machines and transfer it to the virtual machines that really need it.
This allows your virtual machines to continue to operate, but you will see later in this article that this will also mean a deterioration in their performance.
Remember that you can reserve RAM, manage RAM allocation priority, and limit the RAM usage of your virtual machines with the "Reservation", "Parts" and "Limit" settings to prevent your virtual machines from running out of RAM in the event that your VMware ESXi host's RAM is overcommitted.
To find out how, go to this page : VMware ESXi 6.7 - Configure your virtual machines settings - Memory (RAM).
However, be careful not to reserve too much RAM at the risk of having the opposite effect.
Indeed, the reserved memory (even when it's not used by the VM concerned) can't be recovered by VMware ESXi to transfer it to another virtual machine which may need more at some point.
Source : Memory Overcommitment - VMware Docs.
Using a proprietary Transparent Page Sharing (TPS) technique, VMware ESXi can efficiently use your physical RAM through its VMkernel.
Indeed, when this TPS feature is enabled (which is not the case by default), the VMware ESXi VMkernel is able to detect memory pages that are identical (in green in the diagram below) and allows your virtual machine(s) to share them.
This saves a lot of memory if your virtual machines use the same operating system, the same applications and/or the same data (documents, images, ...).
Note that TPS can share memory pages on a single virtual machine as well as with several virtual machines whose workload is similar.
Of course, this sharing of memory pages is only possible in read-only mode, which means that if one of the virtual machines concerned must modify a memory page (in red in the diagram below), the VMware ESXi VMkernel will create a copy of this memory page (thanks to copy-on-write) and will modify this private copy.
This corresponds to the page hatched in red on the diagram below. Other virtual machines will continue to use the original read-only shared memory page.
Important : for security reasons, Transparent Page Sharing (TPS) is disabled by default since version 5.0 of VMware ESXi for TPS on a virtual machine and since version 6.0 for TPS between VMs.
Note that Transparent Page Sharing (TPS) is only supported for small memory pages (4 KB).
For 2 MB pages, when necessary, VMware ESXi sometimes fragments these pages to generate 4 KB memory pages and thus be able to share them via TPS.
Finally, keep in mind that this Transparent Page Sharing (TPS), if enabled, will only occur when your VMware ESXi host's free RAM becomes too low.
This avoids consuming unnecessary system resources when your VMware ESXi host's RAM is currently underutilized.
If you want to re-enable Transparent Page Sharing (TPS) on VMware ESXi, access its web interface and go to: Host -> Manage -> System -> Advanced Settings.
Then, look for the "Mem.ShareForceSalting" advanced setting.
As you can see in its description, sharing pages between VMs is only possible when the value of this setting is :
Otherwise, page sharing will not be allowed between VMs. Which is the default case (2 = disabled).
Select this advanced setting and click "Change option".
Specify the value "0" to allow Transparent Page Sharing (TPS) between all virtual machines on your VMware ESXi host and click Save.
The "Successfully changed Mem.ShareForceSalting" message appears.
Important : to have Transparent Page Sharing (TPS) enabled on currently powered-on virtual machines, you will need to :
- switch them off, then start them again.
- or migrate them to another VMware ESXi host and then migrate them to the source VMware ESXi host, if you wish to do so without shutting down your virtual machines.
Thanks to the memory ballooning driver (vmmemctl), your VMware ESXi hypervisor will be able to recover RAM by asking your virtual machines to free the memory pages that they don't use or more at the moment.
In diagram "a" below :
In figure "b" below, this shows what happens when VMware ESXi asks this virtual machine which memory pages it can reclaim :
Then, VMware ESXi will redistribute the reclaimed RAM to the virtual machines that currently need it.
Important : in order for VMware ESXi to be able to recover the memory not used by your virtual machines, it's imperative that the VMware Tools are installed on them.
Indeed, when you install the VMware Tools in a virtual machine, you will see that a memory control driver is installed.
It's this driver that is used for the memory inflation (ballooning) described above.
Without this driver, VMware ESXi will not be able to reclaim unused memory from the guest OS since VMware ESXi can't see its contents (from a technical point of view).
Note that you can change the amount of RAM (in MB) that can be claimed by this driver if desired by configuring the "sched.mem.maxmemctl" setting on the desired virtual machine.
You can also adjust the amount of RAM this driver can reclaim per virtual machine by modifying the "Mem.CtlMaxPercent" advanced setting.
To do this, go to "Host -> Manage -> System -> Advanced settings" and look for the "Mem.CtlMaxPercent" setting.
Then, select it and click : Edit option.
Specify the desired percentage (default : 65) and click Save.
Finally, keep in mind that this memory inflation (ballooning) will only be used when the available RAM is too low on your VMware ESXi host despite the use of transparent page sharing (TPS).
Source : Memory Balloon Driver - VMware Docs.
If the previous mechanisms (TPS and ballooning) don't recover enough RAM on your VMware ESXi host, your hypervisor will attempt to compress the virtual memory pages.
Specifically, VMware ESXi will attempt to compress pages to 2KB or less and store these in the affected VM's compression cache.
Note that this RAM compression is enabled by default on VMware ESXi and that accessing compressed memory is faster than accessing swapped memory. Indeed, the memory which is permuted (swap) is stored on the hard disk.
Source : Memory Compression - VMware Docs.
If the various memory management mechanisms explained previously don't recover enough RAM on your VMware ESXi hypervisor, then it will use system swapping.
On VMware ESXi, System Swap involves moving memory pages from RAM to a swap file stored on the host hard drive.
Warning : this means that performance will be greatly degraded since access to a hard disk is much slower than access to RAM. Especially since VMware ESXi will move memory pages of the virtual machine chosen randomly.
This can significantly slow down the virtual machine depending on the pages that have been moved.
That said, according to VMware, it's not technically possible to know what is in the pages of a virtual machine. Hence the random choice of these memory pages to quickly free up RAM.
More precisely, it's the ".wsp" file created at the start of each virtual machine and erased when it's stopped.
By default, this file is stored on the hard disk of the host in the folder of the virtual machine concerned.
Note that you can change the default location of this swap file (.vswp) globally by going to : Host -> Manage -> System -> Swap.
Then, click on : Edit settings.
In the "Edit swap configuration" window that appears, you will be able to :
For the "Datastore" option, you can, for example, select a datastore corresponding to faster storage (such as an SSD) instead of a classic hard disk so that access to swap files (.vswp) is a bit faster.
Note that you can also change this behavior for each virtual machine individually by selecting it, then clicking on : Edit.
Then, go to the "VM Options" tab.
In the "Advanced" section, you will find a "Swap file location" setting for which you can choose between :
Warning : if you have several VMware ESXi hosts and you want to be able to migrate your virtual machines from one VMware ESXi host to another via vSphere vMotion, it's strongly recommended to store the swap file (.vswp) in a datastore visible by the 2 VMware ESXi hosts.
Otherwise, vSphere vMotion performance will be degraded because pages swapped with a swap file will need to be transferred over the network.
As a reminder, this RAM recovery mechanism is only used as a last resort since this will greatly degrade the performance of the virtual machines concerned.
The goal is to prevent a virtual machine or your host from crashing due to lack of RAM on it.
To learn even more about memory management on VMware ESXi, see VMware's official "Understanding Memory Resource Management in VMware® ESX™ Server" PDF.
® InformatiWeb-Pro.net - InformatiWeb.net 2008-2022 - © Lionel Eppe - All rights reserved.
Total or partial reproduction of this site is prohibited and constitutes an infringement punishable by articles L.335-2 and following of the intellectual property Code.