• Best practices for creating a VM
• Persistent memory (PMem) via virtual disks

Since version 6.7 of VMware ESXi, you have the option of using or simulating persistent memory (PMem) (for testing purposes only) to improve the performance of specific compatible virtual machines.

1. What is persistent memory (PMem) and where can I find it ?
2. How to use persistent memory (PMem) ?
3. Simulate persistent memory (vPMem) on VMware ESXi (via SSH)
4. Simulate persistent memory (vPMem) on VMware ESXi (via GUI)
5. Reserved memory and PMem datastore created
6. Information about actual persistent memory modules
7. Create a new virtual machine with an NVDIMM (persistent memory)
8. Information about the created virtual machine
9. Installing Windows Server 2016
10. Partition the persistent memory disk (PMem)
11. Performance comparison
12. Persistent memory retained when restarting a VM
13. Restart the VMware ESXi host
14. Virtual persistent memory (PMem) erased
15. Replace erased NVDIMM
16. Lost data
17. Disable simulated persistent memory

1. What is persistent memory (PMem) and where can I find it ?

Persistent memory (PMem) is non-volatile memory (NVM) that has much lower storage latency than when using a virtual hard drive stored on a physical hard drive or a host's RAID volume.

Persistent memory is available as NVDIMM (Non-Volatile Dual In-line Memory Module) modules that look like regular RAM modules.
The main advantage is the performance when writing and accessing the data contained therein.
But, unlike RAM modules, the data contained in these NVDIMMs (persistent memory) will not be lost when these RAM modules are unpowered.
Indeed, these RAM modules also have flash memory chips in which the data will be stored when the server is stopped. And this same data will then be restored in the RAM chips when it starts up.

These persistent memory NVDIMMs are created by Intel : Intel® Optane™ Persistent Memory.
VMware vSphere 6.7 and later support Intel Optane PMem 100 series modules and from version 7.0U2, Intel Optane PMem 200 series modules are also supported.
Source : vSphere Support for Intel's Optane Persistent Memory (PMEM) (67645).

At Dell, you can find an OEM version of the Intel Optane PMem 200 series modules for your Dell professional server : PowerEdge R450, R550, ...

At HPE (Hewlett Packard Enterprise), you will also find OEM versions of these Intel Optane Pmem 100 series and 200 series modules for your HPE ProLiant server.

Finally, be aware that this type of module is only compatible with professional servers.
Having a high-performance computer (used in a learning environment, for example) will therefore not be sufficient.
However, for testing purposes only, you will see that it's possible to simulate persistent memory by using a portion of your computer's RAM as persistent memory.
In detail that the data contained in your persistent memory will be lost in this case since you will have created "persistent" memory virtually on RAM modules (which are erased when they are no longer powered).

2. How to use persistent memory (PMem) ?

Persistent memory (PMem) can be used in 2 different ways depending on the guest operating system used :

• vPMem (Virtual Persistent Memory) : if the guest operating system (OS) supports NVM technology, you will be able to expose this persistent memory to it through virtual NVDIMMs.
  Which is possible since Windows Server 2016 and Windows 10 (version 1607). For Linux, this is notably supported on CentOS 7.4 : vSphere - Architecture.
  Note that this requires at least version 14 (ESXi 6.7) of virtual hardware for the virtual machine.
• vPMemDisk (Virtual Persistent Memory Disk) : if the guest operating system (OS) doesn't support this NVM technology, then you will need to create a persistent memory virtual hard disk which will be stored in the PMemDS datastore and which will be exposed via a virtual SCSI controller.

Note that persistent memory will be reserved when you create the desired virtual machine, persistent memory virtual hard disk, or NVDIMM (depending on which case you are in).

For more information, refer to the "Persistent Memory" page of the official VMware documentation.

3. Simulate persistent memory (vPMem) on VMware ESXi (via SSH)

If you don't have real servers (Dell, HPE, ...) and you want to learn how to use this persistent memory, here is how to simulate "persistent" memory (vPMem) using a part of the random access memory (RAM) of your host server.

Note that this technique should only be used for testing purposes and not in production.
Indeed, this virtual "persistent" memory (vPMem) will be erased automatically when shutting down or restarting your VMware ESXi host, unlike real persistent memory (PMem) which would be kept thanks to their flash memory chips.

As you can see, in our case we have 256 GB of random access memory (RAM) at the moment.

To reserve some of your RAM as persistent memory, you will need to connect from the command line (via SSH) to your VMware ESXi host.
To do this, you must first enable the "TSM-SSH" service of VMware ESXi, as well as change its startup policy so that it starts automatically with the host.

Then, connect in SSH to your VMware ESXi host.
On Windows, you can download "PuTTY" which is free and very light.

By default persistent memory is not configured, as you can check with this command :

Bash

esxcli system settings kernel list -o fakePmemPct

Plain Text

Name         Type   Configured  Runtime  Default  Description
-----------  -----  ----------  -------  -------  ------------------------------
fakePmemPct  uint8  0           0        0        Amount of fake persistent
                                                  memory (in pct of all volatile
                                                  memory)

To reserve some of your random access memory (RAM) as persistent memory, you will need to use the command below specifying what percentage of your random access memory (RAM) you want to reserve for use as persistent memory.
In our case, we have 256 GB of random access memory (RAM), so we have reserved 25% of it. Which is equivalent to 64 GB.

Bash

esxcli system settings kernel set -s fakePmemPct -v 25

To verify that this reservation as persistent memory has been made, use the command below :

Bash

esxcli system settings kernel list -o fakePmemPct

As you can see, the "fakePmemPct" parameter has been configured correctly (Configured = 25).
However, a restart of your VMware ESXi host will be necessary to be able to use it, because for the moment the "Runtime" column is at 0.

Plain Text

Name         Type   Configured  Runtime  Default  Description
-----------  -----  ----------  -------  -------  ------------------------------
fakePmemPct  uint8  25          0        0        Amount of fake persistent
                                                  memory (in pct of all volatile
                                                  memory)

Restart your VMware ESXi host from its web interface (VMware Host Client) or via SSH by typing this command :

Bash

reboot

Then, verify that persistent memory has been created using the command below.

Bash

esxcli system settings kernel list -o fakePmemPct

This time, the "Runtime" column is worth 25 and not 0.

Plain Text

Name         Type   Configured  Runtime  Default  Description
-----------  -----  ----------  -------  -------  ------------------------------
fakePmemPct  uint8  25          25       0        Amount of fake persistent
                                                  memory (in pct of all volatile
                                                  memory)

Creating persistent memory on your VMware ESXi host will also cause a "PMemDS-[unique identifier]" datastore to be created.
To see it, use the command below.

Bash

esxcli storage filesystem list

As you can see, there is indeed a "PMemDS-[unique identifier]" datastore whose mount point is "/vmfs/volumes/pmem:[another unique identifier]".

Plain Text

Mount Point                                             Volume Name                                  UUID                                      Mounted  Type             Size           Free
------------------------------------------------------  -------------------------------------------  ----------------------------------------  -------  ------  -------------  -------------
/vmfs/volumes/609fbb78-44142142-8cea-d45d64be91fe       datastore1                                   609fbb78-44142142-8cea-d45d64be91fe          true  VMFS-6   504121786368   498797117440
...
/vmfs/volumes/pmem:612243b5-ee848161-9bd4-d45d64be91fe  PMemDS-00a4f7f2-9594-6a47-9137-5c2ddd1621ca  pmem:612243b5-ee848161-9bd4-d45d64be91fe     true  PMEM      68648173568    67272441856

4. Simulate persistent memory (vPMem) on VMware ESXi (via GUI)

Although it's possible to simulate persistent memory (vPMem) using the commands mentioned above, you can also do it via the web interface of your VMware ESXi hypervisor.
To do this, go to : Host -> Manage -> System -> Advanced settings.

In the box at the top right of the table, simply enter "fake" to find the "VMkernel.Boot.fakePMemPct" advanced setting.
Then, select this "VMkernel.Boot.fakePMemPct" setting and click on : Edit option.

Specify the percentage of random access memory (RAM) you want to reserve for use as "persistent memory", then click : Save.

The "Successfully changed VMkernel.Boot.fakePMemPct" message appears.

After changing this setting, restart your VMware ESXi host.

Confirm the host reboot.

5. Reserved memory and PMem datastore created

After reserving a portion of your RAM as persistent memory and restarting the VMware ESXi host, you will see that your VMware ESXi host has less RAM than before.
However, you will also see that VMware ESXi now detects the presence of persistent memory.

Important : as a reminder, the use of the above technique should only be used for testing purposes and not in production.
This is because in this case, since the persistent memory data will be physically stored in your random access memory (RAM) modules, these data will be lost during shutdown or restart of the VMware ESXi host.
Unlike using real persistent memory modules where this would not be the case.

If you go to the "Storage" section of VMware ESXi, you will find the new "PMemDS-[unique identifier]" datastore created automatically by VMware ESXi and which corresponds to the amount of persistent memory present in your server (or the part of the RAM that you have reserved as "persistent" memory, in this case).

If you click on the name of this datastore "PMemDS-[unique identifier]", you will see that the type of this one is "PMEM" and not "VMFSx".
Additionally, you will see that its mount point is "/vmfs/volumes/pmem:[another unique identifier]".

Because we have simulated persistent memory usage instead of using real persistent memory, the configuration of persistent memory will be lost every time the VMware ESXi host is shut down or restarted.
This means that each time you start or restart your VMware ESXi host, this "PMemDS" datastore will disappear by itself and a new "PMemDS" datastore will be recreated automatically.
The only problem is that its name and location (mount point) will have changed. This means that if virtual NVDIMMs or persistent memory virtual hard disks were still attached to your virtual machines, they will not start again until they have been reconfigured to use this new "PMem-DS-[unique identifier different from the previous one]" datastore.