VMware vSphere APIs: array integration (VAAI), you can delegate certain storage-related operations to your storage array if it supports VAAI.

1. What is VAAI (vSphere APIs Array Integration)?
2. VAAI block storage primitives (iSCSI, FC and FCoE)
   1. Atomic Test & Set (ATS)
   2. XCOPY (Extended Copy)
   3. Write Same (Zero)
   4. UNMAP
   5. UNMAP from a guest OS
3. Check storage array compatibility with VAAI
4. Check VAAI support from the VMware vSphere Client
5. Creating a VM without VAAI support
6. Creating a VM with VAAI support
7. Testing the UNMAP primitive
   1. Download HPE StoreVirtual
   2. Deploy HPE StoreVirtual Storage 12.8
   3. Manually download and install the CMC console (optional)
   4. Configure your HPE StoreVirtual Storage server hardware
   5. Create a cluster
   6. Enable space reclamation
   7. Add an iSCSI server to your HPE StoreVirtual Storage server
   8. Download and install the HPE iSCSI Rescan script
   9. Configure the VMware ESXi host iSCSI adapter
   10. Create an iSCSI datastore
   11. Testing UNMAP on a datastore
   12. Testing UNMAP from the guest OS

1. What is VAAI (vSphere APIs Array Integration)?

VAAI provides hardware support to storage providers.
Indeed, storage-related operations (such as cloning or migrating a VM, zeroing the blocks of a virtual disk created in static provisioning, ...) are often costly in terms of resources.
Thanks to VAAI, these operations will be carried out by your storage array. This will reduce the workload on your VMware ESXi hosts.

Note that VAAI can be used by VMware vSphere for all operations that remain internal to a storage array.
So, if you create multiple datastores that are in the same storage array, operations will still be managed by the storage array (thanks to VAAI support).

APIs are made up of several storage primitives that allow certain operations to be performed on the storage array. The more storage primitives your storage array supports, the more the host will be able to delegate some types of storage-level operations.

As you can see in the diagram below:

• without VAAI (left): the host managed reads and writes (via the network for iSCSI) to your array's storage.
• with VAAI (right): the host only needs to ask your array to perform this or that operation for it.
  This also greatly reduces network traffic between your VMware ESXi host and your storage array.

Without VAAI support, all operations performed on storage would be performed by the vSphere VMkernel Data Mover. Which means that the resources of VMware ESXi hosts will be used much more.

Important : originally, since VMware vSphere 4.1, there were only 3 VAAI primitives and this was only supported for block storage (iSCSI, FC and FCoE).
However, since VMware vSphere 5.0, new storage primitives have appeared for NAS and dynamic provisioning of VMware vSphere.

2. VAAI block storage primitives (iSCSI, FC and FCoE)

2.1. Atomic Test & Set (ATS)

When you create a datastore, the typically used file system is VMFS.
In VMFS, many operations require the use of locks to prevent data corruption.
Indeed, given that datastores can be used simultaneously by several hosts, these locks prevent a host from attempting to modify a file or metadata that is being modified by another host.

Without ATS support, VMware vSphere uses SCSI reservations on VMFS volumes as a locking system to prevent 2 hosts from being able to modify the same metadata at the same time.
The downside is that a reservation locks an entire LUN. This can create contention issues if many virtual machines are on the same datastore.

With support for ATS which is an enhanced locking system, locks are managed by the storage array.
Additionally, using ATS, it is possible to modify only a disk sector on a VMFS volume without fully locking the LUN.

2.2. XCOPY (Extended Copy)

Without VAAI support, the cloning or migration of a virtual machine (Storage vMotion) is managed by the vSphere VMkernel Data Mover.
This requires the host to read and write large amounts of data to the storage array. Which uses the resources (CPU and others) of the host, as well as the network between the host and the storage array.

However, if your storage array supports the "XCOPY" storage primitive (which is part of VAAI), your host will only need to ask your array to copy the necessary data.
In other words, your host can ask for example to copy a certain "vmdk" file in its place without having to read it entirely to rewrite it on the same storage bay.
This greatly reduces the workload on your VMware ESXi host, as well as the use of network bandwidth.

2.3. Write Same (Zero)

Without VAAI support, when you create a virtual hard disk with an immediately zeroed static provisioning type, a SCSI command is used to zero all the blocks needed to create it.
Which again consumes CPU resources, as well as the DMA buffer and which also fills up the HBA queue.

With VAAI support, your VMware ESXi hosts will be able to zero a large number of blocks using the SCSI WRITE_SAME command. This avoids unnecessary transfer of data (via the network for iSCSI) between the host and the storage bay.

Note that this primitive helps speed up provisioning-related tasks when:

• virtual hard disk cloning with static provisioning with immediate block zeroing.
• allocation of new blocks for virtual hard disks with thin provisioning.
• initializing previously unwritten blocks for VHDs with static provisioning.

Note that storage arrays do not write all zeros directly to disk.
Some storage arrays use a metadata file. Which can give different performances.

2.4. UNMAP

The VAAI "UNMAP" primitive allows a VMware ESXi host to notify the storage array of freed space after migrating or deleting a virtual machine. UNMAP is also known as "Dead Space Recovery".

Previously, under vSphere 5.0, this UNMAP operation was performed automatically (for example: just after migrating a virtual machine). However, this caused performance issues.
Then this operation had to be done manually.
However, since vSphere 6.5, this UNMAP primitive is automatic again if you format your datastore in VMFS6. Indeed, for VMFS6 datastores, an automated UNMAP bot works continuously in the background to inform the array of the space freed on them.

Source : Space Reclamation Requests from VMFS Datastores - VMware Docs.

2.5. UNMAP from a guest OS

Since vSphere 6.0, a guest OS can also notify an array of freed space when it is installed on a dynamically provisioned virtual hard disk.

Native support for UNMAP was limited to Windows Server 2012 R2. Linux distributions could not perform this UNAMP operation because the SCSI version used by the virtual hardware was version 2 and the minimum required version was version 5.
Thanks to SPC-4 support that appeared with vSphere 6.5 for virtual hardware, Linux distributions are now able to perform its UNMAP operations.

Important : it is recommended that the block size used for guest operating system partitions be 1 MB for UNMAP to function properly.

Source : VMware vSphere APIs: Array Integration (VAAI) | VMware.

3. Check storage array compatibility with VAAI

To check if VAAI is supported by your storage array, as well as find out which VAAI primitives it supports, you can access the "VMware Compatibility Guide - Storage/SAN" page.

For the example, we will search for the "HPE LeftHand" storage bay:

• Partner Name : Hewlett Packard Entreprise
• Keyword : LeftHand

Click on the name of your storage array.
In our case: Lefthand.

Choose your VMware ESXi version from the list at the top left and see the information about the firmware version it has.
In our case, we can see that it supports all VAAI primitives, including UNMAP:

• UNMAP
• HW Assisted Locking = ATS
• Full Copy = XCOPY
• Block Zero = Write Same (Zero)

4. Check VAAI support from the VMware vSphere Client

To check if your storage array supports VAAI, select your VMware ESXi host and go to "Configure -> Storage -> Storage Adapters".
Next, select the "iSCSI Software Adapter" and go to the "Devices" tab (at the bottom of the page).

If it says "Supported" in the "Hardware Acceleration" column, VAAI is supported by the storage bay where the desired iSCSI drive is located.
Otherwise it will say "Not Supported".

Important : the fact that it is indicated "Supported" does not mean that UNMAP (which is one of the functionalities of VAAI) is also supported.
To verify this, you will need to log in via command line (SSH) to your VMware ESXi host.

As you can see :

• Hardware Acceleration (VAAI) is supported for iSCSI drives created on StarWind Virtual SAN.
• Hardware acceleration (VAAI) is not supported for iSCSI drives created on Windows Server.

Source : Does W2K12R2 Storage Server integrate into VMware with VAAI?

To find out exactly which VAAI features are supported, go to "Configure -> System -> Services" and start the "SSH" service.

Connect to your VMware ESXi host using SSH.

Type the command:

Plain Text

esxcli storage core device vaai status get

As you can see :

• VAAI features (ATS, Clone and Zero) are supported for our "eui..." iSCSI disks created on StarWind Virtual SAN.
  But, the VAAI "Delete" feature (which corresponds to UNMAP) is not supported.
• VAAI features are not supported for our iSCSI "naa..." disk created on Windows Server.
  Hence the value "VMW_VAAIP_MASK" which appears.

Source : Listing VAAI Filters and Plug-ins – teaser.

Plain Text

eui.0ca9aa96ed2401f9
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

...

eui.47f31a4cc14db26c
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

naa.60003ff44dc75adcafe8cc13f634ba70
   VAAI Plugin Name: VMW_VAAIP_MASK
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: unsupported
   Delete Status: unsupported

5. Creating a VM without VAAI support

If you want to test the difference between without and with VAAI support, simply create a virtual machine on storage that supports it and create another on storage that does not support it.
You will notice that thanks to VAAI you will benefit from better performance than without it.

To get started, select your VMware ESXi host and click: Actions -> New Virtual Machine.

In the "New Virtual Machine" wizard that appears, select "Create a new virtual machine" and click Next.

Select an iSCSI drive that does not support hardware acceleration (VAAI).

Use Thick Provision Eager Zeroed for its virtual hard disk.

Wait while the virtual machine is created.

Calculate the time it took to create this virtual machine based on the start and end times.

6. Creating a VM with VAAI support

Now create a new virtual machine by selecting storage provided by a storage array supporting VAAI primitives.
In our case, we select our "SW iSCSI DS 1" datastore created on an iSCSI disk provided by our StarWind Virtual SAN storage array.

Again, use Thick Provision Eager Zeroed for its virtual hard disk.

Wait while this virtual machine is created.

If you compare the time spent, you will see that the creation of the same virtual machine will take less time with VAAI support, because the zeroing of the virtual hard disk will be managed by the storage array and not by the VMware host ESXi.

Note: in our case, given that everything is virtualized, there is no real difference during our tests. But in a professional environment with real storage arrays, you will be able to see the difference by performing these tests.