• Add a VMkernel interface
• Limit outbound bandwidth

Since VMware vSphere 6.0, you have the option of using multiple TCP/IP stacks. This allows you to better separate different types of network traffic of your VMware ESXi hypervisor by using different default gateways, if desired.

1. TCP/IP stacks on VMware vSphere 6.0 and later
2. Default TCP/IP stack
3. vMotion stack
4. Create a custom TCP/IP stack

1. TCP/IP stacks on VMware vSphere 6.0 and later

As mentioned earlier, since VMware vSphere 6.0, you can use multiple TCP/IP stacks.
Most of the time, this will allow you to separate vMotion traffic (virtual machine migration) from the rest of your hypervisor traffic (such as : vSAN traffic, host management traffic, or fault tolerance via vSphere Fault Tolerance).

As you can see, each TCP/IP stack has :

• its own memory heap
• its own ARP tables
• its own routing tables (IPv4/IPv6)
• its own default gateway IP addresses

However, before vSphere 6.0, you could only specify one default gateway IP address per VMware ESXi host.
You therefore had to separate the different types of network traffic by using different virtual switches and/or port groups.

Source : vSphere TCP/IP Stacks - VMware Docs.

2. Default TCP/IP stack

If you access the web interface of your VMware ESXi 6.7 hypervisor and go to "Networking -> TCP/IP stacks", you will see that the default TCP/IP stack is the one used by default.

As you can see, on VMware ESXi 6.7, there are 3 default TCP/IP stacks :

• vMotion stack : isolates traffic related to virtual machine migration via vMotion between your VMware ESXi hosts.
  Using this TCP/IP stack will disable the vMotion service on the default TCP/IP stack, since the vMotion service will now use this "vMotion stack" stack.
• Default TCP/IP stack : this stack can be used for all services supported by VMware ESXi, but only one default gateway IP address can be used for all services placed on this TCP/IP stack.
• Provisioning stack : allows you to isolate traffic related to virtual machines, such as cloning VMs, creating snapshots on them, cold migration (VM off), ...

To learn more about these TCP/IP stacks and the services available on VMware vSphere (management, vMotion, ...), see the "VMkernel Networking Layer - VMware Docs" page.

If you click on its name, you will see that the VMkernel interface (VMkernel NIC) that uses it is : vmk0.

If you view the information for this VMkernel interface (VMkernel NIC) "vmk0", you will see that it's part of a "Management Network" port group.
In other words, it's the interface for managing your VMware ESXi hypervisor.
Hence the fact that it uses the default TCP/IP stack of your hypervisor. Note that the system name of this TCP/IP stack is "defaultTcpipStack".

As you can see, in our case, this interface uses the "10.0.0.x" network (thanks to its associated "255.255.255.0" subnet mask).

3. vMotion stack

As explained earlier, the most common case where you will use multiple TCP/IP stacks is implementing virtual machine migration via vMotion.
For this, you will use the "vMotion stack" stack created by default, but not used by default.

In our case, this "vMotion stack" is used by our VMkernel interface "vmk1" and the subnet used is "10.10.0.x" (as you can see in its IPv4 routing table).
In our case, we don't have a default gateway (router) on our network dedicated to the migration of virtual machines via vMotion (hence the appearance of the address "0.0.0.0" as the gateway).

However, you can set another default gateway for this vMotion stack by clicking "Edit Settings", then filling in the desired field : "IPv4 gateway" or "IPv6 gateway".
Don't forget to also fill in the DNS configuration to use in this case.

In our case, the "vmk1" interface has the "10.10.0.11" IP address and therefore uses a "10.10.0.x" network which is different from the default one (10.0.0.x).
Hence the use of a different TCP/IP stack since the default gateway can't be the same for these 2 networks.

4. Create a custom TCP/IP stack

Although not necessarily necessary, you have the option of creating a custom TCP/IP stack if you wish.
However, this is only possible from the command line.
You will therefore first need to start the "TSM-SSH" service of your hypervisor by going to : Host -> Manage -> Services.

Then, connect in SSH to your VMware ESXi hypervisor using PuTTY (for example).

To create a custom TCP/IP stack, simply use the following command, indicating the desired name via the "-N" parameter :

Bash

esxcli network ip netstack add -N="iSCSI"

The custom TCP/IP stack created appears in the "Networking -> TCP/IP stacks" section of your VMware ESXi hypervisor.

Source : Create a Custom TCP/IP Stack - VMware Docs.

As you can see, this new TCP/IP stack has its own configuration, its own routing tables, ...
Note that the settings of a TCP/IP stack can only be modified when at least one VMkernel interface (vmkX) uses it.
Hence the fact that the "Change settings" option is greyed out at the moment.

To remove the custom TCP/IP stack created earlier, simply replace the keyword "add" with "remove" in the command used earlier.
Which gives this.

Bash

esxcli network ip netstack remove -N="iSCSI"

The desired custom TCP/IP stack disappears from your hypervisor's list of TCP/IP stacks.