Witness Traffic Separation introduced the ability to tag “witness” traffic on an alternate VMkernel interface on vSAN data nodes.
Witness Traffic Separation is the feature that enabled 2 Node “Direct Connect” in vSAN 6.5, and was also added to later editions of vSAN 6.2 (vSphere 6.0 Update 3 & higher).
Some additional logic was added in vSAN 6.7 to support Witness Traffic Separation in Stretched Clusters as well.
MTU Size
Traditional vSAN Clusters require the MTU size to be uniform across all vSAN VMkernel interfaces. vSAN Stretched Clusters have had the same requirement, due to the vSAN data network connecting to the vSAN Witness Host. When VMkernel ports communicate with each other and have mixed MTU sizes, some potential communication challenges include dropped packets, retransmits, and the like.
But what about configurations using Witness Traffic Separation?
The “backend” vSAN data network doesn’t communicate with the vSAN Witness. Communication with the vSAN Witness is over a completely different VMkernel interface that has been tagged for “witness” traffic.
It only makes sense that completely different MTU sizes on the vSAN data network and the network used to communicate with the vSAN Witness would work without issue.
The Need for Mixed MTU
One common ask for “Mixed MTU” when using Witness Traffic Separation, is when connectivity to the vSAN Witness has a limited MTU, or one that is less than desired for the backend vSAN data network.
Examples of this, could be:
- vSAN Stretched Cluster
- 2 Node vSAN
- Backend Direct Connect vSAN with Jumbo Frames
- Frontend vSAN connectivity to vSAN Witness in a central DC, on OVH, or on VMC, connected over an IPSEC VPN w/max MTU of 1500
- May have to use an MTU of something like 1372
With versions of vSAN prior to 6.7 Update 1, the MTU must match on all vSAN VMkernel interfaces, regardless of being tagged for vSAN or Witness traffic types to be supported.
Support for Mixed MTU with 6.7 Update 1
VMware isn’t supporting mixed MTUs for interfaces that communicate directly with each other, but rather mixed for the different vSAN traffic types.
The vSAN Health Check is updated in vSAN 6.7 Update 1 to recognize Witness Traffic Separation deployments and allow for a different MTU for the vSAN data and vSAN witness networks.
Here is an illustration showing the vSAN data network (VMkernel vmk2) with Jumbo Frames configured and vSAN Traffic enabled on a host.
On this same host, the Management VMkernel (vmk0) is tagged for “witness” traffic with an MTU of 1500.
On the vSAN Witness, the Management VMkernel (vmk0) is tagged for vSAN Traffic with an MTU of 1500.
Remember that the vSAN Witness has “vSAN Traffic” tagged, though it communicates with the “witness” tagged interfaces on the data nodes when using Witness Traffic Separation. In this installation, vmk1 is not being used, this is also a valid configuration.
Looking at the vSAN Health Check, it can be seen that the hosts are properly communicating with the vSAN Witness, even though they have different MTU settings.
Notice in the vSAN Health Check, that communications to or from the vSAN Witness only occur on vmk0 in this illustration. Vmk0 on each of the data nodes, as well as the vSAN Witness, make up the “frontend” vSAN (witness) network. The data nodes are using vmk2 for “backend” vSAN Traffic.
To summarize in an effort to avoid confusion:
| Scenario | Mixed MTU Supported |
|---|---|
| vSAN Stretched Clusters pre 6.7 Update 1 | No |
| vSAN Stretched Clusters 6.7 Update 1 or higher with Witness Traffic Separation | Yes |
| vSAN 2 Node Clusters 6.2 or higher | No |
| vSAN 2 Node Clusters with Witness Traffic Separation 6.2-6.7 | No |
| vSAN 2 Node Clusters with Witness Traffic Separation 6.7 Update 1 or higher | Yes |
Better flexibility
Customers, if they choose, can use Jumbo Frames on the vSAN data network, with no requirement to extend Jumbo Frames to the vSAN Witness. By allowing the ability to configure a different MTU for the “frontend” and “backend” vSAN networks, vSAN connectivity requirements can better align with the capabilities of the network and their requirements.
