vSAN Hyperconverged Infrastructure Products Software-Defined Storage

Understanding Mixed MTU support in Stretched & 2 Node vSAN 6.7 U1 clusters with Witness Traffic Separation

Witness Traffic Separation introduced the ability to tag “witness” traffic on an alternate VMkernel interface on vSAN data nodes.

Witness Traffic Separation is the feature that enabled 2 Node “Direct Connect” in vSAN 6.5, and was also added to later editions of vSAN 6.2 (vSphere 6.0 Update 3 & higher).

Some additional logic was added in vSAN 6.7 to support Witness Traffic Separation in Stretched Clusters as well.

MTU Size

Traditional vSAN Clusters require the MTU size to be uniform across all vSAN VMkernel interfaces. vSAN Stretched Clusters have had the same requirement, due to the vSAN data network connecting to the vSAN Witness Host. When VMkernel ports communicate with each other and have mixed MTU sizes, some potential communication challenges include dropped packets, retransmits, and the like.

But what about configurations using Witness Traffic Separation?

The “backend” vSAN data network doesn’t communicate with the vSAN Witness. Communication with the vSAN Witness is over a completely different VMkernel interface that has been tagged for “witness” traffic.

It only makes sense that completely different MTU sizes on the vSAN data network and the network used to communicate with the vSAN Witness would work without issue.

The Need for Mixed MTU

One common ask for “Mixed MTU” when using Witness Traffic Separation, is when connectivity to the vSAN Witness has a limited MTU, or one that is less than desired for the backend vSAN data network.

Examples of this, could be:

  • vSAN Stretched Cluster
    • Backend vSAN data network supports Jumbo Frames – Desired by the customer
    • Frontend vSAN witness network does not support Jumbo Frames
      • Low speed connection to an alternate datacenter, OVH, or VMC running the vSAN Witness.
  • 2 Node vSAN
    • Backend Direct Connect vSAN with Jumbo Frames
    • Frontend vSAN connectivity to vSAN Witness in a central DC, on OVH, or on VMC, connected over an IPSEC VPN w/max MTU of 1500
      • May have to use an MTU of something like 1372

With versions of vSAN prior to 6.7 Update 1, the MTU must match on all vSAN VMkernel interfaces, regardless of being tagged for vSAN or Witness traffic types to be supported.

Support for Mixed MTU with 6.7 Update 1

Mixed MTUVMware isn’t supporting mixed MTUs for interfaces that communicate directly with each other, but rather mixed for the different vSAN traffic types.

The vSAN Health Check is updated in vSAN 6.7 Update 1 to recognize Witness Traffic Separation deployments and allow for a different MTU for the vSAN data and vSAN witness networks.

Here is an illustration showing the vSAN data network (VMkernel vmk2) with Jumbo Frames configured and vSAN Traffic enabled on a host.

Mixed MTUOn this same host, the Management VMkernel (vmk0) is tagged for “witness” traffic with an MTU of 1500.

Mixed MTU

On the vSAN Witness, the Management VMkernel (vmk0) is tagged for vSAN Traffic with an MTU of 1500.

Mixed MTU

Remember that the vSAN Witness has “vSAN Traffic” tagged, though it communicates with the “witness” tagged interfaces on the data nodes when using Witness Traffic Separation. In this installation, vmk1 is not being used, this is also a valid configuration.

Looking at the vSAN Health Check, it can be seen that the hosts are properly communicating with the vSAN Witness, even though they have different MTU settings.

Mixed MTUNotice in the vSAN Health Check, that communications to or from the vSAN Witness only occur on vmk0 in this illustration. Vmk0 on each of the data nodes, as well as the vSAN Witness, make up the “frontend” vSAN (witness) network. The data nodes are using vmk2 for “backend” vSAN Traffic.

To summarize in an effort to avoid confusion:

Scenario Mixed MTU Supported
vSAN Stretched Clusters pre 6.7 Update 1 No
vSAN Stretched Clusters 6.7 Update 1 or higher with Witness Traffic Separation Yes
vSAN 2 Node Clusters 6.2 or higher No
vSAN 2 Node Clusters with Witness Traffic Separation 6.2-6.7 No
vSAN 2 Node Clusters with Witness Traffic Separation 6.7 Update 1 or higher Yes

Better flexibility

Customers, if they choose, can use Jumbo Frames on the vSAN data network, with no requirement to extend Jumbo Frames to the vSAN Witness. By allowing the ability to configure a different MTU for the “frontend” and “backend” vSAN networks, vSAN connectivity requirements can better align with the capabilities of the network and their requirements.



2 comments have been added so far

Leave a Reply

Your email address will not be published. Required fields are marked *