Hyperconverged Infrastructure Software-Defined Storage vSAN

vSAN 2 Node with Direct Connect

vSAN 2 Node with Direct ConnectTraditional vSAN 2 Node configurations require connectivity between the vSAN tagged VMkernel ports, and the vSAN Witness Appliance’s VSAN tagged VMkernel port.

Also remember that vSAN traffic will use the default gateway of the Management VMkernel interface to attempt to reach the Witness, unless additional networking is put in place, such as advanced switch configuration or possibly static routing on ESXi hosts and/or the Witness Appliance

New in 6.5
VMware vSAN 6.5 supports the ability to directly connect two vSAN data nodes using one 10Gb networking cable or, preferably, two connections for redundancy.

This is accomplished by tagging an alternate VMkernel port with a traffic type of “Witness.” The data and metadata communication paths can now be separated. Metadata traffic destined for the Witness vSAN VMkernel interface, can be done through an alternate VMkernel port.

I like to call this “Witness Isolated Traffic” (or WIT), I think we calling are it “Witness Traffic Separation” (or WTS).

With the ability to directly connect the vSAN data network across hosts, and send witness traffic down an alternate route, there is no requirement for a high speed switch for the data network in this design.

This lowers the total cost of infrastructure to deploy 2 Node vSAN. This can be a significant cost savings when deploying vSAN 2 Node at scale.

The How
To use ports for vSAN today, VMkernel ports must be tagged to have “vsan” traffic. This is easily done in the vSphere Web Client. To tag a VMkernel interface for “Witness” traffic, today it has to be done at the command line.
To add a new interface with Witness traffic is the type, the command is:

esxcli vsan network ipv4 add -i vmkX -T=witness

We can configure a new interface for vSAN data traffic using this command (rather than using the Web Client)

esxcli vsan network ipv4 add -i vmkX -T=vsan

When looking to see what our vSAN network looks like on a host, we use:

esxcli vsan network list

It should look something like the image to the right:

Notice that vmk0, the management VMkernel interface in this example, has Witness traffic assigned.

In the example shown, vmk0, on each data node, requires connectivity to the VMkernel port vmk1 on the Witness Appliance. The vmk2 interface on each data node could be direct connected.

Also keep in mind that it is cleaner & easier to have the direct connected nics on their own vSphere Standard Switch, or possibly a vSphere Distributed Switch. Remember that vSAN brings the vSphere Distributed Switch feature, regardless of what version of vSphere you are entitled to.

If we have dual nics for redundancy, we could possibly assign vMotion traffic down the other 10Gbps nic. Couple that with NIOC, and we could ensure that both vSAN and vMotion could have enough resources in the event of one of the two nics failing.

Here’s a video I put together demonstrating the vSAN 2 Node Direct Connect feature:

I’m in the process of updating the Stretched Cluster & 2 Node guide for vSAN 6.5 to include this, some recommendations for it, as well as some more new content. Stay tuned.

Comments

28 comments have been added so far

  1. Sweet. Thank you! This helped me very much. Thank you for the video. It helped show me how to setup the networking. The part that had me stumped for awhile, was how to configure only 1 nic per port group. I finally found the setting editing the distributed port group settings. Then clicking on Teaming and failover I configured 1 uplink active and the other to standby.

  2. Actually 2 Node vSAN has been supported since 6.1.
    2 Node vSAN with Direct Connect was introduced with vSAN 6.5, but is supported on vSAN 6.0 Patch 3 or higher (vSAN 6.2).

    Thank you,
    Jase

  3. Hi Jase,

    Does it require vcenter also to be version above 6 update 3 for a 2 node direct connect setup?

    Thanks

  4. Hi Jase!

    What about the Idea to use the management VMkernel for the Witness traffic also in a 4-Node stretched Cluster?
    I’m just thinking of it – because, then there would be no need to route the VSAN Network (Layer 3) to the Witness Host?

    In our case, the 10Gbit VSAN Network could still act as an “isolated LAN” and would need no uplink to the core-network…

    Thanks,
    Patrick

  5. Since direct connect is used and the witness traffic is carried over management vmk, what to do with the WitnessPg vmk in witness esxi host?

  6. Hi guys, about the direct connect, where do I find some HCL information about 10GB SFP+ direct connect without switches and over long distance +-3km. I think to interlink 2 VSAN nodes with Transceiver, SFP+, 10GbE, LR, 1310nm Wavelength over fibre.

    Is it supported?

  7. Hi Mr. McCarty,
    Thank you for your very usefull explanation regarding this capability of VSAN.
    So as I understand, the witness appliance is required anyway, even in this case of two nodes attached directly by the 10 GigaB NIC.
    Is that right??
    Thank you in advance for your gently reply.
    Best regards
    Dante Carlini

  8. Hi Jase
    I have configured this but it seems incorrect with messages saying 0 witness detected etc

    Only thing I did was ran the CLI commands after everything was setup
    Do the hosts require rebooting ? Or is there a NW rest required of some sort ?
    Tks

  9. Pradeep,

    Sorry for the late response, just seeing this. Your vCenter should be at least vCenter 6.0 Update 3 for this to work with vSphere 6.0 Update 3.

  10. Cezar,

    As long as the connection medium supports it, and there is <5ms latency between hosts, you should be fine.

  11. Dante,

    You do have to have a vSAN Witness Host (could be our free appliance) for 2 Node vSAN.

  12. Valter,

    You can use the data node vmk0 for Witness Traffic, and that would need to communicate with the vSAN Witness’s vmk1 (WitnessPg). If you would like to use the Witness’s vmk0, that’s fine, just untag “vSAN Traffic” on the Witness vmk1 and tag it on the Witness vmk0. (Fully supported)

  13. Apologies for the late replies folks, it would appear there is a routing issue for comment moderation.

  14. FQ,

    You may want to reboot your VCSA before doing any additional troubleshooting.

    Run the health check again, and everything should clear up.

    Let me know how it goes.

  15. Thanks for the great article. I am new to vSAN, and having some issues here and there. One thing I noticed, is that when I deploy my witness appliance, it sets up a Portgroup called “witnessPg” under “witnessSwitch.” The VMkernel adapter under this is vmk1. When I run esxcli vsan network list I see that this adapter is set for traffic type VSAN. Is this correct? Or do I also need to set this adapter to witness type traffic?

  16. Hey Mike,

    In a 2 Node Direct Connect configuration, the vSAN Witness Appliance is still going to have vmk1 (WitnessPg on witnessSwitch) set to “vSAN Traffic”.
    The only VMkernel interfaces that will have “Witness Traffic” set, are the alternate VMkernel interfaces (connected to a switch and routing to the vSAN Witness vmk1).
    The backend vSAN Node interfaces are still set to “vSAN Traffic”.

    I hope this helps.

  17. Could the commands in the article be fixed?
    This is what you have
    To add a new interface with Witness traffic is the type, the command is:

    esxcli vsan network ipv4 add -i vmkX -T=witness

    We can configure a new interface for vSAN data traffic using this command (rather than using the Web Client)

    esxcli vsan network ipv4 add -i vmkX =T=vsan

    Should be

    esxcli vsan network ipv4 add -i vmkX -T=vsan

  18. Jase,

    Can I use L2 networking for the witness traffic? I know that is not recommenced in a general terms to avoid the potential vSAN traffic on management/witness up-link in case of failure….but for small internal cluster should not matter I think.

    This is the official documentation:

    *Local Witness with 2 Node Configurations: A typical Option 3 configuration would include the
    Witness being remote, but running the vSAN Witness Appliance on alternate infrastructure in the
    same site is supported if so desired. In this configuration a Layer 2 configuration would likely be used,
    which may not require any static routing. This vSAN Witness Appliance may NOT run on the 2 Node
    cluster, but may run on alternate ESXi 5.5 or higher infrastructure.

Leave a Reply

Your email address will not be published.