vCloud Director 5.1 VXLAN Configuration

By: Rawlinson Rivera, Senior Technical Marketing Manager at VMware

This is a repost from Rawlinson’s personal blog, Punching Clouds. 

The new vCloud Director 5.1 delivers many new features and enhancements, one in particular is the introduction and support of Virtual Extensible LAN (VXLAN). VXLAN is a technology that enables the expansion of isolated vCloud architectures across layer 2 domains beyond the limits imposed by the IEEE 802.1Q standard. By utilizing a new MAC-in-UDP encapsulation technique, a VXLAN ID adds a 24 bit identifier, which allows networks to push beyond the IEEE 802.1Q limit to a possible 16 million logical networks. Figure 1 below illustrates the changes added to the Ethernet frame by VXLAN.

Figure 1: Ethernet Frame with VXLAN Encapsulation

While the conventional IEEE 802.1Q standard works perfectly, when trying to meet greater scalability demands VXLAN surpasses the IEEE 802.1Q limitation by offering scalable capabilities of up to 16 million possible networks. Because of the scalable and flexible capabilities offered by VXLAN, this technology is something to be consider for large scalable Cloud (vCloud) networks. For a quick crash course on VXLAN, take a look at Duncan Epping’s post “Understanding VXLAN and the value prop in just 4 Minutes…”

Configuring VXLAN in vCloud Director 5.1 required some initial steps that are outside of the vCloud Director 5.1 management interface which I want to illustrate here.

First a couple of facts:

A VXLAN network pool is automatically created in vCloud Director 5.1 whenever a Provider vDC is created. If the hosts of any given cluster are not prepared to use VXLAN first, the VXLAN network Pool in vCloud Director will display an error. I would recommend identifying all of the pre-requisites for the use of VXLAN from a network as well as the software dependency perspective before creating new Provider vDC in vCloud Director 5.1.

In order to prepare the resource clusters (hosts) to use VXLAN, log in the vCloud Networking and Security appliance (previously knows as vShield Manager). The preparation of the networks as well as the hosts requires the identification and assignment of the Segment ID Pool and the Multicast addresses. Below are the steps necessary to prepare and configure VXLAN for vCloud Director 5.1.

Step 1: Log into the vCloud Networking and Security appliance. Select the Datacenter. Then, select the Network Virtualization tab on the right side of the screen and click the Preparation hyperlink. This will reveal the Connectivity and Segment ID screen, as illustrated in figure 2.

Figure 2: Network Virtualization Settings

Step 2: Click the Edit button on the right end of the screen, and enter the required Segment ID Pool, and Multicast address that will be used by vCloud Networking and Security appliance. The Segment ID’s cannot be mapped directly to any one Multicast Address, as the possibility of one-to-one mapping doesn’t exist. This Segment ID and Multicast Address configuration is defined in ranges. Figure 3 illustrates the Segment ID and Multicast Address options.

Figure 3: Segment ID Pool and Multicast Address

Step 3: Click on the Connectivity button in the Network Virtualization tab to prepare the resource clusters (hosts) to be part of the VXLAN with vCloud Director. Choose the Distributed switch that is to be associated with the resource cluster, and enter the VLAN ID for the desired network segment that will be used to overlay the VXLAN traffic coming from the Distributed Switches. Figure 4 illustrates the configuration options.

Figure 4: Resource Cluster

Step 4: Specify the NIC teaming policy that applies to the respective Distributed Switch configuration, and the MTU settings. The MTU settings for VXLAN default to 1600 bytes due to the VXLAN ID encapsulation technique which increases the size of the packets. This behavior is similar for the configuration of vCDNI in vCloud Director.  vCDNI required the minimum MTU packet configuration of 1524. Overall, the important thing to understand here is the requirement to use jumbo frames across all network devices. Figure 5 illustrates the NIC teaming policies available as well as the default MTU settings and click Finish.

Figure 5: VXLAN Attributes

After choosing and completing the specification for the Distributed Switches, the VXLAN vmkernel modules are pushed and enabled on to all of the hosts that are part of the selected cluster. New dvPort Groups and vmknic interfaces are added and automatically created on the Distributed Switch associated to the VXLAN. The new dvPort group can be identified by the unique naming convention vxw-vmknicPg-dvs-xx-xx-xx-xx. Figure 6 offers an example of the adapter configuration.

Figure 6: VXLAN VMkernel Interfaces

A troublesome results of the automated network configuration process for the vmknics, is that all interfaces will be automatically assigned an IP address based on DHCP. This behavior can become a configuration management issue; unless there is a DHCP server on that network segment (normally the management network), all of the newly created interfaces will receive an IPv4 address within the 169.254/16 prefix that is valid for communication with other devices connected to the same physical link.

This configuration will not work as an IPv4 local addresses are not suitable for communication with devices not directly connected to the same physical or logical link, and are only used where stable, routable addresses are not available. As a result of this configuration the status of the hosts preparation will be displayed as “Not ready” in the vCloud Networking and Security appliance interface. Figure 7 illustrates the issue discussed above.

Figure 7: vmknics IP Address Status

The solution to this issue is simple: update the vmknics interface with automatically assigned IP with valid addresses. This can be achieved in a manual or automated format. Figure 8 illustrate the results of a successful configuration.

Figure 8: VXLAN Successful Preparation Results

Step 5: At this point, all the required network, and hosts preparation for the use of VXLAN with vCloud Director 5.1 have been completed. In order to start using the VXLAN feature in vCloud Director 5.1,  create a Provider vDC.  A VXLAN Pool is automatically created. Figure 9 illustrates the existence of VXLAN capable network pool in the management interface of vCloud Director.

Figure 9: VXLAN Network Pool in vCloud Director 5.1

There you have it, folks. You can now proceed with the creation and configuration of Organization, and vApp networks to harness the scalable features delivered by VXLAN in vCloud Director 5.1 infrastructures.

Enjoy!

Rawlinson is a Senior Technical Marketing Manager in the Cloud Infrastructure Technical Marketing Group at VMware, focused on Storage Virtualization technologies. Previously he was an architect focused on Cloud and vSphere enterprise architectures for VMware fortune 100, 500 customers. Rawlinson is amongst the first VMware Certified Design Experts (VCDX#86), author of multiple books and a retired Professional Skater.