Operational best practices

After customers successfully design the virtual network infrastructure, the next challenge for the customer is how to deploy the design and how to keep the network operational. VMware provides various tools, APIs, and procedures to help customers deploy and manage their network infrastructure effectively. Following are some key tools that are available in the vSphere platform

• CLI
• vCenter API
• Virtual Network Monitoring and Troubleshooting
  • NetFlow
  • Port Mirroring

In the following section, we will briefly discuss how vSphere and network administrators can utilize these tools to manage their virtual network. For more details on these tools please refer to the vSphere documentation.

Command Line Interface

vSphere administrators have several ways to access vSphere components through vSphere interface options that include vSphere Client, vSphere Web Client, and vSphere Command- Line interface. The vSphere CLI command set allows you to perform configuration tasks using a vCLI package installed on supported platforms, or using vMA. Please refer to Getting Started with vSphere CLI document for more details on the commands at the following link http://www.vmware.com/support/developer/vcli. The entire networking configuration can be performed through the CLI and thus helps administrators to automate the deployment process.

vCenter API

The networking setup in the virtualized data center involves configuration of virtual and physical switches. To automate this configuration process, VMware has provided APIs that allow network switch vendors to get information about the virtual infrastructure. This information regarding the virtual infrastructure helps network switch vendors in automating the configuration of the physical switches. For example, vCenter can trigger an event after the vMotion of a virtual machine is performed. After receiving this event trigger and related information, the network vendors can reconfigure the physical switch port policies such that when the VM moves to another host the VLANs/Access Control Lists (ACLs) configurations are also migrated along with the VM. Multiple networking vendors have provided this automation between physical and virtual infrastructure configuration through the integration with vCenter APIs. Customers should check with their networking vendors and find out if such automation tool exist that will bridge the gap between physical and virtual networking and simplify the operational challenges.

Virtual Network Monitoring and Troubleshooting

Monitoring and Troubleshooting network traffic in a virtual environment requires similar tools that are available in the physical switch environment. With the release of vSphere 5, VMware provides network administrators the ability to monitor and troubleshoot the virtual infrastructure through the features such as NetFlow and Port Mirroring.

NetFlow capability on a Distributed Switch along with a NetFlow collector tool helps monitor application flows and measures flow performance over time. It also helps in capacity planning and ensuring that I/O resources are utilized properly by different applications, based on their needs.

The port mirroring capability on a Distributed Switch is a valuable tool that helps network administrators in debugging network issues in a virtual infrastructure. The granular control over monitoring ingress, egress or all traffic of a port helps administrators fine-tune what traffic is sent for analysis.

Conclusion

vSphere Distributed Switch provides customers the right amount of features, capabilities and operational simplicity for deploying the virtual network infrastructure. As customers move on to build private or public cloud, VDS provides the scalability numbers for such deployments. The advanced capabilities such as NIOC and LBT are key for achieving better utilization of I/O resources and for providing better SLAs for the virtualized business critical applications and multi-tenant deployments. The support for standard networking visibility and monitoring features such as Port mirror and NetFlow help administrators manage and troubleshoot virtual infrastructure through familiar tools. VDS also is an extensible platform that allows integration with other networking vendor products through the open vCenter APIs.

This is the final entry in the series of VDS best practices blog. I would love to get your inputs on all the discussed VDS design options. As I mentioned earlier, customers are not limited to use the discussed design options. Depending on the needs and available infrastructure, customers can either tweak these deign options or come up with a new design for their deployments. Thanks for reading through these long posts.

Blade Server in Example Deployment

Blade servers are server platforms that provide higher server consolidation per rack unit along with benefits of lower power and cooling costs. Blade chassis that hosts the blade servers have proprietary architectures and each vendor has its own way of managing resources in the blade chassis. It is difficult to talk about all different blade chassis available in the market and explain their deployments in this document. In this section, we will focus on some generic parameters that customers should consider while deploying VDS in a blade chassis environment.

From the networking point of view all blade chassis provide the following two options:

• Integrated switches: In this option, the blade chassis allows built-in switches to control traffic flow between blade servers within the chassis and external network.
• Pass – Through Technology: This is an alternative method of network connectivity that allows the individual blade servers to communicate directly with external network.

In this design, the integrated switch option is described where the blade chassis has a built-in Ethernet switch. This Ethernet switch acts as an Access layer switch as shown in Figure 1.

This section discusses the deployment where the ESXi host is running on Blade server. Two types of Blade server configuration will be described in the following section

• Blade Server with Two 10 Gigabit Ethernet network adapters
• Blade Server with hardware assisted multiple logical network adapters

For each of the above two configurations, the different VDS design approaches will be discussed.

Blade Server with Two 10 Gigabit Ethernet network adapters

This deployment is quite similar to the Rack Server with two 10 Gigabit Ethernet network adapters deployment where each ESXi host was presented with two 10 Gigabit network adapters. As shown in Figure 1, the ESXi host running on a blade server in the blade chassis is also presented with two 10 Gigabit Ethernet network adapters.

Figure 1 Blade Server with 2- 10 Gig NICs

In this section two design options are described; one is a traditional static approach and other one is a VMware recommended dynamic configuration with NIOC and LBT features enabled. These two approaches are exactly similar to the deployment described under Rack Server with two 10 Gigabit network adapter section. Only blade chassis specific design decisions will be discussed as part of this section. For all other VDS and switch related configuration, readers are encouraged to refer the Rack sever with two 10 Gigabit network adapter section of this document.

Design Option 1 – Static Configuration

The configuration of this design approach is exactly same as described in the Design option1 section under Rack server with two 10 Gigabit network adapters. Please refer to the Table 1 below for the dvportgroup configuration details. Let’s take a look at the blade server specific parameters that need attention during the design.

Table 1 Static design configuration

The network and hardware reliability considerations should be incorporated during the blade server design as well. In these blade server designs, customers have to focus on the following two areas:

• High availability of blade switches in the blade chassis
• Connectivity of the blade server network adapters to internal blade switches.

High availability of blade switches can be achieved by having two Ethernet switching modules in the blade chassis. And the connectivity of two network adapters on the blade server should be such that one network adapter is connected to first Ethernet switch module and other network adapter is hooked to the second switch module in the blade chassis.

Another aspect that needs attention in the blade server deployment is the network bandwidth availability across the mid plane of blade chassis and between blade switches and aggregation layer. If there is oversubscription scenario in the deployment then customers have to think about utilizing traffic shaping and prioritization (802.1p tagging) features available in the vSphere platform. The prioritization feature allows customer to tag the important traffic coming out of the vSphere platform. These high priority tagged packets are then treated according to priority by the external switch infrastructure. During congestion scenarios, the switch will drop lower priority packets first and avoid dropping the important high priority packets.

This static design option provides the flexibility to the customers of choosing different network adapters for different traffic types. However, while doing the traffic allocation on limited two 10 Gigabit network adapters administrators end up scheduling multiple traffic types on a single adapter. As multiple traffic types flow through one adapter, the chances of one traffic dominating others goes up. To avoid the performance impact due to the noisy neighbors (dominating traffic type), customers have to utilize the traffic management tools provided in the vSphere platform. One of the traffic management features is NIOC, and that feature is utilized in the design option 2 described below.

Design Option 2 – Dynamic Configuration with NIOC and LBT

This Dynamic configuration approach is exactly same as described in the Design option2 section under Rack server with two 10 Gigabit Ethernet network adapters. Please refer to the Table 2 below  the dvportgroup configuration details and NIOC settings. The physical switch related configuration in the blade chassis deployment is the same as described in the rack server deployment. For the blade center specific recommendation on reliability and traffic management please refer to previous section.

Table 2 Dynamic design configuration

VMware recommends this design option that utilizes the advanced VDS features and provides customer with a dynamic and flexible design approach. In this design, I/O resources are utilized effectively and Service Level Agreements are met based on the shares allocation.

Blade Server with Hardware assisted Logical network adapters (HP-Flex10 like deployment)

Some of the new blade chassis supports traffic management capabilities that allow customers to carve I/O resources. This is achieved by presenting logical network adapters to the ESXi hosts. Instead of two 10 Gigabit Ethernet network adapters, ESXi host now sees multiple physical network adapters that operate at different configurable speeds. As shown in Figure 2, each ESXi host is presented with eight Ethernet network adapters that are carved out of two 10 Gigabit Ethernet network adapter.

Figure 2 Multiple Logical network adapters

This deployment is quite similar to the Rack server with eight 1 Gigabit Ethernet network adapter deployment. However, instead of 1 Gigabit network adapters the capacity of each network adapter is configured at the blade chassis level. In the blade chassis, customers can carve out different capacity network adapters based on the need of each traffic types. For example, if iSCSI traffic needs 2.5 Gigabit of bandwidth, a logical network adapter with that amount of I/O resources can be created on the blade chassis and presented to the blade server.

As for the configuration of the virtual switch VDS and blade chassis switch infrastructure goes, the configuration described in the design option 1 under the Rack server with eight 1 Gigabit network adapters is more relevant for this deployment. The static configuration option described in that design can be applied as is in this blade server environment. Please refer to Table 3 for dvportgroup configuration details and the switch configurations descried in that section for physical switch configuration details.

Table 3 Static Design configuration

Now the question is whether NIOC capability adds any value in this specific blade server deployment. NIOC is a traffic management feature that helps in scenarios where multiple traffic types flow through one uplink or network adapter. If in this particular deployment only one traffic type is assigned to a specific Ethernet network adapter then the NIOC feature will not add any value. However, if multiple traffic types are scheduled over one network adapter then customers can make use of NIOC to assign appropriate shares to different traffic types. This NIOC configuration will make sure that the bandwidth resources are allocated to the traffic types and SLA is met.

To illustrate this through an example, Let’s consider a scenario where vMotion and iSCSI traffic is carried over one 3 Gigabit logical uplink. To protect the iSCSI traffic from network intensive vMotion traffic, administrators can configure NIOC and allocate shares to each traffic type. If both traffics are equally important then administrators can configure shares with equal values (10 each). With this configuration, when there is a contention scenario, NIOC will make sure that iSCSI process will get half of the 1Gigabit uplink bandwidth and avoid any impact of vMotion process.

VMware recommends that the Network and Server administrators work closely together while deploying the traffic management features of the VDS and Blade Chassis. A lot of co-ordination is required during the configuration of the traffic management features to achieve the best end-to-end QoS result.

This concludes the different design options for the Rack and Blade server deployments with different network adapter configurations. Would love to get your feedback on these different design options and design guidelines. In the next blog entry I will talk about some operational aspect of VDS. Please stay tuned. 

Rack Server with Two 10 Gigabit Ethernet network adapters

The two 10 Gigabit Ethernet network adapters deployment model is becoming very common because of the benefits they provide through I/O consolidation. The key benefits include better utilization of I/O resources, simplified management, and reduced CAPEX and OPEX. While this deployment provides these benefits, there are some challenges when it comes to the traffic management aspects. Specially, in highly consolidated virtualized environments where more traffic types are carried over fewer 10 Gigabit Ethernet network adapters, and it becomes critical to prioritize traffic types that are important and provide the required SLA guarantees. The NIOC feature available on the VDS helps in this traffic management activity. In the following sections you will see how to utilize this feature in the different designs.

As shown in Figure 1, the rack servers with two 10 Gigabit Ethernet network adapters are connected to the two access layer switches to avoid any single point of failure. Similar to the Rack server with eight 1 Gigabit Ethernet network adapters section, the different VDS and Physical switch parameter configurations are taken into account during this design. On the physical switch side, the new 10 Gigabit switches might have support for FCoE that allows convergence for SAN and LAN traffic. This document only covers the standard 10 Gigabit deployments that support IP storage traffic (iSCSI/NFS) and not FCoE.

In this section two design options are described; one is a traditional approach and other one is a VMware recommended approach.

Figure 1 Rack server with 2 – 10 Gig NICs

Design Option 1 – Static Configuration

The static configuration approach for rack server deployment with 10 Gigabit Ethernet network adapters is similar to the one described in the design option 1 of rack server deployment with eight 1 Gigabit Ethernet adapters. There are few differences in the configuration where the numbers of dvuplinks are changed from eight to two, and dvportgroup parameters are different. Let’s take a look at the configuration details on the VDS front.

dvuplink configuration

To support the maximum two Ethernet network adapters per host, the dvuplink port group is configured with 2 dvuplinks (dvuplink1, dvuplink2). On the hosts the dvuplink1 is associated with vmnic0 and dvuplink2 is associated with vmnic1.

 dvportgroups configuration

As described in the Table 1, there are five different dvportgroups that are configured for the five different traffic types. For example, dvportgroup PG-A is created for the management traffic type. Following are the other key configurations of dvportgroup PG-A:

• Teaming Option: Explicit Failover order provides a deterministic way of directing traffic to a particular uplink. By selecting dvuplink1 as an Active uplink and dvuplink2 as standby uplink the management traffic will be carried over dvuplink1 unless there is a failure of dvuplink1. It is also recommended to configure the failback option to “No” to avoid the flapping of traffic between two NICs. The failback option determines how a physical adapter is returned to active duty after recovering from a failure. If failback is set to No, a failed adapter is left inactive even after recovery until another currently active adapter fails, requiring its replacement.
• VMware recommends isolating all traffic types from each other by defining separate VLAN for each dvportgroup.
• There are various other parameters that are part of the dvportgroup configuration. Customers can choose to configure these parameters based on their environment needs.

Table 1 below provides the configuration details for all the dvportgroups. According to the configuration, dvuplink1 carries Management, iSCSI, and Virtual Machine traffic while dvuplink2 handles the vMotion, FT, and Virtual Machine traffic. As you can see, Virtual machine traffic type makes use of two uplinks, and these uplinks are utilized through the load based teaming (LBT) algorithm.

In this deterministic teaming policy, customers can decide to map different traffic types to the available uplink ports depending on the environment needs. For example, if iSCSI traffic needs higher bandwidth and other traffic types have relatively low bandwidth requirements, then customers can decide to keep only iSCSI traffic on dvuplink1 and move all other traffic to dvuplink2. When deciding on these traffic paths, customers should understand the physical network connectivity and the paths bandwidth capacity.

Physical switch configuration

The external physical switch, where the rack servers’ network adapters are connected to, is configured with trunk configuration with all the appropriate VLANs enabled. As described in the physical network switch parameters sections, following switch configurations are performed based on the VDS setup described in Table 1.

• Enable STP on the trunk ports facing ESXi hosts along with “port fast” mode and “bpdu” guard.
• The teaming configuration on VDS is static and thus no link aggregation is configured on the physical switches.
• Because of the mesh topology deployment as shown in Figure 1, the link state-tracking feature is not required on the physical switches.

 Table 1 Static design configuration

This static design option provides the flexibility in the traffic path configuration but it cannot protect against one traffic type dominating others. For example, there is a possibility that network intensive vMotion process can take away most of the network bandwidth and impact virtual machine traffic. Bi-directional traffic shaping parameters at portgroup and port level can provide some help in managing different traffic rates. However, using this approach for traffic management requires customers to limit the traffic on the respective dvportgroups. Limiting traffic to a certain level through this method puts a hard limit on the traffic types even when the bandwidth is available to utilize. This underutilization of I/O resources because of hard limits is overcome through the NIOC feature, which provides flexible traffic management based on shares parameter. The design option 2 described below is based on the NIOC feature.

Design Option 2 – Dynamic Configuration with NIOC and LBT

This dynamic design option is the VMware recommended approach that takes advantage of the NIOC and LBT features of the VDS.

The connectivity to physical network infrastructure remains same as described in the design option 1. However, instead of allocating specific dvuplinks to individual traffic types, the ESXi platform utilizes those dvuplinks dynamically. To illustrate this dynamic design, each virtual infrastructure traffic type’s bandwidth utilization is estimated. In a real deployment, customers should first monitor the virtual infrastructure traffic over a period of time to gauge the bandwidth utilization, and then come up with bandwidth numbers.

Following are some bandwidth numbers estimated per traffic type:

• Management Traffic (< 1 Gig)
• vMotion (2 Gig)
• FT (1 Gig)
• iSCSI (2 Gig)
• Virtual Machine (2 Gig)

These bandwidth estimates are different from the one considered with rack server deployment with eight 1 Gig network adapters. Let’s take a look at the VDS parameter configurations for this design. The dvuplink portgroup configuration remains same with two dvuplinks created for the two 10 Gigabit Ethernet network adapters. The dvportgroup configuration is as follows.

dvportgroups configuration

In this design all dvuplinks are active and there are no standby and unused uplinks as shown in Table 2.  All dvuplinks are thus available for use by the teaming algorithm. Following are the key configurations of dvportgroup PG-A:

• Teaming Option: Load based teaming is selected as the teaming algorithm. With LBT configuration, the Management traffic initially will be scheduled based on the virtual port ID hash. Based on the hash output the management traffic will be sent out over one of the dvuplink. Other traffic types in the virtual infrastructure can also be scheduled on the same dvuplink With LBT configuration. Subsequently, if the utilization of the uplink goes beyond 75% threshold, LBT algorithm will be invoked and some of the traffic will be moved to other underutilized dvuplinks. It is possible that Management traffic will get moved to other dvuplinks when such event occurs.
• There are no standby dvuplinks in this configuration so the failback setting is not applicable for this design approach. The default setting for this failback option is “Yes”.
• VMware recommends isolating all traffic types from each other by defining separate VLAN for each dvportgroup.
• There are several other parameters that are part of the dvportgroup configuration. Customers can choose to configure these parameters based on their environment needs.

As you follow the dvportgroups configuration in Table 2, you can see that each traffic type has all the dvuplinks as active and these uplinks are utilized through the load based teaming (LBT) algorithm. Let’s take a look at the NIOC configuration.

The Network I/O Control (NIOC) configuration in this design not only helps provide the appropriate I/O resources to the different traffic types but also provides SLA guarantees by protecting from one traffic type dominating others.

Based on the bandwidth assumptions made for different traffic types, the shares parameters are configured in NIOC shares column in Table 2. To illustrate how share values translate to bandwidth numbers in this deployment, let’s take an example of 10 Gigabit capacity dvuplink carrying all five traffic types. This is a worst-case scenario where all traffic types are mapped to one dvuplink. This will never happen when customers enable the LBT feature, because the LBT will move traffic type based on the uplink utilization. This example shows how much bandwidth each traffic type will be allowed on one dvuplink during a contention or oversubscription scenario and when LBT is not enabled

• Management: 5 shares;        (5/75) * 10 Gigabit = 667 Mbps
• vMotion: 20 shares;               (20/75) * 10 Gigabit = 2.67 Gbps
• FT: 10 shares;                          (10/75) * 10 Gigabit = 1.33 Gbps
• iSCSI: 20 shares;                      (20/75) * 10 Gigabit = 2.67 Gbps
• Virtual Machine: 20 shares; (20/75) * 10 Gigabit = 2.67 Gbps
• Total shares: 5 + 20 + 10 + 20 + 20 = 75

As you can see, for each traffic type first the percentage of bandwidth is calculated by dividing the share value by the total available share number (75), and then the total bandwidth of the dvuplink (10 Gigabit) is used to calculate the bandwidth share for the traffic type. For example, 20 shares allocated to vMotion traffic translate to 2.67 Gbps of bandwidth to vMotion process on a fully utilized 10 Gigabit network adapter.

In this 10 Gigabit Ethernet deployment, customers can provide bigger pipes to individual traffic types without the use of trunking or multipathing technologies. This was not the case with eight 1 Gigabit Ethernet deployment.

There is no change in physical switch configuration in this design approach. So please refer to the physical switch settings described in design option 1 in previous section.

Table 2 Dynamic design configuration

This design option utilizes the advanced VDS features and provides customer with a dynamic and flexible design approach. In this design I/O resources are utilized effectively and Service Level Agreements are met based on the shares allocation.

In the next blog entry I will talk about the Blade center deployments.

Rack Server in Example Deployment

After looking at the major components in the example deployment and key virtual and physical switch parameters, let’s take a look at the different types of servers that customers can have in their environment. Customers deploy ESXi host either on a Rack Server or a Blade Server. This section discusses the deployment where the ESXi host is running on Rack server. Two types of Rack server configuration will be described in the following section

• Rack Server with Eight 1 Gigabit Ethernet network adapters
• Rack Server with Two 10 Gigabit Ethernet network adapters

For each of the above two configurations, the different VDS design approaches will be discussed.

Rack Server with Eight 1 Gigabit Ethernet network adapters

In a Rack Server deployment with eight 1Gigabit Ethernet network adapters per host, customers can either use traditional static design approach of allocating network adapters to each traffic type or make use of advanced features of VDS such as Network I/O Control (NIOC) and Load Based Teaming (LBT). The NIOC and LBT features help provide a dynamic design that utilizes I/O resources efficiently. In this section both the traditional and new design approaches are described along with their pros and cons.

Design Option 1 – Static configuration

This design option follows the traditional approach of statically allocating network resources to the different virtual infrastructure traffic types. As shown in the Figure 1, each host has eight Ethernet network adapters and four of those network adapters are connected to one of the first Access layer switches while the other four network adapters are connected to the second Access layer switch to avoid single point of failure. Let’s take a look in detail how VDS parameters are configured.

Figure 1 Rack Server with eight 1 Gigabit Ethernet network adapters

dvuplink configuration

To support the maximum eight 1 Gigabit Ethernet network adapters per host, the dvuplink port group is configured with eight dvuplinks (dvuplink1…. dvuplink8). On the hosts the dvuplink1 is associated with vmnic0 and dvuplink2 is associated with vmnic1… so on. It is a recommended practice to change the names of the dvuplinks to something meaningful and easy to track. For example, dvuplink1 that gets associated with vmnic on a motherboard can be renamed as “LOM-uplink1”.

If the hosts have some Ethernet network adapters as LAN On Motherboard (LOMs) and some on expansion cards, then for a better resiliency story, VMware recommends to select one network adapter from LOM and one from expansion card while configuring NIC teaming. To configure this teaming on a VDS, administrators have to pay attention to the dvuplink and vmnic association along with dvportgroup configuration where NIC teaming is enabled. In the NIC teaming configuration on a dvportgroup, administrators have to choose the different dvuplinks that are part of a team. If the dvuplinks are named appropriately according to the host vmnic association, administrators can select “LOM-uplink1” and “Expansion-uplink1” while configuring the teaming option for a dvportgroup.

 dvportgroups configuration

As described in the Table 1 there are five different portgroups that are configured for the five different traffic types. Customers can create up to 5000 unique portgroups per VDS. In this example deployment, the decision of creating different portgroups is based on the number of traffic types.

According to the Table 1, dvportgroup PG-A is created for the management traffic type. There are other dvportgroups defined for the other traffic types. Following are the key configurations of dvportgroup PG-A:

• Teaming Option: Explicit Failover order provides a deterministic way of directing traffic to a particular uplink. By selecting dvuplink1 as an Active uplink and dvuplink2 as standby uplink, the management traffic will be carried over dvuplink1 unless there is a failure on dvuplink1. Note that all other dvuplinks are configured as unused. It is also recommended to configure the failback option to “No” to avoid the flapping of traffic between two network adapters. The failback option determines how a physical adapter is returned to active duty after recovering from a failure. If failback is set to No, a failed adapter is left inactive even after recovery until another currently active adapter fails, requiring its replacement.
• VMware recommends isolating all traffic types from each other by defining separate VLAN for each dvportgroup.
• There are several other parameters that are part of the dvportgroup configuration. Customers can choose to configure these parameters based on their environment needs. For example, Customers can configure PVLAN to provide isolation when there are limited VLANs available in the environment.

As you follow the dvportgroups configuration in Table 1, you can see that each traffic type is carried over a specific dvuplink except the virtual machine traffic type that has two active uplinks dvuplink7 and dvuplink8. Virtual machine traffic type uses two active links, and these links are utilized through the load based teaming (LBT) algorithm. As mentioned earlier, LBT algorithm is much more efficient in utilizing link bandwidth than the standard hashing algorithm.

Table 1 Static Design configuration

Physical switch configuration

The external physical switch, where the rack servers’ network adapters are connected to, is configured with trunk configuration with all the appropriate VLANs enabled. As described in the physical network switch parameters section, the following switch configurations are performed based on the VDS setup described in Table 1.

• Enable STP on the trunk ports facing ESXI hosts along with “port fast” mode and “bpdu” guard.
• The teaming configuration on VDS is static and thus no link aggregation is configured on the physical switches.
• Because of the mesh topology deployment as shown in Figure 1, the link state-tracking feature is not required on the physical switches.

In this design approach, resiliency to the infrastructure traffic is achieved through Active – Standby uplinks and security is accomplished by providing separate physical paths for the different traffic types. However, with this design, the I/O resources are underutilized because the dvuplink2, dvuplink6 standby links are not used to send or receive traffic. Also, there is no flexibility to allocate more bandwidth to a traffic type when it needs it.

There is another variation to the static design approach that addresses some customer’s need of providing higher bandwidth to the storage and vMotion traffic type. In the static design described earlier, iSCSI and vMotion traffic is limited to 1 Gig. If a customer wants to support higher bandwidth for iSCSI, then they can make use of iSCSI multipathing solution. Also, with the release of vSphere 5, vMotion traffic can be carried over multiple Ethernet network adapters through the support of multi-NIC vMotion, and thus providing higher bandwidth to the vMotion process.

For more details on how to setup iSCSI multipathing please refer to the vSphere Storage guide link at the following website https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html. The configuration of multi-NIC vMotion is quite similar to the iSCSI multipath setup, where administrators have to create two separate vmkernel interfaces and bind each one to a separate dvportgroup. The two separate dvportgroup configuration provides the connectivity to two different Ethernet network adapters or dvuplinks.

Table 2 Static design configuration with iSCSI mutipathing and multi-NIC vMotion

As shown in Table 2, there are two entries each for vMotion and iSCSI traffic type listing the additional dvportgroups configuration required to support the multi-NIC vMotion and iSCSI multipathing processes. For multi-NIC vMotion the dvportgroup PG-B1 and PG-B2 are configured with dvuplink 3 and dvuplink4 as active links respectively. And for iSCSI multipathing the dvportgroups PG-D1 and PG-D2 are connected to dvuplink5 and dvuplink6 as active links respectively. The load balancing across the multiple dvuplinks is performed by the multipathing logic in iSCSI process and by the ESXi platform in vMotion process. It is not required to configure the teaming policies for these dvportgroups.

The FT, Management, and Virtual Machine traffic types dvportgroup configuration and physical switch configuration for this design remains same as described in the design option 1 in previous section.

This static design approach improves on the first design by using the advanced capabilities such as iSCSI multipathing and multi-NIC vMotion. But at the same time this option has the same challenges related to underutilized resources and inflexibility in allocating additional resources on the fly to different traffic types.

Design Option 2 – Dynamic configuration with NIOC and LBT

After looking at the traditional design approach with static uplink configurations, let’s take a look at the VMware recommended design option that takes advantage of the advanced VDS features such as NIOC and LBT.

In this design the connectivity to physical network infrastructure remains same as described in the static design option but instead of allocating specific dvuplinks to individual traffic types, the ESXi platform utilizes those dvuplinks dynamically. To illustrate this dynamic design, each virtual infrastructure traffic type’s bandwidth utilization is estimated. In a real deployment, customers should first monitor the virtual infrastructure traffic over a period of time to gauge the bandwidth utilization, and then come up with bandwidth numbers for each traffic type.

Following are some bandwidth numbers estimated per traffic type:

• Management Traffic (< 1 Gig)
• vMotion (1 Gig)
• FT (1 Gig)
• iSCSI (1 Gig)
• Virtual Machine (2 Gig)

Based on this bandwidth information, administrators can provision appropriate I/O resources to each traffic types using the NIOC feature of VDS. Let’s take a look at the VDS parameter configurations for this design as well as the NIOC setup. The dvuplink portgroup configuration remains same where eight dvuplinks are created for the eight 1 Gigabit Ethernet network adapters. The dvportgroup configuration is as follows.

dvportgroups configuration

In this design all dvuplinks are active and there are no standby and unused uplinks as shown in Table 3.  All dvuplinks are thus available for use by the teaming algorithm. Following are the key parameter configurations of dvportgroup PG-A:

• Teaming Option: Load based teaming is selected as the teaming algorithm. With LBT configuration, the Management traffic initially will be scheduled based on the virtual port ID hash. Depending on the hash output, the management traffic is sent out over one of the dvuplink. Other traffic types in the virtual infrastructure can also be scheduled on the same dvuplink initially. However, when the utilization of the dvuplink goes beyond 75% threshold, the LBT algorithm will be invoked and some of the traffic will be moved to other underutilized dvuplinks. It is possible that Management traffic will be moved to other dvuplinks when such LBT event occurs.
• The Failback option means going from using standby link to using active uplink after the active uplink comes back up in operation after a failure. This Failback option works when there are Active and Standby dvuplink configurations. In this design there are no Standby dvuplinks. So, when an active uplink fails, the traffic flowing on that dvuplink is moved to another working dvuplink. If the failed dvuplink comes back, the LBT algorithm will schedule new traffic on that dvuplink. This option is left default.
• VMware recommends isolating all traffic types from each other by defining separate VLAN for each dvportgroup.   
• There are several other parameters that are part of the dvportgroup configuration. Customers can choose to configure these parameters based on their environment needs. For example, Customers can configure PVLAN to provide isolation when there are limited VLANs available in the environment.

As you follow the dvportgroups configuration in the Table 3, you can see that each traffic type has all dvuplinks active and these links are utilized through the load based teaming (LBT) algorithm. Let’s now look at the NIOC configuration described in the last two columns of Table 3.

The Network I/O Control (NIOC) configuration in this design helps provide the appropriate I/O resources to the different traffic types. Based on the previously estimated bandwidth numbers per traffic type, the shares parameter is configured in NIOC shares column in Table 3. The shares values specify the relative importance of specific traffic type, and NIOC ensures that during contention scenarios on the dvuplinks each traffic type gets the allocated bandwidth. For example, a shares configuration of 10 for vMotion, iSCSI, and FT allocates equal bandwidth to these traffic types. While the Virtual Machines get the highest bandwidth with 20 shares and Management gets lower bandwidth with 5 shares.

To illustrate how share values translate to bandwidth numbers, let’s take an example of 1 Gigabit capacity dvuplink carrying all five traffic types. This is a worst-case scenario where all traffic types are mapped to one dvuplink. This will never happen when customers enable the LBT feature, because LBT will balance the traffic based on the utilization of uplinks. This example shows how much bandwidth each traffic type will be allowed on one dvuplink during a contention or oversubscription scenario and when LBT is not enabled.

• Management: 5 shares;        (5/55) * 1 Gigabit = 90.91 Mbps
• vMotion: 10 shares;               (10/55) * 1 Gigabit = 181.18 Mbps
• FT: 10 shares;                          (10/55) * 1 Gigabit = 181.18 Mbps
• iSCSI: 10 shares;                      (10/55) * 1 Gigabit = 181.18 Mbps
• Virtual Machine: 20 shares; (20/55) * 1 Gigabit = 363.64 Mbps
• Total shares: 5 + 10 + 10 + 10 + 20 = 55

To calculate the bandwidth numbers during contention, you should first calculate the percentage of bandwidth for a traffic type by dividing its share value by the total available share number (55). In the second step the total bandwidth of the dvuplink (1 Gigabit) is multiplied with the percentage of bandwidth number calculated in the first step. For example, 5 shares allocated to management traffic translate to 90.91 Mbps of bandwidth to management process on a fully utilized 1 Gigabit network adapter. In this example, custom share configuration is discussed but customer can make use of predefined High (100), Normal (50), and Low (25) shares while assigning them to different traffic types.

The vSphere platform takes these configured share values and applies them per uplink. The schedulers running at each uplink are responsible in making sure that the bandwidth resources are allocated according to the shares. In case of eight 1Gigabit Ethernet network adapter deployment, there are eight schedulers running. Depending on the number of traffic types scheduled on a particular uplink, the scheduler will divide the bandwidth among the traffic types based on the share numbers. For example, if only FT (10 shares) and Management (5 shares) traffic are flowing through dvuplink 5, then based on the shares value, FT traffic will get double the bandwidth of management traffic. Also, when there is no management traffic flowing, all bandwidth can be utilized by FT process. This flexibility in allocating I/O resources is the key benefit of NIOC feature.

The NIOC limits parameter of Table 3 is not configured in this design. The Limit value specifies an absolute maximum limit on egress traffic for a traffic type. Limits are specified in Mbps. This configuration provides a hard limit on any traffic even if I/O resources are available to use. It is not recommended to use limit configuration unless you really want to control the traffic even though additional resources are available.

There is no change in physical switch configuration in this design approach even with the choice of the new LBT algorithm. The LBT teaming algorithm doesn’t require any special configuration on physical switches. Please refer to the physical switch settings described in design option 1.

Table 3 Dynamic design configuration with NIOC and LBT

One thing to note about this design is that it doesn’t provide higher than 1 Gigabit bandwidth to the vMotion and iSCSI traffic types, as is the case in one of the static design using multi-NIC vMotion and iSCSI multipathing. The Load based teaming algorithm cannot split the infrastructure traffic across multiple dvuplink ports and utilize all the links. So, even if the vMotion dvportgroup PG-B has all the eight 1 Gigabit Ethernet network adapters as active uplinks, vMotion traffic will be carried over only one of the eight uplink. The main advantage of this design is in the scenarios where the vMotion process is not using the uplink bandwidth, and other traffic types are in need of the additional resources. In these situations NIOC makes sure that the unused bandwidth is allocated to the other traffic types that need it.

This dynamic design option is the recommended approach because it takes advantage of the advanced VDS features and utilizes I/O resource efficiently. This option also provides Active-Active resiliency where no uplinks are in standby mode. In this design approach, customers allow the vSphere platform to make the optimal decisions on scheduling traffic across multiple uplinks.

Some customers who have restrictions in the physical infrastructure in terms of bandwidth capacity across different paths and limited availability of the layer 2 domain might not be able to take advantage of this dynamic design option. While deploying this dynamic option, it is important to consider all different traffic paths that a traffic type can take and make sure that the physical switch infrastructure can support the specific characteristics required for each traffic type. VMware recommends that vSphere and Network administrators should work together to understand the impact of this dynamic traffic scheduling over physical network infrastructure before deploying this approach.

Every customer environment is different, and the requirements for the traffic types are also different. Depending on the need of the environment, customer can modify these design options to fit their specific requirements. For example, customers can choose to use combination of static and dynamic design option when they need higher bandwidth for iSCSI and vMotion activities. In this hybrid design four uplinks can be statically allocated to iSCSI and vMotion traffic types while remaining four uplinks can be used dynamically for the remaining traffic types. The Table 4 below shows the traffic types and associated port group configuration for the hybrid design.

Table 4 Hybrid design configuration

In the next blog entry I will discuss the Rack server deployment with two 10 Gigabit network adapters.

Important virtual and physical switch parameters

Before diving into the different design options around the example deployment, let’s take a look at the VDS (virtual) and physical network switch parameters that should be considered in all these design options. These are some key parameters that vSphere and network administrators have to take into account while designing VMware virtual networking. As the configuration of virtual networking goes hand in hand with physical network configuration, this section will cover both the VDS and Physical switch parameters.

VDS parameters

VDS simplifies the challenges of the configuration process by providing one single pane of glass to perform virtual network management tasks. As opposed to configuring vSphere standard switches (VSS) on individual hosts, administrators can configure and manage one single vSphere distributed switch. All centrally configured network policies on VDS get pushed down to the host automatically when the host gets added to the distributed switch. In this section an overview of key VDS parameters is provided.

Host Uplink Connections (vmnics) and dvuplink parameter

VDS has a new abstraction for the physical Ethernet network adapters (vmnics) on each host. This new abstraction is called dvuplinks that gets defined during the creation of the VDS. All the properties including NIC teaming, load balancing, and failover policies on VDS and dvportgroups are applied to dvuplinks and not to vmnics on individual hosts. When a host gets added to the VDS, each vmnic on the host is mapped to a dvuplink. This provides the advantage of consistently applying the teaming and failover configurations to all the hosts irrespective of how the dvuplink and vmnic assignments are made.

The Figure 1 below shows two ESXi hosts with four Ethernet network adapters each. When these hosts are added to the VDS with four dvuplinks configured on a dvuplink portgroup, administrators have to assign the network adapters (vmnics) of the hosts to dvuplinks. To illustrate the mapping of the dvuplinks to vmnics Figure 1 shows one type of mapping where ESXi hosts vmnic0 is mapped to dvuplink1 and vmnic1 to dvuplink2 and so on. Customers can choose different mapping if required where vmnic0 can be mapped to different dvuplink instead of dvuplink1. VMware recommends having consistent mapping across different hosts because it reduces complexity in the environment. 

Figure 1 dvulpink to vmnic mapping

As a best practice, customers should also try to deploy hosts with same number of physical Ethernet network adapters and with similar port speeds. Also, as the number of dvuplink configuration on VDS depends on the maximum number of physical Ethernet network adapters on a host, administrators should take that into account during dvuplink portgroup configuration. Customers always have an option to modify this dvuplink configuration based on the new hardware capabilities.

Traffic Types and dvportgroup parameters

Similar to portgroups on standard switches, dvportgroups define how the connection is made through the VDS to the network. The VLAN ID, traffic shaping, port security, teaming and load balancing parameters are configured on these dvportgroups. The virtual ports (dvports) connected to a dvportgroup share the same properties configured on a dvportgroup. When customers want a group of virtual machines to share the security and teaming policies, they have to make sure the virtual machines are part of one dvportgroup. Customers can choose to define different dvportgroups based on the different traffic types they have in their environment or based on the different tenants or applications they support in the environment.

 In this example deployment, the dvportgroup classification is based on the traffic types running in the virtual infrastructure. Once administrators understand the different traffic types in the virtual infrastructure and identify specific security, reliability and performance requirements for individual traffic types, the next step is to create unique dvportgroups associated with each traffic type. As mentioned earlier, the dvportgroup configuration defined at VDS level is automatically pushed down to every host that is added to the VDS. For example, in Figure 1, you can see that the two dvportgroup PG-A (Yellow) and PG-B (Green) defined at the distributed switch level are available on each of the ESXi host that is part of that VDS.

dvportgroup specific configuration

Once customers decide on the number of unique dvportgroups they want to create in their environment, they can start configuring those dvportgroups. The configuration options/parameters are similar to those available with port groups on vSphere standard switches. There are some additional options available on VDS dvportgroup that are related to teaming setup. These new options are not available on vSphere standard switches. Customers can configure the following key parameters for each dvportgroup.

• Number of virtual ports (dvports)
• Port binding (static, dynamic, ephemeral)
• VLAN Trunking/Private VLANs
• Teaming and Load Balancing along with Active and Standby Links
• Bi-directional traffic shaping parameters
• Port Security

As part of the teaming algorithm support, VDS provides a unique approach to load balance traffic across the teamed network adapters. This approach is called Load Based Teaming (LBT), and it distributes the traffic across the network adapters based on the percentage utilization of traffic on those adapters. LBT algorithm works on both ingress and egress direction of the network adapter traffic as opposed to the hashing algorithms that work only in egress direction (traffic flowing out of the network adapter). Also, LBT prevents the worst-case scenario that could happen with hashing algorithms where all traffic hashes to one network adapter of the team and other network adapters are not used to carry any traffic. To improve the utilization of all the links/network adapters, VMware recommends the use of this advanced feature (LBT) of VDS. The LBT approach is recommended over the Etherchannel on physical switches and route based IP hash configuration on the virtual switch.

Port security policies at port group level allow customer protection from certain behaviors that could compromise security. For example, a hacker could impersonate a virtual machine and gain unauthorized access by spoofing the virtual machines MAC address. VMware recommends to set the MAC address Changes and Forged Transmits to “Reject” to help protect against attacks launched by a rogue guest operating system. Set the Promiscuous Mode to “Reject” unless customers want to monitor the traffic for network troubleshooting or Intrusion detection purpose.

NIOC

Network I/O control (NIOC) is the traffic management capability available on VDS. The NIOC concept revolves around resource pools that are similar in many ways to the ones existing for CPU and Memory. vSphere and network administrators now can allocate I/O shares to different traffic types similar to allocating CPU and Memory resources to a VM. The share parameter specifies the relative importance of a traffic type over other traffics, and provides a guaranteed minimum when the different traffic competes for a particular network adapter. The shares are specified in abstract units numbered 1 to 100. Customers can provision shares to different traffic types based on the amount of resources each traffic type requires.

This capability of provisioning I/O resources is very useful in situations where there are multiple traffic types competing for resources. For example, in a deployment where vMotion and VM traffic types are flowing through one network adapter, it is possible that vMotion activity can impact the virtual machine traffic performance. In this situation, shares configured in NIOC provide the required isolation to the vMotion and VM traffic type and prevents one flow (traffic type) dominating other flow. NIOC configuration provides one more parameter that customers can utilize if they want to put any limits on a particular traffic type. This parameter is called the Limit. The Limit configuration specifies the absolute maximum bandwidth for a traffic type on a host. The configuration of limit parameter is specified in Mbps. NIOC limits and shares parameters only work on the outbound traffic i.e traffic that is flowing out of the ESXi host.

VMware recommends customers to utilize this traffic management feature whenever they have multiple traffic types flowing through one network adapter. This situation of multiple traffic type flowing through a network adapter is more prominent with 10 Gigabit Ethernet network deployments but can happen in 1 Gigabit Ethernet network deployments as well. The common use case for using NIOC in 1 Gigabit network adapter deployment is when the traffic from different workloads or different customer VMs is carried over the same network adapter. As multiple workload traffic flows through a network adapter, it becomes important to provide I/O resources based on the needs of the workload. With the release of vSphere 5, customers now can make use of the new user defined network resource pools capability and allocate I/O resource to the different workloads or different customer VMs depending on their needs. This user defined network resource pool feature provides the granular control in allocating I/O resources and meeting the SLA requirements for the virtualized tier 1 workloads.

 Bi-directional traffic shaping

Apart from NIOC, there is one more traffic-shaping feature that is available in the vSphere platform. This traffic-shaping feature can be configured on a dvportgroup or dvport level. Customers can shape both inbound and outbound traffic using three parameters: average bandwidth, peak bandwidth, and burst size. Customers who want more granular traffic shaping controls to manage their traffic types can take advantage of this capability of VDS along with NIOC feature. It is recommended to involve network administrators in your organization while configuring these granular traffic parameters. These controls only makes sense when there are oversubscription scenarios that are causing network performance issues. These oversubscription scenarios could be caused because of the oversubscribed physical switch infrastructure or virtual infrastructure. So it is very important to understand the physical and virtual network environment before making any bi-directional traffic-shaping configurations.

Physical Network switch parameters

The configuration of VDS and physical network switch should go hand in hand to provide resilient, secure and scalable connectivity to the virtual infrastructure. The following are some key switch configuration parameters customer should pay attention to.

VLAN

If VLANs are used to provide logical isolation between different traffic types it is important to make sure that those VLANs are carried over to the Physical switch infrastructure. To do so, enable VST (Virtual switch tagging) on the virtual switch, and trunk all VLANs to the physical switch ports.

Spanning Tree Protocol (STP)

Spanning Tree protocol is not supported on virtual switches and thus no configuration is required on VDS. But it is important to enable this protocol on the physical switches. STP makes sure that there are no loops in the network. As a best practice, customer should configure the following.

• Use “portfast” on ESXi host facing physical switch ports. With this setting, network convergence on these switch ports will happen fast after the failure because the port will enter the Spanning tree forwarding state immediately, bypassing the listening and learning states
• Use “BPDU guard” to enforce STP boundary. This configuration protects from any invalid device connection on the ESXi host facing access switch ports. As mentioned earlier, VDS doesn’t support Spanning Tree protocol and thus doesn’t send any Bridge Protocol Data Unit (BPDU) frames to the switch port.  However, if any BPDU is seen on these ESXi host facing access switch ports the BPDU guard feature puts that particular switch port in error-disabled state. The switch port is completely shut down and prevents affecting the Spanning Tree Topology.

The recommendation of enabling “portfast” and “BPDU guard” on the switch ports is valid only when customers connect non-switching/bridging devices to these ports. The switching/bridging devices can be hardware based physical boxes or servers running software based switching/bridging function. Customers should make sure that there is no switching/bridging function enabled on the ESXi hosts that are connected to the physical switch ports.

In the scenario where the ESXi host has a guest VM that is configured to perform bridging function, the VM will generate BPDU frames and send out to the VDS. The VDS then forwards the BPDU frames through the network adapter to the physical switch port. When the switch port configured with “BPDU guard” receives the BPDU frame, the switch disables the port and the VM looses connectivity. To avoid this network failure scenario while running software-bridging function on an ESXI host, customers should disable the “portfast” and “BPDU guard” configuration on the port and run the spanning tree protocol.

In case customers are concerned about the security hacks that can generate BPDU frames, they should make use of the VMware vShield App security product that can block the frames and protect the virtual infrastructures from such layer 2 attacks. Please refer to vShield product documentation for more details on how to secure your vSphere virtual infrastructure. http://www.vmware.com/products/vshield/overview.html

Link Aggregation setup

Link Aggregation is used to increase throughput and improve resiliency by combining multiple network connections. There are various proprietary solutions in the market along with vendor-independent IEEE 802.3ad (LACP) standard based implementation. All solutions establish a logical channel between the two end points using multiple physical links. In the vSphere virtual infrastructure the two ends of the logical channel are virtual switch (VDS) and physical switch. These two switches have to be configured with link aggregation parameters before the logical channel is established. Currently, VDS supports static link aggregation configuration and does not provide support for dynamic LACP. When customers want to enable link aggregation on a physical switch, they should configure static link aggregation on the physical switch and select IP hash as NIC teaming on the VDS.

When establishing the logical channel with multiple physical links, customers should make sure that the Ethernet network adapter connections from the host are terminated on a single physical switch. However, if customers have deployed clustered physical switch technology then the Ethernet network adapter connections can be terminated on two different physical switches. The clustered physical switch technology is referred by different names by networking vendors. For example, Cisco calls their switch clustering solution as VSS (Virtual Switching System) while Brocade calls it as VCS (Virtual Cluster Switching). Please refer to the networking vendor guidelines and configuration details while deploying switch-clustering technology.

Link State Tracking

Link state tracking is a feature available on Cisco switches to manage the link state of downstream ports (ports connected to Servers) based on the status of upstream ports (ports connected to Aggregation/Core switches). When there is any failure on the upstream links connected to aggregation or core switches, the associated downstream link status goes down. The server connected on the downstream link is then able to detect the failure and re-route the traffic on other working links. This feature thus provides the protection from network failures due to the down upstream ports in non-mesh topologies. Unfortunately, this feature is not available on all vendors’ switches, and even if it is available, it might not be referred to as link state tracking. Customers should talk to the switch vendors to find out if similar feature is supported on their switches.

The Figure 2 below shows the resilient mesh topology on the left and a simple loop free topology on the right. VMware highly recommends deploying the mesh topology shown on the left that provides highly reliable redundant design, and it doesn’t need link state tracking feature. Customers who don’t have the high-end networking expertise and are also limited with number of switch ports might prefer the deployment shown on the right. In this deployment customers don’t have to run the Spanning Tree Protocol because there are no loops in the network design. The downside of this simple design is when there is a failure on the link between the access and aggregation switch. In that failure scenario, the server will continue to send traffic on the same network adapter even when the access layer switch is dropping the traffic at the upstream interface.  To avoid this black holing of server traffic, customers can enable link state tracking on the virtual and physical switches and indicate any failure between access and aggregation switch layer to the server through link state information.

Figure 2 Resilient loop and no-loop topologies

VDS has default network failover detection configuration set as “Link status only”. Customers should keep this configuration if they are enabling the link state-tracking feature on physical switches. If link state tracking capability is not available on physical switches, and there are no redundant paths available in the design, then customers can make use of Beacon Probing feature available on VDS. Beacon probing function is a software solution available on virtual switches for detecting link failures upstream from the access layer physical switch to the aggregation/core switches. Beacon probing is most useful with three or more uplinks in a team.

Maximum Transfer Unit (MTU)

Make sure that the Maximum Transfer Unit (MTU) configuration matches across the virtual and physical network switch infrastructure.

After covering the important virtual and physical switch parameters and some recommended guidelines for each, we will take a look at the rack server deployments with multiple 1 Gigabit network adapters as well as two 10 Gigabit network adapters in the next blog entry.

After a long break I am back again on this forum. Over the last couple of months I was busy interacting with various customers as part of the VMworld US and Europe conferences and trying to understand how customers are deploying distributed switch in their environment. I have also spent lot of time internally talking to VMware R&D engineers as well as Field experts (SEs and PSOs) on what guidance we would like to provide to our customers regarding the deployment of vSphere Distributed Switch (VDS). After all these discussions, I have come up with some guidelines or best practices that customers have to consider while deploying VDS. These guidelines or best practices are discussed with reference to a sample example deployment. It is important to note that the suggestions/guidelines discussed here are based on the assumptions of the example deployment. Every customer environment is different as well as customer needs are different, and customers have to take that into account while designing the virtual network infrastructure using VDS. VDS is a flexible platform that can support different requirements from different customers. 

So let’s first dive into the different design considerations that you should consider before you go and start deploying VDS in your environment.

Design Considerations

Three main aspects influence the design of a virtual network infrastructure:

1)   Customer’s infrastructure design goals

2)   Customer’s infrastructure component configurations

3)   Virtual infrastructure traffic requirements

Let’s take a look at each of these aspects in a little more detail.

Infrastructure design Goals

Customer’s want their network infrastructure Available 24/7, Secure from any attacks, and Perform efficiently throughout the day-to-day operations. In the case of a virtualized environment these requirements become much more demanding as more and more business critical applications run on consolidated environment. These requirements on the infrastructure translates into design decisions that should incorporate following best practices for virtual network infrastructure:

• Avoid any single point of failure in the network
• Isolate each traffic type
• Make use of traffic management and optimization capabilities

Infrastructure component configurations

In every customer environment, the compute and network infrastructure used differ in terms of configuration, capacity, and feature capabilities. These different infrastructure component configurations influence the virtual network infrastructure design decisions. The following are some of the configurations and features that administrators have to look out for.

• Server configuration: Rack or Blade servers
• Network adapter configuration: 1 Gigabit or 10 Gigabit Ethernet network adapters; number of available adapters; offload function on these adapters if any.
• Physical Network Switch Infrastructure capabilities: Switch clustering

It is impossible to cover all the different virtual network infrastructure design deployments based on the various combinations of type of servers, network adapters and network switch capability parameters. In this paper, the following four commonly used deployments are described that are based on standard rack server and blade server configurations:

• Rack Server with Eight 1 Gigabit Ethernet network adapters Deployment
• Rack Server with Two 10 Gigabit Ethernet network adapters
• Blade Server with Two 10 Gigabit Ethernet network adapters
• Blade Server with Hardware assisted Multiple Logical Ethernet network adapters

It is assumed that the network switch infrastructure has standard layer 2 switch features (High availability, redundant paths, fast convergence, port security) available to provide reliable, secure and scalable connectivity to the server infrastructure.

Virtual Infrastructure traffic

VMware vSphere virtual network infrastructure carries different traffic types. To mange the virtual infrastructure traffic effectively, vSphere and network administrators have to understand the different traffic types and their characteristics. The following are the key traffic types that flow in the VMware vSphere infrastructure along with their traffic characteristics:

• Management traffic: This traffic flows through a vmknic and carries ESXi host to vCenter configuration and management communication as well as ESXi host to ESXi host High Availability (HA) related communication. This traffic has low network utilization but has very high availability and security requirements.
• VMware vMotion traffic: With advancement in the vMotion technology, a single vMotion can consume almost a full 10 Gigabit bandwidth. A maximum of eight simultaneous vMotions can be performed on a 10 Gigabit uplink while on a 1 Gigabit uplink four simultaneous vMotions are allowed. vMotion traffic has very high network utilization and can be bursty at times. Customers have to make sure that vMotion traffic doesn’t impact other traffic types because it could consume all available I/O resources. Another property of vMotion traffic is that it is not sensitive to throttling and thus makes a very good candidate to perform traffic management on.
• Fault Tolerant traffic: When fault tolerance logging is enabled for a virtual machine, all the logging traffic is sent to the secondary fault-tolerant virtual machine over a designated vmknic port. This FT process could require a lot of bandwidth at low latency as it replicates the I/O traffic and memory state information to the secondary virtual machine.
• iSCSI/NFS traffic:  IP storage traffic is carried over vmknic ports and this traffic varies according to disk I/O requests. With end-to-end jumbo frame configuration, more data is transferred with each Ethernet frame reducing the number of frames on the network. This larger frame reduces the overhead on servers and targets and improves the IP storage performance. Congested and lower speed networks can cause latency issues that disrupt access to IP storage. It is recommended to provide high-speed path for IP storage and avoid any congestion in the network infrastructure.
• Virtual Machine Traffic: Depending on the workloads that are running on the guest virtual machine, the traffic patterns will vary from low to high network utilization. Some of the applications running in virtual machines might be latency sensitive as is the case with VOIP workloads.

 The following Table 1 summarizes the characteristics of each traffic type

 Table 1 Traffic Types and characteristics

To understand the different traffic flows in the physical network infrastructure, network administrators use Network Traffic management tools. These network management tools help monitor the physical infrastructure traffic but do not provide visibility into virtual infrastructure traffic. With the release of vSphere 5, VDS now supports the NetFlow feature. The NetFlow feature allows exporting the internal (VM to VM) virtual infrastructure flow information to standard network management tools. Administrators now have the required visibility in to virtual infrastructure traffic. This helps administrators monitor the virtual network infrastructure traffic through a familiar set of network management tools. Customers should make use of the network data collected from these tools during the capacity planning or network design exercises.

Example Deployment components

After looking at the different design considerations, this section provides the list of components that are used in an example deployment. This example deployment is used to help illustrate some standard VDS design approaches. The following are some common components in the virtual infrastructure. The list doesn’t include storage components that are required to build the virtual infrastructure. It is assumed that customer will deploy IP storage in this example deployment.

Hosts

Four ESXi hosts providing compute, memory and network resources according to the configuration of the hardware. Customers can have different number of hosts in their environment based on their needs. One VDS can span across 350 hosts. This capability to support large number of hosts provides the required scalability to build a private or public cloud environment using VDS.

Clusters

A cluster is a collection of VMware ESXi hosts and associated virtual machines with shared resources. Customers can have as many clusters in their deployment as required. With one VDS spanning across 350 hosts, customers have the flexibility of deploying multiple clusters with different number of hosts in each cluster. For simple illustration purpose two clusters with two hosts each are considered in this example deployment. One cluster can have maximum 32 hosts.

vCenter Server

VMware vCenter server centrally manages a VMware vSphere environment. Customers can manage VDS through this centralized management tool. vCenter Server can be deployed on a virtual machine or a physical host. The vCenter Server is not shown in the diagrams but customers should assume that it is present in this example deployment. vCenter Server is only used to provision and manage VDS configuration. Once provisioned, hosts and virtual machines network operate independent of vCenter Server. All components required for network switching reside on ESXi hosts. Even if the vCenter Server fails, the hosts and virtual machines will still be able to communicate.

 In the deployment where vCenter Server is hosted on a virtual machine, customers have to pay more attention to the network configurations. In such deployments, if the networking for virtual machine hosting vCenter server is misconfigured, then the network connectivity of vCenter Server is lost. This mis-configuration must be fixed. However, you need vCenter Server to fix the network configuration because only vCenter Server can configure a VDS. As a workaround to this situation, customers have to reconnect the virtual machine hosting vCenter Server to a standard vSwitch using vSphere Client that is directly connected to the host. To do this, a VSS (vSphere standard switch) must be on a host that is also connected to the management network of hosts.

Network Infrastructure

Physical network switches in the access and aggregation layer provide connectivity between ESXi hosts and to the external world. These network infrastructure components support standard layer 2 protocols providing secure and reliable connectivity.

Along with the above four components of the physical infrastructure in this example deployment, some of the virtual infrastructure traffic types are also considered during the design. The following section describes the different traffic types in the example deployment.

Virtual Infrastructure Traffic Types

In this example deployment, there are standard infrastructure traffic types that include iSCSI, vMotion, FT, Management along with virtual machine traffic type. Customers may have other traffic types in their environment based on the choice of the storage infrastructure (FC,NFS,FCoE). The diagram in the Figure 1 below shows the different traffic types along with associated port groups on an ESXi host. It also shows the mapping of the network adapters to the different port groups.

Figure 1 Different Traffic types running on a Host

After covering the different design considerations and the example deployment, we will turn our attention to the different VDS design options around the example deployment. In the next blog entry I will cover the important virtual and physical switch parameters that customers should consider in all these design options.

Network I/O Control Enhancements

Consolidated I/O or I/O virtualization delivers similar benefits as provided by x86 virtualization in terms of better utilization and consolidation of resources. However, as multiple traffic types flow through a single physical network interface, it becomes important to manage the traffic effectively such that critical application flows don’t suffer because of a burst of low-priority traffic. Network traffic management provides the required control and guarantee to different traffic types in the consolidated I/O environment. In the VMware vSphere 5 platform, NIOC supports traffic management capabilities for the following traffic types or also called as network resource pools:

• Virtual machine traffic

• Management traffic

• iSCSI traffic

• NFS traffic

• Fault-tolerant traffic

• VMware vMotion traffic

• User-defined traffic

• vSphere replication traffic

Similar to CPU and memory resource allocation in the vSphere platform, a network administrator through NIOC can allocate I/O shares and limits to different traffic types, based on their requirements. In this new release of vSphere, NIOC capabilities are enhanced such that administrators can now create user-defined traffic types and allocate shares and limits to them. Also, administrators can provide I/O resources to the vSphere replication process by assigning shares to vSphere replication traffic types. Let’s look at some details on User defined and vSphere replication traffic types.

User-Defined Network Resource Pools

User-defined network resource pools in vSphere 5 provide an ability to add new traffic types beyond the standard system traffic types that are used for I/O scheduling.

Figure below shows an example of a user-defined resource pool with shares, limits and IEEE 802.1p tag parameters described in a table. In this example, Tenant 1 and Tenant 2 are two user-defined resource pools with virtual machines connected to their respective independent port groups. Tenant 1, with three virtual machines, has five I/O shares. Tenant 2, with one virtual machine, has 15 I/O shares. This indicates that during contention scenarios, Tenant 2 virtual machines will have a higher guaranteed share than Tenant 1 virtual machines.

Usage 

When customers are deploying critical applications on virtual infrastructure, they can utilize this advanced feature to reserve I/O resources for the important, business-critical application traffic and provide SLA guarantees.

Service providers who are deploying public clouds and serving multiple tenants can now define and provision I/O resources per tenant, based on each tenant’s need.

Configuration

The new resource pools can be defined at the Distributed Switch level by selecting the resource allocation tab and clicking on new network resource pools. After a new network resource pool is defined with shares and limits parameters, that resource pool can be associated with a port group. This association of a network resource pool with a port group enables customers to allocate I/O resources to a group of virtual machines or workloads. The figure below shows the new Tenant 1 and Tenant 2 resource pools created under user-defined network resource pools.

vSphere Replication Traffic

 vSphere replication is a new system traffic type that carries replication traffic from one host to another. NIOC now supports this new traffic type along with other system and user-defined traffic types.

 Usage

Customers implementing a disaster recovery (DR) solution with VMware vCenter Site Recovery Manager (Site Recovery Manager) and vSphere replication can use this vSphere replication traffic type to provide required network resources to the replication process.

Configuration

A vSphere replication traffic type can be configured on a Distributed Switch under the resource allocation tab. This traffic type is now part of the system network resource pool. Customers can allocate shares and limits parameters to this traffic type.

IEEE 802.1p Tagging

IEEE 802.1p is a standard for enabling QoS at MAC level. The IEEE 802.1p tag provides a 3-bit field for prioritization, which allows packets to be grouped into seven different traffic classes. The IEEE doesn’t mandate or standardize the use of recommended traffic classes. However, higher-number tags typically indicate critical traffic that has higher priority. The traffic is simply classified at the source and sent to the destination. The layer-2 switch infrastructure between the source and destination handles the traffic classes according to the assigned priority. In the vSphere 5.0 release, network administrators now can tag the packets going out of the host.

Usage

Customers who are deploying business-critical applications in a virtualized environment now have the capability to guarantee I/O resources to these workloads on the host. However, it is not sufficient to provide I/O resources just on the host. Customers must think about how to provide end-to-end QoS to the business-critical application traffic. The capability of a Distributed Switch to provide an IEEE 802.1p tag helps such customers meet those requirements for end-to-end QoS or service-level agreements.

 Configuration

IEEE 802.1p tagging can be enabled per traffic type. Customers can select the Distributed Switch and then the resource allocation tab to see the different traffic types, including system and user-defined traffic types. After selecting a traffic type, the user can edit the QoS priority tag field by choosing any number from 1 to 7. Figure below is the screenshot of QoS priority tag configuration for the MyVMTraffic traffic type.

With this post, I have completed the coverage of new networking features in vSphere 5. Also, today VMware has officially announced the genearal availability of vSphere 5.

I will be attending VMworld 2011 during the week of Aug 29th. At VMworld, I have a session on VDS best practices and couple of group discussions. Looking forward to meeting with various partners and customers.

After VMworld, I will focus my attention on writing about the different deployment options with vSphere Distributed Switch (VDS). 

Port mirroring is the capability on a network switch to send a copy of network packets seen on a switch port to a network-monitoring device connected to another switch port. Port mirroring is also referred to as Switch Port Analyzer (SPAN) on Cisco switches. In VMware vSphere 5, a Distributed Switch provides a similar port mirroring capability that is available on a physical network switch. After a port mirror session is configured with a destination—a virtual machine, a vmknic or an uplink port—the Distributed Switch copies packets to the destination.

Port mirroring provides visibility into

• Intrahost virtual machine traffic (virtual machine–to–virtual machine traffic on the same host)

• Interhost virtual machine traffic (virtual machine–to–virtual machine traffic on different hosts)

Figure below shows different types of traffic flows that can be monitored when a virtual machine on a host acts as a destination or monitoring device. All traffic shown by the orange dotted line with arrow is mirrored traffic that is sent to the destination virtual machine.

The terms Ingress source and Egress source are with respect to the VDS. For example, when you want to monitor the traffic that is going out of a virtual machine towards the VDS, it is called Ingress Source traffic. The traffic seeks ingress to the VDS and hence the source is called Ingress. If you want to monitor traffic that is received by a virtual machine, then configure the port mirroring session with the traffic source as Egress Source as shown in the top right corner diagram of the figure below.

Following figure shows the traffic flow when the mirror destination is configured as an uplink port. In this case both the normal traffic as well as mirror traffic flows through the same physical uplink.

When network administrators are concerned about the impact of the mirror traffic on normal traffic, they can choose a separate uplink port to send mirror traffic. The figure below shows the traffic flow when a separate uplink port is configured to carry mirror traffic.

Usage

The port mirroring capability on a Distributed Switch is a valuable tool that helps network administrators in debugging network issues in a virtual infrastructure. The granular control over monitoring ingress, egress or all traffic of a port helps administrators fine-tune what traffic is sent for analysis.

Configuration

Port mirror configuration can be done at the Distributed Switch level, where a network administrator can create a port mirror session by identifying the traffic source that needs monitoring and the traffic destination where the traffic will be mirrored to. The traffic source can be any port with ingress, egress or all traffic selected. The traffic destination can be any virtual machine, vmknic or uplink port.

The following figure is the first screen of port mirror session configuration process. In this step users can define the name of the port mirror session and choose if they want to allow normal I/O on a destination port. They can also choose a VLAN to encapsulate these mirrored packets by selecting the Encapsulations VLAN box.

Once you click next, the configuration screen will provide the option to choose the source that you want to monitor. Depending on what traffic you want to monitor you can choose Ingress, Egress, or Ingress/Egress in the traffic direction pull down menu as shown in figure below. Then specify the Port ID of that particular source VM. To get the corresponding dvPort number or Port ID number of a VM use the following steps:

1.      Switch to the Home > Inventory > Networking view.

2.      Select dvSwitch and choose the Ports tab on the right panel. Scroll down to see the virtual machines and the associated port ID.

Enter the port number in the Port ID field and move it to the right panel and then click next.

In the next step you have the option to configure the destination where you want to mirror the traffic to. There are two options provided in the Destination type pull down menu as shown in figure below.

This completes the creation of the port mirroring session. As shown in below, you can find the details about the port mirror session and also note that the status of the session is disabled.

To enable the port mirroring session, click Edit in the figure above. This will pop up the panel with an option to enable the session. Select the status as enabled as shown in figure below. This enables the port mirror session and VDS will mirror the traffic to selected destination port.

This covers the trouble shooting and monitoring features of vSphere 5. I will talk about the enhancements to the Network I/O control (NIOC) feature in my next post. So please stay tuned. 

As part of the Network Monitoring and Troubleshooting features, vSphere 5 provides NetFlow and Port Mirroring capabilities. In this blog entry I will discuss the NetFlow feature that is available in vSphere 5.

NetFlow

NetFlow is a networking protocol that collects IP traffic information as records and sends them to a collector such as CA NetQoS for traffic flow analysis. VMware vSphere 5 supports NetFlow v5, which is the most common version supported by network devices. NetFlow capability in the vSphere 5 platform provides visibility into virtual infrastructure traffic that includes

• Intrahost virtual machine traffic (virtual machine–to–virtual machine traffic on the same host)

• Interhost virtual machine traffic (virtual machine–to–virtual machine traffic on different hosts)

• Virtual machine to physical infrastructure traffic

Figure below shows a Distributed Switch configured to send NetFlow records to a collector that is connected to an external physical network switch. The blue dotted line with arrow indicates the NetFlow session that is established to send flow records for the collector to analyze.

Usage

NetFlow capability on a Distributed Switch along with a NetFlow collector tool helps monitor application flows and measures flow performance over time. It also helps in capacity planning and ensuring that I/O resources are utilized properly by different applications, based on their needs.

IT administrators who want to monitor the performance of application flows running in the virtualized environment can enable flow monitoring on a Distributed Switch.

Configuration

NetFlow on Distributed Switches can be enabled at the port group level, at an individual port level or at the uplink level. When configuring NetFlow at the port level, administrators should select the NetFlow override tab, which will make sure that flows are monitored even if the port group–level NetFlow is disabled.

The NetFlow configuration screen below shows the different parameters that can be controlled during the setup.

1.      The Collector Settings of IP address and Port should be configured according to the information collected about the collector tool installed in your environment.

2.      The Advanced Settings parameters allow you to control the timeout and sampling rate for the flows. To change the amount of information that is collected for a flow, you can change the sampling rate. For example, a sampling rate of 2 indicates that the VDS will collect data from every other packet. You can also modify the Idle flow export timeout values.

3.      The VDS IP address configuration is useful when you want to see all flow information in the collector tool as part of one VDS IP address and not as a separate host management network IP address. In this example screen shot, because the VDS IP address is not entered, the collector tool will provide flow details under each host’s management network IP address.

You can also monitor only internal flows of the virtual infrastructure by checking “Process Internal flows only” box.

I almost always get the question about the CPU impact of enabling NetFlow feature. Just wanted to address that while I am on this topic. Answer is, it all depends on how many flows you have in your environment and what traffic rate they are operating at. If you think you have lot many flows in your environment and are concerned about CPU resources, you can use the controls provided in the NetFlow setup to choose which flows gets monitored. For example, you can change the sampling rate or choose to monitor only internal flows. Also, you can selectively enable or disable NetFlow on a port group or a port.

As customers move to virtualize Tier 1 applications, they need the proper tools to manage the SLA requirements of these applications. NetFlow feature on vSphere 5 platform helps in monitoring these tier 1 application flows and also helps in capacity planning of network resources.

In the next post I will talk about the Port-mirroring feature

Before I introduce you to the new networking features in vSphere 5, I want to take a moment and introduce myself first. My name is Venky and I work in the Technical Marketing group at VMware. I am responsible for technical marketing activities around vSphere Networking features. I am really excited to share and learn with you on this forum. I would like to kick this journey off by introducing the new networking features.

There are two broad types of networking capabilities that are new or enhanced in the VMware vSphere 5 release. The first type improves the network administrator’s ability to monitor and troubleshoot virtual infrastructure traffic by introducing features such as

• NetFlow V5

• Port mirror (SPAN)

I am sure Network administrators are very happy to hear this. Now they have the required visibility into the virtual infrastructure to monitor and troubleshoot network issues. Also, support for LLDP (standard based discovery protocol) simplifies the network configuration and management in non-Cisco switch environment.

The second type focuses on enhancements to the network I/O control (NIOC) capability first released in vSphere 4.1. These NIOC enhancements target the management of I/O resources in consolidated I/O environments with 10Gb network interface cards. The enhancements to NIOC enable customers to provide end-to-end quality of service (QoS) through allocating I/O shares for user-defined traffic types as well as tagging packets for prioritization by external network infrastructure. The following are the key NIOC enhancements:

• Ability to create User-defined resource pool

• Support for vSphere replication traffic type

• Support for IEEE 802.1p tagging

These Traffic management features are very useful when customers are thinking of deploying Tier1 applications or hosting multiple tenants with different I/O resource requirements.

In the next blog entry I will talk more about the Monitoring and troubleshooting features.