vSphere 5.1 – Network I/O Control (NIOC) Architecture - Old and New

Recently there has been some discussion around the egress traffic management feature of vSphere Distributed Switch (VDS) also called as Network I/O Control (NIOC). Thanks to my colleague Frank Denneman for providing more details about this feature on his blog site and bringing to my attention an architectural change in the vSphere 5.1 release. This change impacts how the Limit parameters are applied at the host level. In this post, I will first describe the old architecture of NIOC and then discuss the change. I will also talk about the impact of this change and what users need to keep in mind while configuring limit parameter.

Let’s first take a look at the NIOC components and architecture in the previous releases of vSphere. The diagram below shows a vSphere host with two 10 gig NICs, VDS components, NIOC configuration table, and different traffic types running on the host.

The following are the key architectural layers in the VDS:

1) Teaming Policy layer: This is the layer where teaming policy defined at the port group level gets applied. Based on the choice of the load balancing algorithm and the active/passive uplink configurations at the port group level, this layer will determine which traffic from the virtual ports (vnic,vmknic) will be sent over the which physical NICs ( uplinks). For example, when the default load-balancing algorithm of “Route based on originating virtual port” is selected, based on the port ID hash output traffic will be sent on Physical NIC1 or Physical NIC2. The diagram above shows VM1 traffic flowing through NIC1 and VM2 through NIC2.

2) Shaper : This is the layer where the Limit parameter configured for a traffic type is enforced. For example, if the virtual machine traffic is limited to 2 Gig any additional virtual machine traffic will be dropped at this layer even if physical NICs have capacity. In the diagram above total VM1 and VM2 traffic can’t exceed 2 Gig of bandwidth.

3) Scheduler: In this layer scheduler schedules traffic on physical NIC based on the share configurations per traffic type. There is one scheduler instantiated per physical NIC, and it handles all the traffic that is sent by the teaming layer. The share parameter is used to decide what percentage of physical NIC bandwidth is allocated to that particular traffic type.

Based on the number of active traffic types scheduled on a particular uplink and their share configuration, the scheduler will make sure the bandwidth is distributed among the traffic types according to the share configurations. If one of the traffic types is not utilizing any bandwidth then that bandwidth gets distributed among other traffic types. Shares configuration thus provides flexibility and better utilization of bandwidth. However, it is important to note that the limit parameters override the shares parameter configuration. If there is a limit parameter configured for a traffic type, it can’t utilize the unused bandwidth of a NIC and go over the assigned limits. As a best practice we recommend not to use limit parameter unless user really wants to control a traffic type.

After looking at the old architecture let’s now talk about the changes in the new vSphere release. As shown in the diagram below, you will find that there is a shaper instantited per physical NIC similar to the scheduler function per NIC. In this release instead of applying limit parameters across the whole teaming level they are now applied per physical NIC level.

So the question is what is the impact of this change? Let’s take the VM traffic type example to illustrate.

As shown in the table in the above diagram, the VM Traffic is limited to 2 gig bandwidth. The VM Traffic consists of VM1 and VM2 communication, and it utilizes both the physical NICs. In this situation each shaper is going to make sure that it doesn’t send more than 2 gig of VM traffic. If you take a look at the overall VM traffic coming out of the host, it will be (2 + 2) 4 Gig. This is different from what we saw in the old architecture where the limit is applied across the two NICs and max VM traffic out of the host is 2 Gig as configured in the table.

This change is going to impact only traffic types that are utilizing multiple physical NICs. You will not see any difference how limits are applied for infrastructure traffic types such as vMotion and iSCSI, unless you are using multi-NIC vMotion or multi-pathing (MPIO) i.e more than two uplinks for a particular traffic type.

When you look at the configuration screen shown below, we still call the limits as “Host Limit (Mbps)”. We might have to change that as “Uplink Limit (Mbps)” to reflect how the limits are applied. But for now, if you are using limit parameters you should find out if that traffic type is flowing through multiple NICs. If yes, then, to calculate the overall host level limit for that traffic type, multiply the configured limit parameter with the number of NICs on which the traffic is flowing. For example, in the diagram above, VM traffic is configured with 2 Gig limit and the traffic is flowing through 2 NICs. The overall host level VM traffic is limited to 2gig x 2 NICs = 4 Gig.