Home > Blogs > VMware vSphere Blog

vSphere 5.1 – Network I/O Control (NIOC) Architecture – Old and New

Recently there has been some discussion around the egress traffic management feature of vSphere Distributed Switch (VDS) also called as Network I/O Control (NIOC). Thanks to my colleague Frank Denneman for providing more details about this feature on his blog site and bringing to my attention an architectural change in the vSphere 5.1 release. This change impacts how the Limit parameters are applied at the host level. In this post, I will first describe the old architecture of NIOC and then discuss the change. I will also talk about the impact of this change and what users need to keep in mind while configuring limit parameter.

Let’s first take a look at the NIOC components and architecture in the previous releases of vSphere. The diagram below shows a vSphere host with two 10 gig NICs, VDS components, NIOC configuration table, and different traffic types running on the host.

NIOC – Old Architecture

The following are the key architectural layers in the VDS:

1)   Teaming Policy layer: This is the layer where teaming policy defined at the port group level gets applied. Based on the choice of the load balancing algorithm and the active/passive uplink configurations at the port group level, this layer will determine which traffic from the virtual ports (vnic,vmknic) will be sent over the which physical NICs ( uplinks). For example, when the default load-balancing algorithm of “Route based on originating virtual port” is selected, based on the port ID hash output traffic will be sent on Physical NIC1 or Physical NIC2. The diagram above shows VM1 traffic flowing through NIC1 and VM2 through NIC2.

2)   Shaper : This is the layer where the Limit parameter configured for a traffic type is enforced. For example, if the virtual machine traffic is limited to 2 Gig any additional virtual machine traffic will be dropped at this layer even if physical NICs have capacity. In the diagram above total VM1 and VM2 traffic can’t exceed 2 Gig of bandwidth.

3)   Scheduler: In this layer scheduler schedules traffic on physical NIC based on the share configurations per traffic type. There is one scheduler instantiated per physical NIC, and it handles all the traffic that is sent by the teaming layer. The share parameter is used to decide what percentage of physical NIC bandwidth is allocated to that particular traffic type.

Based on the number of active traffic types scheduled on a particular uplink and their share configuration, the scheduler will make sure the bandwidth is distributed among the traffic types according to the share configurations. If one of the traffic types is not utilizing any bandwidth then that bandwidth gets distributed among other traffic types. Shares configuration thus provides flexibility and better utilization of bandwidth. However, it is important to note that the limit parameters override the shares parameter configuration. If there is a limit parameter configured for a traffic type, it can’t utilize the unused bandwidth of a NIC and go over the assigned limits. As a best practice we recommend not to use limit parameter unless user really wants to control a traffic type.

After looking at the old architecture let’s now talk about the changes in the new vSphere release. As shown in the diagram below, you will find that there is a shaper instantited per physical NIC similar to the scheduler function per NIC. In this release instead of applying limit parameters across the whole teaming level they are now applied per physical NIC level.

NIOC – New Architecture

So the question is what is the impact of this change? Let’s take the VM traffic type example to illustrate.

As shown in the table in the above diagram, the VM Traffic is limited to 2 gig bandwidth. The VM Traffic consists of VM1 and VM2 communication, and it utilizes both the physical NICs. In this situation each shaper is going to make sure that it doesn’t send more than 2 gig of VM traffic. If you take a look at the overall VM traffic coming out of the host, it will be (2 + 2) 4 Gig. This is different from what we saw in the old architecture where the limit is applied across the two NICs and max VM traffic out of the host is 2 Gig as configured in the table.

This change is going to impact only traffic types that are utilizing multiple physical NICs. You will not see any difference how limits are applied for infrastructure traffic types such as vMotion and iSCSI, unless you are using multi-NIC vMotion or multi-pathing (MPIO) i.e more than two uplinks for a particular traffic type.

When you look at the configuration screen shown below, we still call the limits as “Host Limit (Mbps)”. We might have to change that as “Uplink Limit (Mbps)” to reflect how the limits are applied. But for now, if you are using limit parameters you should find out if that traffic type is flowing through multiple NICs. If yes, then, to calculate the overall host level limit for that traffic type, multiply the configured limit parameter with the number of NICs on which the traffic is flowing. For example, in the diagram above, VM traffic is configured with 2 Gig limit and the traffic is flowing through 2 NICs. The overall host level VM traffic is limited to 2gig x 2 NICs = 4 Gig.

Screen Shot – Host Limits

Please let me know if you have any specific questions on this feature.

Get notification of these blogs postings and more VMware Networking information by following me on Twitter:  @VMWNetworking

8 thoughts on “vSphere 5.1 – Network I/O Control (NIOC) Architecture – Old and New

  1. Pavel V.

    Excellent article.

    But I can imagine possible problem with the new architecture.
    If I configure limits as described I am afraid of situation when one of uplinks failed.
    Suddenly each limit is doubled but on single uplink, isn’t it?


    1. Vyenkatesh Deshpande Post author

      Hi P,

      Glad you liked the post.
      Now to your question.
      The Limits are assigned to traffic types.
      If you take the example of VM traffic where the limit is configured as 2Gig.
      When there is a failure of one uplink (NIC 2) , all VM traffic will be moved to other uplink (NIC1).
      However, the limit still remains 2 Gig on uplink NIC1. It doesn’t go up to 4 gig.
      Hope this helps.
      Let me know if you have any more questions.

    2. Vyenkatesh Deshpande Post author

      Hi Pavel,
      As I mentioned in my earlier comment the Limit doesn’t get doubled when one of the Link fails.
      For example, If limit is configured as 2 Gig and one of the uplink fails instead of overall 4 Gig you now have only 2 Gig allowed.

  2. srinivas

    Hello Vyenkatesh, I have two queries for you.
    1. How do I limit the Org network bandwidth in vCloud Director 5.1 ?
    2. Where is this screen shot ‘hosts-limits-screen’ taken? in vCD? I am using vCD 5.1 and I cannot see this screen on my environment :-(

    1. Vyenkatesh Deshpande Post author

      Hi Srinivas,

      NIOC is not exposed through the vCloud Director.
      The screen shot that I showed is vSphere Next Generation Client.

  3. Phil Cohen

    While it’s true NIOC settings are not exposed to the vCD interfaces, NIOC can still be used and configured as per usual via vSphere. But I would recommend using it very judiciously as you could inadvertently do more harm than good if your share values are not good for your environment.

    A use case I would consider would be on vCD resource groups comprised of blade servers with minimal slots for network cards. So, for example, if your blades only had two nics, you would of course team both NICs into a single vDS that is separating the core network traffic types via VLANs in separate port groups. Then when you build out the vSwitches to support the vCD network pools, you would setup the NIOC resource pools on those switches with share values appropriate to VM Traffic. Keep in mind you would probably want to prioritize VMotion traffic and IP Storage traffic (if used) slightly over the VM traffic and Mgmt traffic as contention there could have an impact on DRS and SDRS cluster load distribution. So test, test, test!!!

    However, once your ratios are acceptable, Org vDC networks and vApp networks will have port groups dynamically created (assuming not using port-group backed pools) to support the traffic of the workloads running in the vCD Org vDCs. Then comes the chore of monitoring and maintaining as needed. :)

  4. The Site

    Hi, i think that i saw you visited my weblog so i came to “return the favor”.I am trying to find things to enhance my web
    site!I suppose its ok to use a few of your ideas!!


Leave a Reply

Your email address will not be published. Required fields are marked *