Home > Blogs > Virtualize Business Critical Applications


RUSH POST: VMware Tools and RSS Incompatibility Issues

VMware is aware of the following reported issues affecting Windows Server 2012 / 2012R2 and newer Guest Operating Systems on VMware vSphere:

  • The Windows Receive Side Scaling (RSS) feature is not functional on virtual machines running VMware Tools versions 9.10.0 up to 10.1.5
  • Some Windows Virtual Machine running these versions of the VMware Tools may experience increase in rate of “Received Dropped Packets”, caused by the fact that all network traffic will be serviced by only one CPU in the Guest. This would occur only if vCPU 0 is overwhelmed and cannot keep up with the rate of the network transmit/receive.
  • This condition results in performance degradation for the applications and services hosted on the affected Virtual Machine.

NOTE: RSS is disabled by default on VMXNet3 virtual NICs in Windows. VMware encourages enabling RSS only for applications requiring high network throughput and large bursts.

VMware is working on creating a fix for this issues.

VMware advises any customer experiencing the issue described to contact VMware’s Global Support Services for immediate assistance.

NOTE: If you are experiencing dropped packets at the NIC level, it may be because of the described bug (but also to other causes such as network congestion). If you are not seeing dropped packets, the driver has sufficient CPU power to process all packets. If you see dropped packets and can confirm that the root cause is with the VMXNET3 driver, we recommend that you evaluate the risks of downgrading versus status quo. If the applications running on the VM are not sensitive to dropped packets, it may not be needed to do anything before we have the permanent fix.

22 thoughts on “RUSH POST: VMware Tools and RSS Incompatibility Issues

  1. Jeremy

    If a Windows Server 2012 R2 running tools 10.0.9 shows 0 receive packets dropped in the vSphere performance tab, and receive side scaling is on, does that indicate the problem is not occurring? Is there a recommended way to check receive dropped packets is occurring?

    Reply
    1. Deji Post author

      The VMkernel System Information Shell command (vsish) is the place to look. Please don’t do this in a production environment (or without guidance from VMware Support) unless you are very familiar with the use of this shell. Incorrect use of “vsish” can create problems in your environment.

      Reply
    1. Deji Post author

      We recommend manually enabling RSS on modern Windows versions IF the VM has more than 1 CPU. This is more relevant for IO-intensive workloads.

      Reply
          1. Joshua Post

            Is this bulletin stating that RSS is not enabled by default with the affected versions of VMware Tools, but doing the netsh and Device Manager changes in the link you posted will still enable it, or is the defect that it still doesn’t work, even with those two changes?

          2. Deji Post author

            The defect is that RSS doesn’t work with those impacted VMware Tools versions, regardless of the state of RSS reported in Windows.

  2. Matts

    This feels a little incomplete. Does this problem affect all 9.10.x and all 10.1.x? Or are we safe running 10.1.0 or 10.0.9? Or does it affect all versions newer than 9.4.15?

    Reply
    1. Deji Post author

      Yes, this is a “Rush Post”, so it will feel a bit incomplete. But, the advisory says “VMware Tools versions 9.10.0 up to 10.1.5”, so versions 10.1.0 and 10.0.9 would fall within that range. In short any version from 9.10.0 all the way up to (and including) 10.1.5 is impacted. HTH

      Reply
  3. Prabhash Jena

    Is the new tools version also impacting PVSCSI controller? After the VMtools upgrade on 2012R2 Terminal servers (RSS disabled) the logon time increased to 5-6 minutes. I uninstalled VMtools and installed the recommended version also changed SCSI from PVSCSI to LSI SAS and logon time has improved (RSS still disabled).

    Reply
    1. Deji Post author

      We are not sure of the correlation between your logon time and PVSCSI. Your logon time will more likely than not be impacted by network stability. Could you try switching back to PVSCSI to see if the problem comes back? Thanks

      Reply
  4. _nd345as

    Thanks for the clarification that this is only true if RSS is manually enabled, which by default it is not. It will be important for people to know such and therefore should be included in the rush post not to necessarily downgrade unless affected.

    Reply
  5. Diego Azevedo

    Just wondering if there have been any updates on this. I was ready to start upgrading Vmware Tools to 10.1.5 on about 300 VMs, but decided to wait on a new release so as to avoid having to do it again later.

    Reply
    1. Deji Post author

      @Diego, we are working on the public release of the patch. If you are currently experiencing the issue described, please contact VMWare’s Technical Support and mention that you are experiencing this issue. They know what to do.

      Thanks

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *


*