Home > Blogs > VMware TAM Blog


Network Troubleshooting with VMware Native Tools

Petr McAllisterBy Petr McAllister

Recently, a network team of one of my customers expressed interest in getting gradual statistics of workloads in the virtual environment. Any network interface cards (NICs) – physical or virtual – were under the definition of virtual environment. However, the focus was more on physical uplinks.

In this post we will look at the steps of collecting the network stats in vSphere environments by moving from very specific stats to a more aggregated analysis. In real life situations, it’s very likely that you’ll have to follow the opposite path – look at the environment from 30 thousand feet and then get closer and watch the item that concerns you.

So, in our troubleshooting scenario, we connect directly to one of the hosts using SSH protocol. You always have a choice of running commands locally on the host or using centralized tools called vSphere Management Assistant (vMA). (Details on the tool can be found here: https://blogs.vmware.com/kb/2010/03/what-is-the-vsphere-management-assistant-vma.html.)

After connecting to the host, we issue the following command: “esxcli network nic list”. This returns the list of physical NICs that are installed on this specific host.

PMcAllister Network NIC List

After checking with the network team and figuring out what exact NIC is the point of interest, we could execute “ethtool –S <NIC Name>” command to get the detailed stats. Here is a possible output:

PMcAllister ETH Tool

Next, we wanted to see life statistics of all physical and virtual NICs on the host in real time. So we started esxtop utility (if you use vMA it will be ‘resxtop’). To switch to the network data, you simply hit the letter “n”. To adjust columns displayed, the letter “f” is used, and then fields can be chosen. A more detailed description of esxtop can be found at Duncan Epping’s blog at http://www.yellow-bricks.com/esxtop/

PMcAllister Port ID

As you can see, stats for both physical and virtual are displayed here. If you’re interested in what “Shadow of vmnicX” lines are – they’re simply heartbeat instances of uplink ports to monitor these uplinks. More details are in William Lam’s blog – http://www.virtuallyghetto.com/2013/01/what-are-shadow-of-vmnic-in-esxtop.html

Now to the next question. Let’s say there is a significant number of hosts and we are required to check network stats on all those hosts. Connecting to each of those hosts and running ‘esxtop’ would be a time-consuming operation. Is there a way to aggregate this information and look at summarized numbers?

Fortunately, my customer owns and runs vRealize Operations Manager 6. I can’t thank my colleague enough who demonstrated to my customer how easy it is to get the following screen in vRealize Operations Manager 6.

Just follow these three simple steps after you log in to vRealize Operations 6:

  • Select the object on the left (can be your whole environment, data center, cluster, etc.)
  • Click on ‘Details’ tab at the top right
  • Select ‘Host Network Diagnose List’ in the middle of the screen

That’s it – you have a summary screen of exactly the same stats that every host would individually report through ‘esxtop’ utility. Isn’t it wonderful?

PMcAllister vRealize Operations Manager

As was mentioned at the beginning of this article, you can go in an opposite direction – look at the vRealize Operations report, select a specific host, and then go to command line at this host and run additional commands there.

Also, we must mention here that VMware Virtual Distributed switch (vDS) provides you with the ability to supply its stats through Netflow protocol. More details can be found here: https://blogs.vmware.com/vsphere/2011/08/vsphere-5-new-networking-features-netflow.html

To learn more about using vRealize Operations Manager, use the FREE VMware Hands-on-Labs at http://labs.hol.vmware.com/. Specifically, “HOL-SDC-1610 Virtualization 101: vSphere with Operations Management 6” and “HOL-SDC-1602 vSphere with Operations Management 6: Advanced Topics.”


Petr McAllister is a VMware Technical Account Manager based out of Vancouver, British Columbia, Canada.

One thought on “Network Troubleshooting with VMware Native Tools

Comments are closed.