Home > Blogs > VMware TAM Blog

Network Troubleshooting with VMware Native Tools

Petr McAllisterBy Petr McAllister

Recently, a network team of one of my customers expressed interest in getting gradual statistics of workloads in the virtual environment. Any network interface cards (NICs) – physical or virtual – were under the definition of virtual environment. However, the focus was more on physical uplinks.

In this post we will look at the steps of collecting the network stats in vSphere environments by moving from very specific stats to a more aggregated analysis. In real life situations, it’s very likely that you’ll have to follow the opposite path – look at the environment from 30 thousand feet and then get closer and watch the item that concerns you.

So, in our troubleshooting scenario, we connect directly to one of the hosts using SSH protocol. You always have a choice of running commands locally on the host or using centralized tools called vSphere Management Assistant (vMA). (Details on the tool can be found here: https://blogs.vmware.com/kb/2010/03/what-is-the-vsphere-management-assistant-vma.html.)

After connecting to the host, we issue the following command: “esxcli network nic list”. This returns the list of physical NICs that are installed on this specific host.

PMcAllister Network NIC List

After checking with the network team and figuring out what exact NIC is the point of interest, we could execute “ethtool –S <NIC Name>” command to get the detailed stats. Here is a possible output:

PMcAllister ETH Tool

Next, we wanted to see life statistics of all physical and virtual NICs on the host in real time. So we started esxtop utility (if you use vMA it will be ‘resxtop’). To switch to the network data, you simply hit the letter “n”. To adjust columns displayed, the letter “f” is used, and then fields can be chosen. A more detailed description of esxtop can be found at Duncan Epping’s blog at http://www.yellow-bricks.com/esxtop/

PMcAllister Port ID

As you can see, stats for both physical and virtual are displayed here. If you’re interested in what “Shadow of vmnicX” lines are – they’re simply heartbeat instances of uplink ports to monitor these uplinks. More details are in William Lam’s blog – http://www.virtuallyghetto.com/2013/01/what-are-shadow-of-vmnic-in-esxtop.html

Now to the next question. Let’s say there is a significant number of hosts and we are required to check network stats on all those hosts. Connecting to each of those hosts and running ‘esxtop’ would be a time-consuming operation. Is there a way to aggregate this information and look at summarized numbers?

Fortunately, my customer owns and runs vRealize Operations Manager 6. I can’t thank my colleague enough who demonstrated to my customer how easy it is to get the following screen in vRealize Operations Manager 6.

Just follow these three simple steps after you log in to vRealize Operations 6:

  • Select the object on the left (can be your whole environment, data center, cluster, etc.)
  • Click on ‘Details’ tab at the top right
  • Select ‘Host Network Diagnose List’ in the middle of the screen

That’s it – you have a summary screen of exactly the same stats that every host would individually report through ‘esxtop’ utility. Isn’t it wonderful?

PMcAllister vRealize Operations Manager

As was mentioned at the beginning of this article, you can go in an opposite direction – look at the vRealize Operations report, select a specific host, and then go to command line at this host and run additional commands there.

Also, we must mention here that VMware Virtual Distributed switch (vDS) provides you with the ability to supply its stats through Netflow protocol. More details can be found here: https://blogs.vmware.com/vsphere/2011/08/vsphere-5-new-networking-features-netflow.html

To learn more about using vRealize Operations Manager, use the FREE VMware Hands-on-Labs at http://labs.hol.vmware.com/. Specifically, “HOL-SDC-1610 Virtualization 101: vSphere with Operations Management 6” and “HOL-SDC-1602 vSphere with Operations Management 6: Advanced Topics.”

Petr McAllister is a VMware Technical Account Manager based out of Vancouver, British Columbia, Canada.

Use Your Shelf Space for Industry Awards…Not Unused Software

Why the VMware America’s TAM organization is rated higher than the Apple iPhone

By Martha Bellens-Martin

Martha BellensMartinAfter being in this business for more than 25 years, I have worked with hundreds of customers—most of them from a pre-sales perspective. That means I have spent a lot of time working with organizations to help them envision how a proposed solution will lower risk and address their pain points.

But one of my personal pain points was walking away after the sale and watching the software gather dust on a shelf. Sometimes it was because the customer lacked the internal capabilities to successfully implement the solution. Sometimes it was because they were not able to fully leverage all of the features that would allow them to realize the power of the whole solution. And sometimes it was because the goals of the business simply changed direction. Whatever the reason, it still bothered me to see those well-thought-out solutions turning into shelf ware.

Now that I have joined the VMware America’s TAM organization, I have the opportunity to make sure those software boxes never hit the shelves. That’s because the TAM organization is completely focused on making sure you realize all the benefits of the software you have purchased from VMware. If you have never worked with a TAM, it might be difficult to envision what the TAM will do for your organization. Of course everything your TAM does would be hard to capture in a single blog post, but the main objectives of your TAM can be summed up with the following points.

1. Ensure you are getting the most value from your solution

Pre-sales is focused on helping you identify a solution to your problems. After the sale, your TAM continues to work with you to make sure the licenses you purchased are delivering the value you envisioned before the sale. We start the process by helping you plan implementation and training as well as working with you to ensure the project aligns to your original goals. And over the course of our engagement, we help make sure the alignment stays in place.

2. Find new ways that your VMware solution can impact the whole business

While you may have purchased a VMware solution for one reason, your TAM can show you ways that your purchase can help the business in other ways. In fact, one of your TAM’s main objectives is to look at how VMware can benefit your business as a whole. TAMs often have a unique perspective on your business due to the fact that they are approaching it with fresh eyes, from a perspective that sees other businesses and other industries.

3. Turn you into an enthusiastic VMware ambassador

Ensuring you are satisfied and receiving the maximum value for your solution helps you, but it also helps us, too. We work hard to ensure your solution exceeds your expectations so you will be thrilled to recommend VMware and VMware TAMs to your network.

VMware TAMs work hard to accomplish all of this, and it pays off for our customers. The way we know it is working is by looking at our Net Promoter Scores (NPS). If you’re not familiar with the NPS benchmark, it is calculated using the answer to a single question: “How likely is it you would recommend a VMware America TAM to a friend or colleague?” (You can visit www.netpromoter.com/know for more details.)

We are very proud of the fact that our TAMs have received an NPS of 67.5 percent. To put that in perspective, one of the most popular and beloved tech gadgets in recent history—the Apple iPhone—has an NPS of 64 percent.*

Find out for yourself why our customers love us more than the Apple iPhone. Contact VMware to learn about how you can tap the expertise of your own VMware TAM.


Martha manages and leads a team of Technical Account Managers for VMware. Martha believes, “too often we see companies make a significant purchase of a solution yet stumble in the implementation or even the full realization of the power of the whole solution.” Martha Bellens-Martin has worked in the Software Solutions industry since 1990, primarily focused on demonstrating Software Solutions to help businesses run more efficiently, and effectively and to drive value to the bottom line. Her specialties include Effective Technical Presentations and Coaching of Technical Resources. Connect with her on

Platform Services Controller (PCS) and vCenter Server 6 Maximums

Petr McAllisterBy Petr McAllister

One of my customers successfully completed the VMware vSphere: Fast Track [V6] class. The customer provided a lot of positive feedback in regards to the class, and also about new functionality in vSphere 6. However, one thing was unclear: The instructor stated there is a maximum of 10 VMware solutions per vCenter. So the question was, “When we run a complex environment with multiple vCenter servers, vRealize Operations servers, vRealize Automation, vRealize Orchestrator, Site Recovery Manager and backup appliances, how can we fit all those solutions under the 10-solution limit?”

Finding the correct answer was pretty straight forward; VMware has published a document called “Configuration Maximums vSphere 6.0,” and the information is right there. The document has very specific content on exactly what my customer was asking:

A VMware Solution is defined as a product that creates a Machine Account and one or more Solution Users (a collection of vSphere services) … At this time, only vCenter Server is defined as a fully integrated solution and counts against these maximums. Partially integrated solutions, such as vCenter Site Recovery Manager, vCloud Director, vRealize Orchestrator, vRealize Automation Center, and vRealize Operations, do not count against these defined maximums.”

It would be easy to conclude my blog post here, but the nature of my topic is a little bit different. Looking through the PSC section of the “Configuration Maximums vSphere 6.0” document can be somewhat confusing. You’ll notice different unit maximums, some of which are specified as “per vSphere Domain,” “per site,” or “per Single PSC.”



The best way to understand PSC maximums is via a diagram found in the VMware Knowledge Base (KB) article, “List of recommended topologies for VMware vSphere 6.0.x,” which is a brilliant source of information on its own.

PMcAllister PSC

Assume User A has access to all four vCenter servers. When User A is authenticated in the Single Sign-on domain (also known as vSphere domain), the user can:

  • Log in to Site A or Site B using the same credentials
  • See all four vCenter servers in the environment (because these vCenter servers are members of the same SSO domain)
  • Accomplish any task on any of the vCenter servers the user has permissions on, and perform operations that involve multiple vCenter servers as inter-vCenter vMotions

Just to be clear, here is another example: If User B has access to only one vCenter server, he/she will still be able to log in—with the same credentials—to any site that is in the same SSO domain and do any operation that User B has permissions for – but only in the permitted vCenter.

Now let’s move on to the “per single PSC” definition. The PSC can be installed as embedded on the same server with other vCenter components, but in this case, the embedded PSC serves only one vCenter server. For any multi-vCenter server and/or multi-site configuration, PSC has to be installed as an external module on a separate machine in order to serve multiple vCenter servers. But the external PSC has maximums that are specified in the “vSphere 6 Configuration Maximums” document. These limits were introduced to ensure your infrastructure functions at a good performance level.

The final term to explain here is “vSphere Site,” which is partially self-explanatory, but it would help to be a little bit more specific. The KB article “VMware Platform Services Controller 6.0 FAQs” has the best definition of a vSphere 6 site:

A site in the VMware Directory Service is a logical container in which we group Platform Services controllers within a vSphere Domain. You can name them in an intuitive way for easier implementation. Currently, the use of sites is for configuring PSC High Availability groups behind a load balancer.”

So in other words we expect the best possible connection between sites (in terms of latency and bandwidth); however, in case of connectivity issues, every site can be autonomous—for a while—serving with full functionality – with the exception of operations that require connection to another site. PSC will get synchronized when the connection is restored. You might have more questions on PSC in vSphere 6, and if you do, the KB article noted above will answer most of those questions, and reading it is certainly a good investment of your time.

Petr McAllister is a VMware Technical Account Manager based out of Vancouver, British Columbia, Canada.

Adding More Benefits for Our Valued Customers: 2015 TAM Service Enhancements

By Steve Kemp

Our global TAM community works relentlessly to improve TAM Services, and we work relentlessly to improve TAM Service excellence and provide even more value to you, our VMware TAM customers. As 2015 draws to a close, we’re taking a moment to reflect on the value-added services and programs we’ve added this past year. The following is just a sampling of what we’ve put into play in 2015:

Expanded service offerings: Two new TAM services, NSX TAM and vRB TAM, focus on helping you succeed with new and advanced technologies, namely VMware network virtualization (NSX) and business intelligence (VMware vRealize Business Suite) offerings.

Direct access to specialized expertise: We have established a team of TAMs who are VMware product and technology experts in specific areas, and gave all of our TAMs a direct line to them. This structured, instant line of access makes all of our TAMs better at fielding requests and providing advice. Plus, it’s an ideal way for your TAMs to quickly get your feedback and requests to the right VMware product team.

On-demand access to high-value information: Three popular new communication tools give you instant access to high-value information about programs, education, break/fixes, and current alerts:

  • The TAM Source Newsletter is not your average newsletter. For starters, you don’t subscribe to it. Your TAM personally curates a custom newsletter just for you from a long list of articles, product announcements and other items, sending you only the subset that is most relevant to your needs. Be sure to ask your TAM about this cool new twist on the traditional newsletter.
  • The VMware TAM Blog was created in response to customer demand, and provides a steady stream of the latest news and information. You can bookmark it, sign up for the RSS feed, or follow us on Twitter (@vmwaretam).
  • We recently began hosting a series of TAM Customer Webcasts covering a variety of topics. Most of these webcasts are available on-demand after they are presented, so you can access them whenever it is convenient. From that same link, you can also view the upcoming live webinar events.

Expanded TAM offerings at VMworld: In case you missed it, the turnout at this year’s VMworld far exceeded expectations. And as usual, TAM Day was one of the most popular events. This year, in response to attendee feedback, we included an expanded, two-hour version of our Ask the Experts luncheon. TAM customers often tell us that this luncheon is the most impactful session they attend at VMworld because it gives them direct access to multiple experts in various product areas. Here, they can influence our product direction and/or get questions answered directly from the experts who are creating our solutions. We’re looking forward to building on these kinds of events to generate even more informative customer conversations at VMworld 2016 in Las Vegas (August 28 – September 1). We are planning to add more small-group sessions and one-on-one meetings with engineers, where you can hear about NDA programs, product roadmaps, and other insider developments. Hope to see you there!

We’re proud to have been a part of your success this year, and we look forward to creating even more value with new and enhanced programs and services in 2016. If you have suggestions for how we can improve our service offerings, please leave a comment or share them with your TAM.

Steve Kemp joined VMware in April 2006 and helped grow the TAM Program from 12 to over 300 TAMs globally.  As Director of the US West TAM Program, he leads VMware’s leading customer success program for most of private sector, leveraging 20 years of high-tech industry experience including software and hardware infrastructure, networking and storage.

Code Stream: Bridging the Gap Between Development and Operations

Kelly DareBy Kelly Dare

When our customers start automating their infrastructure, some of the first internal customers or users of their automation tools are almost always the software developers in their organization. Infrastructure-as-a-Service and software developers are a natural fit, since software developers need a high level of autonomy to get their machines created on their timelines and to their specs. The business typically supports this wholeheartedly, since the software they are developing is often crucial to the business and/or generates revenue.

However, there is a fundamental conflict between the goals of the Developer and Operations (Dev/Ops) groups. Dev wants to release software fast and often, integrating small changes into their code base. Ops wants slower, well-tested releases, because more churn equals more chance for things to go wrong. A great deal of the time, the software development and release process includes some combination of automation and manual steps in a complex workflow—that works well for a slower release model—but when you attempt to move to an accelerated release pace, these complexities and manual steps become bottlenecks, and the process ends up straining your organization.

VMware understands these issues, and has created Code Stream—an automated DevOps tool—as part of the VMware vRealize Automation suite. It enables our customers to release their software more frequently and efficiently, with a high level of collaboration among Dev and Ops teams. If your organization has a Continuous Delivery or DevOps initiative, Code Stream can significantly accelerate your progress in those areas. Using Code Stream does not require you to change anything about your current process. You can begin by modeling your current process, and Code Stream will mature—along with your processes—all the way to a fully automated release cycle if you so choose.

KDare Software Manager Download

With vRealize Automation, you can leverage just about any existing automation process you already have by moving them into the extensible framework of vRealize Automation. Similarly, you can also take advantage of nearly any software lifecycle tools you have already invested in by connecting them into the extensible framework of Code Stream. Your current source control system, testing frameworks, and build/continuous integration tools can remain the same – you just begin to access them via Code Stream rather than multiple interfaces. Code Stream contains Artifactory for intelligent storage of all your binary artifacts, which allows for the use of nearly any provisioning and configuration management tool. You can bring along all your existing development tools such as Puppet, Chef, SaltStack, or even plain old scripts to continue to build upon them in Code Stream.

KDare Code Stream 2

Once your existing model is in Code Stream, you can continue to further automate your software delivery pipeline as much as you please, up to a fully automated model. Users can reference the Release Dashboard at any time to view the current status of any release, as well as drill down into the details of each deployment if needed.

For more information about Code Stream, follow the links below or ask your VMware account team!

Kelly is a Technical Account Manager for VMware based in Austin, Texas and serving accounts in the Austin and San Antonio areas. She has worked in many capacities in the technology field, and enjoys drawing on those varied experiences to assist her customers. When not working, she stays very busy with reading, cooking, crafts, and most of all lots of family time with her husband and three kids – one infant, one preschooler, and one high-schooler!



vRealize Operations Manager – Architecture

Carl OlafsonBy Carl Olafson

vRealize Operations Manager v6.x is a completely redesigned operations management tool. From an architectural standpoint, vRealize Operations Manager is vastly superior to vCenter Operations Manager, which was a two-VM vApp, and could only scale up. As a starting point, vRealize Operations Manager v6.x uses Gemfire cluster technology, and as such can also scale-out for additional capacity. In addition, the Advanced and Enterprise editions allow vRealize Operations Manager High Availability to be enabled (not to be confused with vSphere HA) for fault tolerance. The remainder of this article will be broken down into some key concepts and architecture terminologies.

Cluster Technology and Scale-Up/Scale-Out Capacity

As mentioned, Gemfire is a cluster technology and for vRealize Operations Manager v6.0/v6.1, there is a node cluster limit of 8 in v6.0.x, and 16 in v6.1.x. This gives vRealize Operations Manager scale-out capacity of 8–16 nodes. In addition, each node/VM has scale-up capacity of 4 vCPUs/16 GB vRAM (small) and 16 vCPUs/48 GB vRAM (large). From a best practices standpoint this brings up a couple of items that must be adhered to:

  1. For a multi-node cluster, all nodes must be the same scale-up size (small, medium or large). Gemfire assumes all nodes are equal and distributes load across the cluster equally. Performance problems will occur if you have different sized nodes in your vRealize Operations Manager cluster. And you can adjust node size after the initial implementation as your environment grows.
  2. For a multi-node cluster, all nodes must have Layer 2 (L2) adjacency. Gemfire cluster technology is latency sensitive. From a VMware supportability standpoint, placing nodes in a cluster across a WAN or Metro Cluster is not supported.
  3. Proper sizing of the cluster and utilization of Remote Collectors is key to a successful implementation. The next article will cover this in detail.

Node Types

For vRealize Operations Manager there are two primary types of nodes: cluster nodes and remote collectors.

Cluster Nodes

The cluster nodes participate in the vRealize Operations Manager cluster. There are three distinct sub-types.

  • Master node, which is the first node assigned to the cluster. The master node is also responsible for managing all the other nodes in the cluster.
  • Data nodes, which would make up the remaining nodes of a non-HA cluster.
  • Replica node, which is a backup to the master node should the master node fail. This assumes vRealize Operations Manager HA is enabled.

Examples of vRealize Operations Manager cluster architectures.

COlafson Cluster 1

Remote Collectors

Remote collectors do not participate in the vRealize Operations Manager cluster analytic process. However, the remote collector is an important node when you have a multi-site implementation or are using specific management packs that cannot be assigned to a cluster node. The remote collector only contains the Admin UI and the REST API component that allows it to talk to the vRealize Operations Manager cluster.

Although your cluster is limited to 8–16 nodes (based on version) and determines your overall object collection capacity, you can have an additional 30–50 remote collectors: 30 in version 6.0, and 50 in version 6.1. The remote collector’s object count applies against the cluster, but does not diminish the size or number of cluster nodes. With the release of vRealize Operations Manager v6.1, remote collectors can also be clustered, and an emerging best practice is to move all management packs/adapters to clustered remote collectors. This helps reduce the load on the analytics cluster, and combined with remote collector clustering provides a higher level of fault tolerance and efficiency.

The remote collector is an important design consideration if you are using management packs (like MPSD) or have vCenters across a WAN/Metro Cluster. If your vRealize Operations Manager cluster is going to collect from multiple vCenters over a WAN or utilize management packs, consult a qualified SME on your design for cluster nodes, remote collectors and level of fault tolerance. VMware Professional Services (PSO) provides vRealize Operations services ranging from Architecture to Operational Transformation.

COlafson Multinode Cluster

Load Balancer

A load balancer is another important design consideration for a multi-node cluster. vRealize Operations Manager v6.x does not currently come with a load balancer, but can utilize any third-party stateful load balancer. Utilizing a load balancer ensures the cluster is properly balanced for performance of UI traffic. It also simplifies access for users. Instead of accessing each node individually the user only needs one URL to access the entire cluster and not be concerned with what node is available.

COlafson Multinode Cluster Load Balancer


Carl Olafson is a VMware Technical Account Manager based out of California.