IT managers, stop me if you’ve heard this one: A colleague dials into your weekly office hours and says, “My VMware vSphere® environment isn’t working, and I need help fixing it.” You ask if they’ve contacted VMware support, only to find out that the environment is still running vSphere 5.1. vSphere 5.1 is an out-of-date version that’s no longer entitled to receive support; however, the colleague says it’s mission-critical and the workloads running on it generate revenue for the company. What are the next steps? What does success look like?
VMware TAM to the rescue
As a VMware Technical Account Manager (TAM), this happens more often than you might think to Fortune 500 customers and small businesses, alike. This scenario happened to me most recently earlier this week. The official datasheet says Technical Account Management Services are a lot of things but, as with any customer-facing role, the TAM job is first and foremost about making sure the right customer outcomes prevail. This request for help would have been easy to refuse on the basis that end-of-life products aren’t entitled to technical support or on the basis that only VMware Support provides technical hands-on-keyboard support. Equally easy would have been to say, “That sounds like a job for VMware Professional Services” or to leverage the urgency of the situation to compel an upgrade to vSphere 7.
But stepping back, it was clear none of these options were in the immediate best interest of the customer. The environment was down now, and upgrades would have taken weeks to arrange. Refusing to help before I even understood the nature of the issue falls well short of the duty of a TAM. What did I do? I said, “Fire up a screenshare, and let’s see what’s going on!”
TAMs don’t provide hands-on-keyboard technical support to customers in large part because we have a dedicated team called VMware Global Support to provide that as part of our paid support offering. You can always ask a TAM basic questions, though. What did I see in this instance? Decrepitude. Everywhere! I saw vCenter 5.1 running on an end-of-life version of Windows connecting to an end-of-life version of SQL Server. I saw Windows complaining about having lost its domain membership and, upon the customer dismissing that dialog box, I saw Windows complain about being unable to contact an activation server. All of this was running on pre-Sandy Bridge Xeon hardware. Now is probably as good a time as any to inform the customer that I’m new enough to VMware that I’ve never actually used a Windows-based vCenter before, but I do have experience as a Windows sysadmin and a SQL Server database administrator. We decide to explore a bit further without making any changes.
Windows Event Viewer points us to the vCenter log files directory, and the latest log file is about 100 lines long ending with a repeating series of messages reading (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 53).
Having no idea how a Windows-based vCenter stores its database configuration information, I hazard a guess that it’s probably relying on Windows’ built-in ODBC Data Sources feature. So, we pull that up to find, lo and behold, there’s a System DSN called “vCenter.” I tell the customer to ping the configured database server to see if it responds on the network. I’m familiar enough with my customer’s environment to already know that it’s not going to successfully resolve DNS. Sure enough, the output reads: Ping request could not find host dbserver.domain.com. Please check the name and try again.
At this point the customer already knows what to do and my participation is unneeded. A simple host file entry is enough to temporarily work around the DNS resolution failure. The customer adds a host file entry, restarts the vCenter Server service and 10 minutes later the environment is back up and running.
What was wrong? Remember the error messages about domain membership failure and Windows activation failure? Remember how I said I knew my customer’s environment well enough to know the ping to her database server was going to fail before she did it? When I saw the fully qualified domain name of the database server configured in the ODBC System DSN screen, I had all the confirmation I needed to pinpoint the issue. Having worked with this customer for four years, I knew the Active Directory domain to which this environment had been joined was actually decommissioned a few years ago. That meant the DNS entries this environment relied upon no longer existed. I’ll never know why this environment continued to work for several years after that, but I’ve added it to my long list of issues whose cause is best explained by gremlins.
This is not the end of the story. The real customer outcome will be the final migration of VM workloads from this environment to a newer environment at the same site, but this simple win earned me another “Trusted Advisor” badge with the customer and an invitation to participate in the planning of those migrations.
Work with a VMware TAM
This story is more than an example of the kind of value TAMs deliver to VMware customers. It illustrates that success happens at the intersection of a TAM’s knowledge of the customer’s environment, our technical knowledge and a commitment to the best outcome.
Don’t have a TAM for your organization? Talk to your VMware Sales Representative about VMware Technical Account Management Services to learn more.
