by: VMware Staff II, Cloud SRE Architect Tom Ralph
VMworld attendees are invited to watch a video of his presentation at VMworld 2018.
VMware’s hybrid cloud environment comprises nine data centers, many public clouds, over 200K managed virtual machines (VMs) and 50+PB of raw storage, just to mention a few statistics. And our private cloud offers 99.99% uptime, a centralized cloud operations model, and infrastructure that can be provisioned in less than 22 hours instead of the traditional four to six weeks.
Despite all those accomplishments, our IT teams realized the current vSphere® environment was missing critical new components beneficial for both our users and the environment’s stability. The decision was made to upgrade to 6.7 from 6.5.
Not surprisingly, there were a number of compelling factors behind the upgrade. vSphere 6.7 offered new features and enhancements, advanced performance, greater scalability, native VMware vCenter® High Availability, VMware vSAN™ encryption, and the latest hardware and OS support. Plus, after tapping into the current community of knowledge, we knew it was the right step to ensure an agile enterprise.
Now the Tricky Part—Upgrading with Minimal Downtime
Achieving a near zero-downtime upgrade meant implementing a very specific process involving the following:
Know the products—from studying the release notes and upgrade guide to accounting for issues that only happen at scale, we had to know our products and which features were in use.
Understand the environment—from every possible angle. This included knowing the history and changes that occurred in the environment.
Conduct dry-run upgrades—just like stage performers who rehearse their act over and over again to make the final product look seamless.
Engage in a phased deployment—by environment and by component.
And there were numerous migration considerations to take into account during the upgrade:
We Did Have Our Challenges to Face
As was unfortunately expected, there were a lot of challenges along the way, although the majority of issues were nipped in the bud early on.
vCenter Server
- During the upgrade, the VMware vCenter Server checks that the DNS PTR and A record are valid. If the PTR record isn’t valid or it’s incorrect, the upgrade will fail.
- After a complete upgrade, AD authentication may need reauthorization.
- The FQDN hostname of the vCenter Server must exactly match the SSL certificate, including the letter case. Previous versions of vCenter allowed mix case certs, but the latest version requires all lower case. It is our standard practice to now issue all SSL certs and FQDNs in lower case.
VMware ESXi™
- VMware NSX® VIBs must be updated before any ESXi upgrade is possible.
- ESXi upgrade failures occur due to VIBs not supporting the new ESXi version.
And We Learned Our Lesson(s)
There were a variety of lessons gained throughout the migration. We don’t typically migrate historical performance, event, alarms, and other statistical information as they can slow down the upgrade and we have that data stored elsewhere if we need it.
Automating where possible offers significant advantages, from auto-deploy to host profiles. It’s also important to verify your monitoring at every stage, especially since many components’ names or locations may have changed. Finally, it’s vital to always test your existing scripts. (For more on automation, consult the vCenter Server Appliance Installer table in this vCenter setup documentation.)
Also, don’t forget the health checks. We did a pre-upgrade health check to verify DNS records and SSL certificate validity. After the upgrade, we checked that Active Directory authentication was working, the third-party plug-ins functioned with the new vCenter, and that monitoring was green.
And what was the end result? The VMware IT team was able to successfully and seamlessly upgrade to vSphere 6.7 in record time with remarkably minimal downtime—setting a precedent for future upgrades regardless of size or complexity.
Resources
Watch the vSphere 6.7 upgrade presentation at VMworld 2018.
Here are some Knowledge Base articles to get you started:
- KB 53704 – Important information before upgrading to vSphere 6.7
- KB 2146420 – Estimating vCenter Server 5.5 to vCenter Server Appliance 6.x migration time
- KB 2147711 – Estimating vCenter Server 5.5 or 6.0 to vCenter Server Appliance 6.x migration time
- KB 2112283 – Regenerate vCenter Certificates
- KB 53704 – Important information before upgrading to vSphere 6.7
- KB 54008 – vCenter Server 6.7 upgrade best practices
- KB 2147824 – Migrating VMFS 5 datastore to VMFS 6 datastore
- KB 53710 – Update Sequence for vSphere 6.7 and its compatible VMware products
- KB 54751 – vSphere Web Client hangs after upgrade to vCenter Server 6.7
- KB 55092 – Update Manager fails to work in Web Client / HTML Client
- KB 2113115 – VMware PSC 6.x FAQs
- KB 2147454 – Linux VMware Tools update fails to complete
- KB 52583 – Supported and deprecated CPU models and devices for VMware vSphere 6.7
VMware on VMware blogs are written by IT subject matter experts sharing stories about our digital transformation using VMware products and services in a global production environment. Contact your sales rep or [email protected] to schedule a briefing on this topic. Visit the VMware on VMware microsite and follow us on Twitter.
