For  those visiting VMworld, come and meet VMware Trust and Assurance (which includes VMware Security Response Center) in Las Vegas next week or in Barcelona in three weeks from now. Bring your questions and concerns on security issues in our products and services, and how we address these. We would also like to have feedback on the VMware Security Advisories  and our patch policies.

How to find us? We  are accepting 1:1 meetings at VMworld. If  you would like to schedule a meeting please contact your Technical Account Manager with a general idea of what you would like to speak with us about and we will  schedule time with you. Alternatively just come and meet us; we are stationed in the Listening Post located in the VM Village. This is the lounge area with seats and games on the top floor.

We share the Listening Post with other teams and they would be delighted with your visit as well! They are Support, Customer advocacy, and the Information Experience, Quality Assurance, and Product Globalization teams of the VMware R&D Central Organization.

Today, VMware has released the following new security advisory:

VMSA-2017-0014 – VMware NSX-V Edge updates address OSPF Protocol LSA DoS

The advisory documents a hard to exploit denial of service vulnerability in the implementation of the OSPF protocol in NSX-V Edge (CVE-2017-4920). This issue is present due to incorrect handling of link-state advertisements (LSA). NSX-V Edge 6.2.8 and NSX-V Edge 6.3.3 address the issue.

We would like to thank Adi Sosnovich, Orna Grumberg and Gabi Nakibly for reporting this issue to us.

Please sign up to the Security-Announce mailing list to receive new and updated VMware Security Advisories.

Customers should review the security advisory and direct any questions to VMware Support.

The Black Hat USA 2017 conference includes a talk by Ofri Ziv of Guardicore Labs about using the VIX API to obtain privileged access to a guest operating system with unexpectedly low VIM API permissions. VMSA-2017-0012 contains details of impacted versions and workarounds. The Common Vulnerabilities and Exposures project has assigned CVE-2017-4919 for this issue, with thanks to Ofri Ziv and Itamar Tal of Guardicore for discovering and reporting it.

Permissions

Understanding the privilege escalation here requires understanding the interaction between several vSphere permissions.

• Host.Config.AdvancedConfig. This permission amounts to full host control. Setting host advanced configurations is equivalent to adjusting VIB acceptance levels or disabling firewalls.
  • In proof-of-concept code, authors used this permission to enable the “shared secret” mechanism at the host level. VMware considers use of this permission equivalent to full control (e.g. root) and thus using it in an attack chain does not amount to a privilege escalation.
  • However, some VMware products (SRM, VUM, and VIN) operate with this permission, and the setting is host-global. When such software enables “shared secret”, they allow a lower privilege user to gain guest access for the window when it is enabled.
  • The issue reporters argue that a typical datacenter would frequently enable “shared secret”. VMware believes such enablements are uncommon, and collisions are unlikely and can be mitigated. But as the collision is possible, VMware is treating this as an exploitable issue.
• VirtualMachine.Config.AdvancedConfig. This is the “vulnerable” permission. Virtual machine advanced configurations are a lower privilege level than host advanced configurations, and this setting should have been safe to give to lower-privilege users.
• VirtualMachine.Interact.GuestControl. This is the “guest control” permission; users with this permission may establish VIX API connections and thus present some form of authentication.

To compute a CVSS v3 score, VMware recommends “Access Vector: Local” to represent the need to be on the management network (equivalent to local access in a non-virtual environment), and “Attack Complexity: High” to represent the difficulty in bypassing the Host.Config.AdvancedConfig permission.

Impact and Workarounds

Accessing a virtual machine should require two authentications: a VIM API authentication to provide a “route” to a virtual machine, and a VIX API guest authentication to access the virtual machine itself. The security issue here bypasses only the second authentication; the first authentication remains a requirement.

Before worrying about this security issue, an organization should first confirm that their configuration does assume a separation of privilege, with some users being given rights to configure or access virtual machines but being denied rights to configure hosts, and vice versa. Such least-privilege configurations are a best-practice recommendation, but many deployments have so few vSphere users that all users operate with full permissions – at which point there is no escalation from a low-privilege user to a high-privilege user.

The escalation of privilege here requires both an ESXi host version capable of reading “shared secret” information, and a VMware Guest Tools version willing to accept “shared secret” authentication. All current versions of VMware ESXi are impacted by this issue. VMware Guest Tools prior to version 10.1.0 (released October 2016) are impacted by this issue, as beginning with Tools 10.1.0 we have disabled the “shared secret” authentication mechanism.

VMware has two recommended workarounds. Either is sufficient.

1. Upgrade VMware Guest Tools to at least version 10.1.0, in combination with running ESXi at least version 6.0. Without an in-guest endpoint willing to accept “shared secret” authentication, there is no privilege escalation.
   • However, some older products are incompatible with newer VMware Guest Tools (see “known issues” in the 10.1.0 release notes). Those products are incompatible precisely because they depend on “shared secret” authentication.
   • VMware Guest Tools version 9.10 and later have an in-guest configuration option to disable this functionality, which is enabled by default. Version 10.1.0 changes the default to disabled. See KB 2151027 for further details.
2. Remove the VirtualMachine.Interact.GuestControl permission from vSphere users, applicable to all vSphere versions. This permission enables VIX API access, which is necessary to present any credential at all via the VIX API (whether “shared secret”, username/password, or SAML token).
   • Most vSphere customers do not use the VIX API (and those that do, are probably aware of it). Any existing users of the VIX API should prioritize moving to the replacement Guest Operations VIM API discussed below.

VMware believes the above workarounds are sufficient for most customers. In particular, either a Tools upgrade or removing GuestControl are straightforward to apply in most scenarios. Regarding ESXi releases:

• ESXi 5.5 will retain “shared secret” behavior as a “mis-feature”; the only workaround is to disable the VIX API entirely via the GuestControl permission. This is ultimately a recognition that in vSphere 5.5, guest authentication was designed as a mechanism to run guest operations as a particular user and not as a security barrier. As described below, adequate security mechanisms to provide a barrier are not available architecturally until vSphere 6.0.
• Any future major release of ESXi will have “shared secret” authentication removed entirely, due to VIX API end-of-life as described below.

VIX API: a slow path to deprecation

The VIX API is an old VMware API written in C, primarily intended for automation use cases. Originally created for Workstation 6 (released 2008), VIX version 1.6.2 (released 2010) added vSphere support.

That heritage embedded some toxic assumptions. The automation use case around 2008 envisioned deploying VMs under API control, and “obviously” the user invoking automation would have full control of those VMs to run processing tasks – so the VIX API offered full guest control without authentication.

Around 2010, VMware recognized that holistically, the VIX API bypassed too many important checks offered by the VIM API and needed to be removed. However, as some bloggers had noticed, the VIX API was unique in being the only mechanism to run a command in guest, which turns out to be a very powerful tool for automating virtualized software. Thus, we added Guest Operations support in the VIM API, a project that started vSphere 5.0 and culminated with vSphere 6.0’s support for SAML tokens to share pre-authenticated access (allowing a guest to opt-in and map VIM users to guest users as a bridge between two security domains). Simultaneously, we moved the guest-unauthenticated access to be off by default and enabled only via higher-privilege “shared secret” settings, to prevent new usages of this functionality.

Deprecation of the VIX API was announced in 2011 with VIX version 1.11, indicating “VIX will be unsupported in the second major release after vSphere 5”. Nominally those next two releases were ESXi 5.5 and 6.0, with VIX scheduled for removal in ESXi 6.5. That didn’t happen.

Now for the bad news. As anyone familiar with IT or software engineering knows, anytime a feature is deprecated, somebody will build a new product depending on it before the deprecation window expires. We initially tried disabling VIX’s unauthenticated guest access entirely… and regressed many VMware products, so needed another release. SRM ported to the Guest Operations API in the 6.5 release. VUM dropped the only feature which used VIX (appliance upgrade). vRealize Infrastructure Navigator chose to end-of-life, with the replacement being the Service Discovery Management Pack. As most of those were completed late in the release, we were uncomfortable removing VIX (or even VIX’s “shared secret”) in the 6.5 release, so delayed removal to the first major release after 6.5.

Astute observers will notice the dilemma here. On one hand, bypassing guest authentication is increasingly an untenable security posture. On the other hand, removing this functionality will break mission-critical functionality like SRM (for disaster recovery). And there are no good answers.

Conclusions

Deprecating and removing concerning code is always a tradeoff between the risk of unknown security bugs, and unknown functional regression – a hard tradeoff to make for enterprise products where functional regressions can bring an entire business down. The removal of the entire VIX API will occur promptly as planned; thanks to this security research, the security costs of any further delay are more obvious.

During the 2017 Pwn2Own competition at CanSecWest, two teams succeeded in demonstrating arbitrary host code execution on VMware Workstation. Today, VMware is releasing updated versions of VMware vSphere ESXi, VMware Fusion, and VMware Workstation to address these vulnerabilities. VMSA-2017-0006 contains details on impacted versions and the releases which contain fixes.

No active exploitation

VMware is not aware of any active exploitation of the vulnerabilities revealed in this competition. Though the vulnerabilities seem to apply to all VMware virtual platforms (ESXi, Fusion, and Workstation), demonstration exploit code appears to exist only for VMware Workstation for Windows.

The rules of the Pwn2Own competition stipulate that contestants provide their vulnerabilities exclusively to ZDI (the contest organizer), who in turn provides the vulnerability only to the affected company. We appreciate ZDI and the contestant’s commitment to responsible disclosure practices, enabling VMware to release updates before details of the vulnerabilities become known.

Vulnerabilities Found

The following vulnerabilities were identified and analyzed:

• SVGA I: CVE-2017-4902 critical
  Heap overflow leading to arbitrary code execution
• SVGA II: CVE-2017-4903 critical
  Uninitialized stack value leading to arbitrary code execution
• XHCI: CVE-2017-4904 critical
  Uninitialized stack value leading to arbitrary code execution
• CVE-2017-4905 moderate
  Uninitialized memory read leading to information disclosure

VMware also recommends examining the vSphere Hardening Guide and
vSphere Security Guide. Among the recommendations in the guides is to remove unnecessary virtual hardware. Removing unnecessary virtual hardware increases the complexity of exploitation and can partially mitigate the issues, but cannot be a full mitigation due to the nature of modern graphics functionality. Exercise caution: removing virtual hardware can have adverse effects on functionality or performance, and often requires the virtual machine be powered off for reconfiguration.

Risk Management

The best response is to apply the patches which correct these defects. VMware employs technologies like vMotion and VUM to reduce the disruption of deploying security patches. Further, VMware recognizes that deploying patches does carry operational complexities, and understands that further improving this situation is among our customer’s greatest needs.

One common aspect of all these vulnerabilities is the need to run arbitrary code in the guest to begin the exploit chain. (VMware categorizes guest escapes as “remotely exploitable” only for CVSS scoring purposes, as our security model assumes untrustworthy guests). Normal defensive mechanisms like antivirus and firewalls installed in the guest can prevent an attacker from having the degree of access necessary to attack the hypervisor. Locked-down environments like a production database should already disallow running arbitrary code, and thus mitigate this sort of attack.

Customers should consider the need to update for a full mitigation, the absence of active exploitation, the pace at which updates can safely be deployed, and any other risk mitigations (like IDS applications) which may protect their environments. At this point VMware’s recommendation is that customers expedite updating, though need not take emergency measures like taking environments offline.

Some readers may be interested in a more detailed discussion of VMware’s approach to the security landscape.

Pwn2Own and Hypervisor Security Research

VMware engineers have been attending security conferences on a regular basis for many years. This enables us to respond to any discovered vulnerabilities as quickly as possible, and also allows us to develop a constructive relationship with security researchers from around the world. Our engineers have been in contact with the Qihoo 360 team for quite some time, beginning with last year’s Pwn2Own 2016 competition. This year was our first introduction to the Tencent team.

At CanSecWest 2016, Qihoo 360 presented work on using fuzzers to detect virtual machine escapes. That work was primarily on QEMU-based hypervisors (CVEs disclosed) but also including VMware hypervisors (no vulnerabilities disclosed). For the first time, the Pwn2Own 2016 competition included a hypervisor platform (VMware Workstation running on Microsoft Windows 10) as a target; no teams chose to make an attempt.

At Power Of Community 2016 (Seoul, South Korea) during the PwnFest competition, Qihoo 360 and Lokihardt independently demonstrated the same guest escape for VMware Workstation via drag-and-drop functionality (not included in ESXi). VMware engineers were on hand to receive information about that vulnerability (CVE-2016-7461), leading to the release of VMware Workstation 12.5.2 and VMware Fusion 8.5.2 several days later. Further mitigations and fixes for related bugs were included in the 12.5.3/8.5.4 (VMSA-2017-0003) and 12.5.4/8.5.5 (VMSA-2017-0005) releases.

At CanSecWest 2017, the Qihoo 360 team presented details on how they had found and exploited the drag-and-drop bug several months before at PwnFest. Again VMware engineers were on hand, and communicated with the team before the presentation occurred to discuss what would be covered, and understand what techniques the team used to find and exploit vulnerabilities. The same engineers also received full details and clarifications about the vulnerabilities used in Pwn2Own 2017 directly from the researchers from Qihoo 360 and Tencent.

These connections are important within the security community. The Qihoo 360 and Tencent teams are premier commercial security research teams in this space; as much as they profit (in both reputation and financially) from VMware software, VMware benefits in understanding modern offensive security research techniques and being held accountable to modern security practices. The learning and accountability gained from these events is vital to maintaining the high quality our customers expect from VMware’s flagship hypervisor products.

The security landscape has changed dramatically over the past several years. Whereas twenty years ago it took a single bug to break software, and ten years ago it generally took two bugs (an information leak to break ASLR, and the actual exploit), today’s cutting edge defensive technologies can force an attacker to construct a chain of as many as six bugs to break out of a web browser and its associated sandbox. Modern fuzzers like AFL have become several orders of magnitude more efficient at revealing exploitable bugs. With the principles of responsible disclosure to connect researchers and software authors, the end result is ever-better software.

As an anecdote, a significant number of researchers at the CanSecWest conference used VMware Workstation or VMware Fusion to give live demonstrations of their work (both offensive and defensive). We saw researchers use debuggers to simulate (destructive) exploits, show techniques on multiple operating systems during a single talk, or use a virtual machine to simultaneously run their slides and unsafe (un-patched) software to demonstrate a particular technique. VMware recognizes the responsibility that goes with being an important tool for cutting-edge security research, and the inherent requirements of those security researchers to ensure any malware or vulnerable software they may be researching remains contained within virtual machines. As security knowledge moves forward, so do VMware’s technologies and techniques to keep researchers protected.

The Evolving Security Landscape

A “guest escape” – arbitrary code execution on a virtual machine host – is the worst category of bug for virtualization software (CVSS of 10.0). VMware software has done well over time in defending against malicious software, though this is not the first guest escape demonstrated in VMware’s history (see most notably CVE-2009-1244 “Cloudburst”, which also affected the virtual SVGA device implementation).

The single best defense against this type of security issue is depth. A robust chain of defenses includes using firewalls / network IDS to control access to virtual machines, running anti-virus or other host IDS to block malicious software from running on a virtual machine, and the hypervisor itself to isolate virtual machines from each other. At some point in time, any of these layers may miss something important or have a bug; the goal is to have enough layers that not all can be breached simultaneously.

At VMworld 2016 (Las Vegas and Barcelona), VMware included a session on security risks around “guest escapes”. The message then remains the same as the message now: security risks are both human and technical, and we consistently see the human aspect under-prioritized when the technical aspect becomes newsworthy. Hypervisor arbitrary code execution bugs are still VERY complicated to discover and exploit; they command the highest prizes at competition and only the best-resourced teams have been able to enter the space, though the barrier to entry is lessening. By contrast, our experience has consistently shown that the biggest threat to IT of any kind is misconfiguration and lax operational practices. We should also learn from web browsers: the average user is at greater risk from phishing than from a web browser vulnerability, and the most effective defense is to stay up to date with patches.

VMware’s Software Development Lifecycle applies a similar defense in depth strategy. We use code reviews to get human eyeballs looking for problems, and also deploy static analysis tools, threat modeling, external audits, and testing (including fuzz testing) to minimize the chances of a vulnerability escaping detection.

To characterize the change we are seeing in the security landscape right now, there is a gradual evolution in targets that can be attacked. When VMware models threats, we consider three categories of actor. The “nation-state” actor has vast resources but generally employs them against limited targets; such an actor will find a way to breach security, whether by technical means or something simpler (money, ideology). The “professional” actor has more limited resources, and tends to look for the softest and most profitable target; defending against this actor amounts to staying on or ahead of the security research curve. And the “script kiddie” uses off-the-shelf resources and previously-known issues; defending here generally requires little more than staying up to date, and the biggest risk is installations which fail to deploy existing patches. The Pwn2Own competition shows that the difficulty of hypervisor attacks is moving from the “nation-state” category to the upper end of the “professional” category. This is a trend we have been expecting for a while, as security research tools become more powerful.

As offensive security evolves, so does defensive security. With vSphere 6.5, VMware began deploying sandboxing technology around virtual machines to prevent a single arbitrary code execution from spreading across a host – a technique we have adapted from studying how web browsers and cell phones have evolved to defend against offensive security research. We are proactively disabling or removing legacy features – the loss in compatibility is increasingly outweighed by the reduction in attack surface. And we are investing deeply in ease-of-upgrade, recognizing that prompt security patches do little good if they cannot be deployed to production in time.

Conclusion

Ultimately, our security mindset is to ask “when”, not “if”, a security vulnerability will occur. This means being proactive in looking for vulnerabilities, staying in touch with the security community to be aware of current trends and research, and developing more efficient means to deploy the inevitable fixes as they become ready.

As always, our goal is to provide customers with the tools they need to operate at their most efficient, while retaining the security mechanisms we all depend upon in the modern IT environment.