VMware vSphere & Microsoft LDAP Channel Binding & Signing (ADV190023)

posted

20 Comments

VMware AppDefense IconUpdate (5/13/2020): This post has been updated to reflect current guidance on this topic, and that Integrated Windows Authentication is affected by this change. There is good information in this post but more information can be found in the post “vSphere Authentication, Microsoft Active Directory LDAP, and Event ID 2889.“

—-

Update (2/6/2020): On February 4, 2020 Microsoft changed their guidance for the March 2020 Windows Updates to indicate that the defaults will NOT be changing in that update.

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV190023

We are leaving this post active as a reference but will prune the comments to flag and/or reduce unsupported & dangerous advice and discussion.

We continue to reiterate that supported versions of VMware vSphere work correctly with both LDAP binding settings, our recommendation is always to use encryption to protect data in flight on a network, and changing vSphere outside of guidance from VMware Global Support Services is not recommended or condoned and may affect product functionality and supportability.

—-

Update (1/16/2020): Updates to reflect the results of Integrated Windows Authentication testing.

Microsoft has recently released warnings to its customer base that, in the March 2020 updates to Windows, it intends to change the default behavior of the Microsoft LDAP servers that are part of an Active Directory deployment. These changes will make secure LDAP channel binding and LDAP signing a default requirement when accessing Microsoft Active Directory using LDAP or LDAPS. These changes are a response to a security concern documented in CVE-2017-8563, where bad actors can elevate their privileges when Windows falls back to NTLM authentication protocols.

Any system that connects to Active Directory via LDAP without using TLS will be negatively affected by this change. This includes VMware vSphere. If you are using unencrypted LDAP (ldap://, not ldaps://) or Integrated Windows Authentication to connect vSphere to Microsoft Active Directory please read further.

All supported versions of VMware vSphere have been verified by VMware Engineering to work as expected after these changes, where we expect unencrypted LDAP authentication to succeed with the old defaults, fail with the new defaults, and succeed when using TLS/LDAPS.

Integrated Windows Authentication (IWA) has also been tested by VMware Engineering. IWA uses GSSAPI and Kerberos to authenticate and does not send credentials in the clear. However, IWA uses unsigned LDAP behind the scenes to allow searching users and groups, and this will stop working. This may impact the ability to add users & groups to authentication configurations. VMware KB 78644 discusses some of this, and VMware is investigating changes that may improve this situation.

Important Notes

These issues are the direct result of Microsoft’s changes to Windows. While we at VMware are committed to helping our customers navigate issues like these, we do ask that you please direct feedback about Windows changes & updates to Microsoft themselves. Configuring authentication sources in vSphere is a documented & supported activity that the professionals at VMware Global Support Services (GSS) can assist with, but that comes with the prerequisite that your authentication source is usable and cooperative. Please contact Microsoft Support for assistance with reconfiguring and and all aspects of Microsoft Active Directory.

These issues affect any system using LDAP to access Active Directory and are not limited to VMware vSphere. You need to check the rest of your systems, too. As part of the documentation for these changes Microsoft lists ways to audit clear-text connections to your Active Directory infrastructure which may be helpful.

All screen shots in this post are of vSphere 6.7 Update 3 and the HTML5 vSphere Client. The experience should be similar for vSphere 6.0 and 6.5.

All testing and documentation of vSphere assumes a direct connection between vSphere and the authentication sources. If your authentication sources are proxied or behind an application load balancer you will need to independently assess how you are affected by these changes. Please consult the documentation and vendors of those products and systems.

If you enable auditing on your Active Directory implementation please review Microsoft’s documentation for what an event log entry means, when it will occur, and what information is gathered in the event log entry before opening a support case with Microsoft or VMware.

Who Is Affected?

If you have configured vCenter Server to access Active Directory over LDAP with TLS (LDAPS) or Identity Federation you will not be affected by this. You can check this by viewing your Identity Sources in the vSphere Client:

vSphere LDAP Identity Sources Compatible with LDAP Channel Signing

The red & green text has been added by us as an illustration. Sources using LDAP (ldap://, port 389) are likely to be affected. Sources using LDAPS (ldaps://, port 636) are likely fine if they are direct connections and not through proxies or load balancers.

Sources listing their type as “Active Directory (Integrated Windows Authentication)” will continue to authenticate, but their ability to search the Active Directory for users & groups will break, as it uses unsigned LDAP to do so. This may impact the ability to add users & groups to authentication configurations. VMware KB 78644 discusses some of this, and VMware is investigating changes that may improve this situation.

What Will Happen

Without proactive action on your part, once Microsoft releases these updates and they are applied the affected identity sources will stop working. This will appear as login failures in the vSphere Client:

vSphere Login - Invalid Credentials

as well as errors when attempting to add or reconfigure an identity source:

vSphere Add Identity Source Error

 

Notes & Suggestions on How to Test

We recommend testing in order to gain familiarity with these updates. Microsoft has released guidance on how to enable these settings:

The document on enabling LDAP Signing in Windows Server 2008 indicates that you need to change the “Default Domain Policy” but in order for it to be effective for domain controllers you must also edit the “Default Domain Controllers Policy” or whichever policy applies to the domain controllers, if you’ve assigned a new one. The procedure in that document also appears to be applicable to all versions of Windows, not just 2008. Once the Group Policy edits are in place you can wait until the Group Policy refreshes automatically or use an Administrator-level shell to issue the “gpupdate /force” command.

Keep in mind that updating Group Policy affects other domain controllers & members as well. vCenter Server can configure multiple LDAP & LDAPS authentication sources, and can specify particular domain controllers, so we recommend creating a new & isolated Active Directory instance for testing (you can see my farm animal theme for test domains & forests in this post, which is separate from my main authentication source).

To test that the settings have taken effect use the “ldp.exe” utility (Start->Run->ldp) from the domain controller itself. From the Connection menu, choose Connect, and enter “localhost” and port 389:

Connecting with ldp.exe

From there, go back to the Connection menu and choose “Bind.” Enter your domain credentials and select “Simple bind” as shown here:

Binding with ldp.exe

You should receive the error “Error 0x2028 A more secure authentication method is required for this server” as shown below, which indicates that you have correctly disabled clear-text LDAP binds.

A more Secure Authentication Method is Required Error

If you do not get this error with this procedure the changes have not taken effect!

From here you can test vCenter Server connectivity to Active Directory, witnessing the behavior seen at the beginning of this post. As mentioned earlier, VMware vCenter Server can have multiple Active Directory instances configured, so testing with an isolated instance of Active Directory is recommended. Similarly, deploying a test vCenter Server appliance is recommended. Take a snapshot of the environment and you can restore it if everything goes wrong. Nested virtualization environments, in general, are an excellent way to test functional changes such as this. William Lam has extensive resources on his personal blog for nested ESXi.

For even more testing isolation, if you deploy the Windows Active Directory VM first and configure AD DNS you can use it as the DNS server for a test deployment of vCenter Server.

Possible Course of Action #1: Enable TLS on Active Directory (LDAPS)

Being security-minded, the first & best recommendation is to secure your authentication with TLS. As a matter of practice, all communications on a network should be encrypted. This is especially true of authentication traffic. Please refer to Microsoft documentation for configuring Active Directory Certificate Services and issuing certificates to Active Directory domain controller services.

Configuring vCenter Server to use LDAPS is straightforward and well-documented at docs.vmware.com. There is one twist: you will need the certificate for the domain controller. You can export it from Windows but if you have access to OpenSSL, either installed on a Windows PC or built into a Linux/UNIX host, this sample command is often easier:

echo | openssl s_client -showcerts -connect cows-ad-a1.cows.local:636

will display the certificate you need, between the “—–BEGIN CERTIFICATE—–“ and “—–END CERTIFICATE—–” lines. Copy those lines and everything between them into a text file and use that when prompted. Replace my farm animal test FQDNs with the native DNS name of the domain controller you are connecting to, of course!

Possible Course of Action #2: Proactively Configure These Settings to The Original Defaults

Being security-minded, making a decision that could negatively affect security can be tough. However, there’s a lot more to information security than just changing registry settings. Information security professionals use the “CIA triad” — confidentiality, integrity, and availability – to thoughtfully weigh the risk of a configuration, and the short timeframes and nature of these changes could seriously impact availability. If you’ve already been running in this configuration you likely have compensating controls in place, such as isolation (firewalls, separate networks), to protect against someone observing authentication traffic.

Microsoft has stated that these upcoming patches will change the defaults. Though they haven’t said it directly, this implies that you can proactively set the those settings to the current defaults, using these same processes (set the LdapEnforceChannelSigning registry key to 0, set the group policies to ‘None’). This assumes that Microsoft will not overwrite them with the patch, which is probably reasonable, but we will all want to verify that once the patches are available.

Should you do this? It might be the easiest way forward for now, but it doesn’t improve security, and is likely worth a conversation with your CISO.

Other Considerations

It isn’t a good idea to significantly delay patching. Patching is the ONLY way to remove a vulnerability from a system, and it’s the #1 way organizations and individuals can secure themselves (the #2 way is great passwords & account hygiene). Microsoft is making this big change for a reason, and we don’t yet know what has changed since the original 2017 vulnerability disclosure. It’s very likely that, in a few months, we will learn more about what is driving this. That won’t be good news, whatever it is. By delaying or omitting patches you delay the inevitable and you increase your risk, both from hackers and from well-intentioned humans accidentally applying cumulative updates.

Whatever course of action you choose, this is a great opportunity to make sure that you know and/or can retrieve the password for your vCenter Server SSO instance’s administrator account (often administrator@vsphere.local).

Active Directory is wonderful in that it’s a multi-master, replicating, distributed database, with built-in availability and recoverability features. Use this to your advantage and plan a staged rollout for both this issue and as a future patching strategy. Similarly, do not forget the flexibility vCenter Server has in targeting specific domain controllers, as well as the ability to apply Group Policies to specific domain controllers.

Thank You

As always, thank you for being a VMware customer. We appreciate you and hope that, like with our continued guidance on CPU vulnerabilities, this has been helpful in navigating these larger industry-wide changes.

For questions and feedback about Active Directory, Windows Updates, and these Microsoft Updates please contact Microsoft directly. We love feedback but if it’s about Active Directory and Windows the best we will be able to do is commiserate with you. Feedback is most powerful when it’s direct from a user to the responsible vendor.

For operational issues with VMware products please contact VMware Global Support Services, who are available 24×7 to assist. If there are other things we can help with, or you have questions or feedback about VMware products, please reach out to your account team or Technical Account Manager. For issues directly concerning this post please leave a comment below.

Last thing – subscribe to this blog and follow us on Twitter or on Facebook! We write a lot of technically-oriented articles like this, covering news and current topics, and we’d love to have you with us.

Thank you!