Protecting Active Directory with VCF

(Day 14 of 22 in our Cybersecurity Awareness Month series. Today: protecting Active Directory)

For most organizations, Active Directory is the crown jewels. It’s where you define who can log in to what, it holds the credentials for nearly every account, and it often contains the groups used for authorization across an organization’s systems. And almost universally, an attacker who controls it controls the organization. That’s why so many attacks, ransomware and espionage alike, head straight for the domain controllers.

Most domain controllers run as virtual machines, so protecting Active Directory is partly a job for the virtualization platform. This post focuses on the infrastructure underneath, and what VMware Cloud Foundation (VCF) can do to keep that infrastructure from becoming the easy way in. Microsoft covers the internals of Active Directory well, and the same best practices apply whether a domain controller is physical or virtual. The ideas here apply to other critical systems, too, like certificate authorities, secrets vaults, and backup servers.

And, for what it’s worth, most of these ideas are directly applicable to other workloads, too.

Your Hypervisor Admins Are Indirectly Domain Admins

Microsoft is explicit about trust relationships in its own guidance for virtualized domain controllers, saying that administrator credentials for a host that runs writable domain controllers should be treated as equal to the domain’s default administrator account (Microsoft). Its tier model places anyone with administrative access to the hosts where Tier 0 systems run in Tier 0 themselves (tier model).

It’s not hard to see why. Someone who controls the hypervisor doesn’t need a password to steal a domain, and attackers, including the group behind the BRICKSTORM backdoor, have shown exactly how:

• The disk swap. Attackers powered off a domain controller VM, detached its virtual disk, attached it to another VM, copied the Active Directory database, and put everything back, all without logging in to the domain controller. Nothing running inside the domain controller could detect it (report). A joint FBI and CISA advisory describes another ransomware group doing the same thing (advisory).
• The silent clone. An espionage group cloned domain controllers through vCenter, mounted the clones’ files to extract credentials, never powered the clones on, and deleted them soon after (report).

Neither technique needs a vulnerability in the hypervisor. Both use ordinary administrative features, which is why the defense is about who holds those features and what they can do with them.

Encrypt Domain Controllers

The disk swap and the silent clone both depend on reading a domain controller’s disks, and you can defend against this by using encryption. Threat researchers call VM encryption for Tier 0 VMs the definitive technical block for the offline disk-swap attack (report). VCF gives you a few ways to do it:

• VM encryption encrypts the domain controller’s virtual disks and files with keys from your key provider (how it works). A detached or copied disk is unreadable without the keys. Encrypting an existing VM means powering it off, so do one domain controller at a time.
• BitLocker inside the guest, with its keys sealed in a virtual TPM, encrypts the disks from the inside. A clone gets a copy of the vTPM, though, so it can unlock itself when it starts. Adding a pre-boot PIN stops that (Microsoft). Someone has to enter that PIN after every restart, which is real operational pain, but Active Directory is one of the few places where it may be worth it.
• vSAN encryption protects against stolen drives, but it doesn’t stop an administrator from cloning a VM or attaching its disk to another VM, so it may not be enough on its own here.

A virtual TPM alone doesn’t encrypt the disks. It encrypts the VM’s home files using VM encryption, but you still may want to pair it with BitLocker. It doesn’t need a physical TPM in the host.

Can you use all of them at once? You can, but think about why, and about the performance cost of encrypting the same data several times over; most domain controllers are latency-sensitive. A reasonable middle ground is vSAN encryption, a vTPM, and BitLocker inside the guest with the pre-boot PIN set.

Control Who Holds the Keys

Encryption only helps if the people who shouldn’t read the data also can’t decrypt it. By default, the vCenter Administrator role holds every cryptographic privilege, and those privileges are what allow exporting, cloning, migrating, and opening the console of an encrypted VM (privileges). Consider removing cryptographic privileges from nearly everyone by using the No Cryptography Administrator role for everyday work, and keep a separate account with cryptographic privileges in your privileged access management system, used only when it’s needed. Think through what else depends on those privileges, like cloning, Storage vMotion, and backups, and adjust those workflows, perhaps through VCF Automation. Users with ESX Shell access hold cryptographic privileges, too (docs), so keep SSH and the Shell turned off.

Limit Who Can Touch Domain Controller VMs

Most people who administer VCF don’t need access to domain controllers at all. Use VCF permission models to keep it that way:

• Group domain controllers in their own folder, and use the No Access role to hide that part of the inventory from everyone who doesn’t need it (best practices).
• Remove Clone and Export from standard administrator roles, as threat researchers recommend (defender guide).
• Grant vCenter administration through local VCF single sign-on groups, as the VCF Security Configuration Guide recommends, so a compromise of Active Directory doesn’t automatically hand over the infrastructure that runs Active Directory.
• Turn off what an attacker could use from the outside. Guest customization can change the administrator password, log in automatically, or add an account to a cloned VM, and guest operations can run commands inside the guest. Follow the guidance in the VCF Security Configuration Guide to set “deployPkg enable-customization” to false on deployed domain controllers, and turn off guest operations.
• Keep domain controllers on known hosts you control tightly. Run domain controllers on management domain hosts if you can, and mind the DNS and NTP dependencies covered below. A dedicated cluster for identity and other Tier 0 systems, or at least VM-Host rules that keep domain controllers on a known set of hosts (docs), makes access easier to limit, and saves you from hunting for your domain controllers in the middle of an incident. We recommend using “should” rules so they can move freely during VCF patching and maintenance operations.
• Zero memory when domain controllers power off. The VCF Security Configuration Guide’s volatile key destruction setting, Mem.MemEagerZero, zeroes a VM’s memory when it powers off, so secrets don’t linger in host memory. It adds a little shutdown time, which doesn’t tend to be noticeable and is worth the trade-off even when it is.
• Alert on the attack pattern. A clone of a domain controller, or a domain controller’s disk attached to another VM, deserves an immediate look. Set vCenter alarms on the domain controller folder for clone and power-off events, since the disk swap starts with a power-off. Send vCenter events to Log Management in VCF Operations or to your SIEM, too, and alert when another VM’s reconfiguration adds a domain controller’s disk.

Isolate Domain Controllers on the Network

Every system in a domain needs to reach domain controllers for logins, directory lookups, and DNS, but very few need to manage them. Use VMware vDefend Firewall to tell the two apart: allow the authentication, directory, and DNS traffic that domain members need, and allow administrative access, like Remote Desktop and remote PowerShell, only from your administrative workstations or jump hosts. Those rules are enforced in the hypervisor, at each domain controller’s virtual network interface, so software running inside a compromised guest can’t turn them off.

Limit what domain controllers can reach, too. If they get time from servers on the internet, allow only those servers, and do the same for outbound DNS, since attackers use DNS and other open UDP traffic to sneak data out.

Run Virtual Domain Controllers the Way Microsoft Expects

Microsoft publishes guidance for virtualized domain controllers (Microsoft), and VCF has everything you need to follow it:

• Run at least two domain controllers per domain, on different hosts. A DRS rule of the “Separate Virtual Machines” type keeps them apart, so a single host failure can’t take them all down (docs). Check your vSphere HA settings, too, so they do what you expect when a host fails.
• Turn on the guest security features. Secure Boot, a vTPM, and virtualization-based security all work for domain controllers, and VBS doesn’t need the Hyper-V role installed.
• Reserve all of the memory. In general, memory overcommitment is a fast way to a bad day, and a domain controller is the last place you want to find that out, especially since authentication and DNS latency cause second- and third-order latency in other workloads.
• Let Active Directory handle time. Turn off VMware Tools periodic time synchronization for domain controllers, since they keep time through the Windows Time service. One-time synchronizations still happen at power-on and after events like vMotion, and the guest clock starts from the host’s, so keep ESX on accurate NTP, too (KB 326306). Give domain controllers several good time sources, at least four, and consider whether the PDC emulator should be the only domain controller with an outside source of time, since that makes it a single point of failure.
• Don’t use snapshots as backups. Modern domain controllers use VM-GenerationID, which VCF supports, to detect when a snapshot is reverted or a VM is restored (KB 341662). That safety net prevents database rollback problems (Microsoft); it doesn’t turn a snapshot into a backup. Back up domain controllers from inside the guest with system state backups.

Break the Circular Dependencies

VCF needs DNS and NTP to start, and in many environments Active Directory domain controllers provide both. The VCF startup procedure says to verify that external services like storage, Active Directory, DNS, and NTP are available before bringing the platform up (startup). If those services run only on the VCF infrastructure they support, a full power outage turns into a puzzle: the platform needs DNS and time to start, and the domain controllers that provide them need the platform. A few ways out:

• Make sure infrastructure logins don’t depend on Active Directory. Keep local break-glass accounts for vCenter, ESX, and the other components. Our post on zero trust and identity covers this.
• Spread the dependencies. Put more than one domain controller and DNS server in each site, replicate between sites, and consider at least one domain controller outside the cluster it supports, like a physical server or a separate cluster. Some environments operate a single “dependency ESX host” to hold virtual machines like DNS, NTP, and Active Directory, even using local disk, so that it can start first and help bootstrap a data center that has been powered off.
• Use automatic startup on standalone hosts that run dependencies like domain controllers, so they come back on their own after a power outage.

Plan for Active Directory Forest Recovery

If Active Directory itself is compromised, you may need to recover the entire forest, and Microsoft has a detailed guide for it. A few points stand out for virtualized environments:

• Back up at least two writable domain controllers for each domain (Microsoft), and keep those backups off the platform and away from the domain they protect.
• Restore onto an isolated network. Virtual domain controllers can join an isolated virtual network without changing IP addresses, which is one reason Microsoft recommends them as the first domain controllers to restore (Microsoft).
• Have the keys on hand. Encrypted domain controllers won’t start without their keys, so the key provider backup, its password, and BitLocker recovery keys need to be reachable without Active Directory (Microsoft).
• Practice. Microsoft recommends practicing forest recovery at least once a year (Microsoft), and an isolated network in VCF is a good place to do it. NSX and its ability to create virtual network segments makes this extremely easy.

Protect the Platform, Protect the Directory

Active Directory is only as safe as the infrastructure it runs on. Treat the people and systems that manage that infrastructure as Tier 0, encrypt domain controllers and keep cryptographic privileges with a small group, limit who can see and touch domain controller VMs and who can reach them on the network, follow Microsoft’s guidance for virtual domain controllers, and make sure the platform can start and recover without the directory it hosts.