Encryption Key Provider Design

(Day 16 of 22 in our Cybersecurity Awareness Month series. Today: key provider design)

Encryption makes data protection simpler, because once data is encrypted, keeping it safe comes down to protecting something much smaller, the keys, and making sure they’re available when you need them. Our post on storage encryption showed how easy VMware Cloud Foundation (VCF) makes it to turn encryption on, and VCF makes the keys easy to manage, too. Most of the design decisions are about the key provider: where the keys come from, where they live, who can use them, and how they stay available when something goes wrong. Good key provider design is straightforward, and a little planning up front makes everything after it easier.

The Key Provider Options in VCF

Key providers supply the keys for VM encryption, virtual TPMs, and vSAN data-at-rest encryption. Other types of encryption in VCF, like TLS, vSAN data-in-transit encryption, memory tiering encryption, and ESX configuration encryption, use keys that are ephemeral or generated on each host, so they do not depend on a key provider (key providers). VCF offers two kinds of key providers (comparison):

• A Standard Key Provider connects vCenter to one or more external key management servers (KMS) using KMIP. For VM encryption, vCenter is the only component that logs in to the key server. It keeps track of key IDs instead of the keys themselves, and it retrieves keys and passes them to hosts as needed (components). vSAN works a little differently, with each host contacting the key server directly. Any KMS that supports KMIP should work, and if other devices, like tape libraries, share it, give VCF its own partition and credentials (standard key provider). Some storage providers have mechanisms to read VCF encryption keys from a KMS to allow the storage array to deduplicate and compress data, and this is out of scope for us here.
• The vSphere Native Key Provider is built into vCenter, with no external server. vCenter generates a primary key, called the key derivation key, and pushes it to every ESX host in the cluster (overview). Each host stores it in its encrypted configuration, which is sealed by the host’s TPM when it has one (native key provider). It only serves VCF, so anything else that needs keys, like a tape library, still needs a KMS.

Beginning in VCF 9.0, a Standard Key Provider can use a wrapping key. Instead of the KMS storing a separate key encryption key for every encrypted object, which adds up quickly when every VM has a virtual TPM, it stores a single wrapping key, and the key encryption keys are stored with the objects they protect, wrapped by that key (key providers). That leaves far fewer keys to manage in the KMS, and it also makes automatic key rotation possible. It’s usually the best way to run a Standard Key Provider, and existing encrypted VMs can move to wrapped keys with a shallow rekey, without downtime.

How Key Providers Work at Boot

The two types behave differently when a host restarts, and that drives key provider design decisions.

With a Standard Key Provider, a host keeps encryption keys cached in hypervisor VMkernel protected memory during normal operations. This helps prevent issues if the KMS becomes unreachable (KMSes need to be patched, too), and allows hosts to trigger vSphere High Availability without a vCenter dependency issue (vCenter might have been on the host that failed). Because these keys are only in memory, when a host reboots the cache is lost, and upon boot the ESX host will need to get the keys again. If it can’t get keys back, virtual TPMs and encrypted workloads can’t function.

vSAN is a little different, as vSAN communicates directly from the host to the KMS itself. This avoids a dependency loop, but it does mean that a host’s storage devices aren’t mounted until it can reach the key server (vSAN encryption paper).

VMs that use VM encryption or virtual TPMs get their keys through vCenter, so after a full restart they can’t start until vCenter is up and can reach the key server, too. Hosts that are already running can keep working through a temporary key server outage (best practices); the trouble comes when hosts restart, and a power outage restarts everything at once.

Key Persistence lets hosts with a TPM 2.0 keep their keys across reboots (docs). It’s off by default, because a host that keeps its keys can unlock its data even after it has been stolen. The VCF Security Configuration Guide recommends leaving it off unless you can count on physical security. It’s tempting to turn it on to work around an unreliable or distant key server, but that brings back the physical risk a KMS is there to avoid, so fix the key server instead.

The Native Key Provider doesn’t need Key Persistence, because each host already has the key derivation key. That’s what makes it simple to run, and it’s also why it depends on physical security. When you create one, the “Use key provider only with TPM protected ESX hosts” option is on by default, so turn it off unless every host has a TPM; hosts without one won’t get the key, and they can’t run encrypted workloads.

Mix and Match Key Providers

You don’t have to choose just one type of key provider. A vCenter can have up to 32 key providers, with a default for the vCenter and, if you need it, a different default for each cluster, and even a different default key provider for vSAN encryption.

Because Standard Key Providers have additional security controls, most notably network access controls, they are popular for protecting against data loss in environments where physical security is a concern. For example, in a retail environment, a cluster in the back office might not have the same physical security as a data center. If that cluster were stolen, you wouldn’t want the attacker to be able to unlock and decrypt the storage, which they might be able to do with the Native Key Provider.

However, there are a number of reasons why the Native Key Provider is attractive. It is extremely simple to set up, easy to use, secure, and does not need any extra infrastructure.

A growing number of organizations are combining the two: using a Standard Key Provider for vSAN, so the keys that unlock the drives stay in the KMS, away from anyone who walks off with a host, and using the Native Key Provider as the default for virtual TPMs, which keeps them simple and free of external dependencies (key providers).

The choice isn’t permanent, either. A shallow rekey can move encrypted VMs and vSAN clusters from one key provider to another, in either direction and without downtime, and replaces their key encryption keys along the way (key providers). That means you can turn on encryption with the Native Key Provider and move to a Standard Key Provider later if your needs change, or vice-versa. There’s no reason to wait for a KMS, and the sooner encryption is on, the less unencrypted data ends up on flash devices that can’t be reliably wiped.

Design Key Providers for Availability

A key provider is a dependency for every encrypted workload, so design it like one:

• Run more than one key server. A Standard Key Provider can include multiple key servers from the same vendor, and Broadcom’s documentation recommends making sure the key server is highly available (docs). Key servers from different vendors need separate key providers.
• Keep key servers away from shared failures. Use different hosts, racks, and power, and ideally a different site, so the event that takes down a cluster doesn’t take down its keys, too.
• Spread them out. Key servers don’t have a defined latency requirement, and VCF connects to them only when it needs a new key, when a host restarts, and for periodic health checks, with very little data each time. That makes it practical to put key servers in other sites or even other regions, and higher-latency links, like satellite connections, can work, too. Test the design before you rely on it, including restarting hosts that run many encrypted VMs (standard key provider).
• Open the firewall. KMIP uses TCP port 5696 by default. vCenter needs to reach the key servers, and if you use vSAN encryption, every vSAN host does, too (standard key provider).
• Watch the certificates. vCenter and the key server trust each other through certificates, and an expired certificate breaks that trust. By default, vCenter raises an alarm 30 days before a KMS certificate expires (docs), and you may need to re-establish trust after replacing vCenter’s certificates (KB 396741). Our post on certificates covers the rest.
• Plan for migrations and disaster recovery. Moving encrypted VMs between vCenter instances needs the same key provider on both sides: the same key servers for a Standard Key Provider, or the same key derivation key for the Native Key Provider (docs). Disaster recovery sites need the same provider, too (docs). This is easy to deal with: just import or configure the key provider in both places.
• Choose names carefully. Encrypted objects find their key provider by name, and key providers can’t be renamed (best practices). Give separate providers unique names across all your vCenter instances, and give a shared provider the same name everywhere it’s configured (docs).

Key providers belong to individual vCenter instances and aren’t shared automatically across a VCF fleet, so plan them for each vCenter and keep the configuration consistent. Using the same provider everywhere, by importing the same Native Key Provider backup or connecting to the same key servers, makes cross-vCenter migrations and disaster recovery simpler. Separate providers for each vCenter limit the blast radius if a key is ever compromised, at the cost of more to back up and track, and either approach works.

Avoid Circular Dependencies

The classic key provider mistake is a loop. The key server runs as a VM, the VM lives on an encrypted datastore, and the datastore can’t be unlocked until the key server is running. Everything works fine until the day everything restarts at once, and then nothing does. Broadcom’s documentation specifically warns about this kind of dependency loop with key servers that run as virtual appliances (docs).

A few rules help:

• Keep key servers outside the infrastructure they protect. Physical appliances, a separate cluster, or another site all work, as long as the key servers don’t depend on the keys they hold.
• Watch for drift. A loop can sneak back in later, when someone uses Storage vMotion to move a key server VM onto an encrypted datastore, or turns on vSAN encryption for the cluster the key server runs on (standard key provider). Check placement during regular reviews and before turning on vSAN encryption, and use vCenter permissions to limit who can move key server VMs.
• Look past the key server. Key servers need DNS, time, networking, and often authentication. If any of those run on encrypted infrastructure that needs the key server first, the loop is still there, just bigger. Our post on dependencies covers this in more depth. Some organizations have an individual ESX host set up as a “dependency host” which runs a clustered DNS, NTP, AD, and KMS node, and can be started ahead of the rest of the data center.

Back Up the Keys, and the Backups of the Keys

Losing a key means losing the data. There’s no back door, which is the point of encryption, and it means backups of key material matter as much as backups of the data:

• Back up the Native Key Provider. In the vSphere Client, select the vCenter, go to Configure > Security > Key Providers, select the provider, and choose Back Up, protecting the file with a password (docs). The Native Key Provider is included in vCenter’s file-based backups, but a new provider can’t be used until you back it up at least once, and vCenter raises an alarm every 24 hours until you do.
• Store the backup file and its password separately, somewhere you can reach them when vCenter is down.
• Treat vCenter backups as sensitive. They contain the Native Key Provider’s key, along with the credentials vCenter uses for external key servers (native key provider). Protect them the way you’d protect the keys themselves.
• Back up the key servers. With a Standard Key Provider, the keys live in the KMS, so the KMS vendor’s backup and replication features are part of your design. VCF is designed to leave old keys in the KMS, because a backup or disaster recovery copy might still need them, so decide how long to keep them based on how long you keep backups.
• Be careful deleting key providers. vCenter allows deleting a key provider that’s still in use, and VMs that depend on it keep running but can’t power on again until it’s restored (standard key provider). Rekey VMs to the provider you’re keeping first, and keep the old provider’s backup or keys in case a backup or replica still needs them. The good news here is that, if you made a mistake, you can restore the key provider configuration to fix it, and then try again.
• Save each host’s recovery key. ESX encrypts its own configuration with a key sealed by the host’s TPM. If the TPM can’t release that key, such as after a TPM or system board replacement, the host can’t boot without its recovery key, so the VCF Security Configuration Guide recommends backing up each host’s recovery key somewhere off the host; the “esxcli system settings encryption recovery list” command shows it, and you can access this through PowerCLI without needing to log in to each host.

If you lose vCenter and have neither a Native Key Provider backup nor a vCenter backup, the encrypted VMs can be permanently inaccessible (VM encryption). Keep a record of where every copy of every key lives. NIST notes that cryptographic erase can’t be trusted for media whose keys were backed up or escrowed somewhere you can’t account for (NIST SP 800-88 Rev. 2), so that record matters at the end of the hardware’s life, too.

Rotate Keys on a Schedule

A key that never changes gives anyone who copies it unlimited time to use it. In one incident, attackers used a signing key that was supposed to have been retired years earlier, and the investigators recommended rotating keys frequently and automatically. That was an identity signing key, but the lesson carries over to storage keys, too.

VCF gives you several ways to rotate keys:

• Wrapping keys. A Standard Key Provider’s wrapping key is generated automatically by default and rotates every 365 days, adjustable from 30 to 3,650 days. You can also specify your own wrapping key and rotate it manually (docs). Existing VMs keep their old keys until they’re rekeyed.
• Shallow and deep rekeys. A shallow rekey replaces the key encryption key and can run while a VM is powered on. A deep rekey replaces the data encryption keys and rewrites the data, so it needs the VM powered off with no snapshots (how it works). Rewriting the data also adds wear to flash devices, so save deep rekeys for when you have a reason, such as a suspected key compromise. In general, a shallow rekey is sufficient for most security and regulatory compliance needs.
• vSAN rekeys work the same way at the cluster level, and the VCF Security Configuration Guide suggests a regular schedule of shallow rekeys, such as every 60 days.
• Native Key Provider rotation is manual. If you need to rotate this key, simply create a new provider, make it the default, and rekey (native key provider). Make a backup of the old key provider before you delete it, please!

Pick a schedule your team can keep, automate it where you can, and treat a rekey like any other change: test it first.

Test Key Provider Recovery Before You Need It

The worst time to find a gap in key provider design is in the middle of a recovery. Practice it in a lab:

• Restore a Native Key Provider from its backup into a test vCenter, and make sure someone other than the person who made the backup can find the file and the password.
• Shut a test cluster down completely and bring it back up with the key servers starting the way they would after a real outage. Note how long it takes before encrypted VMs are available.
• Take one key server offline and confirm that the others carry the load.
• Confirm that running encrypted VMs keep working while vCenter or the key servers are down.
• Include encrypted VMs in disaster recovery tests.

Encryption protects data from people who shouldn’t have it. Good key provider design helps keep you safe, secure, and compliant.