Easy Security Win: VCF Storage Encryption

(Day 15 of 22 in our Cybersecurity Awareness Month series. Today: storage encryption)

Some security work is hard; network segmentation means meetings with every application owner, and identity projects take years. Storage encryption is different. In VMware Cloud Foundation (VCF), vSAN encryption takes a few clicks and runs in the background without taking workloads down. It also protects against one of the most common ways data leaks, which is a drive leaving the building with data still on it. Like any security control, it has limits, so let’s take a look at what it does and doesn’t do before you turn it on.

What Storage Encryption Protects Against

Drives leave data centers all the time. They fail and go back to the vendor, they get swapped out during upgrades, and whole servers get retired, sold, or recycled. Every one of those drives holds whatever was written to it, and reliably wiping every drive, through every vendor and contractor that touches it, is harder than it sounds.

Flash drives, like SSDs and NVMe devices, make this much harder, to the point where you can’t count on sanitizing one at all. At their core, flash devices are consumable parts: the high voltages used to write and erase data wear out the memory cells a little each time, which is why drives are rated in Drive Writes Per Day (DWPD). To spread out that wear, drives write new data to fresh locations, distribute writes evenly across the device, and hold spare capacity in reserve to replace worn-out blocks, so old copies of data can linger where software can’t reach them. In one study, overwriting a drive’s entire visible capacity twice usually cleared it, but not always, and some drives’ built-in erase commands didn’t work as advertised (research paper).

The encryption built into drive firmware can’t always be trusted, either. When researchers reverse engineered the firmware of several self-encrypting SSDs, many had flaws that allowed complete recovery of the data without any password or key (IEEE paper), and Microsoft advised BitLocker users to switch to software encryption instead (Microsoft).

The consequences can be expensive. One organization paid large regulatory fines after its decommissioned drives and servers were resold with customer data still on them, even though it had encryption software it never turned on.

Storage encryption turns these problems into a key management problem instead. Key management is a smaller problem. vSAN data-at-rest encryption protects data on storage devices in case a device is removed from the cluster (docs), because the data on a drive can’t be read without its key. vSAN does that encryption in ESX, before data reaches the drive, so it doesn’t depend on drive firmware getting encryption or erasure right. That also means you don’t need self-encrypting drives (SEDs), which cost more and add operational work, since they need controllers that support them and a process for managing each drive’s keys.

Storage encryption also makes retiring hardware simpler. The keys that protect vSAN data come from your key provider, not the drive, so destroying them leaves the data unreadable without relying on the drive’s own erase commands. NIST’s media sanitization guidance calls this cryptographic erase, counts it as a way to purge data, and notes that for virtual and logical storage it may be the only practical one (NIST SP 800-88 Rev. 2). It assumes nothing sensitive was written to the drive before encryption was turned on, though, which is one more reason to turn storage encryption on early.

What Storage Encryption Doesn’t Protect Against

Storage encryption protects data on devices that have left your control. It does much less about attackers who are still inside your environment, because encryption is transparent to anything that’s allowed to read the data:

• Data is plaintext inside the guest. A virtual machine reads and writes its disks normally, so malware inside the guest, or an attacker with a valid login, can read everything the guest can.
• Administrators can still reach data. Even the No Cryptography Administrator role, which can’t change encryption settings, keeps direct console access and can download virtual machines from an encrypted vSAN datastore (vSAN encryption paper). Access control still matters, so use vCenter permissions to limit who can reach sensitive VMs.
• Attackers who control a host aren’t stopped. Ransomware running with administrative access to ESX can encrypt VM files on an encrypted datastore as easily as on any other, because the host can read the data. Keep ESX itself locked down, with lockdown mode and a management network isolated from everything else.
• Backup data is cleartext. Backup products read VM data through the hypervisor and send it to the backup system over the network, typically protected by TLS. The VM data itself stays cleartext, because otherwise the backup system couldn’t deduplicate or compress it, and those are two core space-efficiency tools that keep backup storage costs under control. That means backups are unencrypted at rest unless the backup product encrypts them itself (best practices), so turn on encryption in your backup product.

Storage encryption is still worth doing. It’s one layer of protection, and the other posts in this series cover the other layers.

Start with a Key Provider

Encryption needs keys, and in VCF, keys come from a key provider. There are two kinds (comparison):

• The vSphere Native Key Provider is built into vCenter and needs no external key server. vCenter generates a primary key and pushes it to the ESX hosts in the cluster, which can then create encryption keys even when vCenter isn’t available (overview). Hosts don’t need a TPM, though a TPM helps protect the key on each host.
• A Standard Key Provider connects vCenter to an external key management server (KMS) using the KMIP protocol. The keys stay in the KMS, so a stolen host or drive doesn’t carry its key with it.

The Native Key Provider is the fastest way to start. The trade-off is physical security: the key material is stored on the hosts, so someone who walks off with an entire host may be able to boot it and read its data (key providers). An external KMS holds up better against that, at the cost of another system to run and keep highly available. A common pattern uses both, with a Standard Key Provider for vSAN encryption and the Native Key Provider for virtual TPMs.

Be careful with the Key Persistence feature, too. It caches a Standard Key Provider’s keys on each host, sealed by the host’s TPM, so hosts can restart and unlock their data without reaching the key server. That brings back the Native Key Provider’s physical risk, because a stolen host that boots normally can unlock its data. Key Persistence is off by default, and the VCF Security Configuration Guide recommends keeping it off on hosts without controlled physical access (Standard Key Provider).

Either way, back up the key provider before you rely on it. vCenter requires a backup of a new Native Key Provider before you can use it (docs), and losing the keys means losing the data.

Turn On vSAN Encryption

vSAN offers encryption for data-at-rest and data-in-transit, and they’re independent of each other:

• Data-at-rest encryption encrypts everything the cluster stores, regardless of storage policy (docs). On vSAN Express Storage Architecture (ESA), data is encrypted once, near the top of the storage stack and after compression, so you keep the space savings (vSAN encryption paper). Deduplication works with data-at-rest encryption, too, including global deduplication (docs).
• Data-in-transit encryption encrypts vSAN traffic between hosts, including traffic to witness hosts, using AES-256 with forward secrecy. It doesn’t need a key provider at all, and it changes its keys every 24 hours by default (docs).

Turn on both. In the vSphere Client, they’re under the cluster’s Configure > vSAN > Services > Data Services. You can turn on data-at-rest encryption for an existing cluster while it’s running; vSAN rewrites the data in the background with no VM downtime, as long as the cluster has the hosts and capacity to move data while it works (vSAN encryption paper). Small clusters, like three-node clusters, don’t have a spare host for that and need the Allow Reduced Redundancy option, which lowers redundancy while the rewrite runs, so make sure backups are current before you start. On a cluster that already holds a lot of data, the rewrite can take a while and keeps the vSAN network busy, but it runs in the background, so it doesn’t tend to be noticeable, and it’s worth the trade-off even if it is.

Turning on encryption for an existing cluster encrypts the data that’s there, but it can leave old, unencrypted copies behind on the drives, and for the reasons covered above, there’s no reliable way to clear those off flash devices. If a cluster held sensitive data before you turned on encryption, then when its drives retire, handle them like drives that were never encrypted. It’s simplest, though, to turn on vSAN encryption when you build a cluster, before there’s anything to rewrite.

If you are required to rekey periodically, it’s easy to do. A shallow rekey replaces the key encryption key and finishes quickly. A deep rekey replaces the data encryption keys and rewrites all the data, which adds wear to flash devices and shortens their life, so don’t do it lightly. Save it for when you have a reason, such as a suspected key compromise. A shallow rekey is usually more than enough for security and regulatory compliance needs.

When VM Encryption Makes Sense

VM encryption works on individual virtual machines instead of a whole datastore. You apply it with a storage policy, and it encrypts the VM’s virtual disks along with its NVRAM, swap, and memory snapshot files (how it works), on vSAN or any other datastore. It makes sense for:

• Storage that isn’t vSAN, where VM encryption gives you storage encryption that works the same way on every datastore. It does tend to lower deduplication and compression ratios on external storage arrays, since encrypted data doesn’t deduplicate or compress well. vSAN’s own encryption avoids that, because vSAN compresses and deduplicates data before encrypting it.
• Workloads that deserve extra protection, like domain controllers, certificate authorities, and secrets vaults, where you want encryption that travels with the VM and permissions that control who can decrypt it. Our post on protecting Active Directory covers this.
• Requirements that go beyond encryption at rest. Regulations that call for encryption at rest, like NIST SP 800-53 control SC-28(1), can be met with vSAN data-at-rest encryption, too. VM encryption adds something when a requirement goes further, like separate keys for each tenant or workload, or encryption that stays with a VM’s files if they’re copied off the datastore. For PCI DSS, requirement 3.5.1.2 counts encryption that’s transparent to a running system, like vSAN or VM encryption, toward protecting stored card numbers only on removable media or alongside another protection for the card numbers themselves.

A few cautions:

• Don’t double up. Broadcom strongly recommends against adding VM encryption on top of an encrypted vSAN cluster unless a regulation requires it. It adds CPU load, cancels out deduplication and compression savings, and triggers a health alert (KB 437757). Virtual TPMs on encrypted vSAN are fine. In general, the same advice applies to encryption inside the guest, like BitLocker or LUKS, though for a few critical systems, like Active Directory domain controllers, it can be warranted.
• Leave the management components out. Don’t VM-encrypt vCenter, VCF Operations, or other core management VMs, because Broadcom warns it can lead to a permanent management lockout (KB 437757). They can run on an encrypted vSAN datastore without a problem because vSAN is designed to not have dependency loops.
• Plan for the power-off. Encrypting an existing VM with this type of encryption requires powering it off (docs), and it needs free space of at least twice what the VM uses now (requirements).

Turn On Storage Encryption

Storage encryption is about as easy a security win as you’ll find, especially for vSAN data-at-rest encryption. Set up a key provider and back it up. Turn on vSAN data-at-rest and data-in-transit encryption on every cluster, ideally when you build it. Use VM encryption where it adds something, and leave the management VMs out of it.

Do that, and a drive that leaves your data center is much less likely to take readable data with it.