Introduction
I got a question from a customer a while back about backing up and replicating encrypted VMs – it’s a fair question, “if they’re encrypted, how can I backup and restore”?
This was particularly pertinent as the customer in question was a service provider – so it wasn’t as clean cut and straightforward as it may have first appeared if they have multiple customers with multiple distinct KMS servers and encryption keys, how can they offer a backup or DR service?
VM image-level encryption
Most backup and replication software use changed block tracking (CBT) to keep a map of the updated blocks since the last backup, this is provided by the VMware APIs for Data Protection (VADP), CBT tracks the unencrypted IO of VMs, but is itself encrypted separately on the VMware datastores.
Any backup software that wishes to backup the encrypted VMs must use the VADP and VDDK to either backup the disks in hot-add mode or NBD mode with SSL enabled – note: this will allow the backup software access to the unencrypted IO of the VM and it is the responsibility of the backup software, or destination storage to re-encrypt the IO as it lands on the target in order to maintain compliance.
Guest OS level encryption
All of the above assumes you are encrypting at a VM image level, not within the VM’s guest operating system. When the data within a virtual disk is encrypted at an OS level instead, then ESXi has no visibility of the unencrypted IO for backup, CBT becomes less useful at this stage as it has no access to the raw unencrypted changed blocks so deltas will be much larger.
In addition, if the VM is restored or replicated to a remote site, that VM needs access to the Key Management Server (KMS) in order to be able to boot – if the KMS is not available the VM is unbootable as the OS level encryption will not decrypt the data otherwise – this mandates the need for the remote site to share the same KMS infrastructure as the source site. A shared KMS infrastructure, especially from a service provider perspective is particularly problematic when it comes to trust levels and peering a single KMS instance with multiple distinct customers.
Using image-level encryption to replicate VMs
Let’s focus on image level encryption as a result, and as it is a solution we provide natively with vSphere and vSAN. When using image-level encryption with a replication solution blocks will be decrypted through the VADP and VDDK, replicated across the wire (either WAN or LAN) in an encrypted fashion, via SSL to the destination replication appliance, and then re-encrypted using the target vCenter’s DEK and KEK as it is treated as a net “new” object on the target side.
Replicating VMs in this fashion, such that they will be created with new encryption keys on the target side is very powerful as it negates the need for a common, shared KMS across sites and customers – this allows a new level of multi-tenancy where service providers can set up a replication or backup service for customers and not need to integrate with their KMS infrastructure, but rather just employ their own solution within their target site. In addition to simpler operations and smaller trust zones, given the VM IO is able to be read unencrypted through VADP before being transferred this leads to much smaller replica deltas, reducing bandwidth and RPO times.
It is important to understand that because the IO is allowed to be read unencrypted via VADP by the backup or replication solution for this purpose, it is up to the backup/replication solution to provide in-flight encryption as well as encrypting backups if they are at rest on a non-encrypted VMware datastore to ensure that compliance is maintained for the data at all times.
Conclusion
To sum up, if you want to provide a backup or DR service for multiple customers, who are all using distinct KMS servers for their environments, rather than using OS-level encryption and having to extend your trust zones to encompass their KMS, use VMCrypt or vSAN Encryption and backup/replication software that supports VADP to allow you to offer services without the need for shared KMS infrastructures.
For more information on VADP, check this KB article and for more information regarding vSAN and VMCrypt visit vSphere Central and for details on vSphere Encryption interoperability, check out the docs.
