Tanzu Service Mesh Security Enhancements using Confidential Computing

Performance and Security Optimizations on Intel Xeon Scalable Processors with Intel SGX – Part 3

Contributors

Andrew Babakian — VMware

Saidulu Aldas, Ramesh Masavarapu, Sakari Poussa, Tarun Viswanathan — Intel

Introduction

Intel and VMware have been working together to optimize and accelerate the microservices middleware and infrastructure with software and hardware to ensure developers have the best-in-class performance and low latency experience for building distributed workloads. The focus is on improving the performance of crypto accelerations and making workloads more secure.

The Service Mesh architecture pattern solves many problems, which are well-known and extensively documented, and will not be central to this discussion. However, the focal point of this blog series will include the architectural challenges of Service Mesh in the following top focus areas:

Performance
Security

In Part 1 of this series, we looked at how Tanzu Service Mesh uses eBPF to achieve network acceleration. In Part 2, we showcased how Intel and VMware collaborated to accelerate Tanzu Service Mesh crypto use cases and improve the performance of asymmetric crypto operations.

In this Part 3 blog series, we will discuss one security challenge (concerning the service mesh private key protection mechanism) and our solution.

In the current implementation of Service Mesh, Private keys used by the data plane to perform the TLS handshaking are not encrypted nor protected. They exist in plaintext in memory and the filesystem. All the CA private keys are stored in clear text in Kubernetes.

Figure 1: Stored in plaintext CA private key dump

Typically, the Service Mesh generates the private key as a self-signed key and is stored as a base64-encoded Kubernetes secret. Anyone with kubectl admin access can read it with a simple kubectl” get secret my-private-key -o json” and then do a base64 decode. Kubernetes does have support for using a key management service (KMS) to manage the key, which avoids having to store it in a secret. That helps protect from the rogue Kubernetes admin. However, you must set up and manage a KMS or “trust” the vendors’ KMS. If the KMS runs as a local service for your cluster, then a clever hacker with host access could theoretically read the unencrypted storage to retrieve the key.

Figure 2: TSM and External CA

The solution design with TSM

Intel and VMware are working on securely storing Service Mesh private keys without having to “trust” 3rd party KMS. No modifications to the software are required, and no need to rebuild the container images. Intel® SGX-based memory encryption technology provides CPU enforcement and attestation to assert that the enclave is genuine. It enables software applications to protect sensitive parts of their running code, and that data is encrypted and cryptographically isolated inside enclaves. Even someone with admin privileges to the host system and/or cluster who installs tools to snoop network traffic or read memory will only see encrypted information.

The solution used Trusted Certificate Service that implements the k8s Certificate Signing Request APIs, which matured to V1 in Kubernetes release v 1.22. It creates an enclave, loads the key signing code, and attests to the enclave. Once established, the CA key is created in the enclave. All future certificate signing happens within the enclave, with no visibility from the outside.

Working model:

Figure 3: The solution architecture

As depicted in Figures 3 and 4, The solution architecture and workflow comprises the following:

Trusted Certificate Service uses the External Certificate Authority (CA) feature and configures it to use an External CA for key signing. Once it is fully started:
1. The enclave is created.
2. The key generation and signing code are initialized into the enclave.
3. The private key is created within the enclave.
4. The TSM is configured to use an external CA for key signing.
The private key is created within the enclave, protected by encryption.
The envoy generates its key pair for every application Pod and then requests TSM to sign the certificate with the private key. The certificate signing request passes to the Trusted Certificate Service to sign, which holds the signing key inside its enclave. The signed certificate is then returned through TSM to the proxy, enabling it to start communicating securely on the TSM.