In our last post, we introduced the CredHub Service Broker, a tool that makes it easy to encrypt app credentials at rest without making any changes to app code. We described how to use the service broker with the CF CLI, and what you can expect from the service once you’re up and running.
The feedback so far has been overwhelmingly positive. There's a lot of excitement around the possibilities unlocked by the new service broker. Many of you have since kicked the tires on the beta.
One constructive commenter pointed out that the post was written from the perspective of a developer (guilty as charged). The reader went on to note that the piece wasn't very clear for enterprise architects, or people who are not familiar with Cloud Foundry. Point taken!
So let’s take a look under the hood. After reading this post, you should be confident in how the CredHub Service Broker handles your sensitive credentials.
Pivotal Cloud Foundry is a complex, distributed system. We’ll focus on just the components that matter for the task at hand: explaining how the service broker works.
-
Pivotal Application Service (PAS): This is the whole enchilada, the full orchestrated deployment of the Cloud Foundry app platform, receiving workloads from your developers and managing their lifecycle in the cloud.
-
Cloud Controller: PAS's API server. It's responsible for handing off requests to the relevant components, and is also the component of record for storing your apps.
-
CredHub: The lockbox. CredHub is an encrypted key-value store with access control rules that map credential resources to actors and operations. CredHub supports mutual TLS and OAuth2 for authentication.
-
CredHub Service Broker: The CredHub interface. This is a component separate from the CredHub server itself, which is authorized to create new credentials in CredHub and apply permissions to them. This adheres to an API — the OSBAPI — that Cloud Controller understands how to pass requests off to.
-
Diego Cell: The VM where your apps run. Diego is a complex system unto itself, but for our purposes, the cells, which are the workers that run your workloads, are the only relevant components.
-
App Container: Where the magic happens. One of these is created for every instance of your application. Containers keep your workloads isolated from other applications on the same Diego Cell.
Now that we understand the players, it's time to talk about the game!
How the Service Broker Stores Credentials in CredHub
Here’s what happens when a user (human or machine) puts credentials in CredHub using the Service Broker:
When Cloud Controller receives the create-service request (#1), it takes the credentials and passes them off to Service Broker (#2). Service Broker, who is permitted to write to CredHub, stores that value in CredHub in a known location (#3), and returns the value of that location to Cloud Controller (#4). The Cloud Controller then stores it as the "credentials" corresponding to the new service.
How an App Reads Credentials from CredHub
Next, let’s review what happens when a user requests that an app be allowed to read those credentials:
When Cloud Controller receives the bind-service request (#1), it first informs Service Broker that the binding is occurring (#2).
Service Broker asks CredHub to grant read access for the intended credential to the app (#3).
From there, the Service Broker lets Cloud Controller know that its work is done (#4).
Cloud Controller then includes the CredHub path in the app's metadata, which the Diego Cell includes in the environment when the app is started (#5).
Launcher is a process that runs within an application container, and it is responsible for starting your app. Right before kicking off your workload, it will first ask CredHub for the values corresponding to any CredHub paths the app has been granted red access to. It will then put the resulting values in the app's process environment just before it starts (#6).
"But wait!" you exclaim. "How does CredHub know to trust that Launcher is really launching the intended app?" As you might expect, the Launcher has to prove its identity. It authenticates with CredHub by using a credential that is only available in the app's own container. Here's how it works:
On the container's side (blue), the app instance gets its identity when its container is provisioned. Every time the Diego Cell starts a new container, it uses an intermediate certificate and private key to sign a leaf certificate and key, and places both in the container's file system (#1). That certificate's metadata includes a GUID that identifies the app that will be running inside this container.
Later, when Cloud Controller tells Diego to include a CredHub path in the app's environment (#2), Launcher takes over. Launcher runs within the app's container, and uses the leaf cert and key within the environment to establish an mTLS connection with CredHub (#3). CredHub confirms that the cert was signed by the Diego Cell's intermediate CA. Then, CredHub checks the app GUID against the access control list for the requested credential. This flow ensures that “read” permission has been granted to the requesting app's GUID, for the given credential.
Let’s now step through the permissions granting side (green). After Cloud Controller has received a bind-service request, it passes on the app GUID and the service to Service Broker (#1). The Service Broker, which has permission to add access control entries for the credential it created previously, asks CredHub to grant read permission to the relevant credential, for the requested app GUID (#2).
Applying the Principle of Least Privilege and the “Need to Know” Standard
This is how we ensure the security of your sensitive credentials. Throughout the design of this process, we kept our eyes on the principle of least privilege. We applied the "need to know" standard across the platform to ensure that the fewest number of components would have access to your credentials.
This reduction in credential “touch points” will be apparent to long-time Cloud Foundry users. Before the CredHub Service Broker, credentials were passed through several different CF components. This older model was sufficient for many InfoSec teams. But Pivotal strives to give our customers a superior security posture.
And that’s just what the CredHub Service Broker delivers. We kept two ideas in mind while building the feature:
-
We wanted credentials to only ever be stored at rest in CredHub.
-
We wanted to enforce a strict app identity scheme with only Diego Cells as provisioners.
As a result, we were able to significantly reduce the exposure of your credentials to both network infiltrations and accidental vulnerabilities like log leaks.
The story gets even better when you're using a CredHub-enabled service tile, like Spring Cloud Services, RabbitMQ and MySQLv2. In those cases, Service Brokers themselves are responsible for generating credentials granting access to services, which means even Cloud Controller itself is kept 100% out of the loop: only CredHub, your application, and the Service Broker itself ever get the credential in the clear.
Ready to learn more about the CredHub Service Broker? Read the docs. Then, download the beta tile, and try it out!
Security will be a hot topic at the cloud event of the year: SpringOne Platform Sept 24-27 in Washington, DC. Register now!
