Worried About Header Spoofing and Compromised Networks? Fight Back with TLS in Pivotal Cloud Foundry.

When securing your applications, you need to secure incoming traffic as well. If you’re wondering why, ask yourself the following questions: Is your application traffic encrypted, in case someone is listening in? Is the user who they say they are?

These questions matter. And without sufficient answers, there will be consequences.

It’s not enough to simply use HTTPS to encrypt the communication between the user and their external load balancer. Sure, you might be protected against Man-in-the-Middle attacks on the user side, but you haven’t protected traffic in the event that your network is compromised.

Don’t Fall for the “Network is Secure” Fallacy!

Here’s the reality: compromised networks are fairly common. So common, in fact, that assuming “the network is secure” is one of the fallacies of distributed computing. In an age where the number of dependencies, frameworks, and external services only continue to grow, all it takes is one vulnerability to allow an attacker to execute code remotely. This simple act can give the bad actor free reign over your network.

Those are the risks, but what are the solutions? As is often the case in tech, you need to ask more questions before you get to a useful answer:

• What happens to your traffic once it hits your load balancer?

• Does it flow around your infrastructure in search of a frontend application?

• Does routing hinge on a properly configured DNS entry?

• If your SSL encryption is terminated at the load balancer, are those requests encrypted on the internal network?

If you’re asking these questions early in the development process, take a bow. Most of your peers aren’t thinking about them until the end of a multi-month release cycle. If you’re running on a platform such as Pivotal Cloud Foundry, though, it’s reasonable to expect that it should have the answers to these questions already. After all, modern businesses release software thousands of times a month. A software-driven enterprise needs an automated, programmatic way to encrypt ingress traffic. And they need a way to do it at scale for thousands of apps.

At Pivotal, we want to make the easy thing the secure thing. So we’ve been steadily adding the transport layer security (TLS) protocol to every nook and cranny of PCF. As the successor to SSL, TLS provides both a means of encrypting communications and mutually proving the identity of both the client and the server—meaning you can be sure whoever is sending the request is who they say they are, and vice versa. Developers like it because it’s supported by the OpenSSL toolkit and every major browser.

TLS encryption is a crucial feature for InfoSec teams and compliance-minded engineers everywhere, so let’s take a closer look at how Pivotal Cloud Foundry uses this tech.

Bringing TLS to Every Pathway in PCF

In the PCF 2.3 release blog, my colleague Jared Ruckle quips that “it’s not really a PCF update unless we’ve got a TLS enhancement, right?”. It’s true, though.

Just take a look at our progress over the last two years, shown below.

You’ll notice a big emphasis on the encryption of intracluster communication. For example, requests from the router to your applications and API traffic within the cluster are all encrypted. On-platform services like MySQL and RabbitMQ use TLS to encrypt communication between clients. Many tiles even use TLS between nodes in their respective clusters. With each release, we get that much closer toward every line of communication within PAS being authenticated and encrypted.

IPsec Add-On: For the Compliance-Minded

Long-time PCF customers use the IPsec Add-on for PCF for encryption. When added to your BOSH deployment, the IPsec BOSH add-on will setup a secure tunnel for all traffic sent between the virtual machines in your PCF environment. While TLS works at the application layer, IPsec works further down the stack at the network layer.

IPsec is a handy option if you have strict compliance requirements. Every bit sent into and out of platform VMs is encrypted. So you might want to consider using the IPsec Add-on as a complement to all the wonderful TLS enhancements discussed above.

But there’s still much more that TLS can bring to the platform, namely identity and authentication. Let’s conclude our tour of TLS in PAS with a killer feature: the automated certificate creation per application instance.

mTLS: Certification of Authenticity Included

In PCF 2.1, PAS introduced a mechanism that allows each application instance to have it’s own identity. Using mutual TLS (mTLS), both the client and the server are able to securely identify themselves…and communicate over an encrypted connection.

Our own Eric Malm wrote up a great piece that details several scenarios where PCF leverages mTLS. This informative post gives you a high-level summary and plenty of technical details.

The ability to reliably identify your app instance’s identity leads to some significant benefits for security and stability. Let’s revisit a foundational workflow in Pivotal Application Service:

"Starting in PCF 1.12 the Pivotal Application Service tile issues a unique certificate for each running app instance. This mechanism encodes the identity of the application instance on the platform in several different ways. Further, the certificate is valid for only 24 hours.”

This certificate contains useful metadata, including the instance ID, application ID, org and space ID, and the IP address of the container on the container-to-container network. This certificate—and the data in contains—is important because PCF uses these certificates in a growing number of scenarios.  Here are a couple of examples:

• Beginning with PCF 2.1, the Gorouter that accepts traffic into PAS verifies that the application instance it’s connecting to is in fact the correct one. This verification is performed using the information provided in the TLS handshake.

• Starting in PCF 2.3, that verification goes both ways, using mTLS to ensure that the traffic that the application instance receives originates from the Gorouter.

This workflow is just as useful for intracluster communication in conjunction with CredHub. When an application instance is started, it connects to CredHub to request credentials for services bound to that application. The app instance provides its certificate to confirm its identity. CredHub proceeds to complete the request. These extra steps happen automatically, so they are a nice boost to the security posture of any enterprise.

Eric goes on to point out yet another scenario: inter-application communication, where your apps talking to one another. Here, your app acts as both client and server, something that’s very common in microservice architectures. Because the app instance information is encoded in the app’s certificate and sent with every request, developers can decide to only serve requests from certain applications. We can see Eric’s example, which extracts the requesting app’s GUI and then compares it to a list of pre-approved GUIDs, accepting or denying the request as desired.

Automated Cert Rotation Reduces Your Risk

When you automate things, you protect against human error. What good with these cert rotations be if they were manual? With nearly every application instance container having their own unique certificate, rotating them by hand would take a day’s work in itself.

That’s why we automatically rotate the app instance certificates for you every 24 hours. It’s a tactic taken right from the “Rotate” section of the Three R’s playbook. If any credentials happen to leak out of your organization, these certificates are only valuable for a short period of time. As a result, the window of opportunity for an attack is greatly reduced.

This is all handled by PCF: the platform automatically generates the new certificate and key pair and places them over the old ones in the application instance container. The new certificate can then be loaded into memory and the old one can be discarded. While the Java buildpack already supports automatically reloading newly generated certificates, work is being done to bring similar functionality to other buildpacks. Until then, developers may some simple logic to periodically refresh what certificates they’re using in their application as Eric does in his example.

Here’s a Look at What’s Next for TLS in PAS

These features already give you industry-leading protection. But we’re not done, not by a long shot. Eric provides a couple examples of what’s already in the works:

• The CF Diego and Container Networking teams are actively exploring how best to route traffic between application instances through their Envoy sidecars transparently. This way, developers can move concerns such as TLS configuration, client-side load-balancing, and circuit breakers out of their own application logic. Instead, these can be configuration parameters that describe how their apps should run on PCF.

• Today, the routing improvements in PAS apply to HTTP routing for Linux applications. We plan to bring the same improvements to the TCP routing tier, and to .NET applications running on PAS for Windows.

When TLS is embedded in the platform, your developers worry less about the mechanics of encryption and spend more time on their custom code. So be sure to keep an eye on to release notes as we integrate these and other features!