A new container vulnerability has emerged (CVE-2019-5736) that seemingly affected all the container platforms that use runC, a standardized runtime that allows creation and running of containers. The vulnerability affects Docker, Kubernetes, and even Apache Mesos, which does not use runC.
A bad actor can gain control of your host by exploiting something called privileged containers or privileged mode. Privileged containers are containers that map the user id of the root user (0) to the same user id within the container. Privileged mode is the Kubernetes equivalent, but it is configured at the Pod level. In this CVE, the attacker uses privileged containers to run commands as root within the container. With this level of control, the attacker can overwrite the runC binary on the host which gives them the ability to execute arbitrary code with root-level permissions. When this happens, your data and systems are, to put it mildly, at risk.
Privileged containers are used when the container needs to access native devices, or the network stack. Some monitoring and control applications require this mode, as does the Istio service mesh. However, privileged containers are not recommended for general use-cases. It’s always best to apply a “least privilege” mindset in container deployments. The most secure systems will feature unprivileged access whenever and wherever possible. Privileged containers may sound like a cool feature or something beneficial. In fact, it’s a potentially dangerous condition for enterprise IT. That brings us to Pivotal Application Service (PAS), our flagship app platform.
PAS incorporates an Open Container Interface (OCI) standard implementation of runC. This is the same container execution code that exists in Docker and Kubernetes. Yet PAS is not susceptible to this new exploit. Why?
Pivotal Application Service Features Containers that are “Secure By Default”
PAS is a secure by default platform. It features dozens of embedded controls that increase the security of the platform, and reduce the attack surface. While PAS implements both privileged and unprivileged containers, all application instances and staging tasks within PAS run as unprivileged. This eliminates the threat of root escalations inside the container.
Additionally, PAS uses the full set of Linux namespaces to provide isolation between containers running on the host. Within PAS, the User namespace is never used for running privileged containers. The breakout access of this new exploit is blocked through the proper use of the user namespaces and mapping the root user id to something other than 0.
PAS adheres to this best practice and maps the user id of root (0) to a different user id on the host to prevent any app from assuming the root user id if a breakout occurs. Further, PAS provides seccomp whitelisting. Seccomp is a system call filter that can be thought of as a system call firewall. It restricts the access of containers to a specific set of system calls, and therefore reduces the risk of container breakout. The net result of these security implementations is that PAS is protected against this type of exploit at multiple levels. This is just one example of the security benefits of implementing an secure platform like PAS.
Some organizations in heavily regulated industries require additional security capabilities. To this end, PAS security can also be enhanced further by installing the File Integrity Monitoring (FIM) for PCF tile. This handy add-on helps you comply with standards such as PCI and HIPAA. How? By alerting you anytime something changes in a set of directories in the file system. If a new file is added, or if an existing file is modified or deleted, you’ll be notified in the log. You will even be notified if the permissions are modified. Any such change is logged and can be linked to an event management system for alerting. Even if a threat-actor was able to break out of a container and overwrite runC, despite the protections PAS provides, FIM would flag the activity so that the operator would be notified immediately.
For additional reading on this topic:
PAS Makes Certain Promises About How It Secures Your Containers
The alternative approach can be seen in Kubernetes, where the operator is allowed to build the cluster on almost any Linux OS. The operator is free to configure security at the OS layer, and of Kubernetes itself. While this offers extreme flexibility, it can potentially lead to insecure systems. Ops teams now have to overcome common pitfalls of traditional IT: incomplete planning, poor change control, and poor implementation. Contrast that with the peace of mind that PAS provides. The app platform itself is built on hardened OS images called stemcells, and the connectivity inside and outside the platform is limited to what is necessary.
Repair Your Systems: One of The 3 R’s of Enterprise Security
As a secure by default platform, PAS can provide the entire stack from the OS through the container and the routing tier. At Pivotal, we often refer to this comprehensive approach as the 3 R’s of Enterprise Security: Rotate, Repave, and Repair. (We won’t cover all 3 of the R’s in this post, so for more information, you can refer to this earlier post. But it does make sense to go deep on “repair”, as that is material today’s CVE.)
Repair (i.e. patching) means you can quickly repair your entire stack and patch everything instead of worrying about each individual component without downtime. This is true of the platform itself, because of the aforementioned stemcells. It extends to vulnerabilities discovered in application frameworks, because PAS uses the buildpack model. So, when exploits like this actually do affect PAS, remediation is fast and automated by the platform. Take a look at this Spectre/Meltdown post, How to Apply the Meltdown Fix to All Your Systems in Less than a Day, to see the dramatic effects these capabilities can have.
You Need a Better Security Model for Kubernetes. That’s Pivotal Container Service
Of course, Kubernetes has a very important role to play in enterprise IT. You just need a better way to patch it! So we apply the same core concepts to Pivotal Container Service (PKS). Our goal is to make Kubernetes “just work” when it comes to security and Day 2 operations.
For example. PKS-deployed Kubernetes clusters are built on security hardened stemcells. In fact they are deployed by the same operational management layer (BOSH) as PAS. Repair (patching) and "repave" operations are part of the service. You can read more about how those capabilities were applied to a recent Kubernetes CVE, here.
PKS 1.3.3, which implements the fix to this current exploit, has already been released and it is recommended it be applied as soon as possible. The cluster upgrade functionality and automation enabled by PKS makes applying fixes during events like this seem just like any other morning.
Additionally, Pivotal and Heroku submitted a buildpack implementation to the Cloud Native Computing Foundation that over time will help ease the operational burden of patching applications, dependencies, and containers on any container platform.
“Shift Left” and Reduce Your Risk
Just as you are modernizing your software practices, you have to rethink your approach to security as well. Security is now the responsibility of every engineer—so it helps if your chosen tools and platforms can be ‘secure by default’ wherever possible. Automated and secure by default platforms can help with that transition by reducing misconfiguration and by providing a solid “repair and repave” strategy that reduces the risk of an exploit, and by simplifying and shortening mitigation efforts if an exploit does occur.
For more information on securing Kubernetes, sign up for the upcoming webinar:
6 Things You Need to Know to Safely Run Kubernetes with Cornelia Davis
