Using VMware’s Harbor with PKS (and Why Kubernetes Needs a Container Registry)

Using Docker images? There’s a good chance your organization is using images with active vulnerabilities that could be exploited at this very moment. That’s why so many enterprise InfoSec teams want to ensure that their chosen container management systems meet key security and governance requirements. (It’s the #1 item for a successful container strategy, according to Gartner).

To this end, Pivotal includes Harbor, a docker image registry from VMware, with our recently launched Pivotal Container Service (PKS).

The big idea behind PKS? A recent blog post sums it up:

A recent research paper explains why we’ve emphasized security so much:

Using our framework, we have studied 356,218 images and made the following findings:

(1) both official and community images contain more than 180 vulnerabilities on average when considering all versions;

(2) many images have not been updated for hundreds of days; and

(3) vulnerabilities commonly propagate from parent images to child images.

These findings demonstrate a strong need for more automated and systematic methods of applying security updates to Docker images (…)

This is the reason why we’ve included so many useful add-ons to make PKS rock-solid and secure! For this post, we’d like to drill down into Harbor, the container registry in PKS.

Harbor is used to deploy images into Kubernetes clusters managed by PKS. Source: VMware.

Let’s explore what a container image registry does (and Harbor specifically) in more detail.

Why You Need a Container Registry in Kubernetes

A container registry is the repository for all your container images. Since your core business applications are packaged into containers (built out of container images), you must protect these images just as you would any other important enterprise IT system. That’s where the image registry comes into play.

We’ve talked to hundreds of enterprises about their desire to secure and govern their container management systems. Common requirements include:

• Role Based Access Control. Ensure only people with proper permissions can access and update image registry.

• Audit Trail. All user activity is logged for auditing purposes.

• Vulnerability Scanning. Scan all Container Images when they are pushed to the registry or pulled from the registry. If a vulnerability is flagged, the feature should block all failed images from being pushed to the registry (or pulled from deployment).

• CVE Patch Updates. Check for new CVEs and scan all container images and act on any failed images. (This is an important part of what Pivotal does for our customers across all our products).

• Notary. Sign the container image as its pushed,  and validate the signature before every pull to make sure unauthorized users haven’t tampered with the image hasn’t changed.

Kubernetes by itself doesn’t address these requirements. So we worked with our friends at VMware to include Harbor into PKS. VMware calls Harbor “an enterprise-class image registry server based on Docker Distribution.”

Here’s a look at how Harbor solves for the requirements discussed above.

Role Based Access Control

Harbor integrates with existing enterprise identity management systems. Administrators can control access to the registry and container images using familiar tools based on LDAP or other database such as Oracle / DB2. (Also: Cloud Foundry adopters can use UAA, User Account and Authentication!)

Vulnerability Scanning & CVE Updates

Harbor includes vulnerability scanning features. It works like this:

• Harbor updates its list of known CVEs from well-known CVE databases like Debian Sec Bug Tracker, Ubuntu CVE Tracker, Red Hat Security Data, Oracle Linux Sec Data, Alphine SecDB and NIST NVD.

• Each container image is scanned upon push or pull.

• If a CVE is caught, any further user Pull action is canceled based on the configured tolerance level. The image is flagged to the administrator for further action.

Administrators set a given “tolerance” ( low, medium, high) which corresponds to the severity levels defined in CVE databases.

Notary & Image Signing

In order to ensure your Container Images are tamper proof, Harbor registry provides the ability to Notarize / Sign Container Images when they are pushed to the registry, It also validates the signature every time the signed images are pulled and block image pull when the signature validation fails.

Let’s dive a little deeper, and see how these features work in real life.

Getting Started: Installing Harbor

Before we can try out the security features, we have to install Harbor first. The registry can be installed three different ways: 

1. Deployed on the vCenter environment as an OVA file.

2. Deployed directly on Linux hosts.

3. As a tile in Pivotal Operations Manager (OpsMgr). 

We recommend option 3, as that’s the most consistent operational experience with the rest of the Pivotal Cloud Foundry family.

1.  Start by downloading VMware Harbor Registry for PCF from the Pivotal Network. 

2. Login to OpsMgr and Import Harbor Registry.

3. Start PKS configuration by opening the PKS tile.

4. Select the correct Availability Zone and Network under the Availability Zones and Networks tab. 

5. Under the General tab, enter the FQDN (Fully Qualified Domain Name) that you have configured for Harbor. You’ll obtain this from a DNS provider such as Google Domains (e.g. harbor.mypksdemo.com, where mypksdemo is your domain name of choice).

6. Under the Certificate tab, generate a new certificate with the wildcard domain name that you have created in Step 5 (e.g. *.mypksdemo.com).

7. Under the Credentials tab,  enter the required admin password for your Harbor account. (You will use this password later to log into Harbor.)

8. Under the Authentication tab, select “Internal” as the Authentication mode. You also have options to select an “External LDAP” or use “UAA for Authentication”.

9. Make sure the following options are selected in the next three tabs so that Clair and Notary are installed with Harbor.

   • Container Registry Storage -> “Local File System”

   • Clair Setting -> “Install Clair”

   • Notary Setting -> “Install Notary”

10. Go back to the Installation Dashboard. Click the "Apply Changes” button to complete the installation.

11. Click on the “VMware Harbor Registry” tile. Copy the IP address under status tab. Map this IP to your Harbor DNS name (first mentioned in Step 5, harbor.mypksdemo.com.)

12. Login to Harbor at https://harbor.mypksdemo.com as Admin (use the password that you set under the Credentials tab)

13. Under the Users tab, create these two users: developer and guest.

14. Create a new Project “testproject” under the Projects tab.

15. Click on the “New Member” button and add the developer and guest users to the project. Make sure the developer is assigned the “Developer” role and guest is assigned the “Guest” role.

16. Under the Configuration tab, set the following values:

17. Click on the Configuration section and Administration menu. Download the Registry Root Certificate from “System Settings” tab. This certificate is required for Docker API to establish a connection with the Harbor registry.

18. From docker site, download and install Docker for your local machine.

19. Using a Mac? Under the user home directory, create the following folders and copy the Registry root certificate to these folders. (This is required for Docker to establish a connection with Harbor registry and for signing the images via the by Notary service.)

20. For this example, let’s download the Elasticsearch and Ubuntu images from Dockerhub.

The setup is now complete! Let’s now verify that Harbor’s important security features are working. First up: role-based access control.

Role Based Access Control

Login to Harbor registry as "developer" and push the ubuntu image. (We ’ll push the Elasticsearch image later.) Push the image to our testproject using the following commands

You will get Access Denied error as shown below:

One security feature verified! On to the next one.

Vulnerability Scanning

Login to Harbor, and check the scanning configuration for our testproject. Confirm that the options “Automatically scan images on push” and “Enable content trust” are selected. Now, go back to Projects->testproject and select testproject/ubuntu.

Scroll over the Vulnerability graph, shown above. You can see that a scan was performed, and some vulnerabilities were uncovered. Clicking on the tag name “latest” shows the results of the scan.

 Let’s adjust the vulnerability severity level to “low” and uncheck “Enable content trust option”. We do this on the project configuration page. Now, let’s try to pull the image again. Since we see a few vulnerabilities that are higher in severity (medium & high), Harbor will deny the pull request.  

 Let’s login to Harbor from docker, using the developer account, and try to pull the ubuntu image.

Following error message will be displayed, because the level of vulnerabilities detected by the scan exceeds the project’s configured threshold.

Select “Enable content trust” and set Vulnerability Severity to “High” under testproject configuration page and save the changes. We will be using this configuration in the next section.

Notary / Image Signing

Notarizing images enables content trust. Let’s set up a few environment variables before pushing the image. Recall the elasticsearch image from earlier. We can add environment variables, that enable the image to be “signed.”

Login to Harbor and select elasticsearch under testproject . You can see that the signed column is marked with a green checkmark. Let’s sign the ubuntu image by pushing it again, and entering the corresponding passphrase.

Validate that the ubuntu image is signed in Harbor UI.

Let us now try to pull either image (ubuntu or elasticsearch) based on the scan results. In our environment, the ubuntu image does not have any High severity alerts. Setting the Vulnerability Severity level to “high” will allow us to successfully pull the ubuntu image as a developer. (For your local example, you should choose a combination that works for you, by selecting the right image & Vulnerability Severity level in your environment.) I have set the Vulnerability Severity to “High” for testproject before executing the following commands:

The ubuntu image pulled successfully. However, the elasticsearch image pull fails with the following error (since there are a few “high” vulnerabilities identified by the last scan).

That's the last one, we're 3-for-3!

Harbor is an Enterprise-grade Container Registry

Get excited about containers and Kubernetes. Just don't forget the container registry! This feature helps you reduce risk, and protect systems and data from bad actors. PKS – along with Harbor – is an all-in-one solution for your enterprise.

Ready to try Harbor? Download it from Pivotal. Check out the documentation to learn how Harbor can help you secure your overall Kubernetes architecture.