kubernetes

Isolated to Conformant: Testing Air-Gapped Kubernetes with Sonobuoy 0.14

Today we are releasing Sonobuoy 0.14.0, which delivers on one of our top roadmap goals: support for running Kubernetes end-to-end tests in air-gapped environments. It is now possible to run the end-to-end suite and validate your cluster’s state without Internet connectivity or investment in a custom, ad hoc work around.

 An open-source diagnostic tool, Sonobuoy runs upstream Kubernetes tests to generate reports that help you understand the state of your cluster, and this release extends the tool’s capabilities to help you analyze air-gapped installations. Sonobuoy is the underlying technology powering the Certified Kubernetes Conformance Program, which was created by the Cloud Native Computing Foundation (CNCF) and is used by every Certified Kubernetes Service Provider.

Air-Gapped Installations and Testing

Running critical systems in air-gapped environments, where the system can’t reach out to the Internet, has long been a common practice to limit the attack surface. Although installing Kubernetes in air-gapped environments has been possible since before Kubernetes 1.6, testing those clusters for conformance was difficult.

The end-to-end suite uses numerous test images, which it tries to pull from public Docker registries; without access to those registries, the cluster is unable to run many of the tests required. Even without an air-gapped environment, restrictive networking policies can make it impossible to effectively run the Kubernetes end-to-end test suite.

Since 2017, multiple people in the community have been working together to make the Kubernetes changes necessary to centralize and then customize the registries used during testing. Since Kubernetes 1.13, testing in air-gapped installations using private registries has technically been possible but was still a pain point for most users. The images were localized in the code, but you still had to manually look them up, get them into your private registry, and inform the tests to use your registry.

Sonobuoy solves all three of these problems for you. By tracking the images required to run the end-to-end test suite and automating all the scripting to move the images, the new version of Sonobuoy makes it easier than ever to test your air-gapped installations.

Getting the Images

The end-to-end (e2e) test suite uses over 70 test images, which can change every release. With the newest release, Sonobuoy keeps track of the images and uses the correct image list based on your Kubernetes version.

To view the list of images run:

$ sonobuoy images  

Getting the Images into Your Private Registry

 Even if you had a list of all the images for your version of Kubernetes, you would have to write custom scripts to run all the necessary Docker commands to pull, tag, and push the images to your registry. With Sonobuoy 0.14, all you have to do is this:

$ sonobuoy images pull

$ sonobuoy images push --e2e-repo-config repos.yaml

Voila. Sonobuoy looks up the version of your Kubernetes cluster and, based upon that version, identifies the images you need and takes care of running all the necessary Docker commands for you. Using the --e2e-repo-config flag, Sonobuoy can properly tag and push all of the images to your private registry. It can even clean up your local client with sonobuoy images delete.

The repos.yaml file you provide will be what the Kubernetes tests are looking for. The file contains a list of named registries (see an example).

If you need to physically move the images to your private network rather than pulling and pushing from a single host, Sonobuoy has you covered as well. Just download the images to a tar file:

$ sonobuoy images download

Then transfer them to your air-gapped environment and run the following commands:

$ docker load --input images.tar.gz

$ sonobuoy images push --e2e-repo-config repos.yaml  

Testing with Private Registries

The last thing you need to run conformance in an air-gapped environment are the conformance and Sonobuoy images themselves. We’ve documented those steps for you. For the next step, we’ll assume you placed those images into a reachable registry named privateRegistry.

Now that you’ve got all the images loaded, running the end-to-end tests with private registries requires just one extra command-line flag:

$ sonobuoy run --e2e-repo-config repos.yaml

--sonobuoy-image privateRegistry/sonobuoy:v0.14.0

--kube-conformance-image privateRegistry/conformance:v1.14.0

Now anyone with a Kubernetes cluster can verify that it is conformant with just a few commands, saving hours of engineering time.

Go forth and test!

P.S. If you’ve got Sonobuoy questions and will be at KubeCon + CloudNativeCon Europe 2019 in Barcelona—let's talk in person. Come find us at the VMware booth (D2) to talk all things Sonobuoy. 

A big thank you goes out to the Sonobuoy community for your continuous feedback and contributions for this release — special thanks to ekj1711 for his contribution. Join the Sonobuoy community:

Related Articles