Uncategorized

Fast and Easy Sonobuoy Plugins for Custom Testing of Almost Anything

Today we are releasing Sonobuoy 0.14.2, which delivers on one of our top roadmap goals: increased support for custom plugins, allowing developers and operators to extend Sonobuoy with additional tests and data gathering. It is now possible to easily create a plugin from a Docker image and run it within Sonobuoy without manually editing any YAML files.

Sonobuoy, an open source diagnostic tool, runs upstream Kubernetes tests to generate reports that help you understand the state of your cluster. Sonobuoy is the underlying technology powering the Certified Kubernetes Conformance Program, which was created by the Cloud Native Computing Foundation (CNCF) and is used by every Certified Kubernetes Service Provider.

The enhanced support for custom plugins means that you can easily write and integrate any logic into the workflow that you need; for example:

  • Ensuring your cluster fulfills your organization’s security requirements
  • Testing if your cluster complies with industry standards
  • Checking availability and connectivity of private, custom resources
  • Performing in-cluster benchmarking

The Plugin Model

A Kubernetes cluster is a large and complex entity, and the number of things you may want to investigate are endless. This is why we didn’t bother trying to hardcode a certain set of diagnostics into Sonobuoy, but instead built an architecture that could leverage custom plugins.

A plugin is really just a container along with some metadata about how to run it in the cluster. The plugin metadata tells Sonobuoy whether to run the plugin as a standalone pod (a Job type plugin) or whether to run the plugin on every node (a DaemonSet type plugin). When running in the cluster, Sonobuoy will load each of the plugin configurations from a configmap, create them, and then monitor them for results.

Sonobuoy then aggregates all the results and makes them available to the client. The plugin itself isn’t responsible for any of the reporting; instead, a Sonobuoy sidecar is deployed alongside each plugin to securely return the results to the aggregator.

The only thing the plugin is responsible for doing is creating a results tarball and telling the Sonobuoy worker where to find it; Sonobuoy handles the rest.

Plugin Selection

Sonobuoy has two integrated plugins. They are run by default each time you initiate sonobuoy run:

  • The Kubernetes test suite
  • A systemd log gatherer that runs on every node

In previous releases, if you wanted to run only a single plugin, you had to go through multiple steps:

  • Save the YAML output of sonobuoy gen to a file
  • Edit the file to configure the aggregator to ignore the plugin
  • Run kubectl apply

With this release, you can specify on the command line which plugins to dynamically load by using a new --plugin flag.

For example, to run just the Kubernetes test suite and skip the systemd logs gathering plugin, run the command:

sonobuoy run --plugin e2e

Likewise, if you wanted to run just the logs gatherer, run the command:

sonobuoy run --plugin systemd-logs

These examples are simple but the flexibility can be extremely useful. For example, if you are trying to run Kubernetes tests on a non-systemd system, the older, YAML-based flow prevented you from using Sonobuoy’s other convenient command-line options, such as setting which tests to run or running the plugin synchronously.

Creating and Running a Custom Plugin

Previously, if you wanted to run a custom plugin, you had to go through an even more manual process:

  • Write your own plugin definition from scratch
  • Save the YAML output of sonobuoy gen to a file
  • Add your plugin definition into a specific configmap within the file
  • Edit the file to configure the aggregator to run the plugin
  • Run kubectl apply

In addition, the resulting YAML file you saved was the whole YAML for the run, not just the plugin you configured. This meant that the file already had multiple options hardcoded in it and that the file itself was large enough to confuse any newcomer. Both of those issues made it even harder to reuse the plugins that you had created.

To simplify this process, Sonobuoy now includes a new command: sonobuoy gen plugin. This command generates the plugin definition file, and then you can use the new, dynamic plugin selection mentioned above to run it.

Let’s make our own custom plugin and run it to show how these commands work together. I’ve created a Docker image that executes the commands given to it through arguments and saves the results for the Sonobuoy worker. For our first attempt, let’s just make it echo “hello world.” First, generate the plugin and save it to a file:

sonobuoy gen plugin 
--name=hello-world 
--image schnake/easy-sonobuoy-cmds:v0.1 
--arg="echo hello world" > hello-world.yaml

Then run the plugin just like we did with the e2e or systemd-logs plugin, except this time specify the filename:

sonobuoy run --plugin hello-world.yaml --wait

(Note: The --wait flag runs the command synchronously.)

Retrieve the results, extract them locally, and print them:

outfile=$(sonobuoy retrieve) && 
  mkdir results && tar -xf $outfile -C results &&
  cat results/plugins/hello-world/results/out*

Clearly, printing “hello world” isn’t a particularly useful plugin, but look at how easy it is to use the plugin for other information gathering:

sonobuoy gen plugin 
--name=debug 
--image schnake/easy-sonobuoy-cmds:v0.1 
--arg="curl -s -o /dev/null -w "%{http_code}" http://www.example.org/" 
--arg="kubectl cluster-info" 
--arg="ifconfig" 
--arg="uname -a" 
--type=daemonset > my-debug.yaml

Now this plugin will be running on every node in our cluster (since the type was set as daemonset) and will be checking:

  • Whether it can use cURL to connect to a web page from the node
  • Whether it can reach the API server from the node
  • The network settings in the container
  • The operating system characteristics

Although the plugin is run in the same way as before, it is worth pointing out that you can specify as many plugins as you want, so let’s also run our other plugins too:

sonobuoy run --plugin my-debug.yaml --plugin hello-world.yaml --plugin systemd-logs --wait

Then output the results in the same way:

outfile=$(sonobuoy retrieve) && 
  mkdir results && tar -xf $outfile -C results &&
  cat results/plugins/debug/results/*/out*

And remember, this is just one plugin made from a single small bash script. Sonobuoy plugins can help you answer the questions you find yourself asking every time you set up a new cluster, have failing conformance tests, or try to diagnose cluster issues. Furthermore, integrating a test or script into a Sonobuoy plugin can help centralize your testing and reporting.

So what questions can Sonobuoy help you answer?

P.S. If you’ve got Sonobuoy questions and will be at KubeCon + CloudNativeCon Europe 2019 in Barcelona—let’s talk in person. Come find us at the VMware booth (D2) to talk all things Sonobuoy. A big thank you goes out to the Sonobuoy community for your continuous feedback and contributions for this release — special thanks to krisdock for his contribution.

Join the Sonobuoy community:

Get updates on Twitter (@projectsonobuoy)

Chat with us on Slack (#sonobuoy on Kubernetes) J

oin the K8s-conformance working group: github.com/cncf/k8s-conformance

Related Articles