People sometimes have the impression that continuous, upstream test-driven development is hard—and that can make them feel better about not doing it. But I want to make the case that it’s a lot easier than you think.
First, it’s worth pointing out that testing really is important. Say I write some code that I’m proud of and I want to upstream it right away. I can easily put together a new project on GitHub and add all the code to it. But how do I know that it actually does what it says it does? And given that, how can I expect that anybody will really want to use what I’ve written?
It should be obvious, but it’s worth restating: you can only be confident of code once you have tested it. Yes, it’s always worth testing your code locally, so it’s important to be able to mock out your calls. But that’s not enough—you also need to be continuously testing upstream if you want to ensure contributions are well-trusted.
Luckily, we now have quite a few continuous integration options that can help with this. Most fit one kind of niche or another. One might focus on mobile, another for Windows or maybe even Linux. Not all of the options are free—although typically they are for public open source projects. But I want to highlight two that are free, serve a relatively broad user base and are fairly easy to use: Travis CI and CircleCI. (Travis CI also happens to be the default CI installed when we upstream projects at VMware).
Let’s look at Travis CI first. It supports Linux, Mac OS and Windows, which is better than most CI systems (although its support for Windows and Mac is newer and therefore not as comprehensive as Linux). It works for GitHub in the form of the newer TravisCI.com. Its configuration is just a simple YAML file, has Checks support (more on that shortly) and its build environment uses virtual machines. That last feature can cause it to be a little slow to startup, but it does have more support for languages and operating systems than competing tools.
To install Travis CI into your project, just give permission for the Travis CI application and it’ll automatically show up in your admin and settings as the Travis CI GitHub App. You also need to implement the .travis.yml file, but setup is essentially super easy.
Then, when you create a pull request of code changes, Travis CI automatically creates a build job according to your configuration. Along with the build status is a “Checks” tab that offers you more information about the build environment. This saves you from having to navigate to the Travis CI site, but if you do get a build failure, you can easily click on a link to the site to see full build log and identify the source your problem.
It’s so easy, you really don’t have an excuse for not using it. Of course, depending on what you are trying to do, things can quickly get complex. To help with this, Travis CI lets you add pre-build install steps or run scripts as multiple commands if there are additional packages you need in place prior to running the scripts. You can also get notifications of the build results and select automatic deployment when Travis CI determines that packages are properly built.
One last neat feature of Travis CI is that if you claim your build supports multiple versions of a language, it will test on all those versions. You set that up through its build matrix function and it will run all the tests in parallel.
Now to CircleCI. I’ve seen this used successfully in quite a few projects, especially on ones that relate to containers. It supports Linux and the Mac OS (but for paid accounts only) and covers a decent range of languages—not quite as many as Travis CI, but all the most common ones.
Again, it works with GitHub, has a YAML configuration and does Checks. It also uses containers for its build environment, which helps it run a lot faster than Travis CI. It can run a test in seconds that takes minutes on Travis CI.
Setting it up is not quite as easy, though. You have to sign up on the CircleCI website with your GitHub user account (they do have some non-GitHub support) and give it permission to access your project. That’s all well and good but if you want GitHub Checks support, you have to jump through a series of hoops that aren’t always easy to follow.
Once it’s set up, CircleCI shows up in your repository in the GitHub Apps area, just like Travis CI. The CircleCI config file is much wordier than what you get with Travis CI. One nice addition, however, is that it allows you to put whatever Docker image you want into your build environment. At the same time, it requires you to repeatedly define things that you do all the time, like how you will share results or where you will store your build artifacts that it would be nice to be able to just copy and paste in.
The Checks tab on CircleCI offers a little more detail about the build job than Travis CI offers. And again, you can easily navigate through to the CircleCI website, where you get a pass/fail summary and the ability to drill down and find the build log.
Fundamentally, though, while Travis CI and CircleCI do have their differences, they are both very easy to utilize. Both can help you assure quality not only to yourself but to others. That’s essential if you want others to trust and therefore use your work.
Stay tuned to the Open Source Blog and follow us on Twitter (@vmwopensource) for more deep dives into open source software and projects.
