Continuous integration testing is imperative for software engineering projects. If you are working with an open source project specifically, there are a handful of testing frameworks available for free. Some are better than others and each continuous integration tool has their benefits based on your project’s individual testing and continuous deployment needs.
As the co-maintainer of a small open source project named Tern, I am no stranger to the importance of a sophisticated and functional continuous integration (CI). For the past 9 months, Tern has used CircleCI open source to manage our CI testing for all pull requests. These tests cover functional testing as well as commit message and code linting. Since Tern is a relatively small project and runs a series of scripts and python tests for our CI, our requirements may vary from more resource-intensive projects.
GitHub Actions (GHA) have been generally available since November and are free to use for open source projects. CircleCI has existed since 2011 and also offers a free plan for open source projects. I have not explored the price comparisons in depth between the two offerings.
Prior to exploring what GHA has to offer, Tern used CircleCI to run all of our CI tests for pull requests. We liked that the CI tests ran in parallel, setup was fairly simple, and it was clear to contributors the tests that passed or failed for their changes after their PR was opened. Once GHA became available, I spent time exploring what it would look like to convert Tern’s current CircleCI open source dashboard and testing to GHA. When considering which CI tool to use for testing, we primarily considered the following traits: ability to run all our tests, ease of understanding of results to the contributor who opened the PR and ease of integration/deployment/maintenance.
Both CircleCI and GHA actions utilize the concepts of workflows, jobs, and steps. A step is a defined set of one or more tasks that will be run in the same virtual environment. Jobs are a collections of steps, and a workflow is configured to orchestrate a series of jobs. The first thing that jumped out at me when trying to convert our CircleCI YAML file to work for GHA was the lack of any ability to customize initialization routines for similar steps within a job. CircleCI offers a concept of commands that allows users to define a custom sequence of steps that may be reused across multiple jobs. For example, the following CircleCI command (named setup) is a required set of steps that each job needs to execute.
commands:
setup:
steps:
– run: sudo apt-get install -y attr
– checkout
– run: pyenv global 3.6.5
– run: pip install –upgrade pip
Having the setup command defined then makes it easy to call these initialization routines for each job without re-writing the steps under each job in the YAML file:
test_changes:
executor: ubuntu1604
# Steps to run tests on files changed
steps:
– setup
While this seems trivial at surface level, the ability to utilize commands with CircleCI made the YAML file cleaner and enabled the run of jobs in parallel. It also factors into the way the jobs are represented on a pull request in the GitHub UI. As shown below, you can view all of the jobs individually.
I did not find anything equivalent to commands in GitHub Actions, which meant that if I didn’t want to write out the same list of initialization steps under each job (~8 lines of initialization steps), I had to define all of our tests under one job where each test gets to run as a separate step in the same job. This meant that all my jobs ran serially instead of in parallel and therefore, the output was harder to interpret. (Disclaimer: while I realize I *could* have separated each test into its own job and re-written the initialization steps for each job, it felt like a lot of unnecessary redundancy in the YAML file.)
Writing all the tests under one job in the GHA YAML file also meant that if one of the tests failed, the UI showed the whole job having failed. Granted, this makes sense if your jobs are only accomplishing one task. In my scenario, however, it required extra digging on behalf of the contributor who opened the PR to figure out which test failed.
They have to click on the Details link from the main PR page and navigate to the GitHub Actions page to see specifics. Clicking on the Details link brings you to a page where you can see the specific step/test that failed.
The integration of both of these test frameworks to Tern’s GitHub project was very easy to set up. GHA uses a config file under .github/workflows/name_of_your_file.yaml and if you’re a CircleCI user, you create your YAML file under .circleci/config.yaml. A benefit for GitHub Actions is that you can have any number of YAML files under the workflows directory, whereas in the CircleCI workflow they all have to go in the same file by default unless you want to integrate other tools to help with this.
Both CircleCI and GitHub Actions testing frameworks fall a little short on documentation. CircleCI has more community blogs written to help. GitHub has one large page of documentation for actions that leaves a little bit of guesswork on the user side. I imagine that the longer GitHub Actions are around, the more relevant material there will be answering basic questions. I will also highlight that the GitHub Community Forum was very quick responding to specific questions I had about actions.
The last major difference I noticed between the two CI tests in my brief experimentation is the option to build your own self-hosted runner with GitHub Actions. This may be a big plus if your tests are resource-intensive and you want to use your own testing infrastructure. Comparatively, CircleCI offers Docker image, Linux Virtual Machines, macOS VM images, or windows VM images as their executors.
In closing, I believe that both tools contain the necessary features to make testing pull requests easy, but it will depend on your specific requirements to decide which to use. As it stands now with Tern, we still have both GHA and CircleCI enabled while we decide which to stick with permanently.
Tern is a container inspection tool written in Python code that aims to help users better understand the contents of what’s inside their container. If you’re interested in looking at our CI files, they are available in our GitHub repository. Likewise, if you’re interested in using or contributing to Tern, check out our GitHub account.
To learn more about GitHub Actions and CircleCI configuration and stay up-to-date on new open source projects and developments, explore our blog and follow us on Twitter (@vmwopensource).
