Simple Test Parallelization

Let’s look at a simple approach to parallelizing a test suite for a Ruby app. Parallelizing your specs can be a good strategy to get a speedup on an existing slow suite. It can also be employed early on a greenfield project as part of a commitment to fast tests. The same caveats that Andrew mentions in that article post here as well, namely that parallelization might mask more important design changes you need to make in you suite.

While you could use a gem like parallel_tests, let’s look at what it would take to achieve this without needing to pull in another dependency.

The only requirement to employ this approach is that the parts of your build that you want to parallelize do not share a database, or if they do, that it will not cause test pollution if your specs run at the same time. These parallelizable portions of your build could be Rails engines, a library in lib, an unbuilt gem, or any other isolated piece of your app. If your app doesn’t meet that requirement, something like parallel_tests will likely be more useful.

Let’s assume that you are using engines to organize functionality in your app. In this case you likely are already using a separate database for each engine’s test suite, so let’s use that as a basis for our example. Assuming that you have two engines (engine1 and engine2) and they are both in the engines directory, you could write a rake task that parallelizes your build that looks something like this:

task :build do
  build_pids = []

  %w{engine1 engine2}.each do |engine_name|
    build_pids << fork { exec "cd engines/#{engine_name} && rspec spec" }
  end

  trap(:INT) do
    build_pids.each do |pid|
      begin
        Process.kill(:INT, pid)
      rescue Errno::ESRCH
      end
    end
  end

  Process.waitall.each do |pid, status|
    unless status.success?
      puts "Build failed"
      exit 1
    end
  end

  puts "Build successful"
  exit 0
end

This rake task does three things:

• Uses fork+exec to kick off child processes to run each engine’s build
• Collects the child processes after they have completed and exits non-zero if any of the child build processes were unsuccessful
• Captures INT so that all child build processes will be killed when you Ctrl-C in the terminal

The output can be ugly but may be worth the time savings, especially if you only are going to be running this task as a last check before CI.