Minitest, Ruby’s built-in testing library, has some great out-of-the-box features. One of these is test parallelization. Parallel testing is often added after a suite gets slow enough to hurt. That can be achieved using the parallel_tests gem, which takes advantage of today’s multi-core processors, or using custom solutions for dividing chunks of a suite across several machines. Arguably, test speed should be dealt with by making code design changes, but that’s another story: what interests me most about minitest’s parallelization is the constraints it places upon the design of stateful systems when TDDing from scratch.
You can turn on parallelization for a particular test case:
describe Server do
parallelize_me!
end
or for all tests:
require 'minitest/hell'
As the name implies, the latter approach turns up the test pain level to 11, but it’s the kind of pain that can have positive effects. For ‘fun’, I started to use minitest’s parallelization on a side project, which has a stateful API backed by a relational database. Here are some of the decisions that were forced out by using parallel test examples.
Commitment to fast tests
I thought that running tests in parallel from the start of a project would make me lazy, causing me to neglect slow tests because they’d be running at the same time as others. Surprisingly, the opposite happened: the need to constantly rerun the whole suite to iron out nondeterministic conflicts encouraged me to fix slow tests early. I ended up being able to run the entire suite several times within a matter of seconds in order to check the tests’ ability to run in parallel.
Side note: this is an early-stage project, with a very low quantity of tests! It will be interesting to see how test speed increases as the volume of tests increases.
Avoiding test duplication
Since the unique constraints of my database could be hit by tests with the same fixture data running at the same time, I was encouraged to use more intention revealing test data for each example, avoiding foo and bar, which commonly litter a suite and make tests harder to read.
For IDs, I used Ruby’s SecureRandom library, which provides GUID and hex generation. I sometimes used hex generation when the user-supplied unique display name of something didn’t matter to the test.
Client-side ID generation
Although not strictly forced out from parallel tests, parallel testing got me thinking about how best to interact with the backend, which has a single database being served by multiple concurrent requests (just like any web server).
Using GUIDs instead of autoincrementing IDs can be a smart decision to make if you can (i.e. you don’t need human-friendly URLs), because it means your database server doesn’t need to worry as much about ensuring uniqueness, since the GUID algorithm effectively guarantees it.
TDDing my API from scratch, without external requirements, encouraged me to use GUIDs to simplify the design and to avoid bottlenecks at the database layer. POST requests canonically return the new URL of the resource you’re creating in the Location header of the response. So to test that a thing really got persisted I’d need to:
- POST to /items with a representation of the resource
- Grab the Location header of the response to get the new URL
- GET /items/:newid and ensure the response body matched the representation I sent
This seemed a very laborious process for storing some data. Much less work is:
- PUT to /items/:newid
- GET /items/:newid and ensure the response body matched the representation I sent
Since GUIDs can be treated as unique, it didn’t make much sense for the server to generate them.
Positive effect: the app would now cope with a distributed database system on the back-end, despite starting out on a technology that’s thought of as difficult to scale horizontally (SQLite).
Avoiding database resets
It’s common practice to wipe the whole database when starting a new example, or to run each example in a transaction and roll it back when each example finishes. I wanted real black-box tests, so I didn’t want to use transactions. Yet, deleting the whole database at the start of an example didn’t play nice with minitest’s parallelization, since data that one example required would be deleted by another.
The usual approach when using parallel_tests is to create a database for each process. However, since minitest doesn’t manage databases (nor should it) I chose to keep a single database and find a different solution.
I chose to sandbox all of the tests by creating new entities each time, and only checking for output that indicated that particular entity had been worked on. The product I’m working on is a Continuous Integration server, so I’d be creating CI projects (you might also know them as jobs) and expecting them to appear in an XML feed. The tests had to be OK with other data being present, since the other tests could be working too.
This approach precluded tests that checked that the number of records had increased by one, because in an otherwise acceptable “green” test situation they’d occasionally increase by more than one (another test added a record too), stay the same (another test deleted a record) or decrease (more than one had been deleted).
Constraints are fun
While I wouldn’t recommend going rogue like this on a client project, playing with constraints like truly parallel tests can get you thinking about your normal testing procedure. Some of the above decisions allowed for a much faster test execution time, and always having the assumption that other processes could be working on the database forced out some interesting techniques. Some of the techniques I had to avoid due to parallelization would normally necessitate different workarounds with their own drawbacks. For example, if you always assume the count of an ActiveRecord class will go up, you require exclusive use of the database. If instead you scope your queries to a parent entity, this restriction would be removed.
Hell isn’t so bad after all.
