“Rails is slow, but Rails tests are slower.” Rails may be slow, but I’m here tell you that it’s likely you have only yourself to blame for your slow test suite.
I’ve seen some bad test suites in my day. I was once pulled onto a rescue project that had a total build time of over 24 hours. The health and future of your production code depends on a fast, reliable test suite. On my current project, we believe in fast feedback and we’ve focused a lot of detail and attention to our test suite. It’s paid off in dividends.
We’re six months into a project, and we have a one-minute test suite. That includes both rspec and cucumber (and the time to boot up rails for both). Here’s how we’ve accomplished it:
1. Pyramid
Your test suite is a pyramid. Skinny at the top (journey acceptance tests), fat at the bottom (unit tests). Conceptually, this sounds right, and it’s easy to think that it should just naturally fall out of the BDD outside-in process.
But in reality, it’s not a given. Tools like cucumber make it so easy to spin up new high-level acceptance tests for every edge case you can think of. Taken to an extreme, this can lead to unit-testing at the browser level. That doesn’t mean you were wrong to drive out exceptional paths at the acceptance level. But you should ask yourself this question before you check in: “Will I have any less confidance in my test suite if I don’t check in this acceptance test?” If you have already covered a happy path of a feature, and at least one exceptional path of a feature at the acceptance level, do you then need to cover every other exceptional case at the acceptance level? If you’ve added tests down at lower levels in your stack, then you might already have all the verification value you need.
2. Grooming
We groom our test suite. Not every day. But we watch it and keep it in order. We listen to pain in our test suite (“Why did that change cause so many other tests to break? And why did I have to go to every single test and fix each one of them individually?”). When we revisit old tests, we reconsider it in light of what we know about our application. We keep an eye out for duplicate tests.
3. Refactoring
Our fast test suite practically begs us to refactor. And it turns out, refactoring often leads to simpler designs, objects with fewer methods (and therefore fewer tests), as well as objects that are easier and more natural to test in isolation. Great tests will lead you to refactor your production code, which will lead to even better tests.
4. ActiveRecord Containment
How many tests require you to create data in the database? Ideally, that’s limited to just your acceptance tests and your activerecord model tests. But the reality is never that simple. Although it’s technically possible to accomplish this with any Rails application, it’s not always feasible, or even desired. Libraries you use may make this difficult in some circumstances (e.g., devise).
You might also find yourself in a situation where testing an object in isolation would lead to a very brittle test that knows the entire implementation via stubs and mocks. Worse, you may not see a way to untangle these dependencies. That’s OK. Test the object’s behavior, even if it means integrating with other objects. As your understanding of your application’s problem domain crystallizes, and as more patterns begin to emerge, you will eventually find a way to simplify it, to untangle the dependencies.
5. GC.disable
At one point on our project, our build time crept up to nearly two minutes. We saw ourselves slipping down a very slippery slope. We started running it less, refactoring less, and even finding less motivation to keep our test suite clean. So we threw a chore at the top of our backlog to bring the test suite time back down to a minute. Our PM was naturally wary, asking us to timebox the chore to an hour or two, but we made the case that this was of critical importance to the future of the project, and that the pair working on it would check with the rest of the team after a day.
Near the end of the day, the pair stopped and told the team that they thought they had done all they could, but it only got the test suite down to 1:20. Shaving forty seconds off a test suite in a day is a fantastic feat, and when we looked at what they’d done, we were really impressed. I jokingly suggested disabling GC. And then they actually did it. The test suite time dropped to 48 seconds. I’m not recommending you do this. This is a bit of a nuclear option. If this is the first thing you try to improve your test suite time, then you are missing the point of bringing your test suite time down. But if you feel like you’ve done everything you can to legitimately bring down the time of your test suite, then consider disabling garbage collection. Ruby GC is a beast, with the potential of turning a linear algorithm that creates N objects into a quadratic algorithm. Weigh the pros and cons, and decide for yourself if you can live with the dirty dirty feeling this will give you.
