Speed Up Rails Tests
There are a lot of suggestions as to how to speed up a test suite. But many of them don’t make much of a difference. The following are the changes that I found most useful when attempting to speed up the test suite for a legacy Rails app, along with the amount of time each saved. In brief, they were:
- Record/Stub All API Calls (saved ~10 min)
- Decrease Ruby’s Garbage Collection Frequency (saved ~8 min)
- Clean the DB With Transactions (saved ~5 min)
- Profile the Test Suite After Each Run (saved ~4 min)
- Block requests to external URLs (saved ~4 min)
As a caveat, these changes worked for me because of the particular details of the app I was working on. Obviously, every app is different. So, I figured it might be useful to give a little more detail about the app in question.
The directory structure:
support folder will come up a bunch later.
Record/Stub All API Calls (saved ~10 min)
The test suite was making live API calls. A lot of them. Often to very slow API’s. *cough* NetSuite *cough*. There were actually some pretty good reasons why this had been done. But I decided it should stop.
To do this, I made the following additions. First, to the
Then, to my
Finally, to a new
vcr.rb support file:
Once I did this, I could add a
vcr: true tag to any examples that were
making live API calls and have them recorded by VCR. The next time the
calls were made, the recordings would automatically be played back. I then
could commit the recordings and push the whole thing to CI.
This was far and away the single most important change. Not only did it save more time than any other improvement, but it was also key in improving the reliability of the test suite, and ensuring that we got consistent results across the team. This made the test suite something we could actually trust.
Decrease Ruby’s Garbage Collection Frequency (saved ~8 min)
This change was especially helpful because:
- The app was running ruby 1.9.3
- The app had a rather involved CI build. It had extensive seeding and
configuration scripts. As well as separate databases and test suites
for each of its 4 engines. This meant, at the very least, that
bundle execwas called no less than 8 times per CI build. (Once to
bundle install, and once to
bundle exec rspec, for each engine.)
Reducing the CI build complexity helped with some of the problems I was facing. But even without doing that, speeding up Ruby by modifying its GC (garbage collection) settings gave the app a huge boost. This turned out to be very simple to do. Adding the following variables to our CI container environment shaved 8 minutes off the build time:
This made ruby eat up much more memory, but dramatically increased speed. Fine by us, since this was just on CI! But what about development? Who wants ruby killing their local machine’s RAM just to run a test suite?
Obviously, this wasn’t a long-term fix. The change that fixed ruby’s GC permanently was an upgrade to ruby 2.1.6 – which is dramatically better about garbage collection than 1.9.3. In all, then, configuring ruby’s GC probably made a huge difference to the build time on CI, and was the second most important change overall.
Clean the DB With Transactions (saved ~5 min)
A good deal of time is spent deleting persisted data between examples in Rspec. Most databases support three different cleanup strategies:
Among these, transactions are considerably faster because they make minimal changes. They simply reverse the changes made by your tests, nothing more. Truncation and deletion, by contrast, delete whole tables, or portions of them. Transactions are also preferable because they are exact. They only reverse what was changed by the tests. This makes them safe to use when you are seeding your test database. You don’t have to worry about accidentally deleting important data for the tests with them. Nor do you have to worry about reseeding between examples. And, since the application had some pretty substantial seed data to it, this was significant.
There is a problem with truncation, however: it doesn’t work when you have multiple database connections. Each database connection has to have its own transaction. Why does this matter? Because Capybara (which we use for our integration tests) starts up a Rails server in a new thread so that the browser that it runs can access it. And Active Record spawns a new connection to the database for each thread.
The trick here was to force Capybara to share the same database connection as the app. There is a well-known hack to do just this. Here’s how.
I added the following to the test group in the Gemfile:
Then, I created a new support file for database_cleaner:
Finally, I created a support file for the shared connection code:
Making this change shaved off close to 5 minutes off the runtime.
Profile the Test Suite After Each Run (saved ~4 min)
This was suggested here. It was very easy to do. I simply added the following to the app’s global rspec config file:
This gave me information about the slowest examples and groups at the end of every successful spec run. For example:
On its own, this obviously didn’t make a difference to the run time. But what it did do is make it super clear what specs were slowing down the test suite the most.
This was important because it made me really think twice about whether some of these legacy tests were worth keeping. As it happened, some of them were completely useless. Some were testing functionality that was already tested elsewhere. Others were clicking through the app without making any concrete expectations at all. And others should have just been unit tests in the first place. Away they went.
This allowed me to simply add a
selenium: true tag to the slow
test examples and watch those examples run in a browser.
When I did this, it immediately became clear why they were so slow. Capybara was taking upwards of 40 seconds to fill in a single select field!
The issue seemed to be caused by formulaic, which sells itself as a convenience DSL (domain specific language) for filling in forms with Capybara. Formulaic allows you write things like the following in your tests:
Really neat and clean. Unfortunately, it was taking Capybara forever to fill in “Pennsylvania” in the select fields. Once I realized that this was the problem, it was fairly easy to fix. Just fill in the select fields using Capybara’s form DSL manually, like so:
Making these changes saved 4 minutes on the test suite runtime.
I highly recommend profiling your test suite. It’s super easy, and makes it really easy to see trends that might be slowing you down. Because of it, the current test profile looks like this:
That’s a big improvement!
Block requests to external URLs (saved ~4 min)
Thoughtbot had a blog post where they talked about a slow test suite that was caused by calls to external URLs in feature tests.
Capybara webkit is pretty good about warning you about these kinds of requests when they are made. And there were a LOT of these warnings when the app’s test suite would run:
I thought that I had blocked all of these URLs with the following code, as per the capybara webkit readme:
The persistence of these warnings, however, and the speed issues I was still having, lead me to believe that I actually hadn’t succeeded in blocking these requests after all. To try to figure out what was going wrong, I dug into the pull request that introduced the global configuration I was attempting to use.
The key to figuring out the issue was in this line in the pull request:
Pay attention to the
Capybara::Webkit::Configuration.to_hash call. That is
what pulls all of the URLs out of the configuration block above. Notice that
it has to be manually added as a second parameter
when initializing a new driver. This was significant because the app I was working
on was using a custom webkit driver, and wasn’t manually adding in the
Having spotted the problem, it was easy to fix. Just add the configuration in!
This shaved an additional 4 minutes off of the test run time, and got rid of a lot of annoying warning messages in the process.
So, those are the changes that made the biggest difference to the app I was working on. They were all changes suggested elsewhere by other people. But hopefully others can benefit from either the implementation detailed here, or the relative importance suggested by the amount of time they saved me.
These are some links that were helpful to me when I was going through this process myself: