Fast test suite boot times with Ruby on Rails

Developers need to be able to run tests quickly or they will stop running them.

The biggest bane of test driven development, or whatever variant you practice, is long boot times. Even when you just run one test a slow boot will make it a tedious job. There are a number of ways to reduce startup times in a Ruby on Rails project.

Load less dependencies

Project dependencies need to be loaded every time you start your test suite, less dependencies means faster startup. Keeping project dependencies to a minimum is always a good idea, not just because of boot time.

Let’s jump into an example. We’re assuming you’re using the following, simplified, Gemfile for your dependencies:

gem "rails"
gem "bcrypt"
gem "foreman"
gem "thin"
gem "newrelic_rpm"
gem "airbrake"

First we stop loading gems which aren’t used in the code.

gem "rails"
gem "bcrypt"
gem "foreman", require: false
gem "thin", require: false
gem "newrelic_rpm"
gem "airbrake"

Then we make sure dependencies are only specified for the environments where they’re used in.

gem "rails"
gem "bcrypt"

group :development do
  gem "foreman", require: false
end

group :production do
  gem "thin", require: false
  gem "newrelic_rpm"
  gem "airbrake"
end

At this point you might start getting undefined constant errors about Airbrake. You can decide to either move Airbrake back into all environments or stub its implementation.

We generally do this by defining the module or class ourselves from a support file in the test directory, like so:

module Airbrake
  extend self
  def notify(exception, opts = {})
  end
end

On the one hand you probably never want to send data from your test suite to Airbrake. On the other hand some people are more comfortable running their entire production stack in at least one or two tests.

Reduce time spent setting up tests

Basically you need to do less. There are a few strategies to accomplish this.

One strategy is moving setup code as local to your test as possible so it only runs for the tests where it’s needed. For example, don’t run setup methods specific for functional tests in unit test.

Keep in mind that your test_helper.rb or spec_helper.rb is loaded during every boot. Try to limit slow operations in this file.

Tests that require long setup tasks sometimes need to be run in a variety of contexts: library tests, model tests, and functional tests. Sometimes you can cheat a little bit and test through the entire stack in one massive integration test.

Another strategy is to perform long setup tasks only once and find a way to cache the result to speed up next runs.

In one of our applications we process YAML files with questionnaire definitions and create records for them in the database. In hindsight we should have never done this, but that’s a discussion for another day.

In unit and functional tests we use fixtures to test specific situations. For regression tests we need the production definitions. We speed up loading these by dumping the database tables with INTO OUTFILE and load them with LOAD DATA INFILE. We also hook into Rails fixture loading code so it only happens once for the entire suite.

Finally, never, ever, ever, ever, call external services from your tests. Not only is this slow, but it might accidentally erase production data. In most tests we override Net::HTTP#start to make sure it’s never called by accident.

Factories or fixtures

There is a lot to say about database setup, so it deserves its own little chapter.

Let’s get the elephant out of the room first. Factories are slow! They’re slow by design.

Factories can be used to create database records anywhere in your test suite. This makes them pretty flexible and allows you to keep your test data local to your tests. The drawback is that it makes them almost impossible to speed up in any significant way.

We can understand why factories are slow by looking at why ActiveRecord fixtures are fast.

Fixtures are bulk loaded once. A transaction is created in the database when a test starts. After the test the database is rolled back to its initial state. This means the fixtures don’t have to be reloaded for each test.

Most factory implementations start a test with an empty database and run through all the factories needed for a test each time it’s run. After the test the database is reset to its blank state by removing all records. On top of that records are created through ActiveRecord model instances, which creates a lot of objects in the Ruby VM, which need to be garbage collected.

We try to use fixtures as much as possible. We write little scenarios with them to make tests easier to understand. Paul might be a pro account holder who has paid his bills the last two years, but unfortunately missed a payment this month because his credit card expired.

Using small scenarios like this reduce the chance of someone messing up tests by using the wrong fixture or changing fixtures to not fit their intended scenario. You can even document these scenarios shortly in the fixture files or test helper.

Use a fast test framework

Your test framework needs to be loaded and initialized during every boot.

RSpec is notoriously slow. If you’re willing to part with it, you might want to consider minitest/spec. You can use minitest/spec with Rails out of the box since version 4.0. Another alternative is Peck on Rails.

In doubt, measure!

Optimizations don’t really make sense when you can’t measure the speedups. Most test frameworks print their runtimes, but this doesn’t include boot time. The time tool is your friend.

$ time ruby test/unit/post_test.rb
Loaded suite test/unit/post_test
Started
..
Finished in 0.001313 seconds.

2 tests, 6 assertions, 0 failures, 0 errors
ruby test/unit/post_test.rb  0.15s user 0.08s system 96% cpu 0.238 total

In this example test/unit reports 0.001313 seconds, but the real runtime is 0.238.

To measure particular parts of your code you can either use a profiler or the Benchmark class.

Isolate the slow code and keep the benchmark in place to measure if your fix actually helps.