How I reduced the runtime of an RSpec test suite by 15%

Some time ago I worked on a project whose goal was to improve the speed of automated tests in a Ruby on Rails application. In this post, I'd like to share my learnings as well as some tips that might help you increase the speed of your project's test suite as well. In my case, the runtime of the test suite decreased by 15%.

Note: all code examples in this article are related to Ruby on Rails, RSpec, and FactoryBot, but some of the general principles can be carried over to other testing frameworks and programming languages.

Step 1: Find the slowest tests in your test suite

I started out by making a list of the slowest tests in our suite and analyzing each of them. To find the top 10 slowest tests in your suite, run the following command from your project's root (replace 10 with the number of slow tests you wish to get):

rspec --profile 10

Step 2: Analyze each of the slow tests

With each test that appears slow relative to other tests, it's important to determine whether the test is slow for legitimate reasons or it is slow because the underlying code is buggy or inefficient. What follows is the rubric I used to analyze the slow tests.

The test slow, but it's acceptable if:

• It doesn’t make unnecessary calls to external services, i.e. all necessary external requests are stubbed.

• All objects that are not under test have been replaced with mocks or stubs.

• It writes objects into the database, but this is necessary for the tests to work, i.e. :create can’t be replaced with :build_stubbed or :build.

• Factories used for creating objects don’t create unnecessary Active Record associations.

• It doesn’t test private methods.

• It doesn’t test logic that belongs to other classes.

• It doesn’t test logic that has already been tested by someone else (an external library or a gem).

• If this is a request/integration spec, it doesn’t test edge cases or cases that have already been tested on the unit level.

• More broadly, the test tests behavior, not implementation.

The slow tests I analyzed faired well under all of the above-mentioned points, except for the test data generation rubric where it was hard to assess the test without further investigation. So my hypothesis was that some of the tests were so slow because they created too much test data that wasn't necessary for the test to work properly. In general, unnecessary test data creation is one of the most common reasons why a test suite gets slow: writing to a database is one of the slowest operations a test can perform (this, and calling external APIs).

Step 3: Check how many test objects are created during a test run

✅ Add this snippet to spec_helper.rb:

# spec/spec_helper.rb

config.before(:each, :monitor_database_record_creation) do |example|
  ActiveSupport::Notifications.subscribe("factory_girl.run_factory") do |name, start, finish, id, payload|
    $stderr.puts "FactoryGirl: #{payload[:strategy]}(:#{payload[:name]})"
  end
end

✅ Add a meta tag :monitor_database_record_creation to the test example or test group that you suspect of creating too many objects:

describe '#recipe_complete?' do
  it 'returns true if a recipe is complete', :monitor_database_record_creation do
    # test body
  end
end

✅ Run the test.
The console output will tell you how many objects were created for this particular test along with which strategy was used to create them:

FactoryGirl: create(:recipe)                                                                                                                                                                                        FactoryGirl: create(:step)
FactoryGirl: create(:step)
FactoryGirl: create(:ingredient)
FactoryGirl: create(:step)

❓ At this stage, you might wonder why so many objects are created for this test example or, more specifically, should the step object be created twice for this test example to work. Often such duplicate objects happen to be nothing but "mystery guests", i.e. unnecessary—and hard to spot—objects that were written to the database but weren't used by the test.

In many cases, you should be able to refactor your test and get rid of these "guests" that slow down your test suit.

---

Next, I'll go over the main culprits of unnecessary test data creation and describe how you can deal with them.

Culprit #1: Using :create where :build would do the job

It's likely that your specs predominantly rely on :create strategy. In many of these specs, you might be able to safely replace :create with :build. These are the cases where the test doesn’t assume the object has actually been written to the database, which is often true for model tests.

In my case, I replaced :create with :build in the slowest tests whenever it made sense, and also grepped through several of the unit tests for the most often used models in the application. Such models tend to become responsibility magnets and accumulate many methods, and, consequently, many tests.

Word of caution: some blog posts recommend doing a global find and replace throughout your whole project and replacing all instances of :create with :build. I don't recommend doing that: you are likely to end up with numerous failing tests 🤯 Fix the specs one-by-one. It will take longer, but you will be confident of the end result.

Culprit #2: Relying on default :create strategy for creating associated objects in factories

You are likely to have several factories where you provide associations as well, and this is where things might get tricky.

By default, even if you call the parent factory with :build, the subordinate factory will still be called with :create. This means that in your tests, you will always write associated objects to the database even when it isn't necessary for the test.

factory :step do
  association :recipe
end

FactoryBot.build(:step)

(0.1ms) begin transation
Recipe Create (0.5ms) INSERT INTO "recipes" DEFAULT VALUES
(0.6ms) commit transaction

To avoid this, you can explicitly use the :build strategy in the factories whenever possible.

factory :step do
  association :recipe, strategy: :build
end

Culprit #3: Not providing associated objects in factory callbacks explicitly

Say we have a factory trait for creating a recipe that has two steps:

 ruby 
# spec/factories/recipe_factory.rb
FactoryBot.define do
  factory :recipe do
    # more code
    trait(:with_two_steps) do
      after(:create) do |record|
        record.steps << create_pair(:step,
          account_id: record.account_id,
          body: Faker::Food.description
        )
      end
    end
  end
end

The issue with with_two_steps trait is that after(:create) callback doesn't explicitly specify the recipe for the step object that is being created. When step factory is called from this callback, another recipe object is always created, unbeknown to anyone. Why?

Let's have a look at how the step factory is designed. For each new step an associated recipe object is created:

# spec/factories/step_factory.rb
FactoryBot.define do
  factory :step do
    account_id: Faker::Number.number
      # more code
    association(:recipe) # this always creates a recipe
  end 
end

This is what causes that extra recipe to be created in the example above. By explicitly setting the recipe object in the :with_two_steps trait definition, you can avoid writing an unnecessary extra recipe into the database:

# spec/factories/recipe_factory.rb
FactoryBot.define do
  factory :recipe do
  # more code
    trait(:with_two_steps) do
      after(:create) do |record|
        record.steps << create_pair(:step,
          account_id: record.account_id,
          body: Faker::Food.description,
          recipe: record # set the recipe explicitly
        )
      end
    end
  end
end

This might seem like a minor win, but if this trait is called from multiple tests, a fix like this can remove dozens of unnecessary writes to the database. If multiple factories are buggy in similar ways, we can be talking about hundreds of unnecessary writes.

This might also seem like a convoluted tutorial example, but sadly, bugs related to associations in factories are very common, incredibly tricky to spot, and can be even more convoluted in real-life projects.

A good rule of thumb here is this: whenever possible, avoid defining associations in factory definitions. Create the associated objects test by test, as needed. You’ll end up with much more manageable test data. If this is not possible, make sure you are not creating more data than is strictly necessary.

Culprit #4: Using let! incorrectly

It might seem convenient to define all the test data you need in your test examples on top of your test file in this way:

let!(recipe_1) { ... }
let!(recipe_2) { ... }
let!(step_1) { ... }
let!(step_2) { ... }

it 'test example that uses recipe_1 and recipe_2 objects' do
end

it 'test example that uses just recipe_1 object' do
end

it 'test example that uses step_1 and step_2 objects' do
end

This looks clean and easy to read. However, due to the nature of how let! works, a new instance of all of these test objects will be created before each test example run, even for test cases that don’t require all (or any!) of those objects to exist. In a big test group, this innocent mistake might lead to dozens of unnecessary writes to the database.

To fix this, see if it's possible to create separate contexts for related text examples:

context 'tests that use recipe_1 and recipe_2 objects' do
  let!(recipe_1) { ... }
  let!(recipe_2) { ... }

  it 'test example that uses recipe_1 and recipe_2 objects' do
  end
  # more test examples
end

context 'tests that use step_1 and step_2 objects' do
  let!(step_1) { ... }
  let!(step_2) { ... }

  it 'test example that uses step_1 and step_2 objects' do
  end
  # more test examples
end

---

All of these tips come down to the simple idea of being mindful of the objects your tests generate and never creating more than the bare minimum of data that is necessary for your test to work properly. It's a mindset shift that can take a while to adopt, but that is likely to pay off in the future. After all, a slow test suite kills your team's productivity and makes certain programming approaches like Test-driven development impossible incredibly painful.

I hope this tutorial will help you speed up your project's tests. If you are aware of other common issues that can slow down a test suite, please share them in the comments below.

Happy testing!

Comments

[Augusts Bautra]

One optimization I recommend everyone check is parallel test runner workload balancing in CI. In practice this means making sure parallel_rspec runs save spec run times, the file is cached, and then next runs load the file, resulting in an even-ish distribution of spec load.
If the spread between the first and last parallel spec running process is more than a minute, something is off.

Metrics are very important. In one project I noticed that CI sometimes spikes form 13min average to 20min. Here's a writeup on how that went: dev.to/epigene/how-i-stopped-rspec...

[Rafael Nunes]

Great post!

[Nadia Zhuk]

Thanks!

[Ande]

thanks to you.