Tame Your Flaky RSpec Tests by Fixing the Seed

About This Article

Have you ever experienced tests that occasionally fail when running your RSpec test suite on CI? These are what we call flaky tests.

Simply re-running the test until it passes and calling it a day is a missed opportunity. Let’s take a more sustainable approach to fixing them.

Enable config.order and Kernel.srand

The key idea here is simple: enable the following configuration in your RSpec settings file, which is generated by default but commented out.

# spec/spec_helper.rb
  # Run specs in random order to surface order dependencies. If you find an
  # order dependency and want to debug it, you can fix the order by providing
  # the seed, which is printed after each run.
  #     --seed 1234
  config.order = :random

  # Seed global randomization in this process using the `--seed` CLI option.
  # Setting this allows you to use `--seed` to deterministically reproduce
  # test failures related to randomization by passing the same `--seed` value
  # as the one that triggered the failure.
  Kernel.srand config.seed
=end

Once enabled, this lets you reproduce random test failures deterministically using the seed printed after each test run.

These lines are typically wrapped in =begin and =end comments when generated, so make sure to move them out to enable them.

Reference:
https://github.com/rspec/rspec-core/blob/v3.13.2/lib/rspec/core/project_initializer/spec/spec_helper.rb#L86-L97

=begin
  These lines
  # are all comments
=end

A Hands-On Example of Dealing with Flaky Tests

Let’s walk through an example to see how flaky tests can be identified and fixed.

Setup from rails new

# Create a new Rails app
$ rails new --skip-test rspec-rails-test
$ cd rspec-rails-test/

# Add and set up rspec-rails
$ bundle add rspec-rails --group development,test
$ rails generate rspec:install

Now activate the config.order and Kernel.srand settings by moving them outside of the comment block.

   config.profile_examples = 10
+=end

   config.order = :random
   Kernel.srand config.seed
-=end
 end

Confirm RSpec runs successfully with zero examples:

$ bundle exec rspec
No examples found.

Randomized with seed 37597

Create a Model and Write a Test

Let’s create a simple model named YearMonth with year and month attributes, and some basic validations.

# app/models/year_month.rb

class YearMonth
  include ActiveModel::Model
  include ActiveModel::Attributes

  attribute :year, :integer
  attribute :month, :integer

  validates :year, numericality: { only_integer: true }
  validates :month, numericality: { only_integer: true, in: (1..12) }
end

Now, add a basic test to confirm that the model is valid.

# spec/models/year_month_spec.rb

require "rails_helper"

RSpec.describe YearMonth do
  subject { YearMonth.new(attributes) }

  let(:attributes) { { year:, month: } }
  let(:year) { rand(2100) }
  let(:month) { rand(12) }

  it do
    expect(subject.valid?).to be_truthy
  end
end

Run the Test

$ bundle exec rspec
Randomized with seed 54242
.

Finished in 0.0191 seconds
1 example, 0 failures

If you were unlucky and saw a failure like this, just re-run the test and confirm it eventually passes:

Randomized with seed 30925
F

Failures:

  1) YearMonth is expected to be truthy
     Failure/Error: expect(subject.valid?).to be_truthy

       expected: truthy value
            got: false

Observe Occasional Failures

Actually, this test sometimes fails.
After several runs, you should see an abnormal termination as shown below.

It is like a CI that is terminating normally and cheerfully in the product code, but sometimes notifies you of a failure.

$ bundle exec rspec

Randomized with seed 30925
F

Failures:

  1) YearMonth is expected to be truthy
     Failure/Error: expect(subject.valid?).to be_truthy

       expected: truthy value
            got: false
     # ./spec/models/year_month_spec.rb:11:in 'block (2 levels) in <top (required)>'

Finished in 0.01313 seconds (files took 0.50544 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/models/year_month_spec.rb:10 # YearMonth is expected to be truthy

Randomized with seed 30925

$

In real-world projects, it's useful to automate the test run in a loop and go grab lunch. Here’s a sample shell script:

#!/bin/bash
set -euxo pipefail

i=0
while true
do
  i=$((i + 1))
  bundle exec rspec
done

Reproduce the Failure

The key detail is this line in the output:

Randomized with seed 30925

You can reproduce the exact test order and failure using this seed:

$ bundle exec rspec --seed 30925

Debug the Failure

Now that it’s reproducible, debugging is straightforward. Add a puts statement to inspect the attributes:

   it do
+    puts "attributes: #{subject.attributes.inspect}"
     expect(subject.valid?).to be_truthy
   end

$ bundle exec rspec --seed 30925
attributes: {"year" => 1913, "month" => 0}
F

The month is 0 — that’s not valid. The issue is with rand(12), which generates values from 0 to 11.

let(:month) { rand(12) }

> 10000.times.map { rand(12) }.uniq.sort
=> [0, 1, ..., 11]

Fix the Root Cause

Fix the range to 1..12 instead of 0..11:

-  let(:month) { rand(12) }
+  let(:month) { rand(1..12) }

> 10000.times.map { rand(1..12) }.uniq.sort
=> [1, 2, ..., 12]

Now commit and push your fix:

git commit -m 'fix flaky test'

Summary

• Enabling config.order = :random and Kernel.srand config.seed in spec/spec_helper.rb helps identify and debug flaky tests.
• Use the printed seed value and --seed option to reliably reproduce failures.
• Don’t leave flaky tests lying around — fix them to ensure your CI surfaces real issues, not noise.