DEV Community: Pavel Myslik

Fixing Production Data in Rails: Lessons from a 6,000-Row Backfill

Pavel Myslik — Wed, 25 Mar 2026 13:45:03 +0000

Fixing a bug in production is usually straightforward. But what happens when the data in the database itself is broken? That’s when things get tricky.

I learned this when a routine deploy quietly broke confirmed_at on our Order records. By the time anyone noticed, around 6,000 rows were affected — and dashboards, emails, and downstream services all depended on that field.

The two-line code fix shipped in minutes, but the data backfill required a lot more thought.

The First Naive Attempt

Once we understood the scope of the problem, the first instinct was obvious: fix it fast. Most Rails developers will reach for one of two things here — a quick one-liner in the console, or a small migration that updates the data directly.

Something like this:

Order.where(confirmed_at: nil, status: "confirmed")
     .update_all(confirmed_at: Time.current)

Note: In reality, determining the correct value for confirmed_at was more complicated — we had to derive it from related records and business logic. For the purposes of this article, Time.current keeps the examples simple and focused on the backfill pattern itself.

It looks clean. One line, no fuss. After a two-minute code fix, it feels like the natural next step.

And this is exactly where production backfills could go wrong.

Why This Is Dangerous in Production

At first glance, the one-liner looks harmless — simple, fast, and it works perfectly in development. The problem starts when you run it on real production data.

There are a few risks that are easy to overlook:

No visibility into progress. Once the query starts, you have no idea how many records have been updated or how many are left.
No safe way to restart. If the process stops halfway — timeout, a deploy, a dropped connection — you don't know what state the data is in.
No way to preview the change. There's no dry run. Running it once already changes the data.
Hard to test properly. You can't write a meaningful test for a console one-liner or an inline migration. If the logic is wrong, you find out in production.
Large transactions put pressure on the database. Updating thousands of rows in a single query can lock tables and slow down the app for real users.

The problem isn't the syntax. It's that once it starts, you've handed over control — and when other systems depend on this data being correct, that's a risk you don't want to take.

A Safer Approach

Instead of trying to fix everything in one query, we decided to treat the backfill like real code.

Not a console one-liner.
Not an inline migration.
A small Ruby class that lives in the repository and can be tested like anything else.

Something like this:

lib/backfills/backfill_confirmed_at.rb

The idea is simple:

the script only triggers the backfill
all the logic lives in a dedicated class
the class can be tested safely before anything runs in production

This keeps the backfill predictable, restartable, and — most importantly — verifiable before it touches a single row.

The Backfill Class

# lib/backfills/backfill_confirmed_at.rb

module Backfills
  class BackfillConfirmedAt
    attr_reader :dry_run, :logger

    def initialize(dry_run: false, logger: Rails.logger)
      @dry_run = dry_run
      @logger  = logger
    end

    def run
      log "Starting | mode: #{dry_run ? 'DRY RUN' : 'LIVE'}"

      processed = 0

      orders.find_each do |order|
        if dry_run
          log "[DRY RUN] Would update Order ##{order.id}"
        else
          order.update_columns(confirmed_at: confirmed_at)
          log "Updated Order ##{order.id}"
        end

        processed += 1

      rescue StandardError => e
        log "Error processing Order #{order.id}: #{e.message}"
      end

      log "Done. Processed: #{processed}"
    end

    private

    def log(message) = logger.info "[Backfills::BackfillConfirmedAt] #{message}"

    def orders = Order.where(confirmed_at: nil, status: "confirmed").order(:id)

    # In reality, the more complex logic of calculating the date
    def confirmed_at = Time.current
  end
end

Testing the Backfill Class

One of the biggest advantages of extracting the logic into a class is that you can test it like any other Ruby code — before running anything in production.

Here are a few tests that cover the most important cases:

# spec/lib/backfills/backfill_confirmed_at_spec.rb

RSpec.describe Backfills::BackfillConfirmedAt do
  let(:null_logger) { Logger.new(nil) }

  describe "#run" do
    let!(:order) { create(:order, status: "confirmed", confirmed_at: nil) }

    context "when dry_run is enabled" do
      it "does not update any records" do
        described_class.new(dry_run: true, logger: null_logger).run

        expect(order.reload.confirmed_at).to be_nil
      end
    end

    context "when order is confirmed with missing confirmed_at" do
      it "updates confirmed_at" do
        described_class.new(logger: null_logger).run

        expect(order.reload.confirmed_at).not_to be_nil
      end
    end

    context "when order already has confirmed_at" do
      let(:timestamp) { 2.days.ago }

      before { order.update(confirmed_at: timestamp) }

      it "does not overwrite it" do
        described_class.new(logger: null_logger).run

        expect(order.reload.confirmed_at).to be_within(1.second).of(timestamp)
      end
    end

    context "when order has a different status" do
      before { order.update(status: "pending") }

      it "does not touch it" do
        described_class.new(logger: null_logger).run

        expect(order.reload.confirmed_at).to be_nil
      end
    end
  end
end

Notice that we pass a null_logger to keep the test output clean — no need to see backfill logs while running the test suite.

These tests won't catch every edge case, but they give you enough confidence to run the backfill knowing the core logic has been verified.

Running the Backfill

The script becomes very simple — it only decides whether we run a dry run or the real update:

# script/backfill_confirmed_at.rb

dry_run = ARGV.include?("--dry-run")

Backfills::BackfillConfirmedAt.new(dry_run: dry_run).run

Always Start With a Dry Run

Before touching any production data, always preview what would change first:

rails runner script/backfill_confirmed_at.rb --dry-run

The output will look something like this:

[Backfills::BackfillConfirmedAt] Starting | mode: DRY RUN
[Backfills::BackfillConfirmedAt] [DRY RUN] Would update Order #10042
[Backfills::BackfillConfirmedAt] [DRY RUN] Would update Order #10051
[Backfills::BackfillConfirmedAt] [DRY RUN] Would update Order #10063
...
[Backfills::BackfillConfirmedAt] Done. Processed: 6000

This gives you a chance to verify a few things before anything is written:

Is the scope correct? Spot-check a few IDs directly in the database.
Does the number of affected records match your expectations? If you expected 6,000 rows but the dry run shows 10,000, something is wrong.
Are there any surprising records? Maybe some orders have a status you didn't account for.

Dry run costs you a few minutes.
A botched live backfill can cost you hours.

Then Run It For Real

When everything looks correct:

rails runner script/backfill_confirmed_at.rb

If you want even more control, consider extending the backfill class with a limit: option to test on a small subset first, or a configurable batch_size: for larger datasets.

How to Safely Stop the Running Backfill

One of the hidden benefits of this approach is that you can stop the script at any time — without worrying about leaving the data in a broken state.

Because find_each processes and commits records one by one, each update is independent. If you stop the script after 2,000 records, those 2,000 rows are correctly updated and stay that way. The remaining records are simply untouched.

To stop the script, a simple Ctrl+C is enough. When you're ready to continue, just run it again. The scope automatically skips already-updated records and picks up where it left off.

This is the key difference from update_all or an inline migration. A large single query either completes fully or rolls back entirely. If something goes wrong halfway through, you're back to square one.

With batch processing, interrupting is not a failure. It's just a pause.

The Workflow We Follow Now

Not every data fix needs this treatment. If you're updating a handful of records, a quick one-liner in the console is perfectly fine.

But once you're dealing with thousands of rows — especially when other systems depend on that data — it's worth slowing down and following a simple process.

1. Understand the scope first

Before writing a single line of code, run a COUNT query on production. Know exactly how many records are affected and confirm what the correct value should be.

2. Write the backfill class

Extract all logic into a dedicated class in lib/backfills/. Add a dry_run option and logging from the start.

3. Test it

Write at least a few basic tests before running anything. This is the step that's easiest to skip under pressure, and the one you'll be most grateful for.

4. Dry run on production

Run with --dry-run first and review the output carefully. Verify the record count matches your expectations and spot-check a few IDs directly in the database.

5. Run it for real

Only when the dry run looks correct. Monitor the logs as it runs — that's what they're there for.

Final Thoughts

The irony of this whole situation was that the code fix took two minutes. The data fix took almost a day — not because it was technically hard, but because we wanted to do it right.

That ratio is worth remembering. A careless backfill can easily cause more damage than the original bug. Treating it like real code — with tests, dry runs, and logging — is not over-engineering. It's just respect for the data your users depend on.

Next time you're staring at thousands of broken records, resist the one-liner. Take the extra hour. Your future self will thank you.

What does your backfill process look like? I'm curious whether others reach for a similar pattern — or something completely different. Let me know in the comments.

I Thought My Rails Query Was Fine — Until NULL Ate My Data

Pavel Myslik — Wed, 11 Mar 2026 13:02:16 +0000

I ran into this while working on a task where I needed to process all contracts that were not coming from SAP.

At first, the query looked perfectly fine:

Contract.where.not(source: 'sap')

Then I double-checked the result by counting returned objects — and something didn’t add up.

You write a Rails query.
It looks correct.
It runs without errors.

But it's quietly hiding records from you.

Welcome to one of the most common — and maybe most dangerous — SQL gotchas.

The Setup

Imagine you have a contracts table with a source column — which is nullable.

Some contracts have:

source = 'sap'
source = 'web'
and some have NULL

Now you want to count all contracts that are not from 'sap'.

Sounds simple:

Contract.where.not(source: 'sap').count
# => 68

68 contracts. Looks reasonable.

But to be sure, I double-checked using Ruby:

Contract.reject { |c| c.source == 'sap' }.size
# => 310

Wait... 310?

Where did 242 records go?

The Problem: NULL Is Not a Value

In SQL, NULL does not mean empty.

It means unknown.

That distinction changes everything.

When SQL evaluates:

source != 'sap'

there are three possible outcomes:

'web'     != 'sap'  => TRUE  (included)  
'sap'     != 'sap'  => FALSE (excluded)  
NULL      != 'sap'  => NULL  (excluded)

That third row is the trap.

NULL != 'sap' does not return TRUE.

It returns NULL.

And in SQL’s three-valued logic, the WHERE clause only keeps rows where the condition is TRUE.

FALSE is excluded.
NULL is also excluded.

Silently.

This isn’t just an issue with !=. Any comparison or negation— <, >, NOT LIKE, NOT IN—can silently exclude NULL rows.

The Fix

Here are three common ways to handle this safely, depending on your database and preference:

Option 1: Rails-style `.or` with `where.not`

Contract.where.not(source: 'sap')
        .or(Contract.where(source: nil))
        .count
# => 310

Rails-native syntax.
Includes all NULL values.
Works on any database supported by Rails.

Option 2: Explicitly include NULLs in SQL with `OR`

Contract.where("source != ? OR source IS NULL", 'sap')
        .count
# => 310

Simple SQL pattern.
Works reliably across most databases (PostgreSQL, MySQL, SQLite).

Option 3: Use `IS DISTINCT FROM` (PostgreSQL)

Contract.where("source IS DISTINCT FROM ?", 'sap')
        .count
# => 310

PostgreSQL-specific.
NULL is treated like a regular value.

The Takeaway

NULL didn’t break my query. It behaved exactly as SQL intended.

That’s what makes it dangerous — no error, no warning, no sign anything went wrong. Just missing data, quietly waiting for you to notice.

Once you understand that NULL means unknown, not empty, the behavior starts to make sense.

Until then… it can silently cost you 242 records.

Have you ever shipped a bug caused by NULL hiding in a query?

Stop Using .any? the Wrong Way in Rails

Pavel Myslik — Tue, 24 Feb 2026 15:30:27 +0000

A single block passed to .any? can silently load thousands of records into memory.

No warnings. No errors. Just unnecessary objects.

And most Rails developers don’t notice it.

You’ve probably used both .any? and .exists? in your Rails app without thinking twice. They both answer the same simple question:

Is there at least one record?

But under the hood, they can behave very differently.

In this article, we’ll look at what actually happens when you call each method, when to use which, and how to avoid a common performance trap.

The Basics: Same Query, Same Result

If you just need to check whether a relation contains any records at all, both .any? and .exists? generate the same efficient query.

user.posts.any?
# SELECT 1 AS one FROM "posts" WHERE "posts"."user_id" = 1 LIMIT 1

user.posts.exists?
# SELECT 1 AS one FROM "posts" WHERE "posts"."user_id" = 1 LIMIT 1

No objects are loaded into memory.
No full table scan.

Both methods ask the database a simple yes/no question and return immediately after finding the first match.

The same applies when you chain .where — as long as you don’t pass a block to .any?:

user.posts.where(published: true).any?
# SELECT 1 AS one FROM "posts"
# WHERE "posts"."user_id" = 1 AND "posts"."published" = true
# LIMIT 1

user.posts.where(published: true).exists?
# SELECT 1 AS one FROM "posts"
# WHERE "posts"."user_id" = 1 AND "posts"."published" = true
# LIMIT 1

So if this is all you need, pick whichever reads better in your code.

There’s no performance difference here.

Where Things Get Dangerous: `.any?` With a Block

The moment you pass a block to .any?, Rails completely changes its behavior.

Instead of asking the database, it loads every matching record into memory and filters in Ruby:

user.posts.any? { |post| post.published? }
# SELECT "posts".* FROM "posts" WHERE "posts"."user_id" = 1

That single line:

Loads all posts
Instantiates an ActiveRecord object for each one
Iterates over them in Ruby
Just to return boolean

It might look harmless in development.

But in production?

If a user has 50,000 posts, you just loaded 50,000 objects into memory to check if one of them is published.

Why This Happens

Here’s how .any? is implemented in Rails:

def any?(*args)
  return false if @none

  return super if args.present? || block_given?
  !empty?
end

If a block is given, Rails delegates to Enumerable#any?.

And Enumerable#any? needs the full collection in memory.

You’ve effectively moved filtering from SQL to Ruby.

The Fix: Let the Database Do the Work

Push the condition into SQL:

user.posts.exists?(published: true)
# SELECT 1 AS one FROM "posts"
# WHERE "posts"."user_id" = 1 AND "posts"."published" = TRUE

One row
One column
Stops at the first match
No object instantiation

Same result. Much lower cost.

When `.any?` Is Actually the Right Choice

There is one important exception.

If the relation is already loaded, .any? will not hit the database again.

For example:

users = User.includes(:posts)

users.each do |user|
  user.posts.any? { |post| post.published? } # no extra query
end

Because posts were preloaded, .any? works entirely in memory.

In this case:

.any? → no additional query
.exists? → forces a new SQL query

So using .exists? here could actually introduce unnecessary database calls — potentially even an N+1 pattern.

Rails internally checks whether the relation is loaded.
If it is, .any? behaves like a normal Ruby collection.

A Better Rule of Thumb

Relation not loaded → prefer .exists?
Relation already loaded → .any? is perfectly fine
.any? with a block → avoid on ActiveRecord relations
.present? for existence → avoid on ActiveRecord relations

Final Thought

Before calling .any?, ask yourself:

Am I checking existence — or am I about to load an entire collection?

Small differences in ActiveRecord APIs can have real production impact.

Have you ever spotted this in a real codebase?

This is part of a small series exploring subtle ActiveRecord behaviors that can impact performance.