Django Data Migrations: Getting Them Right on a Live Database

Schema migrations, adding a column, creating a table are the ones everyone thinks of first. Data migrations are the quieter, riskier cousin: transforming or backfilling actual data, on a database that's serving real traffic while you do it. Get a schema migration wrong and you usually find out immediately. Get a data migration wrong on a table with tens of thousands of rows, and you might not find out until a user reports something that doesn't add up.

I've run data migrations against a live platform with a large, active user base, and here's what actually matters when the table you're migrating isn't a toy dataset anymore.

Data migrations are just RunPython, but treat them differently

A data migration in Django is created the same way as any migration, then edited to use RunPython instead of (or alongside) schema operations:

# migrations/0015_backfill_display_name.py
from django.db import migrations

def backfill_display_name(apps, schema_editor):
    User = apps.get_model("accounts", "User")
    for user in User.objects.filter(display_name__isnull=True).iterator():
        user.display_name = user.email.split("@")[0]
        user.save(update_fields=["display_name"])

def reverse_backfill(apps, schema_editor):
    User = apps.get_model("accounts", "User")
    User.objects.update(display_name=None)

class Migration(migrations.Migration):
    dependencies = [("accounts", "0014_add_display_name")]
    operations = [
        migrations.RunPython(backfill_display_name, reverse_backfill),
    ]

Three things in this example matter more than they look:

Always provide a reverse function. migrations.RunPython.noop is acceptable if reversal genuinely doesn't make sense, but think about it deliberately rather than defaulting to noop out of laziness. If a migration goes wrong in production, having a real way back matters.

Use apps.get_model, never import the model directly. This is easy to get wrong and it silently causes real bugs. Importing from accounts.models import User ties the migration to the model's current state in your codebase, not its state at the point in migration history where this migration runs. If a field gets renamed or removed later, old migrations importing the model directly can break entirely when run from scratch on a new environment. apps.get_model gives you the historical version of the model as it existed at that point in the migration graph.

Use .iterator() on large querysets. Without it, User.objects.filter(...) loads the entire queryset into memory before iterating. On a table with 97,000+ rows, that's a meaningful and unnecessary memory spike. .iterator() streams results from the database instead.

Batch large updates — don't process a huge table in one transaction

The example above updates rows one at a time in a loop, which is safe but slow on a large table, and by default a Django migration runs inside a single transaction (on databases that support transactional DDL). A long-running transaction touching hundreds of thousands of rows can hold locks longer than you want, competing with real production traffic for the same table.

Batch it instead:

def backfill_display_name(apps, schema_editor):
    User = apps.get_model("accounts", "User")
    batch_size = 1000
    queryset = User.objects.filter(display_name__isnull=True)

    while queryset.exists():
        batch_ids = list(queryset.values_list("id", flat=True)[:batch_size])
        for user in User.objects.filter(id__in=batch_ids):
            user.display_name = user.email.split("@")[0]
            user.save(update_fields=["display_name"])

This trades a single long transaction for many short ones, keeping individual lock durations low and letting other queries interleave between batches rather than queuing up behind one enormous operation.

Make migrations idempotent where you can

If a migration fails halfway through a deploy, gets interrupted, a database connection drops, you want to be able to re-run it safely, not worry about double-processing rows that already succeeded. The filter(display_name__isnull=True) pattern in the examples above is already idempotent in this sense: re-running the migration only touches rows that still need it, since already-backfilled rows no longer match the filter.

This is worth checking deliberately for every data migration you write: if it ran twice, either by accident or as a genuine retry after a partial failure, would the result be correct? If not, it's worth restructuring until it is.

Separate schema changes from data changes when the table is large or important

It's tempting to combine "add a column" and "backfill it" into one migration for convenience. On a small table, that's fine. On a large, actively-used table, split it into multiple deploys:

1. Migration 1: add the new column as nullable
2. Deploy, let it run
3. Migration 2: backfill the data (using the batching pattern above)
4. Deploy, let it run and confirm the backfill completed
5. Migration 3: add a NOT NULL constraint (and any index) now that every row has a value

This avoids a scenario where a migration that both adds a column and immediately enforces NOT NULL on it locks a large table for an extended period while every existing row gets the default value written — on a small table this is instant and invisible; on a large one it can cause real, user-facing slowdowns during deploy.

Test data migrations against a realistic copy of production, not just a fresh test database

A migration that runs in half a second against Django's test database (usually empty or near-empty) can behave completely differently against a production-sized table. Before relying on a data migration's runtime in production, either test it against a staging environment with a realistic data volume, or at minimum, calculate roughly how long it should take based on rows-per-second in a batch and total row count — and communicate that expectation to your team before a deploy, not after someone asks why the deploy is taking twenty minutes.

Log progress on long-running migrations

For anything that might run more than a few seconds against a production-sized table, add basic logging so you (or whoever's watching the deploy) can see it's actually progressing rather than silently hung:

import logging
logger = logging.getLogger(__name__)

def backfill_display_name(apps, schema_editor):
    User = apps.get_model("accounts", "User")
    batch_size = 1000
    total = User.objects.filter(display_name__isnull=True).count()
    processed = 0

    queryset = User.objects.filter(display_name__isnull=True)
    while queryset.exists():
        batch_ids = list(queryset.values_list("id", flat=True)[:batch_size])
        for user in User.objects.filter(id__in=batch_ids):
            user.display_name = user.email.split("@")[0]
            user.save(update_fields=["display_name"])
        processed += len(batch_ids)
        logger.info(f"Backfilled {processed}/{total} users")

This is a small addition that pays for itself the first time a migration runs longer than expected and someone needs to know whether to wait or intervene.

A short checklist

• RunPython uses apps.get_model, never a direct model import
• A real reverse function is written where reversal is meaningful, not defaulted to noop without thinking about it
• Large querysets use .iterator() rather than loading everything into memory
• Large updates are batched, not run as one enormous transaction
• The migration is idempotent and safe to re-run after a partial failure
• Schema changes and data backfills on large/important tables are split across separate migrations and deploys
• Runtime has been tested or estimated against realistic data volume, not just an empty test database
• Long-running migrations log progress

Data migrations don't get the same scrutiny schema migrations do, mostly because they look like "just a script," but they're the ones most likely to interact badly with real, live data at scale. Treating them with the same care as a production data change because that's exactly what they are to avoids the class of incident that only shows up once real numbers are involved.

Comments

[Raknaos]

The three-migration split is where most teams lose the plot, because on the dev database all three deploys take four seconds and the constraint feels free. The interesting cost is the one you can only see at production size: the NOT NULL that rewrites every row while traffic is on the table.

One thing I'd add from the operational side: idempotency isn't just a property of the migration, it's a property of the retry path. If a batched backfill dies at row 40k, whoever restarts it needs to know whether to resume or re-run from zero, and the migration graph won't tell them. Do you keep a progress row outside the transaction, or accept that a partial backfill gets rolled back and redone?