The migration ran in two seconds.
You wrote it on Friday,
ran it against your local database,
saw green,
and pushed it with the feature.
Your local database has forty rows.
Production has forty million,
and the two second migration
is a lock held for eleven minutes
while every write in the system
queues up behind it.
The deploy reports success.
The site is down.
A migration is the strangest code
you will ever write.
It runs once.
It runs against data
you have never seen.
It gets the least attention
in the whole pull request,
because it sits at the bottom
and it is only schema.
And it is the one change
you cannot undo
by deploying the previous version.
So slow down at the bottom of the diff.
Know the size of the table
before you touch it.
Not roughly.
The number.
Know whether the operation
takes a lock,
which kind,
and whether it blocks reads,
writes, or both,
because your database has opinions
that differ from the last one you used.
Then split the dangerous ones.
Add the column nullable.
Backfill in batches,
with a pause between them,
while the application keeps serving.
Add the constraint afterwards,
once the data already satisfies it.
Three boring deploys
instead of one clever one.
Rehearse it against a copy
of production restored last night,
and time it.
A number from real data
outranks every opinion
in the review.
Ship the schema change
separately from the code
that depends on it.
Old application, new schema,
running together for a day,
so that when something goes wrong
you are debugging one thing.
And write down
what happens if you stop halfway.
Not a rollback script
you will never run.
A paragraph saying whether
the system can survive
in the half migrated state,
because that is the state
you will actually be standing in
at two in the morning.
Small change.
Enormous table.
The table decides
how long it takes.
– Serguey Asael Shinder
Top comments (0)