Small teams don't get a DBA or a staging cluster that matches production. The one rule that carries most of your safety: never change a live schema in place. Expand, migrate, contract.
Why: on a 40M-row table, a naive in-place ALTER can take an exclusive lock for minutes. Minutes where your app can't read or write, and at small-team scale there's no customer tolerance for a "maintenance mode" and no idle replica fleet to fail over to.
The pattern in seven steps:
- Write the undo FIRST and run it against a restored copy of production. An undo that has never executed is a rumor.
- Answer the five-question change card: what breaks if this fails, how do we undo it, who is on-call when it ships, what does the customer feel, why this window.
- Ship the additive change alone - new nullable column or new table. No app code in the same deploy.
- Backfill in batches (5k-50k rows, pause between batches), watching replica lag and p95 latency. Stop on breach.
- Dual-write to old and new paths, then run a parity check (counts + checksums) until divergence is ZERO for 24-48 hours. Not "looks fine" - zero.
- Cut reads, then writes, deliberately. Keep the old path writing one more day - it's your rollback, not a habit.
- Contract in a later PR: drop the old column and the flag after a clean parity window and a calm week.
The five traps: the big-bang ALTER (schema + backfill + cutover in one deploy), the untested undo, app code that tolerates only one schema, verification by vibes, ignoring replica lag during the read cut-over.
Worked example: an eight-person invoicing SaaS ran one in-place RENAME on the live orders table. 14-minute exclusive lock, checkout down twice, 300 tickets, and a rollback that couldn't restore lost writes. The rerun with expand-contract: 200ms additive change, overnight batched backfill, dual-write, 48h clean parity, read cut Tuesday morning, write cut Wednesday, cleanup PR next week. Zero tickets, zero downtime.
Counter-example: a team skipped the parity check because "dual-write is deterministic". A timezone bug meant the new path wrote UTC and the old path local time. A customer found it: invoice with the wrong due date. The parity check is the only part you can check BEFORE customers do.
Free checklist (with the metrics that keep the pattern honest) on our ops notes site - link in the first comment. Pairs with our change advisory board template (the five questions before merge).
Top comments (0)