The deploy went green in ninety seconds. Then the error rate on blue hit 100%, and every request touching /orders returned column "orders.status_v2" does not exist. Two healthy app stacks, no working rollback.
Blue-green only ever protected one thing
Blue-green gives you two copies of your application code and a router that flips between them. It does not give you two copies of your database, because you cannot cheaply fork a live Postgres cluster with continuous writes and merge the forks later. Both environments point at the same host, the same schema, the same tables.
So the moment migrate finishes, blue runs new code against a new schema and green runs old code against that same new schema. The swap is atomic. The schema change is not.
Here is the migration that took us down:
ALTER TABLE orders DROP COLUMN status;
ALTER TABLE orders ADD COLUMN status_v2 text NOT NULL DEFAULT 'pending';
Order matters. Green still selects status. Between the DROP and the moment green stops serving, every green query fails. Green keeps serving for as long as the router takes to drain, plus every in-flight request, plus every retry. The rollback plan said "flip back to blue." Blue was already broken by a migration that ran before the flip.
Rollback means two different things
Code rollback is a pointer change: flip the router, and the old artifact still exists. It works only while the data underneath still fits.
Schema rollback is not symmetric. The reverse of DROP COLUMN is ADD COLUMN, and the values are gone. You cannot un-drop a column, un-truncate a table, or un-run an UPDATE that rewrote rows in place. The database has one state, it moves forward, and it has no previous release to point at.
There is a third, worse case: the migration fails halfway. Postgres runs most DDL in a transaction, so a failed ALTER TABLE inside BEGIN rolls back cleanly. But a file with several statements, some CONCURRENTLY, or explicit commits between steps leaves a schema that is neither the old shape nor the new one. That is the state nobody rehearsed.
Reversible schema, irreversible data
The distinction I should have drawn before that deploy.
A schema change is reversible when old code can still run against the new schema. Adding a nullable column, an index, a table — all reversible, because old code ignores what it does not know about.
A data change is irreversible when the migration rewrites or removes values. Dropping a column, a lossy type cast, a backfill of computed values — the new shape is not derived from the old one, so no reverse migration reconstructs it.
A reversible schema change often accompanies an irreversible data change. That is the trap. ADD COLUMN status_v2 alone is safe. ADD COLUMN plus a backfill plus a DROP COLUMN in one file is a one-way door.
Expand, contract, flag
The safe path has three releases, not two.
Release one: expand. Add the new column as nullable, and dual-write in application code. Old code reads the old column and keeps working. New code writes both. Nothing breaks if you roll back, because both shapes exist.
def write_order(order):
db.execute(
"UPDATE orders SET status = %s, status_v2 = %s WHERE id = %s",
(order.legacy_status, order.status, order.id),
)
Release two: backfill in batches, outside the deploy, with a query you can stop. Backfill must be restartable. Run it in chunks, log the last processed id, and make a rerun a no-op.
Release three: switch reads to the new column behind a feature flag. The flag is the rollback. If reads break, flip the flag, not the deploy. Only after the flag has been on for a full traffic cycle do you contract — drop the old column in a boring release where the code no longer references it.
Do the flag before the drop, not after. A flag is reversible in seconds; a DROP COLUMN is not.
When the migration is already half-applied
Stop the deploy. Do not roll back the app, and do not run a second migration to "fix" the first. Both make the schema harder to reason about.
Find out what actually landed. In Postgres, read the real state instead of trusting your migration table:
SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY ordinal_position;
Then check whether the migration is holding a lock. pg_stat_activity and pg_locks show a blocked ACCESS EXCLUSIVE waiting on a long-running read. Kill the blocker if it is a stale report query, or your next deploy deadlocks.
Then pick a direction and commit. If the old shape still exists, disable the new code path with the flag and leave the schema alone — you are back to release one. If the old shape is gone, the only path is forward: ship code matching the current schema, patch the failing queries today, and accept that you are deploying under pressure. Restore from a backup only if you can afford to lose everything written since the migration, and measure that window first.
The lesson I keep: the database is not a deployment target. It is a shared, single-state dependency, and every migration is a change to production data. Rehearse the migration on a copy with real row counts, and time how long the lock is held.
I write about production failures in Postgres, queues, and distributed systems.
Top comments (0)