DEV Community

Morgan Li
Morgan Li

Posted on

Agent-Written Down Migrations or Human Recovery Runbooks: A Rehearsal-First Debate

Consider a composite of the failure reports that keep circulating in Postgres incident threads, rather than a logged incident from my own systems. A migration adds a column and a backfill, an AI reviewer approves the up-path, and nobody authors the down-path at all. Three hours later the backfill is still running, the lock queue is growing, and the only rollback plan left is last night's base backup. The interesting failure is not that the model was wrong; it is that nobody defined who owned the reversal before the migration started.

That gap is what this post argues about, in the same debate format this account has used before. Two defensible positions exist, each with real supporting evidence, and the resolution is not "be careful" but a decision rule you can encode in a pull request checklist.

Position A: The agent should author both directions

The first position says a review agent is not finished until it has produced a down migration alongside the up migration. Its strongest argument is behavioral rather than technical: pairing forces the reversibility question to be asked at authoring time, when the change is still cheap to reshape. A migration that cannot be expressed as a clean inverse is usually a migration that should be split, staged, or moved out of band. Coverage also improves for mechanical cases, because additive DDL has a genuinely trivial inverse that nobody should be writing by hand in 2026.

There is supporting evidence for this position in ordinary diff review. When a reviewer sees ALTER TABLE orders ADD COLUMN settled_at timestamptz with no counterpart, the asymmetry is visible and cheap to flag. When the down-path is instead a paragraph in a runbook that lives somewhere else, the asymmetry disappears from the diff entirely. Model-assisted review is good at exactly this kind of local symmetry check, which is why the "author both directions" camp tends to win arguments about small, additive changes.

Position B: Humans own rollback; agents may only draft

The second position accepts the symmetry argument but rejects its conclusion for destructive changes. Its central claim is that a down migration which drops a column is not a rollback; it is data loss with extra ceremony. Once values have been overwritten by a backfill, no DDL statement can reconstruct them, and a down.sql file that pretends otherwise creates false confidence in exactly the moment confidence is most expensive. The real recovery artifact for a destructive change is a restore path plus a runbook, and neither is something a code generator should own.

This position also has an organizational argument that is hard to dismiss. Review capacity, not authoring capacity, is the scarce resource on most teams. Generating down migrations for every change produces a large body of reviewable output whose value is concentrated in the small subset of changes that are genuinely reversible. A team that reviews twenty generated inverses to catch one dangerous one has spent attention it could have spent on the up-path's lock profile instead.

Where each position actually breaks

Position A breaks on data-destroying changes, and it breaks silently. The schema after the down migration looks identical to the schema before, so every automated check you are likely to have will pass while the values are gone. This is the single most important failure mode in the whole debate, and it is the reason schema equivalence alone cannot be the gate.

Position B breaks on velocity and on forgotten coverage. If rollback planning is a separate human deliverable, it competes with feature work and loses, which is how teams end up with a migration history full of up-paths and no rehearsed reversal for anything. It also breaks on long-lived branches, where the person who wrote the migration is no longer the person merging it. A rule that depends on human memory of intent degrades as the branch ages.

The decision rule

Author the down migration with an agent only when DDL alone can restore the previous state; anything that destroys values needs a human-authored recovery runbook. That single sentence resolves most of the disagreement, and the table below turns it into per-change-class gates you can paste into a review checklist.

Change class Rehearsal signal Down-path author Merge gate
Additive, nullable, no backfill down restores an identical schema hash agent may author CI rehearsal passes
Backfill or bulk UPDATE write volume exceeds your maintenance budget agent drafts, human rewrites as a batched job explicit row cap plus an off-peak window
Index creation concurrent build lock profile observed agent may author deployed as a separate step
Column drop or type change down cannot restore original values human-authored restore runbook no agent-authored down migration
Constraint or foreign key addition lock wait exceeds the rehearsal timeout agent drafts, human approves staged validation, not one transaction

The rule is deliberately conservative about data and deliberately permissive about structure. Structural changes are where generation pays off, because the inverse is mechanical and the failure is visible in a diff. Data changes are where generation is dangerous, because the inverse is impossible and the failure is invisible until someone queries a column that used to have values.

A rehearsal harness you can run in CI

Everything above is cheap to test if you keep one disposable database around. The script below rehearses an up/down pair and fails when the down-path is not schema-equivalent, or when the up-path changed nothing at all.

#!/usr/bin/env bash
# rehearsal.sh — run an up/down pair against a disposable database.
set -euo pipefail

DB_URL="${REHEARSAL_DB_URL:?set REHEARSAL_DB_URL to a throwaway database}"
UP="${1:?usage: rehearsal.sh up.sql down.sql}"
DOWN="${2:?usage: rehearsal.sh up.sql down.sql}"

# Fail fast instead of queueing behind a lock you cannot see.
export PGOPTIONS="-c lock_timeout=3s -c statement_timeout=15s"

# sha256sum on Linux, shasum -a 256 on macOS.
HASH="sha256sum"
command -v sha256sum >/dev/null || HASH="shasum -a 256"

schema_hash() {
  pg_dump --schema-only --no-owner --no-privileges "$DB_URL" \
    | sed -e '/^--/d' -e '/^$/d' \
    | $HASH | cut -d' ' -f1
}

before="$(schema_hash)"
psql "$DB_URL" -v ON_ERROR_STOP=1 -f "$UP"   >/dev/null
applied="$(schema_hash)"
psql "$DB_URL" -v ON_ERROR_STOP=1 -f "$DOWN" >/dev/null
after="$(schema_hash)"

printf 'before  %s\napplied %s\nafter   %s\n' "$before" "$applied" "$after"

[ "$before" != "$applied" ] || { echo 'FAIL: up migration changed nothing'; exit 3; }
[ "$before" =  "$after" ]   || { echo 'FAIL: down migration is not schema-equivalent'; exit 2; }
echo 'PASS: up applied, down restored the prior schema'
Enter fullscreen mode Exit fullscreen mode

Before you let a backfill near production, estimate the write volume without executing it. Note the absence of ANALYZE in the block below: EXPLAIN ANALYZE on an UPDATE executes the UPDATE, which is a mistake worth naming explicitly.

-- Estimate rows and plan shape for a proposed backfill. Read-only.
BEGIN;
SET LOCAL lock_timeout = '3s';

SELECT reltuples::bigint AS approx_rows
FROM pg_class
WHERE oid = 'public.orders'::regclass;

EXPLAIN (VERBOSE, COSTS)
UPDATE orders SET settled_at = now() WHERE settled_at IS NULL;

ROLLBACK;
Enter fullscreen mode Exit fullscreen mode

Numbered workflow

  1. Provision a throwaway database and never point the rehearsal at anything you intend to keep.
  2. Capture the before-state schema hash with the schema_hash function above.
  3. Apply the up migration under explicit lock_timeout and statement_timeout values, not defaults.
  4. Apply the down migration and fail the build when the hash does not match the before-state.
  5. Classify the change against the decision table and record the matching row in the pull request.
  6. For destructive classes, replace the down migration with a restore runbook and name its owner in the pull request.

If you also want a second opinion on the up-path, the two-reviewer pattern below is a reasonable prompt sketch rather than a validated system. It is pseudocode: I have not benchmarked it, and two passes from the same model family can share the same blind spot.

# Reviewer 1 — author pass (pseudocode)
Input:  up.sql, table sizes, current indexes
Output: risk list, requested rewrite, proposed down migration
Rule:   if the change destroys values, refuse to author a down migration

# Reviewer 2 — adversarial pass (pseudocode)
Input:  the same up.sql, plus Reviewer 1's output
Output: counterexamples, lock-order hazards, missing rollback data
Rule:   assume the down migration will be executed during peak traffic
Enter fullscreen mode Exit fullscreen mode

Where this runs, and one availability note

The rehearsal database must be disposable by design, which makes a throwaway server the natural host for it. MonkeyCode's operator states that the free tier includes ten million tokens of model access and a free server option, which is enough to run both reviewer passes and a short-lived rehearsal database without a procurement conversation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those availability claims come from the operator and I have not independently verified resource ceilings or long-term uptime, so verify the current limits on their site before you depend on them; use the free server for short, disposable rehearsals and never for anything you intend to keep.

Limitations and who should skip this

Schema-hash equality proves nothing about data, and a rehearsal database has none of production's row distribution or contention. A three-second lock timeout that passes on an idle server may be wildly optimistic on a table receiving continuous writes. The harness above is Postgres-specific and depends on pg_dump access, so managed platforms that restrict it need a substitute snapshot mechanism. Finally, if your framework already owns reversible migrations and you trust its inverses, this workflow mostly duplicates work you have already paid for.

Teams that cannot spin up a scratch database, or that operate under rules preventing schema artifacts from leaving their environment, should not adopt the CI half of this pattern; the decision table alone still applies to them. Everyone else gets the useful part: a rule that lets an agent write the parts of rollback that are genuinely mechanical, and keeps humans accountable for the parts that are not. If you want to try the rehearsal loop without provisioning anything first, the operator-provided free model access and free server are a reasonable starting point — just confirm the current limits before you build a habit on them.

Top comments (0)