At 02:14 the pipeline went green, and green was the only signal the on-call engineer had that night. An agent-authored patch added a btree index to a table with eighty million rows, the migration applied in eleven seconds against an empty CI database, and staging locked up nine minutes after the deploy finished. Nobody skipped a step and nobody lied during review; the diff simply looked boring, so it received ninety seconds of attention and a thumbs up.
What follows is an illustrative reconstruction rather than a named company's incident log, because the shape of the failure matters more than the logo on the dashboard. I assembled it from a pattern that repeats every time agents begin writing schema changes, and you should swap in your own timestamps and service names. The reusable part is the structure: a mechanically extracted timeline, contributing factors you can actually fix, and one durable gate that outlives the next agent.
Pull the timeline out of your tools, not your memory
Every postmortem written from memory is fiction with plausible timestamps, and you can feel the fiction when the first three events are all rounded to the nearest five minutes. The cheap fix is to dump raw events into a directory before anyone starts talking, while the deploy logs and agent session transcripts still exist. A thirty-line script run at the top of the incident buys you a timeline that survives scrutiny.
#!/usr/bin/env bash
# scripts/incident-timeline.sh <iso8601-since> [repo]
set -euo pipefail
SINCE="${1:?usage: incident-timeline.sh 2026-11-05T01:00:00Z [repo]}"
REPO="${2:-.}"
OUT="timeline/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"
# 1. every commit that touched migrations or agent config in the window
git -C "$REPO" log --since="$SINCE" --date=iso-strict \
--pretty=format:'%h|%ad|%an|%s' --name-status > "$OUT/commits.log"
# 2. the agent session that produced each of those commits
git -C "$REPO" log --since="$SINCE" --grep='Co-authored-by: agent' \
--pretty=format:'%h %B' > "$OUT/agent-trailers.log"
# 3. deploy and rollback events, whatever your platform calls them
if command -v gh >/dev/null 2>&1; then
gh run list --limit 50 --json databaseId,conclusion,createdAt,headSha \
> "$OUT/deploys.json"
fi
echo "wrote $(ls -1 "$OUT" | wc -l) artefacts to $OUT"
Run it first and read it second, because the ordering in commits.log usually contradicts the story everyone agreed on in the first ten minutes. In this reconstruction the agent's migration commit landed forty minutes before the patch that justified it, which is the detail that reframes the whole incident. You are not looking for a villain in the log; you are looking for a missing check.
The gate that would have caught it
The migration was correct on an empty schema and catastrophic on a populated one, and that gap is entirely mechanical to detect. Give the migration a scratch database seeded with production-shaped rows and a lock timeout that refuses to wait politely. If the statement cannot finish inside two seconds of lock acquisition on a realistic table, the gate fails and the agent gets a usable error message.
#!/usr/bin/env bash
# scripts/migration-gate.sh — runs in CI before any migrate job
set -euo pipefail
: "${SCRATCH_DATABASE_URL:?set SCRATCH_DATABASE_URL to a disposable postgres}"
psql "$SCRATCH_DATABASE_URL" -q -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'
psql "$SCRATCH_DATABASE_URL" -q -f seed/production-shaped.sql # row counts matter
for f in migrations/*.sql; do
echo "==> $f"
PGOPTIONS='-c lock_timeout=2s -c statement_timeout=30s' \
psql "$SCRATCH_DATABASE_URL" -v ON_ERROR_STOP=1 -q -f "$f"
done
echo 'gate: OK'
The seed file is the part people skip, and it is the only part that makes the gate meaningful. A thousand rows will never reproduce an eighty-million-row lock, so seed with cardinality that resembles your real tables even if the values are synthetic. Pair the script with a hard rule in your agent's project instructions so the constraint arrives before the diff does.
<!-- AGENTS.md -->
## Schema changes
- Any file under migrations/ requires a passing migration-gate run in the same PR.
- Never combine a DDL change and a data backfill in one migration file.
- If you cannot name the lock level of a statement, say so and stop.
Contributing factors worth naming out loud
The first factor is that CI proved the migration parses, and everyone quietly upgraded that to proved it works. The second is that the agent had no representation of row counts, so it could not reason about lock duration even in principle. The third is a review culture where a small diff reads as a low-risk diff, which is exactly backwards for schema changes. None of those are individual failures, and all three are fixable in a week.
| Question | Honest answer in the incident | Cost of fixing |
|---|---|---|
| Did CI run the migration on realistic data? | No, empty schema | One seed file, one job |
| Did the agent know the table size? | No | Ten lines in AGENTS.md |
| Was the reviewer told what to look for? | No | One checklist line |
The durable fix, and what it actually costs
The fix I would carry forward is unglamorous: the gate above runs on every pull request that touches migrations/, and the PR template refuses to render unless the check is green. That converts a judgment call into a mechanical one, which is what you want when the author might be an agent at two in the morning. The cost is a few minutes of CI time and a seed file that drifts unless someone owns it, so name an owner in the runbook.
There is a second, quieter fix that matters just as much, and it lives in the postmortem template rather than the pipeline. Require the timeline section to cite artefact paths instead of prose, so the next reader can rerun incident-timeline.sh and see the same ordering. Postmortems that cannot be reproduced turn into folklore within a month, and folklore is how the same migration gets reverted into production six weeks later.
Where an agent workspace fits into this loop
The tedious middle of this workflow is turning a directory of logs into readable prose, and rewriting gate error messages until a tired engineer can act on them at 03:00. That is genuinely agent-shaped work: bounded inputs, verifiable output, no production access required. MonkeyCode is an open source project that offers free model access alongside a free server option, which makes it a reasonable place to draft the timeline narrative and iterate on the gate script without spending anything to find out whether the loop suits you. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I have not independently verified the free token allowance the outreach material quotes, the list of models behind that access, or what the free server actually provides in terms of hardware and lifetime, so confirm those details on the project page before you plan capacity around them. The honest framing is that the free tier lowers the cost of trying the workflow, not that it replaces your CI runner or your incident tooling. Run the gate on your existing infrastructure and use a free workspace for the parts that are read-only and disposable.
Who should not do this
If your platform refuses lock_timeout, or your migrations run through a managed service that owns the connection, the gate script needs rewriting before it means anything. If you are in a regulated environment where agent-authored changes require formal attestation, a free workspace is the wrong place for anything touching evidence. And if your team cannot maintain a seed file, adding a gate will only manufacture a new class of green check that lies to you.
Start with the timeline script on your most recent incident and see whether the extracted ordering matches what you remember. If it does not, you have found the first thing worth fixing, and it will not cost you a migration outage to learn it.
Top comments (0)