Replay the Rename Before Production

You open the incident channel just after two and learn a migration already rewrote a column three services still treat as text. The assistant that drafted the change sounded certain in the pull request, yet nobody replayed it on a database that was safe to break. By the time the statement rolls back, a retry worker has written the old shape again and the dashboard is a stack of type errors. That night becomes a useful postmortem, because the failure was a missing gate rather than a mysterious change in model mood.

This walkthrough treats that night as a composite you can reproduce, and it does not claim a measured customer incident or a public benchmark. You will build a small replay gate that classifies a prompt, runs it on an isolated copy, and blocks production until that copy agrees. The gate is deliberately boring, which is the point, because excitement let the first change skip the only environment that could say no. If you already keep a staging database and a human approver, leave them in place and treat this latch as an extra check.

What the clock showed

The timeline starts at 01:40, when a developer asks an assistant to rename a status column and backfill an enum together. At 01:52 the generated SQL lands in review with a confident note and no mention of the worker that still inserts the old string. At 02:06 someone merges because the diff looks short and the tests only cover the happy path that lives in memory. At 02:14 production applies the change, the worker retries the old shape, and the type errors start to pile up.

Detection stays slow because the error budget watches latency rather than schema drift, so a person finds it on a broken chart. Rollback begins at 02:31, but the down migration assumes the new column is empty, and that assumption is already false. By 02:48 you have restored the old column and you are replaying failed jobs by hand from a log never designed for that work. The incident closes near 04:10, with the rows repaired and with trust thinner than the short commit message had suggested.

What actually failed

Three contributing factors sit under that clock, and none of them is a story about a model turning malicious. The prompt mixed a rename, a backfill, and a deploy instruction, so one answer carried three different risk levels at once. Review then happened against production-shaped confidence instead of a copy that could fail without paging a human. The cheap path and the careful path were also glued together, which made a second look feel like a luxury rather than a gate.

The durable fix is a replay gate you run before merge, and a hosted assistant is allowed only for the classification step. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator presents MonkeyCode as an open-source option with free model access and a free server for this isolated replay. You should treat those two availability claims as the only product facts here, and you should confirm them again before you depend on them.

You ask the free model path for a risk label, then you paste that label with the SQL into a local script. An unsafe label does not block you alone, because a classifier can be wrong, but it does force the statement onto the isolated copy. That free server is a place you can break on purpose, not a promise about hardware size, region, or how long the offer lasts. If the copy is unavailable, the script exits without printing a production command, which is the behavior you want at two in the morning.

The latch you can rerun

The artifact below is proposed example code, and you should treat it as unexecuted until you run it yourself on a disposable database. It writes a small JSON log and prints a psql command aimed only at the URL you explicitly mark as the replay copy. A local keyword check disagrees toward the stricter label, so a cheerful model answer cannot outvote a statement that starts with drop. The script never calls a product API, because no client or route is established here, so you paste the label yourself.

#!/usr/bin/env python3
"""Proposed replay gate. Unexecuted example: use it only on a disposable database."""

import datetime
import hashlib
import json
import os
import sys

RISKY = ("alter", "drop", "truncate", "delete", "update", "grant", "revoke", "rename")
RANK = {"read-only": 0, "reversible": 1, "unsafe": 2}


def local_risk(sql: str) -> str:
    folded = " ".join(sql.strip().lower().split())
    if folded.startswith("select") or folded.startswith("explain"):
        return "read-only"
    tokens = folded.split()
    if any(token in tokens for token in ("drop", "truncate")):
        return "unsafe"
    if any(token in tokens for token in RISKY):
        return "reversible" if folded.startswith("update") else "unsafe"
    return "unsafe"


def main() -> int:
    try:
        doc = json.loads(sys.stdin.read())
    except json.JSONDecodeError:
        print("refused: input must be JSON with sql and prompt", file=sys.stderr)
        return 2
    sql = doc.get("sql", "")
    label = doc.get("model_label", "unsafe")
    local = local_risk(sql)
    final = local if RANK.get(local, 2) >= RANK.get(label, 2) else label
    digest = hashlib.sha256(sql.encode()).hexdigest()[:12]
    os.makedirs("replay-log", exist_ok=True)
    record = {
        "at": datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0).isoformat(),
        "digest": digest,
        "model_label": label,
        "local_label": local,
        "final": final,
    }
    with open(os.path.join("replay-log", digest + ".json"), "w", encoding="utf-8") as handle:
        json.dump(record, handle, indent=2)
        handle.write("\n")
    if final == "read-only":
        print(sql)
        return 0
    if not os.environ.get("REPLAY_DATABASE_URL"):
        print("refused: set REPLAY_DATABASE_URL to the isolated server", file=sys.stderr)
        return 3
    ack = os.path.join("replay-log", digest + ".ack")
    print("-- replay only on the isolated server")
    print(f"-- digest={digest} label={final}")
    print('psql "$REPLAY_DATABASE_URL" -v ON_ERROR_STOP=1 -c ' + json.dumps(sql))
    if not os.path.exists(ack):
        print(f"refused: write {ack} after a human checks the replay", file=sys.stderr)
        return 4
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Save it as replay_gate.py and pass JSON on standard input, including the label from the free model path you use for classification. The sample below is a fixture, not a production statement, and the expected result is a refusal until REPLAY_DATABASE_URL points at the isolated server. You export that variable to the free server connection string, run the script again, and only then read the printed psql line. You run that printed line yourself, you do not let the script execute it, and you ack the digest before anyone mentions production.

python3 replay_gate.py <<'JSON'
{"sql":"SELECT status FROM orders WHERE id = 1;","prompt":"inspect one row","model_label":"read-only"}
JSON

python3 replay_gate.py <<'JSON'
{"sql":"ALTER TABLE orders RENAME COLUMN status TO status_code;","prompt":"rename and deploy","model_label":"read-only"}
JSON

export REPLAY_DATABASE_URL="postgres://replay:[e-mail]/replay"
python3 replay_gate.py <<'JSON'
{"sql":"ALTER TABLE orders RENAME COLUMN status TO status_code;","prompt":"rename and deploy","model_label":"read-only"}
JSON

digest=$(python3 - <<'PY'
import hashlib
sql = "ALTER TABLE orders RENAME COLUMN status TO status_code;"
print(hashlib.sha256(sql.encode()).hexdigest()[:12])
PY
)
printf '%s\n' "reviewed by the on-call owner" > "replay-log/${digest}.ack"
python3 replay_gate.py <<'JSON'
{"sql":"ALTER TABLE orders RENAME COLUMN status TO status_code;","prompt":"rename and deploy","model_label":"read-only"}
JSON
echo "exit=$?"
cat "replay-log/${digest}.json"

The printed command assumes a Postgres copy, so you swap the client if your isolated server speaks something else. You still keep the digest, the label, and the refusal, because those are the parts that survive a different engine. A green exit code means the ack file existed, not that the statement was wise, so you read the SQL again. The placeholder host is only an example, and it is not a documented product address.

When you open the log you should see the model label, the local label, and a final label that kept the stricter one. That record is the postmortem artifact you lacked at 02:06, because it separates a confident label from permission to touch a database. If the final label is unsafe, the process exit code stays non-zero until a human writes an ack file named after the digest. You can wire that ack into your existing review, and you should not let a green unit test stand in for it.

This latch answers the first contributing factor by forcing every mixed prompt through a label you can store next to the SQL. It answers the second by moving the first execution onto a copy whose failure wakes nobody except you. It answers the third by letting the free model path do the cheap classification while the free server absorbs the blast radius. None of those answers removes the need for a backup, a lock, or a person who can read the statement.

Imagine the next prompt is a plain select that a nervous model labels unsafe because the table name sounds like a migration. The stricter vote withholds a bare production print for that select, which is annoying and still cheaper than another 02:14. You add an override only by writing the ack file, and you put a person's name in it for the next reviewer. That small friction is the durable part, because a postmortem that only retells the story will not stop the next short diff.

What this still will not do

The keyword check is crude, and it will miss dynamic SQL, stored procedures, and comments that hide a drop behind a polite select. The script opens no transaction, takes no lock, and only tests that an ack file exists rather than who signed it. Free model access can drift, refuse, or label badly, and nothing here states a quota, model name, hardware shape, or lasting offer. If you need those details, read the current project documentation and treat this page as a workflow, not as a status page.

You should not use this approach when the SQL holds regulated rows, because classification still means sending that text to a model path. You should not use it without a disposable copy, or if that copy might still hold user rows you cannot afford to expose. You should not use it as your only change control when an audited approval board must remain the system of record. You should also skip it for multi-statement files until you split them, because one digest hiding three actions recreates the original mixed prompt.

A postmortem earns its keep when the next merge is harder to rush, and this gate prints a refusal instead of a command. Keep the timeline in the review note, keep the log beside the diff, and keep the human ack as a named file. If you want to try the classification step, confirm the current offer, then point the replay URL at a free server you can break.