The alert fired at 3:14 AM, and your phone lit up the hotel room like a small sun. The error rate on /account/merge had climbed from 0.2% to 34% in eleven minutes, and the background retry queue was growing faster than the workers could drain it. You opened the deploy history, and there it was: a patch merged nine hours earlier, a one-line change with the title "fix case-sensitive email duplicates." It had passed review in under a minute.
The patch was reasonable in isolation. A support ticket complained that two accounts could be created with User@Example.com and user@example.com, so the AI coding agent added email.strip().lower() to the shared user helper before the uniqueness check. The signup tests passed, the diff was small, and the reviewer approved it without opening the merge endpoint. The problem was that the same helper also served the legacy account-merge path, where those two addresses had legally coexisted for years as separate accounts. After the deploy, every merge attempt hit a unique-constraint violation, and the retry loop made it worse by replaying the same failing request against the database.
The DEV community has been arguing all week about what happens when AI writes most of the code and humans only review it. This incident is a concrete answer: the reviewer becomes the bottleneck, and the bottleneck becomes the incident. The timeline below is a composite, but the pattern is common enough that you have probably lived a version of it.
| Time (UTC) | Event |
|---|---|
| 18:02 | Patch merged: email normalization added to a shared helper |
| 18:40 | Deployed to production |
| 03:14 | Alert: error rate on /account/merge above threshold |
| 03:22 | On-call rolls back the deploy |
| 03:41 | Error rate returns to baseline |
| 09:15 | Postmortem: root cause confirmed and regression test written |
Three contributing factors turned a small patch into a 3 AM incident. First, the agent edited a shared helper, but the review only looked at the signup call site, so the blast radius was invisible in the diff. Second, there was no regression test for the legacy merge path, which meant every automated check blessed the change as safe. Third, the retry job had no circuit breaker, so a deterministic failure became a self-amplifying one. The reviewer was not lazy; the reviewer was the only gate, and the gate was a human eyeball scanning a diff.
The durable fix was not to ban AI-generated patches. The durable fix was to build a gate that tests a patch before a human ever reads it. We started by writing a failing test that reproduced the incident:
# test_regression_merge.py
def test_legacy_merge_survives_case_normalization(client):
legacy = client.create_user("User@Example.com")
duplicate = client.create_user("user@example.com")
# Both accounts existed before normalization was introduced.
response = client.post("/account/merge", json={
"primary": legacy.id,
"secondary": duplicate.id,
})
assert response.status_code == 200
Then we built a small replay harness that applies any candidate patch in a clean worktree and runs the regression suite:
#!/usr/bin/env bash
# replay_incident.sh — apply a candidate patch and run the regression suite
set -euo pipefail
git worktree add /tmp/verify HEAD
cd /tmp/verify
git apply /tmp/candidate.patch
pytest -q tests/test_regression_merge.py tests/test_signup.py
The key insight is that the harness turns the AI agent into a proposer rather than a committer. The agent generates a candidate patch, the harness applies it, the tests decide, and only the patches that pass ever reach a human reviewer. This is where the economics matter, because a gate like this only works if you can afford to run it many times.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding agent that offers free model access and a free server option, which means you can run this loop without paying for model access or standing up your own server. The free tier includes 10 million tokens, enough for a meaningful number of candidate patches against a focused regression suite, and the free server option lets you run the agent's backend in your own environment, close to your CI and your data.
The loop itself is a few lines of glue code:
# gate.py — keep only candidate patches that pass the regression suite
from pathlib import Path
import subprocess
def patch_passes(patch_text: str) -> bool:
Path("/tmp/candidate.patch").write_text(patch_text)
result = subprocess.run(
["bash", "replay_incident.sh"],
capture_output=True,
)
return result.returncode == 0
Every candidate that returns True is a patch you can review with context instead of suspicion. Every candidate that returns False is a lesson the agent can consume before you ever see it. Over a few weeks, the gate changes the review conversation from "does this look right?" to "what did the tests miss?", which is a much better question to spend your attention on.
This gate is only as strong as the tests behind it. If your suite does not cover the legacy path, the harness will bless the same broken patch with a green checkmark, so the first step is always to encode the incident as a failing test. The 10 million free tokens are a starting budget, not an infinite resource, and you still want to keep prompts focused and tests fast. The free server option is a development convenience, not a production SLA; if you need guaranteed uptime or regulated-data isolation, plan accordingly. And you should not use this workflow at all if your project has no automated tests, because the gate would be reviewing nothing but the agent's formatting.
The next time an alert wakes you up, the question is not whether the AI wrote a bad patch; it is whether your pipeline gave that patch a free pass. A replay harness costs an afternoon and turns every future candidate into a testable hypothesis. If you want to try the loop with a real agent, MonkeyCode's free tier — 10 million tokens and a free server option — is a low-risk place to start.
Top comments (0)