Five ways my coding agents broke the deploy, and the guardrails that stuck

I let coding agents ship to my own production — a small internal ops app I use every day. They write code, run the gates, and deploy. It mostly works, and I would not go back.

But the failures are not the ones I expected. None of them were "the AI wrote bad code." They were all about concurrency, state, and who is holding what. Here are five that actually happened, with the guardrail that fixed each.

1. A deploy stamped a commit it never built

The deploy script finished with:

git rev-parse HEAD > .next-deploy-sha

The intent: record what is running, so the next deploy can skip a rebuild if nothing changed.

The bug: a build takes 3–4 minutes. If another agent commits during the build, HEAD has moved by the time you write the stamp. You record a sha you never built. The next deploy sees stamp == HEAD, says "already running this sha," and exits in two seconds — and that commit is never built, ever.

Symptom: "the deploy succeeded but the new feature isn't there / the new endpoint 404s." I lost a whole feature to this before I understood it.

Guardrail: capture the sha at the start and write that one.

HEAD_SHA="$(git rev-parse HEAD)"   # at the top
# ... build ...
echo "$HEAD_SHA" > .next-deploy-sha

The general rule: a stamp must describe the input you used, not the state of the world when you finished.

2. git add -A swallowed another agent's half-finished work

Two agents, one working tree. I committed "my" change with git add -A src tests and pulled in 153 files from an in-progress refactor. The gates failed twice (an untracked script, then a dozen lint errors), and for a while nobody could deploy.

Recovery is easy if you know it — git reset --soft <your previous commit> then git reset, and re-add your files individually. The working tree is untouched, so the other agent loses nothing.

Guardrail: in a repo where more than one agent works, git add -A and git add <dir> are banned. List files explicitly. It feels tedious for about a day and then it feels normal.

3. One agent's lint error blocked everyone's deploy

A deploy failed on a gate that had nothing to do with my change: 156 untranslated UI strings from a different feature, already on HEAD.

That is what a repo-wide gate does. It is not "your change is bad," it is the repository is un-deployable until someone fixes an unrelated thing. With one human that is a nuisance. With several agents shipping in parallel it is a traffic jam, and the agent that hits it usually has no idea why.

Guardrail: two parts.

• Make the gate runnable per-feature so it can be cleared before committing (node scripts/i18n-scan.mts MyNewView must print 0).
• When you hit someone else's gate failure, fix it rather than route around it. Routing around means disabling the gate, and then it protects nothing.

4. Builds queued behind a lock nobody owned

One deploy at a time, enforced with a lock directory. Two failure modes showed up within a week:

• A deploy killed mid-flight (PC reboot, process reaper) leaves the lock behind. Every later deploy aborts. I found a three-day-old lock blocking everything.
• A trivial change (one JSON data file, no commit) still took 16 minutes, because it waited on someone else's build and then rebuilt the identical sha.

Guardrail:

• Take over a stale lock. A deploy takes ~4 minutes, so a lock older than 30 minutes has no live owner. Steal it.
• Wait instead of aborting. Polling every 5 seconds up to a cap turns "failed, retry manually" into "slow but finished."
• Skip if the sha is already running and the service answers — now safe, because of guardrail 1.

5. The agent killed itself (twice, in different ways)

My favourite category.

Self-kill A. An agent ran the test suite. One test started the real process-manager binary. That manager's startup routine reaps orphaned child processes by reading a PID file — the production PID file. It dutifully killed every live session, including the agent running the test. Not a crash. Suicide by passing its own test.

Fix: the PID file records the owning process, the reaper refuses to touch children of a live owner, it never kills its own ancestors, and tests point at a scratch file through an env var.

Self-kill B. The reverse: force-killing a process tree left powershell.exe children re-parented to the system. They accumulated for a week — 112 orphans, 1.2 GB, 6 hours of CPU — until "the PC is slow" became the symptom.

Fix: write live child PIDs to disk, verify the names at startup, and clean them up.

Guardrail: any test that can start a real daemon must be pointed at scratch state by default. And before blaming the model for "the PC is slow," check for processes whose parent is dead.

The bonus failure: nobody noticed it was broken

Four agent tasks in a row returned "failed, exited 1." The code was fine. The CLI's OAuth session had expired, and every feature that used it — drafts, reports, translations, reviews — failed in its own way with its own unhelpful message.

This is the real hazard of autonomy: silent, correlated failure. Nothing crashed. The dashboard looked normal. Work just stopped happening.

Now there is an explicit check before accepting any agent job, a banner with a re-login button, and the failure hint names the cause. Same for browser sessions: a job every two hours actually opens the site and confirms the session is alive, instead of checking that a profile directory exists. A directory exists long after a session dies — the panel said "ready" for days while nothing was being posted.

What I'd tell myself at the start

1. The risk is not bad code. The gates catch bad code. The risk is shared mutable state: one tree, one lock, one PID file, one browser profile.
2. Every stamp, lock, and cache needs an answer to "what if the owner died?"
3. Verify liveness, not the existence of an artifact. A file, a folder, or a running process name is not proof that the thing works.
4. Make failures loud and correlated-aware. If six features stopped at once, the shared dependency is the bug — say so in the message.

None of this is AI-specific. Agents just removed the human who used to be sitting between the mistake and the --force.