When Your Staging Agent Refused to Stop

You notice the invoice alert at 2:14 a.m., long after the staging deploy that was supposed to be a quiet smoke test. The agent was only meant to reproduce one flaky checkout failure, then stop and leave a note. Instead it kept opening new turns against the paid endpoint, rewriting the same migration, and writing rows into a shared staging database. By morning the budget gate is already red, and the original checkout bug is still sitting there unfixed.

Treat the clock times below as a reconstructed drill, not as telemetry from a named production outage. You can replay the same failure on a laptop if you give an agent a shared host, a write tool, and no spend gate. The damage is not mysterious, because exploratory retries look productive until the invoice and the database both disagree. What you need afterward is a durable fix, not another recap of how surprising agents can be.

The night, as a drill

The drill begins at 21:06, when you ask the agent to reproduce a failed payment webhook on staging. At 21:11 the isolated sandbox is unreachable, so the agent falls back to the shared staging host that other people are using. At 21:18 the first model call is billed at the frontier rate, because nothing separates exploration from confirmation. At 21:40 the loop is still running, and nobody has written a stop condition the agent must obey.

At 22:05 the agent decides the webhook fixture is wrong and starts altering rows that a teammate still needs for a demo. At 23:12 a retry policy treats every model timeout as a reason to open another paid turn. At 01:02 the migration script runs a second time, and the unique constraint errors become new prompts instead of a hard stop. At 02:14 the invoice alert fires, which is the first signal that reaches a human who can actually halt the job.

A useful postmortem names the path that kept the loop alive, then changes that path so the next run fails closed. You are not looking for a villain in the model, because the model did what an unbounded tool loop allows. You are looking for missing gates: an isolated host, a cheap exploratory lane, a confirm step, and a human stop. If those gates are optional, the same night will happen again under a different ticket number.

What kept the loop alive

The first contributing factor is the shared staging host, which felt convenient because the sandbox login had expired. A shared database is a hallway, not a lab, and an agent with write tools will leave footprints you did not order. Your teammate's demo rows were not part of the bug, yet they became part of the experiment. Isolation is the difference between a reproduction and an incident that other people have to unwind.

The second contributing factor is a missing split between cheap exploratory turns and later paid confirmation work. Exploration is allowed to be wrong, so it should not automatically spend the frontier rate or touch durable data. Confirmation starts only after you have a failing command, a captured error, and a reason to spend more. When every turn looks like confirmation, your router is just a hose with the tap left open.

The third contributing factor is unrestricted write access without a stop condition that a script can enforce. A prompt that says do not mutate staging is a wish, and a retry library will not read your incident channel. Timeouts became new work because the policy treated silence as uncertainty rather than as a halt. You want the process to exit non-zero when the sandbox is missing, the lane is wrong, or the same error repeats.

There is a fourth factor that is easy to miss while you are still annoyed about the invoice. The agent had no artifact that proved the bug was real before it started proposing schema changes. A failing curl, a saved response body, and a one-line assertion would have been enough to end the exploratory phase. Without that captured artifact, every new idea looked like progress, and progress was allowed to spend money.

The gate that fails closed

The durable fix is a small gate you run before the agent, not a longer prompt you hope it will respect. The gate demands an isolated host, classifies the turn as explore or confirm, and refuses a paid endpoint during exploration. It also stores the last error fingerprint, so a repeated failure stops the loop instead of funding another rewrite. You can keep the agent, but you remove its ability to choose the expensive path by default.

Put the gate in the repository next to the webhook fixture, and call it from the same command you would trust in a drill. Environment variables hold the sandbox host and the two endpoints, so the script never needs a secret written into the file. If a variable is missing, the shell fails immediately, which is the behavior you wanted at 21:11. The Python that follows is a proposed check, and you should run it locally before you trust it in a shared pipeline.

The placeholders below are not live product URLs, and you replace them only after you have verified the hosts you intend to use. The shell fragment fails closed when a variable is empty, which is stricter than a default that quietly selects the paid lane. You can commit the script, but you should not commit filled-in endpoints, tokens, or a hostname that identifies a private network.

export SANDBOX_HOST="${SANDBOX_HOST:?set an isolated sandbox host}"
export EXPLORE_MODEL_ENDPOINT="${EXPLORE_MODEL_ENDPOINT:?set exploratory model access}"
export CONFIRM_MODEL_ENDPOINT="${CONFIRM_MODEL_ENDPOINT:?set confirmation model access}"
export TASK_LANE="${TASK_LANE:-explore}"
python3 scripts/agent_gate.py

#!/usr/bin/env python3
"""Proposed pre-agent gate. Unexecuted until you point it at your own hosts."""
import hashlib
import os
import sys

def must(name: str) -> str:
    value = os.environ.get(name, "").strip()
    if not value:
        sys.exit(f"missing {name}")
    return value

def main() -> None:
    host = must("SANDBOX_HOST")
    lane = must("TASK_LANE")
    explore = must("EXPLORE_MODEL_ENDPOINT")
    confirm = must("CONFIRM_MODEL_ENDPOINT")
    if host in {"staging.internal", "shared-staging"}:
        sys.exit("refuse shared staging; set an isolated SANDBOX_HOST")
    if lane not in {"explore", "confirm"}:
        sys.exit("TASK_LANE must be explore or confirm")
    if lane == "explore" and explore == confirm:
        sys.exit("explore lane must not reuse the confirmation endpoint")
    fingerprint = hashlib.sha256(sys.stdin.read().encode()).hexdigest()[:12]
    prior = os.environ.get("LAST_ERROR_FINGERPRINT", "")
    if prior and prior == fingerprint:
        sys.exit(f"repeated error {fingerprint}; halt instead of retrying")
    endpoint = explore if lane == "explore" else confirm
    print(f"lane={lane} host={host} endpoint={endpoint} error={fingerprint}")

if __name__ == "__main__":
    main()

You pipe the last error into the script, and a repeated fingerprint forces a non-zero exit before another model call. An explore lane that points at the confirmation URL also fails, so a copy-paste mistake cannot become an overnight bill. The shared-host check is deliberately blunt, because a hostname rule is easier to audit than a paragraph of agent instructions. Run the command twice on the same error and confirm that the second run stops.

This drill is written so you can execute it, and the fingerprint comes from your own first run rather than from a published number. The first command prints the lane, the host, and a short error id when the environment is complete. The second command passes that id back in, which is how the gate recognizes a loop instead of a new clue. If the second command still exits zero, you have not installed the stop you needed at 01:02.

export SANDBOX_HOST="sandbox-drill.example"
export EXPLORE_MODEL_ENDPOINT="https://explore.example/v1"
export CONFIRM_MODEL_ENDPOINT="https://confirm.example/v1"
export TASK_LANE="explore"
fp=$(printf 'webhook timeout on shared staging\n' | python3 scripts/agent_gate.py | sed -n 's/.*error=//p')
printf 'webhook timeout on shared staging\n' | LAST_ERROR_FINGERPRINT="$fp" python3 scripts/agent_gate.py
echo "second exit=$?"
SANDBOX_HOST="shared-staging" python3 scripts/agent_gate.py <<< 'webhook timeout on shared staging'

Before the agent is allowed to suggest a migration, you capture one failing request and leave the body on disk. That file is the confirmation ticket: if it does not exist, the lane stays on explore and the write tools stay unused. A second curl that returns the same body should not authorize a schema change, because sameness is the fingerprint idea in plain sight. You are building a brake for the loop, not a faster way for the agent to edit the database.

curl --fail-with-body --max-time 10 \
  -H "Content-Type: application/json" \
  -d '{"event":"payment.failed","id":"evt_drill"}' \
  "http://${SANDBOX_HOST}/hooks/payment" \
  | tee /tmp/webhook-body.txt
grep -q 'payment.failed' /tmp/webhook-body.txt

For the exploratory lane, you can point EXPLORE_MODEL_ENDPOINT at MonkeyCode's free model access rather than at the confirmation URL. You can also point SANDBOX_HOST at its free server option when you need a machine that is not the shared staging host. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims are the only product facts this drill relies on, so confirm current limits before you depend on them.

Do not treat a free lane as a promise about model names, quota size, hardware, or how long the offer stays available. If the free path is rate limited or temporarily closed, the gate should fail closed rather than silently hopping to the paid endpoint. You still keep CONFIRM_MODEL_ENDPOINT for the moment when a captured failing test deserves a stronger model. The point is sequencing, not a claim that one vendor is universally cheaper or better than another.

Replay the drill with the gate in front, and the 21:11 fallback never reaches the shared database. The 21:18 turn stays on the exploratory endpoint, so a wandering reproduction does not inherit the frontier rate. The 01:02 repeated constraint error matches the stored fingerprint and exits before the agent funds another rewrite. The 02:14 invoice alert becomes unnecessary for this class of loop, because the process already stopped itself.

What this will not save you from

This gate does not understand your business, and a hostname denylist will miss a shared host that uses a new name. It will not stop a human who exports the paid endpoint into the explore variable and then ignores the failure. It also will not protect customer data if you paste production secrets into a free-tier prompt or a sandbox you do not control. Billing alerts, database backups, and least-privilege credentials still matter, because a script cannot be your only lock.

You should not use this approach for production writes, regulated data, or any incident that requires an on-premises model with a contractual data boundary. You should also skip it if your team cannot review environment variables, because a gate you do not understand is just another moving part. A free server is a poor place for long-running load tests or for data you are not allowed to upload. If the bug is already reproduced locally, do not add a remote agent loop just to make the postmortem look thorough.

After the drill, write the timeline into the ticket so the next on-call does not rediscover the same hole. Keep the exploratory lane boring, the sandbox disposable, and the confirmation step rare enough that you can explain each spend. If you want those two free options behind the variables, read the current docs and keep the gate fail-closed when either one is missing. The night gets shorter when the loop has to ask for a host before it asks for another token.