Classify the Failure Before You Retry the Agent

Friday, 16:40. A checkout service. An agent had one job: stop a tax helper from rounding 0.5 in two different directions depending on the runner locale.

The first run produced a 41-line diff and a red CI job. The operator hit retry. The second run produced a different 41-line diff. Also red. The model was not the fault. The fixture assumed TZ=UTC, and the sandbox did not export it.

Retry felt cheap. It hid the class of failure. The next run should not start until that class has a name.

What retry actually spends

A second agent pass is not a free coin flip. It spends context, reviewer time, and a clean execution slot. If the failing signal is an unset timezone, another sample from the same prompt distribution will not invent TZ.

Treat run N as evidence. Treat run N+1 as a policy. The policy depends on a failure class, not on how confident the last assistant message sounded.

This article is a glossary, a four-leaf tree, and a small classifier you can run against a frozen receipt. The workflow still holds if you never leave your laptop.

Glossary

Use these terms as they are written. Do not collapse them into “the model was bad.”

1. Receipt. The frozen record of one run: prompt hash, tool calls, exit codes, test output, env names (not values), and the diff. No receipt, no retry.
2. Failure class. The category of why the artifact was unacceptable. One class per run. Mixed classes mean the receipt is incomplete.
3. Transient fault. The repo and the oracle were fine. The runner died, DNS failed, or a rate limit tripped. Retry can be valid after a backoff.
4. Oracle mismatch. Tests, linters, or hidden CI gates encode a requirement the ticket never stated. Another generation will wander.
5. Environment drift. Missing toolchain, wrong cwd, unset locale, absent system package, or a permission the sandbox does not grant.
6. Spec gap. The ticket left an output unconstrained. The agent invented a requirement. Retry without tightening the ticket repeats the invention.
7. Secret surface. Env files, tokens, customer fixtures, or staging credentials that must not leave the trusted host.
8. Isolation boundary. The machine allowed to execute tools: laptop, CI job, or a disposable remote runner.
9. Loop halt. The condition that forbids iteration N+1. Halt is a decision, not a vibe.
10. Regenerative retry. Starting over with the same class, same oracle, and same environment. This is the default anti-pattern.

If a word in that list does not appear in the receipt, you cannot claim the matching leaf.

Capture a receipt before you touch the prompt

Do this on the failing run. Do it even when the UI already shows a stack trace.

1. Create a run directory that cannot be overwritten by the next attempt.
2. Store command, cwd, and non-secret env names.
3. Store stdout, stderr, and the test command’s exit code.
4. Store the diff as a file, not as chat scrollback.
5. Hash the prompt file so “we changed nothing” is checkable.

# Example receipt capture. Label: unexecuted template.
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
REC="receipts/${RUN_ID}"
mkdir -p "${REC}"

cp prompt.md "${REC}/prompt.md"
shasum -a 256 "${REC}/prompt.md" > "${REC}/prompt.sha256"

# Names only. Do not dump values; secrets leak through receipts too.
env | awk -F= '{print $1}' | sort > "${REC}/env.names"

{ echo "cwd=$(pwd)"; echo "cmd=${TEST_CMD:-pytest -q}"; } > "${REC}/meta.txt"

set +e
${TEST_CMD:-pytest -q} > "${REC}/test.stdout" 2> "${REC}/test.stderr"
echo $? > "${REC}/test.exit"
set -e

git diff --stat > "${REC}/diff.stat"
git diff > "${REC}/diff.patch"

A receipt is small. A missing receipt is how regenerative retry gets a second chance it did not earn.

The tree: four leaves, one retry policy

Walk the questions in order. Stop at the first yes.

1. Is there a receipt for run N? If no: halt. Capture the receipt. Do not prompt again.
2. Did the failing signal come from outside the repo? Runner kill, network, rate limit, disk full. If yes: Leaf A — transient.
3. Did a test, linter, or CI gate fail on a rule the ticket never named? If yes: Leaf B — oracle or spec.
4. Did a command fail because of cwd, locale, missing binary, permissions, or an unset env name? If yes: Leaf C — environment or isolation.
5. Otherwise: Leaf D — task or context. Only this leaf may justify a new model pass, and only after the task is shrunk.

The tree does not ask whether the assistant “sounded sure.” Confidence is not a side-effect class, and it is not a failure class either.

Leaf A — Transient: retry the runner, not the prompt

Worked example. Receipt test.stderr ends with Could not resolve host: pypi.org. test.exit is 1. diff.patch is empty. The oracle never ran.

Policy.

1. Record the external signal in REC/class.txt as transient.
2. Do not edit the prompt.
3. Retry the same command on the same isolation boundary after a backoff, or on a runner with network that the ticket actually needs.
4. Halt if the same host error repeats twice. Promote the network dependency into the ticket or mock it.

# receipts/20260914T164012Z/class.txt
class=transient
signal=dns_pypi
retry_prompt=no
retry_runner=yes
halt_after=2

A new sample from the model will not fix DNS. Spending a generation here is regenerative retry dressed as diligence.

Leaf B — Oracle or spec: freeze the rule, then regenerate

Worked example. The ticket said “round half up for tax lines.” CI runs ruff with a repo-local plugin that forbids Decimal.quantize unless a comment nonce is present. The agent’s diff is arithmetically right and style-illegal. Run two, without reading the plugin, swaps in round() and fails the property test.

Policy.

1. Quote the failing gate in the ticket. One sentence. One path.
2. If the gate is wrong, change the gate on a separate commit. Do not ask the agent to guess which authority wins.
3. If the ticket was thin, add the missing example (0.5 -> 1 on a single line item) before the next model pass.
4. Halt if you cannot point to a file that encodes the rule. No file means spec gap, not “try a longer prompt.”

# Added to the ticket, not to chat memory
Oracle: `tests/unit/test_tax_round.py::test_half_up_on_line`
Forbidden substitute: builtin round()
CI gate: ruff plugin `tools/ruff_quantize.py` (comment nonce not required for tax.py)

Retry without an oracle edit is how two red diffs accumulate with no shared thesis.

Leaf C — Environment or isolation: name the host constraints

Worked example. test.stderr contains locale.getencoding() -> 'ISO-8859-1' and a fixture compares against a UTF-8 golden file. env.names has no TZ and no LC_ALL. The patch “fixes” the helper by encoding ASCII. That patch is a lie about production, where the service runs under C.UTF-8.

Policy.

1. Write the required env names and toolchain into the receipt and the job spec.
2. Re-run the existing patch under that environment before asking for a new patch.
3. If the laptop cannot reproduce CI, change the isolation boundary. Do not compensate with a cleverer prompt.
4. Halt if the workload has a secret surface. A disposable runner is not a vault.

This is the leaf where a free remote server is a method, not a slogan. After the missing names are listed, you need a machine that is not your laptop and not production. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option are one way to rent that disposable isolation boundary once the receipt already names the drift. They do not classify the failure for you, and they do not belong in Leaf B while the oracle is still unsigned.

# Re-evaluate the SAME diff under declared constraints.
# Label: unexecuted example.
export TZ=UTC LC_ALL=C.UTF-8
git apply receipts/20260914T164012Z/diff.patch
pytest -q tests/unit/test_tax_round.py

If the old diff turns green under the named env, keep it. Do not generate a third.

Leaf D — Task or context: shrink, then spend a model pass

Worked example. Receipt shows tests and env are fine. The agent rewrote a 900-line pricing module because the ticket said “fix rounding.” The diff mixes tax, discounts, and a drive-by rename. Review cannot bind the change to the oracle.

Policy.

1. Cut the allowed path set. One directory, or one function, in the ticket.
2. Cap the diff. If git diff --stat exceeds the cap, reject without reading the prose in the assistant message.
3. Only now start run N+1, on an isolation boundary that matches Leaf C’s constraints.
4. Halt if run N+1 still crosses the cap. That is a scoping failure, not a sampling failure.

allowed_paths=
  src/checkout/tax.py
  tests/unit/test_tax_round.py
max_diff_lines=80
retry_prompt=yes_after_shrink

Leaf D is the only leaf that should spend another model call. Free model access is relevant here after the shrink, because the cost you are protecting is review attention, not a leaderboard score.

Artifact: classify a receipt directory

The script below is a heuristic. It is not an incident commander. Feed it a receipt path; it prints a leaf and a halt flag. Label: example code, not a measured production evaluator.

#!/usr/bin/env python3
"""Classify one frozen agent receipt. Proposal / example."""
from __future__ import annotations

import argparse
import re
from pathlib import Path

TRANSIENT = re.compile(
    r"(could not resolve host|connection reset|too many requests|no space left)",
    re.I,
)
ENV_DRIFT = re.compile(
    r"(command not found|permission denied|no such file|locale|TZ=|LC_ALL)",
    re.I,
)


def read(p: Path) -> str:
    return p.read_text(errors="replace") if p.exists() else ""


def classify(rec: Path) -> dict:
    stderr = read(rec / "test.stderr") + read(rec / "test.stdout")
    exit_s = read(rec / "test.exit").strip()
    diff = read(rec / "diff.patch")
    prompt = read(rec / "prompt.md")

    if not (rec / "test.exit").exists():
        return {"leaf": "halt", "class": "no_receipt", "retry_prompt": False}

    if TRANSIENT.search(stderr):
        return {"leaf": "A", "class": "transient", "retry_prompt": False}

    ticket_has_oracle = "Oracle:" in prompt or "oracle:" in prompt
    tests_failed = exit_s not in {"", "0"}
    if tests_failed and not ticket_has_oracle:
        return {"leaf": "B", "class": "oracle_or_spec", "retry_prompt": False}

    if ENV_DRIFT.search(stderr):
        return {"leaf": "C", "class": "environment", "retry_prompt": False}

    diff_lines = sum(1 for line in diff.splitlines() if line.startswith(("+", "-")))
    if diff_lines > 80:
        return {"leaf": "D", "class": "task_too_wide", "retry_prompt": False}

    return {"leaf": "D", "class": "task_or_context", "retry_prompt": True}


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("receipt")
    args = parser.parse_args()
    rec = Path(args.receipt)
    result = classify(rec)
    print(f"leaf={result['leaf']}")
    print(f"class={result['class']}")
    print(f"retry_prompt={str(result['retry_prompt']).lower()}")
    halt = result["leaf"] == "halt" or result["retry_prompt"] is False
    print(f"halt_prompt_loop={str(halt).lower()}")


if __name__ == "__main__":
    main()

python3 classify_receipt.py receipts/20260914T164012Z
# expected shape:
# leaf=C
# class=environment
# retry_prompt=false
# halt_prompt_loop=true

The important output is retry_prompt=false on leaves A–C. That is the whole method.

Decision table

Leaf	Evidence in the receipt	Change the prompt?	Change the runner/env?	Change the ticket/oracle?
Halt	Missing test.exit or missing diff	No	No	No; capture first
A Transient	Host/network/rate/disk outside repo	No	Yes, or backoff	Only if dependency is required
B Oracle/spec	Gate fails a rule the ticket omitted	No until oracle is quoted	No	Yes
C Environment	Toolchain, cwd, locale, permissions	No until env is named	Yes	Job spec only
D Task/context	Oracle and env hold; diff is unbound	Yes, after path/diff cap	Only to match C	Shrink scope

If two columns would be “yes,” you skipped a question. Split the run.

Limitations, and who should not use this

The classifier is a regex over one directory. It will mis-label a test that prints permission denied as copy in an assertion message. It will miss a flaky clock that still exits 0. It does not know your compliance boundary.

Do not use this tree as an excuse to park customer fixtures on a shared runner. A free server is the wrong isolation boundary when the receipt’s env.names includes credential material, when the patch needs production data, or when the org cannot accept a third-party execution host. Do not use Leaf D to launder Leaf B: a smaller prompt will not invent the missing oracle.

Teams that already have a single CI shape and a one-page ticket template may only need leaves B and C. Teams that retry from the chat box after every red X should walk the full tree once per incident, on paper, before wiring any script.

The method also does not measure model quality. It measures whether you spent run N+1 on a class that generation can change.

Halt is the artifact

The useful output of an agent loop is not a longer chat. It is a receipt, a class, and a halt bit.

If you apply the tree to one red run this week, keep class.txt next to diff.patch. The second prompt is allowed only when the first file says Leaf D and the patch cap still holds.