Before You Blame the Model: A Retry Probe That Ends the Argument in One Afternoon

The disagreement that survives a pairing session is rarely about the patch itself; it is about the sequence the patch creates. Retry loops are the usual suspect, because every individual attempt looks defensible while the overall behavior quietly changes the task on each pass. A probe that records attempt index, payload shape, outcome, and elapsed time turns that ambiguity into a two-minute read of a JSONL file. The decision worth keeping is procedural rather than stylistic: instrument the sequence before anyone argues about the cause.

The patch that started the session

The patch under review was a small generator: it asked a model for a JSON object, validated the object, and retried up to three times when validation failed. The senior's objection was a single sentence, and it was not about correctness at all — attempt two had no reason to succeed. The session script below is a reconstructed teaching artifact rather than a transcript, and the stub in step 3 is the only part a reader has to supply.

The reviewer's claim was testable, which is what made the session useful. Either the retry re-sent the same task and deserved another chance, or it re-rolled the task and deserved deletion. Nobody in the room could tell which one the code did by reading it, and that gap is the whole article.

Three dead ends, in the order they happened

1. Rewriting the prompt

The first instinct was to tighten the instruction so that attempt one would succeed and the retry would never fire. This changes the sample rather than the code path, so it produces a cleaner demo and identical production behavior. It also hides the defect behind a prompt that happens to work on the current inputs.

2. Reaching for a larger model

The second attempt was to swap in a stronger model and observe that the validation error disappeared. That result is real but not portable, because the loop's structure is model-agnostic and the failure returns the moment the model changes again. Comparing models before understanding the loop answers a question nobody asked.

3. Arguing from documentation

The third dead end was citation: the client library documents retries, the vendor documents determinism caveats, and both statements are true and irrelevant. Documentation describes intent at the boundary, while the probe measures behavior inside the loop. The session only moved forward when the argument stopped being about what should happen.

The decision that was kept

One policy line ended the discussion and outlived the patch: no claim about a generated retry survives review without a per-attempt record. The record has four fields, and none of them are optional — attempt index, payload key set, outcome, and elapsed milliseconds. With those four columns, a reviewer can distinguish a genuine retry from a re-roll without reading the prompt at all.

The signal that matters most is payload drift between attempt one and attempt two. If the key set changes, the loop is sampling a new task on each pass and calling it a retry, which is a design error rather than a model limitation. If the key set is stable and the failure repeats, the validator or its contract is the suspect. Nothing else in the session produced a decision that survived the week.

Build the probe in five steps

Step 1 — Record one row per attempt

# probe_retry.py — writes one JSONL row per attempt. Standard library only.
import json, sys, time, uuid
from dataclasses import dataclass, asdict

@dataclass
class Attempt:
    request_id: str
    attempt_index: int
    model: str
    outcome: str          # "ok" | "retry" | "give_up"
    elapsed_ms: int
    payload_keys: list

def run_once(generate, validate, model, max_attempts=3):
    request_id = uuid.uuid4().hex[:12]
    rows = []
    for i in range(1, max_attempts + 1):
        started = time.perf_counter()
        try:
            payload = generate(model)
            outcome = "ok" if validate(payload) else "retry"
        except Exception as exc:
            payload, outcome = {"_error": type(exc).__name__}, "retry"
        rows.append(Attempt(
            request_id=request_id,
            attempt_index=i,
            model=model,
            outcome=outcome,
            elapsed_ms=int((time.perf_counter() - started) * 1000),
            payload_keys=sorted(payload.keys()) if isinstance(payload, dict) else [],
        ))
        if outcome == "ok":
            break
    else:
        rows[-1].outcome = "give_up"
    return rows

Step 2 — Replace the stub, keep the signature

The generate and validate callables are injected on purpose, so the harness never imports a client and never needs credentials to run. In practice generate wraps one model call and returns a parsed dict, while validate returns a boolean. Label any unexecuted version of this file as a sketch, because a probe that was never run is an opinion with indentation.

Step 3 — Drive forty requests and flush to disk

# __main__ block — 40 requests per model, one JSONL file per model.
if __name__ == "__main__":
    model = sys.argv[sys.argv.index("--model") + 1]
    runs = int(sys.argv[sys.argv.index("--runs") + 1])
    for _ in range(runs):
        for row in run_once(generate, validate, model):
            print(json.dumps(asdict(row)), flush=True)

Step 4 — Reduce the file to five numbers

# summarize.py — the only five numbers a pairing session needs.
import json, sys
from collections import defaultdict

def summarize(rows):
    requests = defaultdict(list)
    for row in rows:
        requests[row["request_id"]].append(row)
    total = len(requests)
    ok = sum(1 for rs in requests.values() if rs[-1]["outcome"] == "ok")
    reshaped = sum(
        1 for rs in requests.values()
        if len(rs) > 1 and rs[1]["payload_keys"] != rs[0]["payload_keys"]
    )
    return {
        "requests": total,
        "success_rate": round(ok / total, 3),
        "attempts_per_request": round(sum(len(rs) for rs in requests.values()) / total, 2),
        "give_up": sum(1 for rs in requests.values() if rs[-1]["outcome"] == "give_up"),
        "reshaped_retries": reshaped,
    }

for path in sys.argv[1:]:
    print(path, summarize([json.loads(l) for l in open(path) if l.strip()]))

Step 5 — Run both configurations on the same day

python probe_retry.py --model model-a --runs 40 > a.jsonl
python probe_retry.py --model model-b --runs 40 > b.jsonl
python summarize.py a.jsonl b.jsonl

Running the second configuration on free infrastructure

Free model access has exactly one useful role in this workflow: it lets the same probe run against a second model without a budget conversation. MonkeyCode's free model access and free server option are availability claims supplied by the operator, and they are sufficient for an A/B run of forty requests per side. The free server option matters for a mundane reason — probe runs are long and boring, and a laptop that sleeps mid-run writes a file that mimics a rate-limit failure.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two rules keep the comparison honest. First, keep the request count identical on both sides, because forty-versus-twenty invites a success-rate gap that is really a sample-size gap. Second, write the model identifier into every row, since free model lists rotate and a result without an identifier cannot be reproduced next month. Quotas and model names change often enough that a number quoted in a blog post is worse than no number at all, so read the console before planning a run.

Decision table

Probe observation	Reading	Action
attempts_per_request near 1.0, success_rate near 1.0	The loop rarely fires; the objection was stylistic	Keep the patch, close the thread
attempts_per_request above 1.3, reshaped_retries = 0	Stable failure, same task, likely validator contract	Fix the contract, keep the retry
reshaped_retries > 0	Each attempt samples a new task	Delete the retry, make the request explicit
High give_up on both models	Not model-specific	Fix the code path before touching the model
give_up differs sharply between models	Model-sensitive	Record the identifier, avoid generalizing the claim

Limitations, and who should skip this

The harness runs in a single process, so it misses concurrency effects and any failure that only appears under parallel load. Forty requests produce a useful ratio and a useless confidence interval, which is fine for a design argument and not fine for a release gate. The stub's validator may itself be the defect, in which case the probe faithfully measures the wrong thing.

Teams that gate releases on latency percentiles should use their existing load tooling instead of a script like this. Anyone whose retry crosses a side-effecting boundary such as payments or email should point the probe at a dry-run sink before the first run. During live incident triage, five minutes of manual reading usually beats a harness that has not been written yet.

When the probe does earn a place in the repository, the free tier is enough to run that first side-by-side comparison before the harness is worth automating.