DEV Community: Dakota Liu

Case Study: A Verdict Contract That Keeps a Triage Bot Honest When the Model Changes

Dakota Liu — Tue, 27 Oct 2026 13:19:54 +0000

You can move a small bot to a different model in an afternoon, but only if that bot asks the model for almost nothing. That is the lesson behind the reference design below, and it sits oddly against this week's DEV front page, where several popular AI posts argue about what should sit between a developer and a model. A narrow, testable contract does more for you than a cleverer prompt, because the contract is the part that can fail loudly on your terms.

Background: the endpoint moves before your code does

Model endpoints change faster than the tools that depend on them. A snapshot gets retired, a JSON mode gets stricter or looser, and a workflow that looked stable starts returning prose where your parser expects an object. Most teams only discover this in production, because their single test of the model is the happy-path call they made on the day they wired it up.

This case study is a reference design, not a production report. The project is a small GitHub issue triage bot that labels incoming issues and decides whether a human should look at them. The code below is a working skeleton you can run yourself, and the results section is deliberately shaped as an interpretation table rather than a wall of numbers I did not measure.

Goal: one narrow output contract

The bot's job is deliberately boring, and that is the point. It reads an issue title and body, assigns one label from a fixed list, returns one to three reason codes, and escalates to a human when it is unsure. Nothing downstream of the bot reads free text, so the whole decision path is a single small object you can validate before it touches a queue.

Three constraints shaped the design:

The label set must be finite and versioned, because label drift is the failure mode nobody notices.
Every decision must carry at least one reason code, so a reviewer can audit the bot without re-reading the model's prose.
Any unparseable reply must degrade to a human queue, never to a guessed label.

The verdict contract

Save this as verdict.schema.json. If you cannot write your bot's output this tightly, the model is doing too much work for you.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["label", "confidence", "reasons", "needs_human"],
  "properties": {
    "label": { "enum": ["bug", "docs", "question", "feature", "duplicate", "spam"] },
    "confidence": { "enum": ["low", "medium", "high"] },
    "reasons": {
      "type": "array", "minItems": 1, "maxItems": 3,
      "items": { "enum": ["repro_steps_present", "stack_trace_present", "version_missing", "asks_how_to", "no_actionable_detail", "links_prior_issue"] }
    },
    "needs_human": { "type": "boolean" }
  }
}

Implementation: one call function, one validator, one repair

The most useful structural decision here is that exactly one function knows about HTTP. Everything else in the bot works with a string and a parsed object, so switching providers means editing one file. Keep the transport dumb and let the validation layer carry the personality.

# client.py - the only file that knows what an endpoint looks like
import json, os, urllib.request


def raw_completion(system, user, repair_note=''):
    base = os.environ['LLM_BASE_URL'].rstrip('/')
    payload = json.dumps({
        'model': os.environ['MODEL_ID'],
        'temperature': 0,
        'messages': [
            {'role': 'system', 'content': system},
            {'role': 'user', 'content': user + repair_note},
        ],
    }).encode('utf-8')
    req = urllib.request.Request(
        base + '/chat/completions',
        data=payload,
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + os.environ['LLM_API_KEY']},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.loads(resp.read())['choices'][0]['message']['content']

If your endpoint speaks a different request shape, change this function and nothing else. That constraint is what makes a swap drill cheap, because the rest of the codebase stays constant while the backend moves underneath it.

Validate first, repair once, then escalate

One repair attempt catches most formatting slips without hiding a model that genuinely does not follow instructions. Two attempts is usually one too many, because a second repair pass doubles the cost of exactly the cases that are already going badly.

# verdict.py
import json
from jsonschema import Draft202012Validator

SCHEMA = json.load(open('verdict.schema.json', encoding='utf-8'))
VALIDATOR = Draft202012Validator(SCHEMA)

SYSTEM = (
    'You classify GitHub issues. Reply with one JSON object and nothing else. '
    'label in {bug, docs, question, feature, duplicate, spam}; '
    'confidence in {low, medium, high}; reasons: 1-3 codes from '
    '{repro_steps_present, stack_trace_present, version_missing, asks_how_to, '
    'no_actionable_detail, links_prior_issue}; needs_human is a boolean.'
)

FALLBACK = {'label': 'question', 'confidence': 'low',
            'reasons': ['no_actionable_detail'], 'needs_human': True}


def parse(raw):
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError:
        return None, 'reply was not JSON'
    errors = sorted(VALIDATOR.iter_errors(obj), key=lambda e: list(e.path))
    if errors:
        return None, 'schema: ' + '; '.join(e.message for e in errors[:3])
    return obj, ''


def classify(issue, call):
    user = issue['title'] + '\n\n' + issue['body']
    obj, why = parse(call(SYSTEM, user))
    if obj:
        return obj, 'first_pass'
    note = '\n\nYour previous reply was rejected (' + why + '). JSON only.'
    obj, why = parse(call(SYSTEM, user, note))
    if obj:
        return obj, 'repaired'
    return FALLBACK, 'fallback'

Note that the fallback is not an error path you hope never runs. It is a first-class outcome that the drill counts, because a rising fallback rate is the earliest honest signal that a candidate model is not compatible with your contract.

The drill is cheap to run because the candidate side does not need to be the endpoint you pay for in production. I point candidate calls at MonkeyCode's free model access and keep the scheduled job small enough for the free server option, so a swap evaluation costs review time instead of budget. MonkeyCode is an open-source project, and the operator advertises a free token allowance alongside that free server option; read both figures off the project page before you plan around them, because free-tier numbers age faster than any code in this article.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The swap drill

A drill is four numbered steps, and none of them need a new tool. The discipline comes from running the same golden set through the old backend and the candidate backend, then comparing outputs row by row instead of eyeballing a few examples.

Curate golden.jsonl from issues you already closed, with one id, the human-assigned label, title, and body per line. Aim for at least a few dozen rows and make sure every label in your enum appears.
Run the drill against the model currently in production and save the CSV as your baseline.
Run the identical drill against the candidate, changing only MODEL_ID and the base URL.
Review the disagreements by hand before you look at the aggregate score, because a model can be wrong in a more useful way than your old one.

# drill.py
import csv, json, time
from client import raw_completion
from verdict import classify


def run(golden_path, out_path, call):
    rows = []
    for line in open(golden_path, encoding='utf-8'):
        issue = json.loads(line)
        started = time.monotonic()
        verdict, status = classify(issue, call)
        rows.append({
            'id': issue['id'],
            'expected': issue['label'],
            'got': verdict['label'],
            'agrees': verdict['label'] == issue['label'],
            'status': status,
            'needs_human': verdict['needs_human'],
            'seconds': round(time.monotonic() - started, 2),
        })
    with open(out_path, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)
    n = len(rows)
    print('rows=' + str(n))
    print('agreement=%.3f' % (sum(r['agrees'] for r in rows) / n))
    print('schema_ok=%.3f' % (sum(r['status'] != 'fallback' for r in rows) / n))
    print('escalated=%.3f' % (sum(r['needs_human'] for r in rows) / n))


if __name__ == '__main__':
    run('golden.jsonl', 'drill.csv', raw_completion)

Wrap the environment plumbing in a script so a drill is one command and not a five-line incantation you will eventually mistype.

#!/usr/bin/env bash
# run_drill.sh
set -euo pipefail
export LLM_BASE_URL=${LLM_BASE_URL:?copy the base URL from the provider docs}
export LLM_API_KEY=${LLM_API_KEY:?set the key for this endpoint}
export MODEL_ID=${MODEL_ID:?set the model id the endpoint documents}
stamp=$(date -u +%Y%m%dT%H%MZ)
python drill.py --golden golden.jsonl --out drill-${stamp}.csv

A decision table you can argue about

Thresholds turn a vague feeling into a reviewable decision. Set them once with your team, then let the table decide instead of whoever is most tired that afternoon.

Gate	What it measures	Suggested rule	Why it matters
Schema-valid rate	Rows where `status` is not `fallback`	at or above 0.98	A contract the model cannot hit is a contract your queue cannot trust
Label agreement	`agrees` compared with the human label	at or above the baseline, or explain each drop	Small drops on rare labels are normal; drops on common ones are not
Escalation delta	Change in `needs_human` rate	no more than five points higher	A hedging model quietly doubles human load
Latency	Per-row `seconds`, looked at as a distribution	fits your job window	The worst case matters more than the average for a nightly job
Manual read	Ten disagreement rows	always required	Nothing replaces reading the failures

Running it on the free server option

A scheduled triage job is a good candidate for a free server, because it sleeps most of the day and a slow start costs you minutes rather than requests. Keep the deployment boring: one directory, one cron entry, and logs you can read with tail.

# crontab -e on the free server option
17 3 * * * cd /srv/triage && ./run_drill.sh >> drill.log 2>&1

Do not assume the free server behaves like the box under your desk. Measure startup time, confirm that the job finishes inside whatever window the host gives you, and keep the state in files you can rebuild from scratch. If a run is interrupted, the next run should still produce a complete CSV, which is why the drill writes one file per timestamp instead of appending to a shared one.

Results: what the output actually tells you

I am not going to print agreement scores I did not measure, so treat the table below as the reading guide for your own drill output. The pattern of results matters far more than any single number, and each pattern points at a different fix.

Drill outcome	Most likely cause	Action
Agreement and schema rate both hold	Candidate is compatible	Promote it and keep the golden set growing
Schema rate falls, agreement holds	Contract is too loose for this model's formatting habits	Tighten the prompt, keep the single repair pass
Only `duplicate` agreement falls	Model lacks repository context	Feed prior issue titles into the prompt or escalate that label
Escalation rate climbs	Model hedges on ambiguous issues	Revisit your confidence wording before blaming the model
Latency p95 blows past the window	Wrong sizing for the job, not a wrong model	Split the drill into batches or move the schedule

The original bot moved from background to reference design because every swap question turned into the same question: can you test this without rewriting the bot. Once the drill existed, that answer was a CSV comparison a reviewer could audit in ten minutes.

Lessons learned

A narrow contract is portable; a wide one is a rewrite waiting for a new endpoint. Keep the schema small enough that you can recite it in a review.
Free tiers are excellent for candidate evaluation and poor for assumptions. Verify allowances and server limits the same day you plan around them.
One repair pass is a formatting safety net; two passes are a hidden cost multiplier on your worst requests.
The fallback path deserves tests of its own, because it runs exactly when everything else is misbehaving.
Golden sets rot, so add a row every time a human overrides the bot, and re-baseline after each promotion.

Who should not use this approach

If you need guaranteed determinism, a formal SLA, or full control over where inference happens, a free-tier endpoint and a schema check will not get you there. Teams handling regulated data should confirm the host's data handling terms before sending a single issue, and high-volume workloads will likely outgrow a free server quickly. Anyone unwilling to curate a few dozen labelled examples should also skip the drill, because without a golden set the comparison is just two opinions.

If you want to try the workflow without committing a paid key, run the drill against MonkeyCode's free model access first and keep the scheduled job on the free server option until the numbers earn something bigger. Check the project page for the current allowance and server terms on the day you start, and let the decision table, not the announcement, tell you when to swap.

Case Study: Sizing Inputs Before the Call, Not After the Bill

Dakota Liu — Mon, 26 Oct 2026 11:48:58 +0000

The cheapest token is the one you never spend, so the most useful budget control sits before the request, not in the invoice you read on Monday. This case study walks through a small changelog bot that kept running out of quota mid-week, and the pre-flight sizing layer that turned a soft budget into a hard boundary. The artifact is about 120 lines of dependency-free Python you can drop into any worker that pays per token.

Background: one paragraph per merged pull request

The project was deliberately boring. A worker polled a repository every ten minutes, collected merged pull requests, and asked a model for a one-paragraph changelog entry that was posted to an internal channel. Volume was roughly thirty to forty pull requests a day, which sounds trivial until you look at the inputs. Many pull request threads contained long review discussions, quoted replies, and pasted stack traces, so the same item could be 400 tokens on a quiet day and 9,000 tokens on a noisy one.

The worker ran on free model access, which made throughput cheap but made quota the binding constraint rather than latency. When the daily allowance ran out, the worker did not stop cleanly. It hit context-length errors on the largest threads, retried with the same oversized input, and burned the remaining budget on requests that could never succeed.

Goal: three properties the fix had to have

The daily cap must be a boundary the process cannot cross, not a number you discover after the fact.
No pull request may be silently dropped when the budget runs out; it must be parked with a machine-readable reason.
Every decision must be reproducible from files on disk, so a restart costs nothing and a re-run tells the same story.

Notice what is absent from that list: accuracy improvements, model upgrades, prompt cleverness. The interesting failure was arithmetic, and arithmetic failures get fixed with arithmetic.

Step 1: normalize before you measure

Half the token bill in this bot was quoted text and bot chatter that never influenced the summary. Normalizing first is free, deterministic, and shrinks the estimation problem as well as the request.

import re

QUOTE = re.compile(r"^\s*>.*$", re.M)
COLLAPSE = re.compile(r"\n{3,}")
SPACES = re.compile(r"[ \t]{2,}")

def normalize(text: str, drop_markers=("<!--", "ci-bot:", "dependabot")) -> str:
    text = QUOTE.sub("", text)
    kept = [ln for ln in text.splitlines() if not any(m in ln for m in drop_markers)]
    text = SPACES.sub(" ", "\n".join(kept))
    return COLLAPSE.sub("\n\n", text).strip()

Run normalize on your own corpus for a week and log len(before) - len(after). If the reduction is under ten percent, your inputs are already clean and this step is noise; if it is above thirty percent, you were paying for quoted replies.

Step 2: estimate with two ratios, not one

A single characters-per-token ratio is wrong the moment your input contains code. Fenced blocks tokenize densely around punctuation and indentation, so the estimator below splits the text and applies a different divisor to each region, then adds a fixed overhead for message framing.

FENCE = re.compile(r"```

[\s\S]*?

```")
MESSAGE_OVERHEAD = 24

def estimate(text: str, factor: float = 1.0) -> int:
    code_chars = sum(len(m) for m in FENCE.findall(text))
    prose_chars = len(text) - code_chars
    raw = code_chars / 3.2 + prose_chars / 4.3 + MESSAGE_OVERHEAD
    return int(raw * factor)

The factor argument is the whole calibration story. After every real call you write the reported usage back into a small JSON file, recompute actual / estimated over the last fifty calls, and store the median as factor. You are not trying to be exact; you are trying to be consistently biased in one known direction.

Step 3: choose a plan from a table, not from a guess

Once you have an estimate, the routing decision is deterministic. The thresholds below are the ones this bot uses, and they are conservative on purpose because an underestimate costs a failed call while an overestimate costs a slightly worse summary.

Estimated tokens	Plan	Max model calls	Failure mode avoided
≤ 1,500	`single`	1	nothing, this is the happy path
1,501 – 4,000	`trim`	1	context-length errors on long threads
> 4,000	`map_reduce`	5	silent truncation of review discussion

def plan(text: str, limits=(1500, 4000), max_sections=4) -> tuple[str, list[str]]:
    est = estimate(text)
    if est <= limits[0]:
        return "single", [text]
    if est <= limits[1]:
        return "trim", [head_tail(text, head=1200, tail=600)]
    return "map_reduce", split_sections(text, max_sections)

head_tail keeps the first 1,200 characters and the last 600, because the opening message usually states the intent and the tail usually holds the resolution. split_sections cuts on markdown headings and falls back to equal chunks, hard-capped at four so the worst case is five calls per item.

Step 4: reserve the budget before the request leaves the process

The reservation happens in memory and on disk at the same time, and a rejected reservation is a normal outcome rather than an error. This is the difference between a budget you hope to respect and one you cannot violate.

import json, pathlib, datetime as dt

LEDGER = pathlib.Path("budget.json")

def load() -> dict:
    if LEDGER.exists():
        return json.loads(LEDGER.read_text())
    return {"day": "", "reserved": 0, "spent": 0, "deferred": []}

def reserve(state: dict, cost: int, daily_cap: int) -> bool:
    today = dt.datetime.now(dt.timezone.utc).date().isoformat()
    if state["day"] != today:
        state.update(day=today, reserved=0, spent=0)
    if state["reserved"] + cost > daily_cap:
        return False
    state["reserved"] += cost
    LEDGER.write_text(json.dumps(state, indent=2))
    return True

When reserve returns False, the worker appends the pull request to deferred with the estimate, the plan name, and the shortfall, then moves on. That list is the single most useful debugging artifact in the whole system, because it tells you whether you are under-budgeted or over-chunking.

Step 5: reconcile actuals against reservations

After each successful call, subtract the reservation and add the real usage, then refresh factor once a day. This keeps the ledger honest without requiring exact prediction, and it makes the weekday shape visible: this bot's Monday is heavier than its Thursday, which is obvious in hindsight and invisible in a monthly total.

Where MonkeyCode fits

This worker calls a model through MonkeyCode's free model access, and the operator states that the free tier includes a free allowance of ten million tokens plus a free server option for running the worker itself. Treat those two numbers as claims to verify against the current product page before you size anything, since tier details are exactly the kind of thing that drifts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option mattered less for compute and more for process hygiene: because the ledger lives in a single JSON file outside the process, a restart loses nothing and a redeploy re-reads the same state. That property is worth having regardless of where the worker runs.

The harness: three tests anyone can re-run

Estimator bias. Feed fifty recorded inputs through estimate and compare to the usage numbers your own calls reported; assert the median ratio sits between 0.8 and 1.25 and fail the build outside that band.
Planner monotonicity. Assert that estimate never decreases as you append text, and that a plan never produces more sections than max_sections.
Cap invariant. Drive reserve with a synthetic cost sequence and assert reserved never exceeds daily_cap, including across a simulated date rollover.

Each test runs offline against fixtures, so you can validate the budget layer without spending a token.

Results

I am not going to quote a savings percentage here, because the number is entirely a function of your corpus and would be theater. What the change does guarantee, by construction of the code above, is narrower and still worth stating: the process cannot exceed its daily reservation, oversized items are split into at most five calls instead of retried whole, and every skipped item leaves a row explaining itself.

Limitations and who should not use this

If your inputs are uniformly small, the estimator is overhead; a fixed cap is enough.
If your provider bills per request rather than per token, reservation arithmetic buys you little.
If your summaries must be exhaustive rather than representative, head_tail trimming will lose information you care about.
If your corpus shifts weekly, a static factor will drift and you should recalibrate on a schedule rather than once.

Closing note

If you want to try the shape, start with the estimator harness and fifty recorded inputs from your own worker, then add the planner only after you know your bias direction. The parts are boring on purpose, and boring is what you want in the component that decides whether the rest of the system gets to run.

Case Study: A Doc-Drift Ledger That Only Sends Changed Symbols to a Model

Dakota Liu — Tue, 20 Oct 2026 09:56:42 +0000

Most documentation rot is invisible until someone follows a docstring into a TypeError. The fix I built for a small internal library is a ledger, not a chatbot: it hashes every public symbol's signature and docstring, then asks a model about the handful of symbols whose signature moved while the prose did not. The useful part is what never leaves your machine — unchanged symbols cost zero tokens, and every model reply is cached against a hash pair you can inspect in Git.

Background: the docstring that outlived its argument

The library in question exports roughly forty functions across a dozen modules, and three of them had quietly changed their parameter names during a cleanup. Nothing failed in CI, because docstrings are not assertions, and nobody reread the module by hand in that sprint. When a teammate wired fetch_user(uid=...) into a script and got a keyword error, the docstring was still advertising uid.

That is the shape of the problem worth solving: not "docs are bad", but "a narrow class of doc drift is mechanically detectable and cheap to detect". Full-file review passes are noisy and expensive, while a symbol-level diff gives you a short, defensible list you can hand to a reviewer or a model.

Goal and explicit non-goals

The goal was one command that answers a single question: which exported symbols changed shape without their documentation following along?

In scope: Python modules, signature changes, missing or stale docstrings on public symbols, a machine-readable report, a non-zero exit code for CI.
Out of scope: prose pages in a wiki, translation drift, tutorial code blocks, and any claim that a model can certify documentation as correct.
Non-negotiable: the tool must run offline against a fake client, so the test suite never depends on a network call.

You should read that last point as the design constraint that made everything else simple. If the diffing layer is deterministic, the model becomes an optional annotation on top of a result you already trust.

Step 1: Extract symbols and hash them

Start with the extractor. It walks module-level definitions only, which keeps the ledger small and the false-positive rate low.

# drift/symbols.py
import ast
import hashlib
from pathlib import Path


def _signature(node: ast.AST) -> str:
    if isinstance(node, ast.ClassDef):
        bases = ", ".join(ast.unparse(b) for b in node.bases)
        return f"class {node.name}({bases})"
    args = [a.arg for a in node.args.args]
    if node.args.vararg:
        args.append(f"*{node.args.vararg.arg}")
    if node.args.kwarg:
        args.append(f"**{node.args.kwarg.arg}")
    return f"def {node.name}({', '.join(args)})"


def _digest(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def collect(root: Path) -> dict[str, dict[str, str]]:
    ledger: dict[str, dict[str, str]] = {}
    for path in sorted(root.rglob("*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for node in tree.body:
            if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                continue
            key = f"{path.as_posix()}::{node.name}"
            sig = _signature(node)
            ledger[key] = {
                "sig": sig,
                "sig_hash": _digest(sig),
                "doc_hash": _digest(ast.get_docstring(node) or ""),
            }
    return ledger

Commit the generated JSON next to your code, and treat it exactly like a lockfile. The diff between the committed ledger and the freshly collected one is your entire work queue.

Step 2: Turn two ledgers into a candidate list

The comparison rule matters more than the code around it. A symbol is a candidate only when its signature hash changed and its docstring hash did not, because a human who already edited the prose has answered the question.

# drift/diff.py
def candidates(base, head):
    out = []
    for key in sorted(set(base) | set(head)):
        old, new = base.get(key), head.get(key)
        if old is None:
            out.append({"key": key, "change": "added", "symbol": new, "ask": True})
        elif new is None:
            out.append({"key": key, "change": "removed", "symbol": old, "ask": False})
        elif old["sig_hash"] != new["sig_hash"] and old["doc_hash"] == new["doc_hash"]:
            out.append({"key": key, "change": "signature", "symbol": new, "ask": True})
    return out

Removals are reported but never sent to a model, since the correct answer there is a human decision about deprecation wording. Added symbols are also a weak signal, so I gate them behind a flag instead of asking about every new helper.

Step 3: Ask the model about candidates only

The prompt is deliberately boring, and it demands JSON so the reply can be validated before it touches the report. Note that the payload contains signatures and docstrings, not whole files.

PROMPT = """You review Python docstrings against a changed signature.
For each item answer only whether the docstring still describes the new signature.
Reply with JSON: {"verdicts":[{"key":str,"verdict":"ok"|"stale","reason":str}]}

Items:
{items}
"""


def build_prompt(cands):
    items = [
        {
            "key": c["key"],
            "signature": c["symbol"]["sig"],
            "docstring": c["symbol"].get("doc", ""),
        }
        for c in cands
        if c["ask"]
    ]
    return PROMPT.format(items=json.dumps(items, indent=2))

Keep the response schema strict and drop any verdict you cannot parse. A model that answers in prose has told you nothing you can put in a report, and silently repairing its output is how false confidence gets into a repo.

Step 4: Cache verdicts by hash pair

Caching is where the economics live. A verdict is keyed by (key, sig_hash, doc_hash), so a rerun after an unrelated commit performs zero model calls.

# drift/cache.py
def cache_key(cand):
    return f"{cand['key']}|{cand['symbol']['sig_hash']}|{cand['symbol']['doc_hash']}"


def pending(cands, cache):
    return [c for c in cands if c["ask"] and cache_key(c) not in cache]

Store that cache in CI artifacts or a small committed file, whichever your team already trusts. The committed option is easier to audit, because a reviewer can see exactly which verdicts were accepted and when.

Step 5: Make the whole pipeline testable with a fake client

The tool has to be provable without a network, so the model client is an injectable one-method interface. The test below asserts two things that actually matter: irrelevant symbols never enter the prompt, and a second run makes no calls at all.

# tests/test_drift.py
import json
from pathlib import Path
from drift.run import run


class FakeClient:
    def __init__(self, replies):
        self.replies = list(replies)
        self.prompts = []

    def complete(self, prompt: str) -> str:
        self.prompts.append(prompt)
        return self.replies.pop(0)


def write(path: Path, text: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(text, encoding="utf-8")


def test_only_changed_symbols_are_sent(tmp_path):
    src = tmp_path / "src"
    write(src / "a.py", 'def fetch(uid):\n    """Fetch by uid."""\n    return uid\n')
    write(src / "b.py", 'def ping():\n    """Ping."""\n    return 1\n')
    base = run(src, cache={}, client=None, write_ledger=tmp_path / "ledger.json")

    write(src / "a.py", 'def fetch(user_id):\n    """Fetch by uid."""\n    return user_id\n')
    reply = json.dumps({"verdicts": [{"key": "a.py::fetch", "verdict": "stale", "reason": "param renamed"}]})
    client = FakeClient([reply])
    report = run(src, cache={}, client=client, base_ledger=base, write_ledger=tmp_path / "ledger2.json")

    assert "b.py::ping" not in client.prompts[0]
    assert report["stale"] == ["a.py::fetch"]

Add a second test that reruns the same tree against the populated cache and asserts client.prompts == []. That pair of tests is the whole contract, and it runs in milliseconds on a laptop with no credentials configured.

Running it nightly without babysitting a laptop

A cron entry finishes the loop, because the interesting case is the commit that landed at 2 a.m. from someone else's branch.

17 3 * * * cd /srv/drift && /usr/bin/python -m drift nightly --root src --fail-on stale >> drift.log 2>&1

This is also where MonkeyCode's free model access and free server option became relevant to the project rather than a footnote. The nightly step needs a machine that stays awake and a model endpoint whose cost does not scale with how often you run the job, and the variant I deployed uses the free tier for both. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two practical notes on that. First, the free allowance advertised on the plan page at the time of writing was ten million tokens, and the operator-supplied framing matters: quotas and terms change, so verify the current page before you build a budget on it. Second, keep the endpoint in an environment variable rather than in code, so swapping providers is a config change and not a refactor.

export MONKEYCODE_API_KEY="..."        # never commit this
export MONKEYCODE_BASE_URL="..."       # from your workspace settings
export DRIFT_MODEL="..."               # whatever your workspace lists

If you want to try the pattern before provisioning anything, the candidate-only design works with any OpenAI-compatible client, so you can prototype locally and move the cron job later.

Results: what the ledger actually changed

On the fixture above, a signature rename produces exactly one candidate, one model call, and one cached verdict; a rerun produces zero calls. Those numbers come from the test suite, not from a benchmark, and I have not run a controlled cost study on a large repository.

The qualitative result is the one I would defend. Reviewers stopped reading forty-symbol reports and started reading a two-line queue, and the stale verdicts arrived with the offending signature attached, so the fix took seconds. The tool also caught a case I expected it to miss: a renamed keyword argument whose docstring example still used the old name in a code block.

Decision table

Situation	Use the ledger?	Why
Python library with docstrings on public symbols	Yes	Hashes are cheap and drift is real
Single-file script under 300 lines	No	Just read it
Docs live in separate Markdown pages	Not yet	You need anchor extraction first
You need legally exact wording	No	Model verdicts are advisory
Heavy churn on private helpers	No	Noise dominates; restrict to exports

Limitations and who should skip this

Hash equality is not meaning equality. A docstring that was already wrong before the rename keeps its hash and stays invisible to the ledger.
Python only, for now. The extractor is ast-based, and porting to TypeScript means a different parser plus your own signature stringifier.
Renames look like deletion plus addition. Wire in git log --follow or accept one noisy commit per rename; I accepted the noise.
Skip this if your team has no committed ledger discipline. A ledger in .gitignore gives you a tool that flags everything on every run.
Skip this if your docstrings are generated. Generated prose is already consistent with signatures, so you would pay for verdicts nobody reads.

Lessons learned

The design that survived contact with a real repository was the least clever one: detect locally, ask rarely, cache aggressively, and test with a fake client. Determinism at the diffing layer is what made the model call optional, auditable, and cheap enough to run nightly on a free tier. If you rebuild this, start with the fake-client test first — the pipeline falls out of it almost mechanically.

Case Study: A Token Ledger and Concurrency Lock for a Free-Tier Changelog Bot

Dakota Liu — Mon, 19 Oct 2026 08:38:03 +0000

The model call is the easy half of shipping a free-tier tool; the hard half is a ledger that tells you, before you call anything, whether you can still afford to. This case study walks through one small internal job end to end: a changelog drafter that turns merged pull request titles into release notes, with a budget guard that refuses work instead of overrunning a free allotment. Everything below is runnable, and the tests never touch the network or spend a single token.

Background: a Friday task nobody wanted to own

Our repository merges roughly forty pull requests a week, and somebody had to turn those into readable release notes before the Friday deploy. The obvious fix was a scheduled job that drafts the notes, but it arrived with three constraints that shaped the whole design. First, there was no budget line, so the job had to run on free model access. Second, there was no on-call rotation, so a failed run had to be boring instead of alarming. Third, the release could not depend on it, which meant the job needed a way to say "I am out of budget" and exit cleanly.

The constraint people underestimate is not the model call itself. It is accounting: once you are spending a fixed allotment you cannot top up, every run becomes a withdrawal, and you need a record of withdrawals that survives a crash. Without that record, a retry loop, a duplicated cron entry, or a provider outage can quietly consume the month in an afternoon.

Goal and non-goals

The goal was deliberately narrow, and writing the non-goals first saved a week of scope creep.

Reserve budget before every model call, using a conservative estimate for the prompt and the reply.
Reconcile the reservation with the provider's reported usage once the call returns.
Refuse a run when the estimate would cross a safety line, and exit with a non-zero code the scheduler ignores.
Survive a crashed run without permanently locking the budget.

Explicit non-goals: no per-tenant billing, no real-time dashboards, no retry orchestration inside the budget guard, and no attempt to predict long-run spend from a single week of data. Those belong to a different tool with a different failure profile.

Where free model access and a free server actually fit

The project advertises a free tier that includes free model access and a free server option, and it states an allotment of ten million tokens for that free tier. Treat both figures as advertised terms rather than guarantees, because quotas and free-tier conditions change and you should confirm the current numbers on the project page before you design around them. What matters for this article is the shape of the constraint, not the exact number: a finite allotment, a reset period you do not control, and a model client you did not write.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option matters because it removes the "where does the cron live" argument. What it does not remove is state: if the free server restarts and your ledger file disappears, your accounting disappears with it. Check whether your plan gives you a persistent volume, and if it does not, keep the ledger somewhere that survives a restart before you schedule a single run.

Implementation

Step 1: two tables, one writer

The ledger is SQLite, because a single job with a single writer needs transactions and not a database cluster. The budget table holds the allowance and the period key, and the runs table holds one row per attempt.

# ledger.py
import sqlite3, time

SAFETY = 0.80          # stop before the last fifth of the allotment
PERIOD = "%Y-%m"       # adjust if your terms reset on a different boundary

SCHEMA = """
CREATE TABLE IF NOT EXISTS budget (
  job       TEXT PRIMARY KEY,
  allowance INTEGER NOT NULL,
  period    TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS runs (
  id            INTEGER PRIMARY KEY AUTOINCREMENT,
  job           TEXT NOT NULL,
  started_at    REAL NOT NULL,
  finished_at   REAL,
  est_tokens    INTEGER NOT NULL,
  actual_tokens INTEGER,
  status        TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS runs_job_time ON runs (job, started_at);
"""

def connect(path):
    conn = sqlite3.connect(path, timeout=5, isolation_level=None)
    conn.execute("PRAGMA journal_mode=WAL")
    conn.executescript(SCHEMA)
    return conn

def period_key(now):
    return time.strftime(PERIOD, time.gmtime(now))

Step 2: reserve before you call anything

The reservation is where the interesting cases live, so it runs inside BEGIN IMMEDIATE. That makes two concurrent schedulers serialize, and the second one sees the first one's charge instead of racing past it.

def sweep_stale(conn, job, max_age_s=1800, now=None):
    now = now or time.time()
    conn.execute(
        "UPDATE runs SET status='aborted', finished_at=? "
        "WHERE job=? AND status='running' AND started_at < ?",
        (now, job, now - max_age_s),
    )

def reserve(conn, job, est_tokens, now=None):
    now = now or time.time()
    conn.execute("BEGIN IMMEDIATE")
    try:
        sweep_stale(conn, job, now=now)
        row = conn.execute(
            "SELECT allowance, period FROM budget WHERE job=?", (job,)
        ).fetchone()
        if row is None:
            conn.execute("ROLLBACK")
            raise RuntimeError(f"no budget row for job={job!r}")
        allowance, period = row
        if period != period_key(now):
            conn.execute("UPDATE budget SET period=? WHERE job=?", (period_key(now), job))
            spent = 0
        else:
            spent = conn.execute(
                "SELECT COALESCE(SUM(COALESCE(actual_tokens, est_tokens)), 0) "
                "FROM runs WHERE job=? AND status <> 'aborted'",
                (job,),
            ).fetchone()[0]
        if spent + est_tokens > int(allowance * SAFETY):
            conn.execute("ROLLBACK")
            return None
        run_id = conn.execute(
            "INSERT INTO runs (job, started_at, est_tokens, status) "
            "VALUES (?, ?, ?, 'running')",
            (job, now, est_tokens),
        ).lastrowid
        conn.execute("COMMIT")
        return run_id
    except BaseException:
        if conn.in_transaction:
            conn.execute("ROLLBACK")
        raise

Step 3: settle with the provider's number

When the call returns, you replace the estimate with the number the provider reports. When the call fails, you pass None, which leaves the estimate in place through COALESCE and prevents a retry storm from spending unseen budget.

def settle(conn, run_id, actual_tokens, status="ok"):
    conn.execute(
        "UPDATE runs SET finished_at=?, actual_tokens=?, status=? WHERE id=?",
        (time.time(), actual_tokens, status, run_id),
    )

Step 4: the job itself stays boring

The runner has one branch that matters: if reserve returns None, print why and exit with code 2. Do not retry, do not sleep, do not page anyone, because the next scheduled window is soon enough.

# changelog_job.py
import sys, ledger

def main(db_path, est_tokens, client, prs):
    conn = ledger.connect(db_path)
    run_id = ledger.reserve(conn, "changelog", est_tokens)
    if run_id is None:
        print("budget guard: refusing this run", file=sys.stderr)
        return 2
    try:
        text, used = client.draft(prs)
    except Exception:
        ledger.settle(conn, run_id, actual_tokens=None, status="failed")
        raise
    ledger.settle(conn, run_id, actual_tokens=used)
    print(text)
    return 0

Step 5: test the guard with a fake client

Because the guard is normal Python around a normal database, you can test the entire decision path offline. The fake client below returns a fixed usage count, so the assertions describe budget arithmetic rather than model behavior.

# test_ledger.py
import time, ledger

class FakeClient:
    def __init__(self, text="## Changes\n- nothing to report", used=137):
        self.text, self.used, self.calls = text, used, 0
    def draft(self, prs):
        self.calls += 1
        return self.text, self.used

def fresh(tmp_path, allowance=1000):
    conn = ledger.connect(str(tmp_path / "budget.db"))
    conn.execute(
        "INSERT INTO budget (job, allowance, period) VALUES (?, ?, ?)",
        ("changelog", allowance, ledger.period_key(time.time())),
    )
    return conn

def test_refuses_when_estimate_crosses_safety_line(tmp_path):
    conn = fresh(tmp_path, allowance=1000)
    assert ledger.reserve(conn, "changelog", 400) is not None   # 400 <= 800
    assert ledger.reserve(conn, "changelog", 401) is None       # 801 > 800

def test_actual_replaces_estimate(tmp_path):
    conn = fresh(tmp_path, allowance=1000)
    run_id = ledger.reserve(conn, "changelog", 400)
    ledger.settle(conn, run_id, actual_tokens=120)
    assert ledger.reserve(conn, "changelog", 680) is not None   # 120 + 680 = 800
    assert ledger.reserve(conn, "changelog", 1) is None

def test_stale_run_stops_holding_budget(tmp_path):
    conn = fresh(tmp_path, allowance=1000)
    now = time.time()
    ledger.reserve(conn, "changelog", 700, now=now - 7200)
    assert ledger.reserve(conn, "changelog", 700, now=now) is not None

def test_failed_run_keeps_the_estimate(tmp_path):
    conn = fresh(tmp_path, allowance=1000)
    run_id = ledger.reserve(conn, "changelog", 300)
    ledger.settle(conn, run_id, actual_tokens=None, status="failed")
    assert ledger.reserve(conn, "changelog", 500) is not None   # 300 + 500 = 800
    assert ledger.reserve(conn, "changelog", 1) is None

Run the suite and inspect the ledger with these commands.

python -m pytest test_ledger.py -q
sqlite3 budget.db "SELECT id, status, est_tokens, actual_tokens FROM runs ORDER BY id DESC LIMIT 5;"

Results: what the guard actually catches

I am not quoting live token counts here, because free-tier terms change and your numbers will differ from mine. What is reproducible is the behavior contract, and you can verify each row in the table below against the tests above.

Failure mode	Without the ledger	With the ledger
Duplicate cron entry	Two drafts run, both spend budget	Second reservation serializes and sees the first charge
Worker killed mid-run	Row stays `running`, budget blocked forever	Stale sweep marks it `aborted` after the age limit
Optimistic estimate	Real spend drifts above the plan	`settle` replaces the estimate with reported usage
Provider outage and retries	Each retry looks free and is not	Failed runs keep the estimate as their charge
New reset period	Manual bookkeeping, usually forgotten	Period key mismatch resets the spend window

The most useful output is not a number but a refusal. A run that exits with code 2 and a one-line reason on stderr is a system that knows its own limits, and that is far easier to hand to a teammate than a job that silently eats the month.

To get your own measurements, seed the budget row with your real allowance, run the job on schedule for a week, and compare est_tokens against actual_tokens in the ledger. If your estimates are consistently twenty percent low, raise the safety multiplier instead of rewriting the prompt.

sqlite3 budget.db "SELECT ROUND(100.0*SUM(COALESCE(actual_tokens,est_tokens))/10000000,2) || '% of a 10M allotment' FROM runs WHERE job='changelog' AND status<>'aborted';"

Limitations and who should not use this

This design is honest about what it cannot do, and several teams should pick something else.

If you cannot read usage from the provider, the ledger is estimates all the way down, and you should treat the safety line as the real limit rather than a margin.
If your free server has no persistent disk, the ledger must live off-box or be exported and committed after every run, otherwise a restart zeroes your accounting.
If your reset boundary is not a calendar month, change PERIOD and period_key together, then add a test for the boundary you actually have.
If you have several writers or tenants, SQLite's single-writer model will frustrate you quickly, and a small Postgres instance is the cheaper decision.
If a missed changelog blocks a release, this guard is the wrong trade: refuse less, alert more, and keep a human escape hatch.

Lessons learned

Three things survived contact with the schedule, and none of them were about prompting. First, put the ledger in place before the prompt, because accounting you add later never quite matches the code paths that already exist. Second, fail closed on budget and fail open on output, so the release notes can be written by hand on the rare Friday the guard says no. Third, keep the model client behind one narrow method, which is what made the offline tests possible in an afternoon.

If you want to try this shape yourself, the free tier's free model access plus the free server option is enough to run exactly one job like this one, so start with the ledger, confirm the current quota terms, and write the prompt last.

Case Study: A Checklist Bot Built to Keep Working When the Model Is Down

Dakota Liu — Sun, 18 Oct 2026 07:17:05 +0000

Case Study: A Checklist Bot Built to Keep Working When the Model Is Down

The cheapest model call is the one your system can survive without. That sentence became the design rule for a small internal tool I finished over two weekends: a bot that turns a pull-request diff into a short reviewer checklist. The interesting part is not the prompt; it is the rule that the tool must return something useful at every level of degradation, including the level where no model answers at all. Below is the whole project end to end, including the budgeter, the fallback ladder, the fixture test plan, and the parts I would not do again.

The constraint that shaped everything

The project ran against two operator-supplied facts about MonkeyCode: free model access, and a free server option for hosting small services. The operator describes the free allowance as 10 million tokens; treat that number as a starting point you re-check on the product's own pricing page, because allowances and terms move. The tool I built assumes only that the model is cheap, rate-limited at some unknown point, and occasionally slow or absent — three assumptions that survive almost any change to the free tier.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

There was also a mood constraint. The last week of DEV discussion has been full of people admitting they are tired of AI woven into every hot path, and the "design for churn" argument keeps making the same point from a different angle: a dependency you cannot remove is a liability. So the goal was an AI tool that a reviewer could ignore, disable, or outlive without losing the PR workflow.

Goal and scope

The target repository is a mid-sized Python service with roughly 40 endpoints. A reviewer opening a diff of 300 changed lines routinely misses the boring questions, so the bot posts a comment with a small checklist. I wrote down four requirements before any code:

One model call per diff, maximum, with a hard timeout and no retries.
A deterministic checklist from path and size rules whenever the model is unavailable or the diff is too large to fit a budget.
No partial context: if the diff does not fit whole, the bot does not call the model at all.
Every degradation level is observable in the comment footer, so trust in the tool can be audited.

That third rule is the one people argue about, and it is the one I would defend hardest. Sending the model half a diff produces confident nonsense about code it never saw, which is worse than a boring path-based checklist.

The degradation ladder

Numbered levels make the fallback path testable instead of aspirational:

Level 0 — empty: no file changed, return a one-line note, no model call.
Level 1 — heuristic: path and size rules only. Used when the diff overflows the character budget.
Level 2 — heuristic after model fault: timeout, network error, or a response that fails the shape check.
Level 3 — model: a single call, a single timeout, a response that parses as a checklist.

You can see the design bias immediately: three of the four levels never touch the network. The model is an accelerator bolted onto a boring, working baseline, which is exactly the posture that makes AI fatigue tolerable.

Implementation

Step 1: Rank hunks before you count anything

Token budgets are the wrong first question for a diff. Ranking is cheaper and more honest: a migration ranks above an auth change, which ranks above an endpoint change, while lockfiles and docs sink to the bottom.

# budget.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Hunk:
    path: str
    size: int
    text: str

RANK = {"migration": 0, "auth": 1, "api": 2, "test": 8, "docs": 9, "lock": 10}

def kind_of(path: str) -> str:
    lowered = path.lower()
    return next((k for k in RANK if k in lowered), "src")

def rank(hunks: list[Hunk]) -> list[Hunk]:
    return sorted(hunks, key=lambda h: (RANK.get(kind_of(h.path), 5), -h.size))

def pack(hunks: list[Hunk], char_budget: int) -> tuple[str, int]:
    ordered = rank(hunks)
    total, kept, dropped = 0, [], 0
    for h in ordered:
        cost = len(h.text)
        if total + cost > char_budget:
            dropped += 1
            continue
        kept.append(h)
        total += cost
    return "\n".join(h.text for h in kept), dropped

Note that pack reports how much it dropped. That count is the whole trigger for rule three: dropped > 0 means no model call, ever.

Step 2: One call, one timeout, no retry loop

The handler is deliberately dumb. It has no queue, no backoff, and no second attempt, because a reviewer is waiting and a stale checklist is noise.

# handler.py
import os
from budget import Hunk, pack
from rules import heuristic_checklist, looks_like_checklist

CALL_TIMEOUT_S = float(os.getenv("CALL_TIMEOUT_S", "8"))
CHAR_BUDGET = int(os.getenv("CHAR_BUDGET", "24000"))
MODEL_ENABLED = os.getenv("MODEL_ENABLED", "0") == "1"

def checklist_for(hunks: list[Hunk], call_model) -> tuple[str, str]:
    if not hunks:
        return "L0-empty", "No observable change; nothing to review."
    packed, dropped = pack(hunks, CHAR_BUDGET)
    baseline = heuristic_checklist(packed)
    if not MODEL_ENABLED or dropped > 0:
        return "L1-heuristic", baseline
    try:
        out = call_model(packed, timeout=CALL_TIMEOUT_S)
    except TimeoutError:
        return "L2-timeout", baseline
    if not looks_like_checklist(out):
        return "L2-invalid", baseline
    return "L3-model", out

The return value carries the level string straight into the PR comment footer, so anyone reading the comment knows which path produced it without opening a dashboard.

Step 3: Deploy on the free server

Keep the model behind an environment variable so you can disable it with a config change rather than a redeploy. A minimal process launch looks like this:

python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
python -m pytest -q tests/            # run the fixtures first
MODEL_ENABLED=1 CHAR_BUDGET=24000 CALL_TIMEOUT_S=8 python -m bot.server --port 8080

The same command runs on the free server option and on my laptop, which matters more than it sounds: the ability to reproduce a production comment locally turned two debugging sessions into two minutes.

The test plan that actually caught bugs

Five fixtures, each a saved diff under tests/fixtures/, each asserting a level and a shape:

Fixture	Expected level	Assertion
`empty.diff`	L0	comment has no checklist block
`lockfile_only.diff`	L3 or L1	no endpoint paths mentioned
`oversized.diff`	L1	footer says `L1-heuristic`
`model_timeout`	L2	baseline checklist still present
`model_garbage`	L2	response rejected by shape check

The oversized and garbage fixtures found real problems. The first revealed that my packer silently truncated a single enormous hunk, and the second showed that my shape check accepted prose bullets with no action items. Both are now assertions, not vibes.

Results

Qualitative results were better than the quantitative ones, which is the honest summary. Reviewers used the L1 path-based checklist more than the model output, because it named the migration and the auth module reliably and never hallucinated an endpoint. The model path added value mainly on medium diffs that touched several modules at once.

I am not quoting an accept-rate or a latency percentile here. Any numbers from a two-weekend project on an unknown network would be decoration, and the fixture suite lets you produce your own numbers in under an hour on your repository. If you want one metric, count how often the footer says L2-invalid — that ratio tells you more about your prompt than any satisfaction survey.

Limitations and who should skip this

If your diffs are routinely larger than the character budget, most comments will be L1 and the model integration earns nothing.
If your reviewers expect prose explanations of intent, path-based rules will read as shallow, and that is a fair complaint.
If your repository has no stable directory conventions, ranking by path is close to random.
If you need guaranteed availability and a support contract, a free tier is the wrong foundation regardless of vendor.
Skip this entirely if a human already writes a consistent checklist; you would be automating a solved problem.

Lessons learned

Write the fallback before the prompt. The fallback defines what "useful" means, and the prompt is judged against it.
Never send a partial diff. Silent truncation is a hallucination factory with a clean audit trail.
Put the degradation level in user-visible output. It converts quiet quality drift into something a reviewer can argue with.
Budget in characters, not tokens, at the boundary. You cannot count tokens for a model you have not called yet.

If you want to try the same shape, start with the fixture suite and the ranking function on a free server, keep the model call behind a flag, and only then spend any of the free token allowance on prompts. MonkeyCode's free model access and free server option are what made a two-weekend proof viable for me, and the same arrangement is a reasonable place to test whether the idea survives your own repository.

Case Study: A Record/Replay Cassette That Makes an AI Agent Test Deterministic

Dakota Liu — Sat, 17 Oct 2026 06:20:21 +0000

The first failure in this build had nothing to do with model quality: the agent's test suite went red because a network call timed out, then green again ten minutes later with no code change. Record every model response once into a plain JSONL file, replay it forever in CI, and fail loudly whenever the prompt or request shape drifts. That single rule turned a flaky agent job into a deterministic one, and the replay side costs nothing to run.

Below is the full case study of a small classify-and-route agent: the constraints, the cassette module, the drift guard that saved the most debugging time, and the parts I would not recommend.

Background: an agent that could not be tested

The project was small on purpose. One job reads a failing test log, asks a model to pick the likely owner module, and posts a suggestion to a review thread. Three functions, one prompt file, and roughly forty lines of glue code — the kind of thing you write in an afternoon and then maintain for a year.

The problem showed up in week two. Every assertion compared model output against expected strings, so the suite depended on sampling, provider latency, and whatever the current model snapshot preferred that morning. A red build told you nothing, because you could not separate a regression from a slow Tuesday.

The usual fixes are worse than the disease. Mocking the client returns whatever you typed, so the test asserts your own fiction. Skipping the test in CI means nobody notices when the prompt file quietly loses a section.

Goal and constraints

Deterministic replay. Same input, same output, with no network and no credentials needed.
Loud drift. A changed prompt or request shape must fail with a readable diff instead of passing on stale data.
Cheap recording. The one pass that touches a real model should not require a paid key or a dedicated box.
Reviewable artifacts. A human should be able to read the recorded fixture inside a normal pull request.

Implementation

1. Hash a normalized request, not the raw payload

Providers attach ids, timestamps, and usage counters to responses, and tool-call blocks carry randomly generated identifiers. If you hash the raw payload, identical requests produce different keys, and your cassette grows one duplicate entry per run.

import hashlib, json

VOLATILE = {"id", "created", "system_fingerprint", "usage"}

def normalize(request: dict) -> dict:
    body = {k: v for k, v in request.items() if k not in VOLATILE}
    body["messages"] = [
        {"role": m.get("role"), "content": m.get("content")}
        for m in body.get("messages", [])
    ]
    return body

def request_key(request: dict) -> str:
    blob = json.dumps(
        normalize(request), sort_keys=True, separators=(",", ":"), ensure_ascii=False
    )
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()

Put a template_sha field into the request as well. Editing prompts/classify.md then changes the key automatically, which is exactly the behaviour you want.

2. Store entries as append-only JSONL

One line per call keeps diffs readable and lets you append without rewriting the file. Record mode is idempotent: if the key already exists, return the stored response instead of paying for a second call.

# cassette.py
import json, time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable

class CassetteMiss(RuntimeError):
    def __init__(self, key: str, mode: str):
        super().__init__(f"no recorded response for key {key[:12]} (mode={mode})")
        self.key = key

@dataclass
class Cassette:
    path: Path
    mode: str = "replay"              # "record" | "replay"
    call: Callable[[dict], dict] | None = None
    entries: dict = field(default_factory=dict)

    def __post_init__(self):
        self.path = Path(self.path)
        if self.path.exists():
            for line in self.path.read_text(encoding="utf-8").splitlines():
                if line.strip():
                    entry = json.loads(line)
                    self.entries[entry["key"]] = entry

    def send(self, request: dict) -> dict:
        key = request_key(request)
        if self.mode == "record":
            if key in self.entries:
                return self.entries[key]["response"]      # idempotent re-record
            if self.call is None:
                raise RuntimeError("record mode requires a call function")
            response = self.call(request)
            entry = {
                "key": key,
                "recorded_at": int(time.time()),
                "request": normalize(request),
                "response": response,
            }
            self.entries[key] = entry
            with self.path.open("a", encoding="utf-8") as fh:
                fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
            return response
        try:
            return self.entries[key]["response"]
        except KeyError:
            raise CassetteMiss(key, self.mode) from None

3. Turn a miss into a readable diff

A missing key is the most common failure, and a bare hash tells you nothing useful. Because you stored the normalized request next to the response, you can show the closest entry and print only the fields that changed.

def explain_miss(cassette, request):
    current = normalize(request)
    best, best_score = None, -1
    for entry in cassette.entries.values():
        score = sum(1 for k, v in current.items() if entry["request"].get(k) == v)
        if score > best_score:
            best, best_score = entry, score
    if best is None:
        return "cassette is empty; record a new entry"
    diffs = [
        f"{k}: recorded={best['request'].get(k)!r} current={current.get(k)!r}"
        for k in sorted(set(best["request"]) | set(current))
        if best["request"].get(k) != current.get(k)
    ]
    return "\n".join(diffs)

Nine times out of ten the diff names template_sha, and you immediately know a prompt edit caused the miss rather than a provider outage.

4. Wire the cassette into pytest

# tests/conftest.py
import os, pytest
from pathlib import Path
from cassette import Cassette

@pytest.fixture
def cassette(request):
    mode = os.environ.get("CASSETTE_MODE", "replay")
    path = Path(__file__).parent / "cassettes" / f"{request.node.name}.jsonl"
    return Cassette(path, mode=mode)

Run plain pytest and the suite replays. Run CASSETTE_MODE=record pytest -k classify only when you intentionally change behaviour, then review the cassette diff exactly like any other fixture file.

5. Guard freshness so green stays honest

Replay suites are dangerously easy to keep green forever. Add one test that compares the recorded template hash with the current prompt file, and make the failure message actionable.

# tests/test_freshness.py
import hashlib, json
from pathlib import Path

def test_prompt_matches_cassette():
    template = Path("prompts/classify.md").read_bytes()
    first = Path("tests/cassettes/test_classify_issue.jsonl").read_text(
        encoding="utf-8"
    ).splitlines()[0]
    recorded = json.loads(first)["request"]["template_sha"]
    assert recorded == hashlib.sha256(template).hexdigest(), (
        "prompt changed; re-record the cassette on purpose"
    )

Where free model access and a free server fitted in

The record pass is the only step that needs a live model, and it is also the step that runs least often. That is precisely where MonkeyCode's free model access and free server option were useful in this build: the recording run happened on the hosted free server rather than behind a paid key on someone's laptop, and the resulting JSONL was committed for everyone else.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two cautions matter more than the convenience. Treat any free tier as a moving target and check the project's current terms for which models are available and what limits apply, because availability and quotas change. Also remember that a free model is a reasonable recording source and a poor benchmarking source, so do not publish quality comparisons built on it.

Checks you can run without trusting me

No network on replay. Install pytest-socket and run pytest --disable-socket; the suite should still pass with zero outbound calls.
No credentials on replay. Unset your provider API key, run pytest, and confirm nothing fails with an auth error.
Drift is loud. Change one sentence in prompts/classify.md and watch the freshness test fail with the message above.
Diffs stay small. Adding a new test case adds one JSONL line, so review cost grows linearly instead of exploding.

Decision table

Situation	Record	Replay	Live call
Unit tests in CI	no	yes	no
Prompt iteration	yes	—	yes
Post-deploy smoke test	no	no	yes
Comparing two models	yes, per model label	yes	—

Limitations and who should not use this

Open-ended generation. If your product ships freeform prose, exact replay tests the plumbing rather than the quality, so you still need a rubric or a human review step alongside it.
Stale cassettes hide real regressions. The freshness test catches prompt edits, not upstream behaviour changes, so schedule a deliberate re-record on a fixed cadence.
Secrets leak easily. Redact before writing anything to disk, and never commit a cassette that contains raw user data or tokens.
Tiny projects. If you have one test and one provider call, a single hand-written mock is simpler and this machinery costs you an afternoon for no benefit.

Lessons from the build

Normalize before you hash, because provider ids will betray you otherwise.
Store the request beside the response, since debugging is mostly a diffing exercise.
Make drift a failing test rather than a note in a README nobody reads.
Keep recording separate from asserting, so nobody re-records just to turn CI green.

If you want to try the record pass before wiring a paid key, start with the project's free model access and free server option, commit the cassette, then delete your provider credentials and confirm the replay suite still passes.

Case Study: A Nightly Maintenance Worker That Rebuilds Its Own State After Every Restart

Dakota Liu — Fri, 16 Oct 2026 05:20:52 +0000

A nightly maintenance worker needs very little: a process that starts on schedule, one file of durable state, and a cheap way to label what it finds. What it really needs is to stay boring under failure, so you can ignore it for a month and still trust its output.

This case study covers one small project end to end — a worker that scans a handful of repositories for dependency drift, labels each finding, and writes a short digest. The scanning is easy. The hard part is the state handoff that lets the worker restart at any moment, on a host you do not administer, without double-posting or losing its cursor.

The trend feed this week is full of AI feature tours and open AI fatigue, and both reactions are reasonable. The useful counter-move is duller: wrap the probabilistic part in a deterministic shell, so a bad model day cannot corrupt your data.

Goal and non-goals

The goal fits in one sentence: given a list of repositories and pinned versions, produce one digest per day, at most once, with a resumable cursor and an auditable reason for every label.

The non-goals matter just as much, because they are what keeps the design small:

No web dashboard, no database, and no queue broker.
No multi-worker parallelism in version one.
No upstream writes — the worker only reads tags and appends to its own state file.
No strict dependence on any single model endpoint for correctness.

That last point is the design hinge. Classification quality is allowed to vary; the digest itself must not.

Decide where the worker lives before you write any code

Most teams skip this table and regret it later. Score your own options against your own constraints.

Option	Survives laptop sleep	Secret exposure	Restart cost	Reasonable fit
Your laptop cron	No	Low	High, manual	Prototyping only
CI scheduled job	Yes	Medium, in CI secrets	Low	Repos already in CI
Free server instance	Usually	Medium, in env vars	Low if stateless	One small worker, tolerant of restarts
Paid VM with a supervisor	Yes	Medium	Low	Delivery SLAs, regulated data

If your job touches regulated data, or another team depends on its output at a fixed hour, skip the free tier row entirely. The rest of this article assumes you accepted restarts as normal and designed for them.

The state file is the only durable contract

Write every piece of resumable knowledge into one JSON document, and treat that document as the interface between runs. Anything not in the file is reconstructed on startup.

{
  "schema_version": 1,
  "last_completed_run": "2026-10-15T02:10:04Z",
  "cursor": {"repo": "acme/etl", "last_tag": "v3.14.0"},
  "lease": {
    "owner": "worker-7f3c",
    "acquired_at": "2026-10-15T02:09:41Z",
    "heartbeat_at": "2026-10-15T02:09:52Z"
  },
  "recent_labels": [{"repo": "acme/etl", "tag": "v3.14.0", "label": "routine"}]
}

Two rules keep this file trustworthy. First, bump schema_version whenever the shape changes, and refuse to run on an unknown version. Second, only ever commit the file with an atomic rename, so a kill signal cannot leave a half-written document on disk.

import json
import os
import tempfile


def save_state(path, state):
    directory = os.path.dirname(os.path.abspath(path)) or "."
    fd, tmp = tempfile.mkstemp(dir=directory, prefix=".state-", suffix=".tmp")
    try:
        with os.fdopen(fd, "w") as handle:
            json.dump(state, handle, sort_keys=True)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp, path)  # commit point
    except BaseException:
        os.unlink(tmp)
        raise

The os.replace call is the commit point, and everything before it is invisible to readers. If the process dies during the write, you lose a temporary file and nothing else. If it dies after the rename, the new state is simply the current state.

A lease that fails loudly instead of double-running

Two workers racing on the same cursor is the classic way a nightly job becomes a nightly outage. A lease with a heartbeat solves this without any coordination service, as long as you accept a bounded risk of a stale takeover.

from datetime import datetime, timedelta, timezone

LEASE_TTL = timedelta(minutes=15)


def parse_ts(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def lease_blocks_run(state, now, owner):
    lease = state.get("lease")
    if not lease:
        return False, "no lease recorded"
    age = now - parse_ts(lease["heartbeat_at"])
    if lease["owner"] == owner:
        return False, "we already own the lease"
    if age > LEASE_TTL:
        return False, "stale lease from %s" % lease["owner"]
    return True, "held by %s for %s" % (lease["owner"], age)

Heartbeat every few minutes, always with the current UTC timestamp, and log the reason string on every exit path. When someone asks why a run stopped, that line is the whole answer.

Model calls behind a deterministic fallback

The labeling step is where a model is genuinely useful and where a free tier is genuinely risky. Keep the call behind a function that can always return a value, and let a rules-based classifier own the default path.

ALLOWED = {"routine", "risky", "unknown"}


def classify(title, call_model, rules_fallback):
    try:
        label = call_model(title, timeout=20)
        if label in ALLOWED:
            return label, "model"
    except Exception as exc:
        print("model tier unavailable: %s" % exc)
    return rules_fallback(title), "rules"

Record the source (model or rules) next to every label. That single field turns a vague quality complaint into a queryable ratio, and it makes provider changes measurable instead of anecdotal.

The code above is a proposal you can run locally with a stub call_model, not a published benchmark. Nothing in this design depends on a specific endpoint, model name, or provider — which is exactly why it survives a free tier changing underneath you.

Where the free tier fits this design

The worker needs two things outside your laptop: a host for the process and an endpoint for the labeling step. MonkeyCode provides free model access and a free server option, which matches both needs in the shape this design already expects. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat free availability as operator-supplied and time-sensitive. Token allowances, rate limits, data retention, and instance lifetimes on free tiers change, so read the current terms on the official site before you schedule anything that matters, and keep the rules-based fallback in place so a policy change degrades quality instead of halting the job.

Results: what each failure does to the run

The value of this structure is not speed; it is a short, predictable list of failure behaviors. Verify each row yourself with the harness below.

Failure	Expected behavior
Killed mid-write	Temporary file discarded, previous state intact
Two workers start together	Second worker exits with the lease reason logged
First worker hangs	Lease goes stale after the TTL, next run takes over
Model endpoint unavailable	Labels fall back to rules and are marked as such
State file schema changed	Preflight exits non-zero instead of guessing

Notice what is absent: no partial digest, no duplicated post, no silent data loss. Every failure mode is either a clean refusal or a degraded-but-labeled result.

Two commands to run before you trust it

Write a preflight check and run it as the first step of every scheduled execution. It should parse the state file, compare schema_version, verify the lease, and check clock skew against a known time source.

python -m worker.preflight --state .state/worker.json && \
  python -m worker.run --once

Then test idempotency directly, because this is the property that actually breaks in production. Run the worker twice in a row against an unchanged input set, and diff the state file after each pass.

python -m worker.run --once && cp .state/worker.json /tmp/after-first.json
python -m worker.run --once && diff /tmp/after-first.json .state/worker.json

A clean run should update last_completed_run and nothing else. If the cursor moves on the second pass, your idempotency assumption is wrong, and no amount of logging will save you.

Limitations and who should not use this

This approach trades throughput and guarantees for simplicity, and that trade is not always right.

Do not use it if your job must run at an exact minute with an SLA, because free hosts restart, throttle, and cold-start without warning.
Do not use it for regulated or customer-identifying data, because you do not control retention on a free endpoint.
Do not use it for multi-writer workflows, because a single lease file assumes one logical writer at a time.
Do not use it if you need sub-minute latency or persistent local disk, since both assumptions are gone on ephemeral hosts.
Do not skip the fallback classifier, because a free model tier is a shared resource with shared availability patterns.

If your job sits in any of those categories, the table at the top of this article points you elsewhere, and that is a legitimate outcome.

What I would change next

The next iteration is to make the digest itself a contract: fix the key order, the label vocabulary, and the timestamp format, then assert them in preflight the same way you assert schema_version. After that, add one integration test that starts two processes against a shared state directory and asserts that exactly one of them reaches the labeling step. Those two checks catch the overwhelming majority of failures you will actually see in the first year.

If you want to try the loop without provisioning anything, start by running the preflight check on a free server instance and letting it complete a single pass before you add a schedule.

Case Study: A Golden-Set Gate That Lets a Free Model Tier Serve One Boring CI Job

Dakota Liu — Thu, 15 Oct 2026 11:26:46 +0000

Short answer up front: you can move one narrow, non-customer-facing job onto a free model tier, provided you freeze the output contract first and gate every swap with a golden set. This case study walks through a CI log-triage CLI, the small harness that decides which model may serve it, and the parts I intentionally kept manual.

Background: one boring job with an unusually strict contract

triage reads a failed CI job's raw log and prints exactly one bucket, one short reason, and one evidence line number. The buckets are deliberately boring: flaky_test, missing_env, dep_conflict, oom, timeout, permission, and unknown. A discussion bot posts that result into the pull request thread, so a malformed object is more damaging than a slightly wrong label. The job had been running against a paid API at low but non-zero volume, which is precisely the profile worth testing on a free tier.

MonkeyCode is an open-source project offering free model access and a free server option, both of which the operator describes as current parts of its offering. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am deliberately not quoting a token allowance or any machine detail here, because those change often and a stale number in a tutorial is worse than no number at all. Check the project's live terms before you size anything around it.

Goal, and the things I refused to change

The goal was narrow: keep triage's stdout contract byte-identical while the model behind it became negotiable.

One JSON object on stdout, exactly three keys, exit code 0 on success.
Every unknown bucket still routes to a human, because that is where silent wrongness hides.
A model swap must be a reviewable diff rather than a runtime experiment.
The whole check has to finish in under a minute so it can run on every push.

The non-goals matter just as much, because scope creep is how a free-tier experiment turns into an incident.

No streaming, no chat surface, and no multi-turn agent loop around the classifier.
No claim of general parity with the paid model, only parity on this one frozen task.
No automatic promotion of a new model; a human merges that change on purpose.

Decision table: which jobs deserve a free tier

Use this as a filter before you write any adapter code at all.

Job	Customer-visible	Output strictness	Move it?
CI log triage	No, internal bot only	One enum plus two fields	Yes
PR title cleanup	No	Free-form text	Yes, if nobody blocks on it
Release notes draft	Yes, public changelog	Tone plus accuracy	No, keep a human reviewer
Support reply generation	Yes, sent to customers	Legal and brand risk	No

The pattern is simple: strict, machine-checkable output plus an internal audience is where a free tier earns its place.

The artifact: a golden set and a small gate

The harness is the durable part of this project, not the prompt. Set up the layout first, then keep every number in version control next to the code.

triage/
├── adapter.py       # the only file that knows about a provider
├── golden.py        # scoring plus the swap gate
├── baseline.json    # the accepted score of the current model
└── golden/
    ├── 01-timeout.log
    ├── 01-timeout.expected.json
    └── ...

Step 1: freeze the contract in code

Write the validator before you talk to any model, because it becomes the definition of done for the whole task.

#!/usr/bin/env python3
# golden.py — score a model against a frozen contract, then gate the swap.

BUCKETS = ['dep_conflict', 'flaky_test', 'missing_env', 'oom',
           'permission', 'timeout', 'unknown']

def validate(obj):
    # Return None when the object satisfies the contract, else a short reason.
    if not isinstance(obj, dict):
        return 'not_object'
    if set(obj) != {'bucket', 'reason', 'evidence_line'}:
        return 'keys'
    if obj['bucket'] not in BUCKETS:
        return 'bucket_enum'
    reason = obj['reason']
    if not isinstance(reason, str) or not reason.strip():
        return 'reason_empty'
    if len(reason) > 160:
        return 'reason_long'
    line = obj['evidence_line']
    if not isinstance(line, int) or isinstance(line, bool) or line < 1:
        return 'evidence_line'
    return None

Step 2: build a small golden set, including impossible cases

Thirty anonymized logs were enough for this task, but five of them were written to be genuinely ambiguous on purpose.

Twenty-five logs with one obvious cause and a stable expected bucket.
Five logs where two causes are equally plausible, expected bucket unknown.
The expected answer stored as JSON next to each log and reviewed like a fixture.
Hostnames, tokens, and repository names anonymized before anything leaves the machine.

Those five ambiguous rows are the most valuable entries in the set, because they catch a model that guesses confidently.

Step 3: keep exactly one provider-aware file

Everything that knows about a network endpoint lives in adapter.py, and the shape below is illustrative, so match your provider's actual API.

# adapter.py — keep this file small and boring.
import os, json, urllib.request

def complete(prompt: str, timeout: int = 60) -> str:
    # Shape below is illustrative; match your provider's real request format.
    req = urllib.request.Request(
        os.environ['FREE_MODEL_URL'],
        data=json.dumps({'prompt': prompt}).encode(),
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + os.environ['FREE_MODEL_KEY']},
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read())['text']

Step 4: score schema first, accuracy second

Schema failures are cheap to measure and expensive to ignore, so they belong first in the report.

import json

PROMPT = '''You classify CI failure logs.
Return one JSON object only, no prose and no markdown fence.
Keys: bucket, reason (160 chars max), evidence_line (integer, 1 or more).
If two explanations are equally likely, use the bucket unknown.

LOG:
{log}'''

def call_model(text):
    raw = complete(PROMPT.format(log=text)).strip()
    if raw.startswith('```

'):                       # models love fences
        raw = raw.split('\n', 1)[1].rsplit('

```', 1)[0]
    return json.loads(raw)

def score(cases):
    failures, bucket_ok, abstain_ok, abstain_total = [], 0, 0, 0
    for path, expected in cases:
        try:
            obj = call_model(path.read_text())
        except Exception as exc:                    # network, JSON, timeout
            failures.append((path.name, 'call:' + type(exc).__name__))
            continue
        bad = validate(obj)
        if bad:
            failures.append((path.name, bad))
            continue
        bucket_ok += obj['bucket'] == expected['bucket']
        if expected['bucket'] == 'unknown':
            abstain_total += 1
            abstain_ok += obj['bucket'] == 'unknown'
    return {
        'cases': len(cases),
        'schema_pass': 1 - len(failures) / len(cases),
        'bucket_acc': bucket_ok / len(cases),
        'abstain_recall': abstain_ok / abstain_total if abstain_total else 1.0,
        'failures': failures,
    }

Step 5: make the gate exit non-zero

A gate that only prints numbers is a dashboard, and dashboards get ignored during a busy week.

GATE = {'schema_pass': 1.00, 'bucket_drop': 0.05, 'abstain_recall': 0.80}

def verdict(result, baseline):
    reasons = []
    if result['schema_pass'] < GATE['schema_pass']:
        reasons.append('schema_pass below 1.00')
    if result['bucket_acc'] < baseline['bucket_acc'] - GATE['bucket_drop']:
        reasons.append('bucket_acc dropped more than 0.05')
    if result['abstain_recall'] < GATE['abstain_recall']:
        reasons.append('abstain_recall below 0.80')
    return reasons

Wire verdict into main, write the raw scores into results/<date>.json, and exit with code 1 whenever that reason list is non-empty.

Running the gate on the free server instead of a laptop

A check that only runs on your laptop is a check that stops running the week you travel. The operator's free server option fits this stage well, because the harness needs no GPU, no private data, and no long-lived state beyond a results directory.

# nightly, on whatever host you already trust
python golden.py --baseline baseline.json --out results/$(date -u +%F).json
echo exit=$?   # 0 means the current model still passes

Keep the golden logs anonymized, keep the results inside the repository, and never let the nightly job swap the model by itself.

Results: what the harness actually decides

The exact figures below are an illustrative dry run, so reproduce them against your own baseline before you trust any of it.

Signal	Illustrative dry run	Gate rule
schema_pass	1.00	must equal 1.00
bucket_acc	baseline 0.83, candidate 0.80	no worse than baseline minus 0.05
abstain_recall	0.80, four of five ambiguous logs	at least 0.80
wall clock	roughly forty seconds for thirty logs	under one minute

The useful outcome is not the score itself, but the failure list. Most rejected candidates failed on keys or reason_long rather than on the bucket label, which means prompt hardening, not a better model, was the real fix.

Lessons learned

Freeze the contract before you shop around, because a moving target cannot be evaluated fairly.
Score schema conformance first, since a malformed object breaks automation more quietly than a wrong label.
Keep at least five impossible cases, because confident guessing is the failure mode that hurts most.
Confine provider knowledge to one small file so a swap never touches business logic.
Make the gate exit non-zero, because an advisory check is a suggestion and suggestions lose to deadlines.

Who should not use this approach

Skip it if the output is customer-visible prose, if a wrong answer carries legal or financial blast radius, or if nobody will maintain the labeled examples over time. Skip it as well for latency-critical paths, where a free tier may not offer the tail behavior you need. The method fits internal, boring, machine-checkable jobs and very little else.

A closing note on where this fits

If you already keep a golden set for a paid model, pointing that same harness at a free tier costs one afternoon and buys you a real answer about your own contract. Run your own numbers on your own logs, then let the gate make the call instead of your optimism.

Case Study: A Contract Fixture That Stops a Widened HTTP Client

Dakota Liu — Tue, 13 Oct 2026 01:57:52 +0000

You should refuse an agent-written HTTP client until a local contract fixture agrees with every status that client treats as success. A helpful model will often widen success to any 2xx response, and that shortcut hides conflicts, rate limits, and partial writes. This case study walks through one small project that freezes those decisions in a replay harness before you merge the diff. The harness stays useful even if you draft the client by hand and never open an assistant at all.

Background

Suppose you needed a thin client for a partner orders API, and you asked a coding assistant to add create_order plus basic error handling. The draft treated every status from 200 through 299 as success, and it logged 409 and 429 as generic failures. In this scenario the service already retries 429, and it treats 409 as a conflict that must surface to the caller. A widened client would mark a conflict as created, and a later worker would double-charge or double-ship.

Goal

The goal is a one-directory project that records expected status classes and rejects a client that disagrees with that record. You are not trying to mock the whole network, and you are not trying to score the model. You want a binary gate: the classifier either matches the fixture or the patch does not land. A second goal is to keep the gate runnable on a small free server, so the check does not depend on your laptop.

Implementation

The project has three files: a JSON fixture, a classifier module, and a small test runner you can read in one sitting. You write the fixture yourself from the partner's published status table, and you let the assistant draft only the classifier. The runner never calls the network, so a green result means agreement with your record, not proof that the partner API is healthy.

Fixture shape

Store one object per case, with the status, a short body, and the class your service already understands. Keep the file in git so a later agent cannot fix the tests by editing expectations in the same patch. If the partner changes a code, you change the fixture in a separate commit that a human reviews.

{
  "cases": [
    {"name": "created", "status": 201, "body": {"id": "o1"}, "expect": "success"},
    {"name": "conflict", "status": 409, "body": {"code": "dup"}, "expect": "conflict"},
    {"name": "rate_limited", "status": 429, "body": {"retry_after": 2}, "expect": "retry"},
    {"name": "server", "status": 503, "body": {"code": "down"}, "expect": "retry"},
    {"name": "bad_request", "status": 400, "body": {"code": "form"}, "expect": "fail"}
  ]
}

Classifier under test

The classifier is the only function the assistant is allowed to rewrite in this small project. It must return one of four labels, and it must not swallow unknown statuses into success. The version below is the contract you want, written as a reference implementation you can compare against a generated patch. Label this listing as the reference, and treat any assistant draft as untrusted until the harness agrees.

def classify(status, body):
    if status == 201:
        return "success"
    if status == 409:
        return "conflict"
    if status in (429, 503):
        return "retry"
    if 400 <= status < 500:
        return "fail"
    raise ValueError(f"unclassified status {status}")

The widened draft you reject

The following function is the failure this case study is built to catch, and you should not ship it. It marks 409 as success, which would hide a duplicate order from the caller that needs to resolve the conflict. Run the harness against this function and you should see the conflict case fail before any merge command runs. Point the harness import at this function only long enough to see the rejection, then restore the reference classifier.

def classify_widened(status, body):
    if 200 <= status < 300 or status == 409:
        return "success"
    if status in (429, 503):
        return "retry"
    return "fail"

Harness

The runner loads the fixture, calls classify, and exits non-zero on the first mismatch it finds. It also fails if the classifier raises on a status you already recorded, because a crash is not a mapped error. You can drop this file next to the classifier and run it with the standard library alone. The example is complete and unexecuted in this article, so you should run it yourself before you trust the printout.

import json
import sys
from client import classify

def main():
    fixture = json.load(open("contract.json", encoding="utf-8"))
    failures = []
    for case in fixture["cases"]:
        try:
            got = classify(case["status"], case["body"])
        except Exception as exc:
            failures.append(f"{case['name']}: raised {exc}")
            continue
        if got != case["expect"]:
            failures.append(
                f"{case['name']}: expected {case['expect']} but got {got}"
            )
    if failures:
        print("\n".join(failures))
        return 1
    print(f"ok: {len(fixture['cases'])} cases matched")
    return 0

if __name__ == "__main__":
    sys.exit(main())

How you run the gate

Create a scratch directory, save the fixture as contract.json and the runner as harness.py, and save the reference classifier as client.py. If you want a generated classifier, paste only client.py into the assistant and tell it not to edit the fixture or the runner. Then rerun the same command and treat a non-zero exit as a rejected patch you do not merge. A widened classifier that maps every 2xx and 409 to success fails the conflict case and stops the merge.

mkdir -p order-client && cd order-client
python3 harness.py
# reference classifier prints:
# ok: 5 cases matched

The command below repeats the widened function so you can see the bad label without changing client.py. A printed success on status 409 is the mismatch, and it is enough to reject the draft. You do not need a network call for that comparison, because the fixture already names the expected class.

python3 - <<'PY'
def classify_widened(status, body):
    if 200 <= status < 300 or status == 409:
        return "success"
    if status in (429, 503):
        return "retry"
    return "fail"

print(classify_widened(409, {"code": "dup"}))
# widened draft prints: success
# fixture expects: conflict
PY

A decision table for review

Use this table when you review a generated classifier, and do not accept a label that is missing from the expected column. The last row is the fail-closed rule: an unlisted status must not become success, even if the body looks harmless. You can add rows only in a human-reviewed fixture commit, and the harness should then fail until the classifier learns the new class.

Status	Expected class	Acceptable return	Reject if you see
201	success	success	conflict, retry, fail
409	conflict	conflict	success
429	retry	retry	success or fail
503	retry	retry	success
400	fail	fail	success or retry
any other	unclassified	raise	success

Where free model access fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is relevant here as a place to draft client.py with free model access and a free server option. The fixture stays in your repo, and the server only runs the harness you already reviewed. Those two availability claims come from the operator for this draft, and they are not a promise about model names, quotas, or hardware. They also say nothing about how long free access lasts, so you should confirm the project page before you depend on it.

If you use that server, copy the fixture up with the runner and do not let the session rewrite contract.json. A model that can edit the expected labels can make a bad client look green, which is the failure this project exists to catch. The same gate still works on your own machine if the free server option is missing or the terms have changed.

Results you can claim

You can claim a match only after your own run prints the five-case line, because this article did not execute the listing for you. You cannot claim that a particular model passes, that latency improved, or that the partner API matches the fixture. Treat the listing as a complete example you run locally, not as a production incident report or a measured model evaluation. If your run prints a mismatch, the patch is rejected until you fix the classifier or change the fixture in a reviewed commit.

Lessons

Freeze status classes in a file the assistant is not allowed to edit during the same change.
Treat unknown statuses as errors rather than success, so a new partner code fails closed instead of looking created.
Keep the runner on the standard library so a small free server without extra packages can still execute the gate.
Separate a drafted file from an agreed fixture, and record only the fixture result as your merge reason.

Limitations and who should skip this

This harness does not prove authentication, pagination, idempotency keys, or clock-related retry headers in the partner response. It does not replace a contract test against a staging server, and it will not catch a body change that keeps the same status. You should not use it as your only check for payments or any client where a wrong success label moves money or records. Teams that already run consumer-driven contract tests against a live stub can skip this smaller gate.

If you cannot keep the fixture under human review, the approach collapses, because a rewritten expectation file will bless a widened client. Do not point this harness at production traffic, and do not treat a green local run as permission to skip staging. The five cases are illustrative, and your real table should come from the partner document you can cite, not from a model's memory.

Closing

Start with cases that already have a meaning in your service, and add a case only after a human reads the partner document. When a draft disagrees, fix the classifier or update the fixture in its own commit, and do not reconcile both files in one prompt. The merge reason is the harness exit code, not the confidence of the draft that produced the classifier. That split is the whole lesson of this case study, and it holds whether you type the client or accept a generated one.

Case Study: An Allowlist Gate That Rejects an Agent Patch Before You Apply It

Dakota Liu — Sun, 11 Oct 2026 21:48:50 +0000

You should reject an agent-written patch before it touches your tree, unless a written allowlist and a named test command both pass. Free generation makes extra files cheap to propose, while a wrong file remains expensive to undo. This case study walks you through a tiny Python gate you can run on any disposable workspace. The gate is a review tool, not a claim that any model or host is safe.

What you are protecting

You are protecting a three-file health service, not a production platform and not a secret-bearing repository. The service exposes /health, and you want a second route named /ready with one focused test. An agent can draft that change quickly, yet it can also edit deploy files, lockfiles, or a hidden env sample. Your job is to keep the applied diff inside a card you wrote before the session started.

Background

Picture a small folder that already contains app.py, test_app.py, and README.md, with no database and no cloud credentials. You ask an assistant to add readiness behavior, and you expect a unified diff plus a short report. The report must name the exact test command you will run after the patch applies cleanly. If either artifact drifts outside the card, you stop and you do not apply the patch.

An agent draft is still an untrusted patch, even when the prose around the diff sounds finished and careful. You need a local rule that does not depend on how confident the report appears in the chat window. This exercise does not rank models, and it does not borrow a scoreboard from a public thread. You are freezing one small contract that names allowed paths, forbidden paths, and a test command.

Goal

The goal is a fail-closed checker that exits nonzero unless three conditions hold at the same time. The diff may touch only paths on the allowlist, and it must not touch any forbidden path. The agent report must contain the exact test command that you already wrote on the card. You treat missing fields as failure, because a silent default is how an unreviewed file lands in the tree.

The session card

You write the card before you prompt, and you store it beside the checker rather than inside the agent's scratch space. You keep the card in the repository you control, so a later edit to the prompt cannot quietly widen the allowlist. You treat the byte cap as your own courtesy limit, not as a limit advertised by any host. You can change the goal line, but you should not add deploy files just because a draft looks tidy.

# session_card.yaml - proposal, not a production policy
goal: Add GET /ready that returns 200 and a ready status
allowed_paths:
  - app.py
  - test_app.py
forbidden_paths:
  - .env
  - .env.example
  - credentials.json
  - deploy/
  - .git/
required_test_command: python -m unittest test_app.py
max_diff_bytes: 12000

Implementation

The checker below is a labeled, unexecuted example that you should read carefully before you run it. It uses only the Python standard library, so you do not add a dependency just to review a patch. It reads the card, the diff, and the report, then prints one JSON line for a completed check. You should treat a parser error or a bad argument count as a failed review before any patch is applied.

#!/usr/bin/env python3
"""session_gate.py - unexecuted example. Fail closed on an agent diff."""

import json
import sys
from pathlib import Path


def load_card(path: Path) -> dict:
    card = {"allowed_paths": [], "forbidden_paths": []}
    key = None
    for raw in path.read_text(encoding="utf-8").splitlines():
        line = raw.split("#", 1)[0].rstrip()
        if not line.strip():
            continue
        if line.startswith("  - "):
            if key is None:
                raise SystemExit("list item before a key")
            card.setdefault(key, []).append(line.strip()[2:].strip())
            continue
        if ":" not in line:
            raise SystemExit("card line missing colon")
        key, value = line.split(":", 1)
        key, value = key.strip(), value.strip()
        if value:
            card[key] = int(value) if value.isdigit() else value
        else:
            card.setdefault(key, [])
    required = ("goal", "required_test_command", "max_diff_bytes")
    missing = [name for name in required if name not in card]
    if missing or not card["allowed_paths"]:
        raise SystemExit(f"incomplete card: {missing or 'allowed_paths'}")
    return card


def changed_paths(diff_text: str) -> list[str]:
    paths = []
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:].strip()
            if path and path != "/dev/null":
                paths.append(path)
    return paths


def main() -> int:
    if len(sys.argv) != 4:
        print("usage: session_gate.py CARD DIFF REPORT", file=sys.stderr)
        return 2
    card = load_card(Path(sys.argv[1]))
    diff_text = Path(sys.argv[2]).read_text(encoding="utf-8")
    report = Path(sys.argv[3]).read_text(encoding="utf-8")
    reasons = []
    if len(diff_text.encode("utf-8")) > int(card["max_diff_bytes"]):
        reasons.append("diff exceeds max_diff_bytes")
    allowed = set(card["allowed_paths"])
    forbidden = tuple(card["forbidden_paths"])
    for path in changed_paths(diff_text):
        if path in forbidden or any(path.startswith(prefix) for prefix in forbidden):
            reasons.append(f"forbidden path: {path}")
        elif path not in allowed:
            reasons.append(f"path outside allowlist: {path}")
    command = str(card["required_test_command"])
    report_lines = [line.strip() for line in report.splitlines()]
    if command not in report_lines:
        reasons.append("report missing required_test_command")
    if "+++ " not in diff_text:
        reasons.append("diff has no file headers")
    result = {"ok": not reasons, "reasons": reasons, "command": command}
    print(json.dumps(result, ensure_ascii=True))
    return 0 if result["ok"] else 1


if __name__ == "__main__":
    raise SystemExit(main())

Commands you run

You create two fixtures, one clean and one dirty, then you run the gate against each fixture. These commands are the reproducible artifact, and they do not call a network or a model. You keep the fixtures in a local fixtures directory, and you do not fetch them from an untrusted gist. You run each command in a clean shell so a previous exit code cannot confuse the result you record.

Local fixtures

Save this as fixtures/ready_ok.diff. It is a proposal hunk, not a patch this article applied to a live service.

--- a/app.py
+++ b/app.py
@@ -1,2 +1,5 @@
 def health():
     return {"status": "ok"}
+
+def ready():
+    return {"status": "ready"}
--- a/test_app.py
+++ b/test_app.py
@@ -1,2 +1,8 @@
 import unittest
+from app import ready
+
+
+class ReadyTest(unittest.TestCase):
+    def test_ready(self):
+        self.assertEqual(ready()["status"], "ready")

Save fixtures/ready_ok.md as one exact line, with no extra words on that line.

python -m unittest test_app.py

Save fixtures/ready_bad.diff as a forbidden-path example. The token below is a placeholder, not a credential you should reuse anywhere.

--- a/.env
+++ b/.env
@@ -0,0 +1 @@
+API_TOKEN=example-not-real

A clean diff should mention only app.py and test_app.py, and the matching report should repeat python -m unittest test_app.py exactly. A dirty diff should add .env or deploy/app.yaml, or the matching report should omit the required command. You apply the patch only after the clean run prints "ok": true and you have read the diff yourself. You discard the dirty fixture even if the prose around it sounds confident, because the gate exists to ignore tone.

python3 session_gate.py session_card.yaml fixtures/ready_ok.diff fixtures/ready_ok.md
python3 session_gate.py session_card.yaml fixtures/ready_bad.diff fixtures/ready_bad.md

Apply only after a clean gate

patch -p1 --dry-run < fixtures/ready_ok.diff
patch -p1 < fixtures/ready_ok.diff
python -m unittest test_app.py

You use --dry-run first so a malformed hunk cannot rewrite a file before you have inspected the reject list. You run the unittest only after a real apply in that same workspace, not in a different checkout with different code. You stop immediately if the dry run, the apply, or the unittest fails, and you do not let the agent edit in place. You save the JSON line next to the diff so a later reviewer can see why you applied or refused the change.

Expected results

Specified exits

This section describes the specification of the example, not a measured production result and not a benchmark. A clean fixture should exit 0 and print a JSON object whose ok field is true. A dirty fixture should exit 1 and list either a forbidden path or a missing command. An incomplete card should exit through SystemExit before any diff is judged, because a missing allowlist must not become allow everything.

Decision table

Fixture	What changed	Report	Expected
clean	`app.py`, `test_app.py`	exact command line	exit 0
forbidden	adds `.env`	exact command line	exit 1
silent report	allowed files only	command omitted	exit 1
broken card	any diff	any report	abort before judgment

You can treat the table as the test plan, and you should not claim a pass until your machine prints the same exits. A row that aborts is still a failure, even when the diff itself would have been small and boring. You record the JSON beside the fixture name, so a later run can be compared without trusting memory. You should rerun the table after any parser edit, because a green memory of yesterday is not evidence about today's script.

Extra cases worth trying

You should also try an empty diff, which this checker rejects because it finds no file headers at all. A delete hunk that uses a null header can slip past this scan, so you should treat that pass as unfinished. If that null-header case passes, you revise the path parser before any agent draft is allowed near the tree. No latency number, token count, or pass rate is claimed here, because this example was not executed against a live service for this article.

Where free model access fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is relevant only as the place you might draft the patch and run the gate, if you choose that tool. The operator-supplied notes for this draft say that free model access and a free server option exist. This article does not name models, quotas, hardware, duration, or permanence, because those details were not verified against a primary source dated 2026-10-12. You should read the current product docs before you rely on any limit, and you should assume a free workspace can disappear or reset.

A split you can actually run

A practical split keeps the product mention from doing the review work that you still owe the tree. You may use free model access to propose app.py and test_app.py only, with the card pasted into the prompt as a constraint. You may use a free server as a disposable workspace to run session_gate.py and python -m unittest, away from your laptop secrets. You still copy the diff back by hand, and you still refuse the patch when the JSON says ok is false.

Confirm the docs first

If you want to try that split, start from the current MonkeyCode docs and confirm what the free options include today. Run the two fixture commands before you send a real repository, and stop if the docs disagree with this draft. That confirmation step is the only invitation this case study makes, and it is not a promise about price or capacity. You can ignore the product entirely and still keep the card, the checker, and the fail-closed rule.

Limitations

You should read these limits before you reuse the script, because a teaching checker hides several sharp edges. None of these limits is a reason to skip the gate, but each one is a reason not to overclaim what it proves. You can replace the parser later, and you should do that before anyone treats the script as team policy.

The tiny card parser is not a YAML library, and nested values or quoted colons will break it.
Path checks use the +++ b/ header only, so a clever or malformed diff can hide a rename.
A matching test command in the report does not prove the test passed or even exists on disk.
The byte cap is a local courtesy limit you chose, not a quota published by any vendor or host.
The gate cannot see side effects that an agent already performed on a server before you downloaded the diff.
Forbidden prefixes are literal strings, so a prefix such as deploy/ does not cover every obfuscated path.
The patch command is a local Unix assumption, so a Windows shell may need an equivalent apply tool you already trust.

Who should not use this approach

You should not use this gate as a security boundary for a hostile model, a malware sample, or a secret-filled repository. You should not point a free server at production credentials, private customer data, or a remote you cannot revoke. You should not skip the dry-run patch and the unittest just because the JSON line looks calm and short. Teams that need signed reviews, policy-as-code, or a real diff parser should use those systems instead of this teaching script.

Lessons

Write the allowlist before the prompt, because an agent cannot honestly invent your risk boundary after the fact.
Require the test command as an exact line, so a vague sentence about tests cannot pass the gate alone.
Fail closed on a missing card field, since a default of allow-all is the bug you are trying to remove.
Keep vendor claims out of the result line, and record only the exit code and JSON your own command printed.
Treat a free workspace as disposable, and never store long-lived keys there even for a short teaching demo.

You leave this exercise with a card, a checker, two fixture commands, and a rule you can explain to a reviewer in one minute. The rule is simple enough to repeat, and it says that no allowlist match and no named test means no applied patch. That habit stays useful even if you later replace the script with a stricter diff tool on your team. You can rerun the same fixtures next week, and any JSON change then belongs to your checker rather than a headline.

Case Study: A Retry Budget That Refuses to Sleep Past Its Cap

Dakota Liu — Sat, 10 Oct 2026 18:22:39 +0000

You should freeze a retry budget before any generated client is allowed to sleep, because unbounded backoff can pin a worker for hours. A small in-process budget can express attempt caps, elapsed caps, and delay clamps as data you test without a network. This case study walks that budget from a frozen decision table to commands that prove each stop condition. The files are runnable teaching code, not a report of a live outage or a measured production improvement.

You are looking at one module and one unittest file, which is enough to expose the failure without a platform narrative. The worker calls a flaky read and then updates a local row, so it must stop when the frozen budget says stop. A generated draft that retries every exception will hide a permanent validation error behind a long and useless sleep. You prevent that outcome by writing the contract first and by treating every later draft as an untrusted patch.

Background

The pressure usually arrives as a short task that says to wrap the call, add retries, and move on to the next ticket. A sample loop from a coding assistant often uses an open-ended loop, doubles the delay without a ceiling, and catches a broad exception. That shape can survive a happy-path demo, and then it can hold a thread through a long upstream outage. It also retries mistakes you cannot fix by waiting, such as a rejected identifier or a failed local precondition.

You do not need a new framework, a queue, or a hosted workflow engine to close that particular gap. You need a policy object, a wrapper that accepts injected time, and tests that fail if sleep is called after a cap. This case study uses those three pieces and leaves webhook signatures, money rounding, and export idempotency for other write-ups. Keeping the scope that small is what makes the contract reviewable in one sitting by someone who did not write the loop.

Goal

The goal is a wrapper that returns the operation result or raises a budget error when a frozen limit is hit. You should be able to explain that behavior to a teammate without asking them to read the loop line by line. A reviewer should be able to reject a patch by pointing at one row in the decision table below. The tests should stay on the standard library so a clean Python process is enough to reproduce the result.

Frozen rules

You freeze the rules before implementation, and you keep them stable while any draft is proposed or edited. Changing a rule means changing the table and the tests in the same edit, not silently fixing a red assertion. The list below is the whole policy for this case study, and anything absent from it is out of scope on purpose.

The attempt cap counts every try, must be at least one, and includes the first call rather than only the retries.
After failure number n, the delay is the base delay times two to the n minus one, clamped at the configured maximum.
If time already spent plus the next delay would pass the elapsed ceiling, you raise the budget error and skip sleep.
Only exception types you listed as retryable may sleep, and every other error propagates on the attempt that raised it.
Sleep and the clock are injected arguments, so the tests never call the real sleep function or read the wall clock.

Invalid construction is part of the same contract, not a later cleanup you leave for a second patch. A zero attempt cap, a negative duration, or a maximum delay below the base delay raises ValueError immediately. That early failure matters because a generated client should not run at all with a policy you already know is nonsense.

Decision table

You review generated code against the rows below, not against how confident or fluent the draft happens to sound. Each row names the inputs that matter and the only result you should accept for that situation. If a draft needs a new row, you add the row before you accept the code, so the table stays the source of truth.

Situation	Attempts used	Next delay	Elapsed plus delay	Required result
Success on first try	1	none	not used	return the value
Retryable error with room left	below cap	clamped exponential	within ceiling	sleep, then retry
Retryable error at attempt cap	equals cap	ignored	not used	raise budget error
Retryable error at time cap	below cap	computed	above ceiling	raise budget error, no sleep
Non-retryable error	any	none	not used	re-raise the original error
Invalid policy values	none	none	not used	raise `ValueError` at construction

Implementation

The module below is the reference implementation for this worked example, and you can save it as retry_budget.py. You should change it only when the decision table changes, because the tests encode those rows rather than a style preference. The listing has no narrative comments, so a reviewer has to match behavior to the table instead of trusting prose in the file. Treat the listing as example code you can run locally, not as a versioned library with compatibility promises.

class RetryBudgetExceeded(Exception):
    """Raised when a frozen attempt cap or elapsed cap blocks another try."""


class RetryPolicy:
    def __init__(self, max_attempts, max_elapsed_ms, base_delay_ms, max_delay_ms):
        if max_attempts < 1:
            raise ValueError("max_attempts must be at least 1")
        if max_elapsed_ms < 0 or base_delay_ms < 0:
            raise ValueError("time values must be non-negative")
        if max_delay_ms < base_delay_ms:
            raise ValueError("max_delay_ms is below base_delay_ms")
        self.max_attempts = max_attempts
        self.max_elapsed_ms = max_elapsed_ms
        self.base_delay_ms = base_delay_ms
        self.max_delay_ms = max_delay_ms


def delay_for_attempt(policy, attempt):
    scaled = policy.base_delay_ms * (2 ** (attempt - 1))
    if scaled > policy.max_delay_ms:
        return policy.max_delay_ms
    return scaled


def run_with_budget(policy, operation, sleep, clock, retryable=(TimeoutError,)):
    started = clock()
    attempt = 0
    while True:
        attempt += 1
        try:
            return operation()
        except retryable as exc:
            if attempt >= policy.max_attempts:
                raise RetryBudgetExceeded("attempt cap reached") from exc
            wait = delay_for_attempt(policy, attempt)
            if clock() - started + wait > policy.max_elapsed_ms:
                raise RetryBudgetExceeded("elapsed cap reached") from exc
            sleep(wait)

Three details are where generated drafts usually drift, even when the function names look identical to this listing. The delay exponent uses the attempt that just failed, so the first wait equals the base delay rather than double that value. The elapsed check reads the clock after the failed call, which means a slow attempt can exhaust the budget by itself. Sleep runs only after both checks pass, which is why the time-cap test can assert that the sleep log stayed empty.

Tests you can run

Save the next listing as test_retry_budget.py in the same directory as the module so the import stays boring. The tests record sleeps in a list and feed the clock from a short script, which keeps the file deterministic on a shared runner. Nothing in this file needs a network, a clock permission, or a third-party package, which is intentional for a clean environment. You can add cases later, but you should not delete an assertion to help a draft turn green.

import unittest
from retry_budget import (
    RetryBudgetExceeded,
    RetryPolicy,
    delay_for_attempt,
    run_with_budget,
)


def raise_error(exc):
    raise exc


class ScriptedClock:
    def __init__(self, readings):
        self.readings = list(readings)

    def __call__(self):
        if not self.readings:
            raise AssertionError("clock exhausted")
        return self.readings.pop(0)


class RetryBudgetTests(unittest.TestCase):
    def test_success_does_not_sleep(self):
        slept = []
        policy = RetryPolicy(3, 1000, 100, 400)
        value = run_with_budget(policy, lambda: "ok", slept.append, lambda: 0)
        self.assertEqual(value, "ok")
        self.assertEqual(slept, [])

    def test_attempt_cap_sleeps_only_between_tries(self):
        calls = {"n": 0}

        def fail():
            calls["n"] += 1
            raise TimeoutError("slow upstream")

        slept = []
        policy = RetryPolicy(2, 10000, 100, 400)
        with self.assertRaises(RetryBudgetExceeded):
            run_with_budget(policy, fail, slept.append, lambda: 0)
        self.assertEqual(calls["n"], 2)
        self.assertEqual(slept, [100])

    def test_elapsed_cap_does_not_sleep(self):
        slept = []
        policy = RetryPolicy(5, 1000, 100, 400)
        clock = ScriptedClock([0, 950])
        with self.assertRaises(RetryBudgetExceeded):
            run_with_budget(
                policy,
                lambda: raise_error(TimeoutError("late")),
                slept.append,
                clock,
            )
        self.assertEqual(slept, [])

    def test_validation_error_is_not_retried(self):
        slept = []
        policy = RetryPolicy(4, 5000, 50, 200)
        with self.assertRaises(ValueError):
            run_with_budget(
                policy,
                lambda: raise_error(ValueError("bad sku")),
                slept.append,
                lambda: 0,
            )
        self.assertEqual(slept, [])

    def test_delay_clamps_and_policy_rejects_bad_values(self):
        policy = RetryPolicy(3, 5000, 100, 150)
        self.assertEqual(delay_for_attempt(policy, 3), 150)
        with self.assertRaises(ValueError):
            RetryPolicy(0, 1000, 10, 10)


if __name__ == "__main__":
    unittest.main()

From the directory that holds both files, you run the standard-library commands in the next block. You should see five tests pass, with no installer step and no outbound call required by the example itself. If a draft removes the elapsed check, the elapsed-cap test fails because the sleep list is no longer empty. If a draft retries ValueError, the validation test fails because the wrapper no longer preserves the original error type.

python -m py_compile retry_budget.py test_retry_budget.py
python -m unittest test_retry_budget.py -v
python test_retry_budget.py -v

If the runner exposes only python3, you swap the command name and you leave the module arguments unchanged. A compile failure is a syntax problem, while a unittest failure is a contract problem, and you should not mix those two signals. You keep the verbose flag so the printed names match the review notes in the next section.

What a red test means

A red elapsed-cap test means the wrapper slept, or raised the wrong type, after the scripted clock crossed the ceiling. A red validation test means a permanent error was retried or wrapped so tightly that the original type disappeared. A red clamp test means the delay formula forgot the maximum and let exponential growth pick the wait by itself. You fix the wrapper or the table, and you do not loosen the assertion to manufacture a pass on a remote runner.

How you review a generated patch

You read the diff before you celebrate a green test run, because a pass can hide a weakened test. A draft can also pass for the wrong reason if it retries less by accident while breaking a row you care about later. Your review looks for a short list of drifts that show up often when a model writes retry loops from memory. You comment on the table row that broke, and you ask for a smaller patch rather than a broader rewrite.

Reject a bare exception handler that retries validation errors, cancellations, or an interrupt you did not mark as retryable.
Reject a call to real sleep, a random source, or the wall clock inside the wrapper, even if the tests still pass.
Reject an elapsed check that counts only the planned delay and ignores time already spent inside the operation.
Reject a first-wait formula that doubles too early, so the first failure sleeps twice the configured base delay.
Reject a test edit that deletes the empty-sleep assertion or widens the retryable tuple to the base exception type.
Reject a new dependency when the standard library already compiles and runs both files on a clean interpreter.

You paste the failed assertion back to the assistant instead of asking it to try harder without evidence. That habit keeps the case study short enough to finish in one sitting, even if the first draft is wrong. It also keeps a small remote runner useful, because the feedback is a unittest name rather than a screenshot or a log dump. You stop when the five tests pass and the diff still matches the table, not when the draft adds extra features.

A practical drafting workflow

You can implement the module by hand, and that remains a valid ending if you never call an assistant at all. When you do want a draft, you keep the assistant on a short leash and you follow the order below. The order is the method, and skipping ahead to a finished client is how the open-ended sleep loop returns.

What you hand the assistant

You paste the decision table and the unittest file, and you withhold the reference module so the draft cannot copy it.
You ask for one dependency-free implementation that satisfies the tests and does not call the real sleep function.
You diff that draft against the frozen rules before you run it, and you discard edits that invent new policy.
You run the unittest command, then you reject any change that weakens an assertion merely to obtain a pass.
You keep the table in the review notes so the next generated tweak has the same fence as the first.

Where a free drafting setup fits

MonkeyCode's free model access fits the drafting step, because you can request the patch without a paid seat for this exercise. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server option fits the unittest step, because that same command can judge the patch on a clean Python process. You should confirm current access terms on the project page before you depend on them for anything beyond this exercise.

This article states no model names, token quotas, hardware shapes, or permanent limits, since those details change and were not verified here. You still review the diff yourself, and you still discard a green run that came from a weakened test. A free runner does not make the table optional, and a fluent draft does not outrank a failed assertion. If the current free access is unavailable, you run the same command on any local Python 3 interpreter and the case still holds.

Results you should expect

If you run the files as written, the expected result is a local pass of the five tests in this case study. The attempt-cap test shows one sleep between two tries, and then a budget error instead of a third call. The elapsed-cap test shows a budget error with an empty sleep list when the scripted clock is already at 950 milliseconds. The clamp check shows a third-failure delay of 150 milliseconds rather than the unclamped value of 400 milliseconds.

These checks describe the contract, not throughput, hosting cost, or the behavior of any live inventory service. They also do not prove that a particular model will draft a correct patch, because that outcome depends on the prompt and the review. You should record a failure as a failed assertion name, not as a story about the assistant being careless or clever. When the command prints five successful tests, you have reproduced the case study, and you do not need further metrics.

Limitations

This wrapper is single-process and single-threaded, and it does not cancel a call that has already started. It does not share a budget across workers, and it does not add jitter to spread out a thundering herd. It also does not make the following write idempotent, so a successful retry can still double-apply a non-idempotent update. The injected clock must move forward in milliseconds, because a backward jump can mask an elapsed-cap failure.

You should not use this module for card charges, device control, or any loop where a wrong retry is irreversible. You should not ask a model to invent the caps from a vague prompt that only says to make the client resilient. If you cannot fill the decision table with real numbers, you are not ready to generate the loop at all. Teams that need cross-host coordination should write a shared limiter with its own contract, not a longer copy of this file.

Lessons

You get a worker that fails visibly when the stop rule is data you can assert in a test name. You get a calmer review when the table is small enough to read without a meeting or a design document. You get portable checks when sleep and time are arguments, which is why a modest remote process can host them. The product note is optional to the lesson, because the module, the table, and the command still stand if you delete it.

If you lack a local interpreter this week, you can rehearse the same two-file case on a free coding setup. MonkeyCode's free model access and free server are enough for that rehearsal, provided you still keep the table as the fence. Merge only the diff that preserves the five tests, and leave every other generated convenience out of the branch. That is the whole invitation: try the workflow on the exercise above, not a claim that a tool replaces the contract.

Case Study: A Three-Line Cart Priced in Minor Units, Not Floats

Dakota Liu — Fri, 09 Oct 2026 14:53:55 +0000

You should lock currency scale, rounding mode, and step order before a coding agent touches invoice math. A three-line cart can look correct in a demo and still miss the ledger by one cent after tax. This case study builds a small pricing helper, a decision table, and tests you can rerun without a shared database. You can remove every tool name later and the contract, the code, and the checks still stand on their own.

Background

Your checkout charges a card and then renders a PDF invoice that finance reconciles against the ledger the next morning. The cart in this study has one 19.99 line, a quantity of three, a ten percent discount, and an 8.25 percent tax rate. An unlabeled float draft can return success and still disagree with minor-unit pricing of that same cart. That drift is the bug you freeze out of the prompt, rather than a bug you hope a later review will catch.

The carts below are a constructed example for you to rerun, not a customer incident and not a published benchmark. You should treat every total in this article as an assertion you can recompute, not as a metric from a live store. If your own rate or step order differs, you change the frozen rule before you change the code. The point is the sequence of decisions, not a claim that this helper already runs in production.

Why the draft is the risky part

Agent drafts are now a normal way to get a first implementation, which is useful only after the money policy is already written down. A trending demo can show a total on screen and still hide the cent that finance will dispute during reconciliation tomorrow morning. You apply that caution to a cart you can recompute by hand, instead of to a demo whose total you cannot audit.

Goal

You want a pure function that prices one cart in integer minor units and fails closed when the currency scale is unknown. You want the discount applied to the net, then tax applied to the discounted net, with rounding after each step. You do not want a tax jurisdiction engine, a payment capture client, or a claim that generated code is already reviewed. Success is a green local test run, plus the same file passing on a disposable server you can delete the same day.

Contract you freeze first

Before you prompt, you write the pricing rules into the repository so the agent cannot invent a friendlier policy. The rules below are a product choice for this helper, not a claim about every accounting standard in the world. You keep them short so a reviewer can reject a diff that violates one line without rereading the model output. A chat transcript is not the contract, because the next session will not scroll back to the sentence you liked.

Five rules

Represent money as integer minor units at the boundary, and use decimal quantization only inside the helper.
USD uses scale 2, JPY uses scale 0, and KWD uses scale 3; any other code raises UnknownCurrency.
Round half up after the line extension, after the discount, and after the tax, and never only at the end.
Reject an empty cart, a non-integer quantity, a quantity below one, and any rate below zero or above one.
Return net, discount, tax, and total as minor-unit integers so a renderer cannot reintroduce binary floats.

Sample numbers you pin

You also freeze the sample numbers so a passing test cannot quietly change the fixture to match a wrong implementation. The expected USD total for the sample cart is 5842 minor units, which is 58.42 after the rules above. If an agent returns 58.41 or 58.43, the test fails, and you do not rewrite the assertion to match the draft. That refusal is the point of freezing the contract before any implementation is allowed to exist.

Worked arithmetic for the sample cart

The line extension is 19.99 times three, which is 59.97 dollars, or 5997 minor units, before any discount is applied. Ten percent of 59.97 is 5.997, and half-up rounding at scale 2 makes the discount 6.00, or 600 minor units. Tax is 8.25 percent of the remaining 53.97, which is 4.452525, and half-up rounding at scale 2 makes tax 4.45. Adding 53.97 and 4.45 yields 58.42, so the fixture asserts 5842 and fails if a draft returns either neighboring cent.

Implementation

You keep the helper in one module so the agent has a narrow place to write and you have a narrow place to review. The scale table is explicit, and an unknown currency raises before any multiplication can hide the mistake inside a total. Rates are parsed as decimals, and a rate below zero or above one raises InvalidRate instead of clamping to a convenient value. Line extensions are quantized before they enter the net, which stops a later step from inheriting an extra fraction.

Save this as invoice_total.py next to the test file:

from decimal import Decimal, ROUND_HALF_UP, localcontext
from dataclasses import dataclass


SCALES = {"USD": 2, "JPY": 0, "KWD": 3}


class UnknownCurrency(ValueError):
    pass


class InvalidRate(ValueError):
    pass


class InvalidQuantity(ValueError):
    pass


def _quantizer(currency):
    try:
        scale = SCALES[currency]
    except KeyError as exc:
        raise UnknownCurrency(currency) from exc
    return Decimal("1").scaleb(-scale)


def _quantize(amount, currency):
    with localcontext() as ctx:
        ctx.rounding = ROUND_HALF_UP
        return amount.quantize(_quantizer(currency))


def parse_rate(raw):
    rate = Decimal(raw)
    if rate < 0 or rate > 1:
        raise InvalidRate(raw)
    return rate


@dataclass(frozen=True)
class Line:
    unit_price: str
    quantity: int


@dataclass(frozen=True)
class Cart:
    currency: str
    lines: tuple
    discount_rate: str
    tax_rate: str


def invoice_total(cart):
    if not cart.lines:
        raise ValueError("empty cart")
    net = Decimal("0")
    for line in cart.lines:
        if isinstance(line.quantity, bool) or not isinstance(line.quantity, int):
            raise InvalidQuantity(str(line.quantity))
        if line.quantity < 1:
            raise InvalidQuantity(str(line.quantity))
        unit = _quantize(Decimal(line.unit_price), cart.currency)
        net += _quantize(unit * line.quantity, cart.currency)
    discount = _quantize(net * parse_rate(cart.discount_rate), cart.currency)
    taxable = _quantize(net - discount, cart.currency)
    tax = _quantize(taxable * parse_rate(cart.tax_rate), cart.currency)
    total = _quantize(taxable + tax, cart.currency)
    return {
        "currency": cart.currency,
        "net_minor": _minor(net, cart.currency),
        "discount_minor": _minor(discount, cart.currency),
        "tax_minor": _minor(tax, cart.currency),
        "total_minor": _minor(total, cart.currency),
    }


def _minor(amount, currency):
    scale = SCALES[currency]
    shifted = amount * (Decimal(10) ** scale)
    return int(shifted.to_integral_value(rounding=ROUND_HALF_UP))

You should treat the listing as a reproducible example, not as a record of a production incident or a measured outage. Read the quantize calls before you trust the function, because a missing quantize is how the one-cent drift returns. If your finance partner wants half-even rounding, you change the frozen rule and the fixtures together in one review. You do not let an agent swap the rounding mode in a drive-by refactor while it is cleaning up the file.

A float expression you do not promote

A one-shot float expression is not this contract, because it skips the per-step quantize that the rules require. You may print that expression while learning, but you must not paste its output into the fixture as a convenience. If the printed float total differs, that difference is a reason to keep the decimal path, not a tie to split. Leave the comparison in a comment or a scratch cell so it cannot become an accidental second source of truth.

# Scratch only. Not the contract, and not an assertion.
print(round(19.99 * 3 * 0.9 * 1.0825, 2))

Test plan

You pin the USD sample, a JPY zero-scale cart, an unknown currency, and a negative discount that must not be clamped. You also pin the half-up boundary, because a later edit can switch rounding and still pass the happy-path cart. The commands below assume an activated virtual environment and pytest, and they do not call a network or a payment provider. Run them locally first so a later server run is a confirmation, not your only source of truth.

python -m venv .venv
python -m pip install pytest
python -m pytest test_invoice_total.py -q

You commit the tests before you ask for an implementation, so a green run cannot be manufactured by editing the expected total. If the import path differs on your machine, you change the import in the test, not the expected minor units. Keep both files in the same directory for this example so pytest can import the helper without a package install. Put the decision table in that same change so a reviewer sees the policy and the checker together.

Save this as test_invoice_total.py:

from decimal import Decimal

import pytest

from invoice_total import (
    Cart,
    InvalidQuantity,
    InvalidRate,
    Line,
    UnknownCurrency,
    _quantize,
    invoice_total,
)


def test_usd_sample_cart_is_5842_minor():
    cart = Cart(
        currency="USD",
        lines=(Line("19.99", 3),),
        discount_rate="0.10",
        tax_rate="0.0825",
    )
    assert invoice_total(cart) == {
        "currency": "USD",
        "net_minor": 5997,
        "discount_minor": 600,
        "tax_minor": 445,
        "total_minor": 5842,
    }


def test_jpy_zero_scale_tax():
    cart = Cart("JPY", (Line("1000", 1),), "0", "0.10")
    got = invoice_total(cart)
    assert got["tax_minor"] == 100
    assert got["total_minor"] == 1100


def test_unknown_currency_fails_closed():
    cart = Cart("BHD", (Line("1.000", 1),), "0", "0")
    with pytest.raises(UnknownCurrency):
        invoice_total(cart)


def test_negative_discount_is_rejected():
    cart = Cart("USD", (Line("10.00", 1),), "-0.01", "0")
    with pytest.raises(InvalidRate):
        invoice_total(cart)


def test_rate_above_one_is_rejected():
    cart = Cart("USD", (Line("10.00", 1),), "0", "1.01")
    with pytest.raises(InvalidRate):
        invoice_total(cart)


def test_empty_cart_is_rejected():
    with pytest.raises(ValueError):
        invoice_total(Cart("USD", (), "0", "0"))


def test_zero_quantity_is_rejected():
    cart = Cart("USD", (Line("10.00", 0),), "0", "0")
    with pytest.raises(InvalidQuantity):
        invoice_total(cart)


def test_half_up_boundary_on_usd_and_kwd():
    assert _quantize(Decimal("1.005"), "USD") == Decimal("1.01")
    assert _quantize(Decimal("1.004"), "USD") == Decimal("1.00")
    assert _quantize(Decimal("1.2345"), "KWD") == Decimal("1.235")

Decision table

You paste a decision table into the pull request so the reviewer checks policy, not only whether the file compiles. A green test with a rewritten fixture is a failed review, even when the agent explains the change with confidence. You keep the table next to the code, because a chat log is not something the next teammate will reliably find. The rows below match the tests, and you should update a row only when finance changes the written rule.

Case	Frozen behavior	Why it is here
USD sample cart	total_minor is 5842	Pins step order and half-up
JPY 1000, 10 percent tax, no discount	tax_minor is 100 and total_minor is 1100	Scale 0 must not invent cents
Currency BHD	UnknownCurrency	Unknown scale fails closed
discount_rate -0.01	InvalidRate	No silent clamp
tax_rate 1.01	InvalidRate	A rate above one is not a bonus
Empty lines	ValueError	Zero is not a valid invoice here
Quantity 0	InvalidQuantity	A free line is not a silent skip
Unit price 1.005 USD	Quantizes to 1.01	Half-up boundary stays explicit
Amount 1.2345 KWD	Quantizes to 1.235	Scale 3 is represented, not guessed

Repeatable review workflow

You repeat the same five steps whenever an agent is allowed to touch this helper, including a second session next week. The order matters, because a prompt written before the tests exist invites the model to choose both the policy and the proof. A disposable server is only step four, and it runs the file you already trusted on your laptop. You merge only after a person reads the diff against the five rules, not after the tool reports a clean run.

Commit the rules, a stub that raises NotImplementedError, and the tests that pin 5842 for the sample cart.
Prompt for an implementation that must not edit the tests, the scale table, or the expected minor units.
Run pytest in the local virtual environment and stop if the failure is a changed assertion rather than bad code.
Run that same pytest command on the free server option when you want a second machine that you can discard.
Read the diff for unknown scales, float literals, and any rounding call that is not half up at each step.

Where free model access fits

You can ask MonkeyCode's free model access for the draft, and you do that only after the rules and fixtures are committed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server option matters when you want the same pytest file executed away from a shared continuous-integration runner. Neither option supplies the rounding policy, the sample total, or a reason to skip reading the diff.

Your prompt should quote the five rules and ask for code that makes the existing tests pass without editing them. You reject any draft that adds a currency by guessing its scale, even when the guess matches a public reference you trust. You also reject a draft that switches to floats for simplicity, because that is the failure this study is built to prevent. If the tool cannot see your repository, you paste the contract into the prompt and you still review the patch yourself.

Results you can actually claim

This example is not a production billing migration, so you should not quote it as an uptime, revenue, or accuracy study. What you can verify is narrower and fully local: the USD fixture expects 5842 and the JPY fixture expects 1100. Unknown currencies must raise, and a negative rate must raise, which you can see by running the two rejection tests. If an assertion fails, you fix the implementation or you revise the frozen rule, and you do not average the two totals.

Limitations and who should skip this

This helper is the wrong tool when tax depends on address, product class, or a jurisdiction service you have not integrated. You should not use it for foreign-exchange conversion, because no rate source and no timestamp rule are frozen here. Teams that need an audited ledger, cash rounding to five cents, or tax-inclusive prices should not adopt this step order. Free model access and a free server option can change, and this article states no quota, hardware profile, or permanence promise.

You should not point an agent at a live payment database and ask it to backfill historical totals with this function. A backfill needs its own idempotency and reconciliation plan, which is outside this cart helper on purpose. If you cannot name the currency scale, you are not ready to automate the total, with or without a model in the loop. Skip the workflow when the only goal is a demo screenshot, because the tests are the product rather than the plot.

Lessons learned

The useful lesson is that invoice math fails at policy boundaries, not at the moment a function finally returns a number. You save review time by forbidding unknown scales, rather than by asking an agent to explain a confident but wrong cent. A second run on another machine confirms execution, and it does not replace the fixture you froze in git. Keep the decision table in the repository so the next session starts from the contract instead of from scrollback.

What you do next

If your money rules already live in the repository, rehearse the draft on free model access before anyone touches a ledger. A free server is enough for that second pytest run, and you should still merge only after a human reads the diff. Start from the contract in this case study, then replace the sample cart with one your finance partner has already signed.