DEV Community: Finley Zhu

Workshop: Catch Shared-Host Drift With a Five-Probe Floor Card in 85 Minutes

Finley Zhu — Sun, 20 Sep 2026 12:00:24 +0000

Shared inference hosts fail quietly when a probe set is missing, not when a dashboard still looks green. A five-probe floor card records the cheapest behaviors you still require after a host, prompt, or routing change. This workshop times that card at eighty-five minutes, with rerunnable files and a pass-or-fail ledger. Students leave with a JSON pack, a local scorer, and a printed floor they can recheck after every lab swap.

Why a floor card beats a chat anecdote

Anecdotes from one lucky prompt hide regressions that only appear on the second or third frozen case. Shared hosts also move under you, because capacity, routing, and hidden system prompts are not a public version pin. You need a frozen input set, a deterministic scorer, and a stored floor, or next week's run cannot be compared at all. The method below treats the model as an untrusted function with a tiny, documented surface.

Measurement talk in developer circles often outruns the tests that still discriminate. When a host starts returning fluent prose, empty objects, or a constant label, yesterday's demo stops being evidence. A floor card is deliberately small so a classroom can finish it, not so a vendor can be ranked. If the card cannot fail, it cannot teach anything useful about drift.

What this workshop is not

This is not a leaderboard, a latency study, or a claim about production model quality on a named system. It is a teaching lab that detects silent capability loss on one narrow, field-shaped task. If your team already runs a versioned eval platform with owners and an SLA, skip this outline and keep that platform. If you cannot freeze inputs because the task is open-ended prose, shrink the contract before you schedule the lab.

Workshop clock

Keep a visible timer for eighty-five minutes and refuse to expand the probe set during the first pass through the files.

0–10 min — Frame the failure. Name one task, one JSON contract, and one host URL environment variable.
10–25 min — Write the probe pack. Freeze five inputs, expected fields, and scoring modes on disk.
25–45 min — Run the worked example. Score against a file fixture before any remote host is allowed.
45–65 min — Exercise: point at a shared host. Keep the scorer identical and record a second floor card.
65–80 min — Exercise: break one probe. Mutate a prompt or response shape and show the ledger turning red.
80–85 min — Recap limits. List who should not ship this card as a quality gate.

Artifact layout

Students should create a directory they can zip, copy to another machine, and rerun without editing expects.

probe-floor/
  probes.json
  score_probes.py
  floor_card.json
  Makefile

The Makefile is the only class entrypoint, which stops ad-hoc flags from becoming the real curriculum.

.PHONY: fixture remote diff

fixture:
    python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json

remote:
    python3 score_probes.py --probes probes.json --base-url "$$PROBE_BASE_URL" --out floor_card.remote.json

diff:
    python3 score_probes.py --diff floor_card.json floor_card.remote.json

Probe pack schema

Keep scoring modes boring on purpose. Exact field match plus JSON parse success catch more host drift than a long prose rubric.

{
  "task": "semver_bump_from_diff",
  "contract": {
    "type": "object",
    "required": ["bump", "reason"],
    "properties": {
      "bump": {"enum": ["major", "minor", "patch", "none"]},
      "reason": {"type": "string", "minLength": 8, "maxLength": 160}
    }
  },
  "floor": {"min_pass": 5, "max_parse_fail": 0},
  "probes": [
    {
      "id": "P1_docs_patch",
      "input": {"diff_summary": "docs: fix typo in README install block"},
      "expect": {"bump": "patch"}
    },
    {
      "id": "P2_optional_field_minor",
      "input": {"diff_summary": "feat: add optional timeout_ms to ClientConfig"},
      "expect": {"bump": "minor"}
    },
    {
      "id": "P3_removed_field_major",
      "input": {"diff_summary": "breaking: remove ClientConfig.retry_count"},
      "expect": {"bump": "major"}
    },
    {
      "id": "P4_empty_diff_none",
      "input": {"diff_summary": ""},
      "expect": {"bump": "none"}
    },
    {
      "id": "P5_chore_none",
      "input": {"diff_summary": "chore: reformat imports with no behavior change"},
      "expect": {"bump": "none"}
    }
  ]
}

Five probes are a floor, not coverage, and they exist to fail closed when a host collapses. Watch for fluent prose, empty JSON, or a constant minor returned for every distinct case. If a pair wants a sixth probe during the first hour, park it in a notes file instead of changing the pack.

Worked example students can rerun

Label: this example is a local teaching fixture, not a measured vendor benchmark and not a claim about any hosted model. The file backend returns canned JSON so the scorer can be graded without a network round trip. Remote calls below use a lab default path; change that path to match whatever route your host actually documents.

# score_probes.py — teaching example, not a production eval platform
from __future__ import annotations

import argparse, json, sys, urllib.request
from pathlib import Path

SYSTEM = (
    "Return only JSON with keys bump and reason. "
    "bump must be major, minor, patch, or none."
)

def load_probes(path: Path) -> dict:
    return json.loads(path.read_text())

def complete_file(probe: dict) -> str:
    bump = probe["expect"]["bump"]
    return json.dumps({"bump": bump, "reason": f"fixture:{probe['id']}"})

def complete_http(base: str, probe: dict, timeout: float = 30.0) -> str:
    payload = json.dumps({
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(probe["input"])},
        ]
    }).encode()
    req = urllib.request.Request(
        base.rstrip("/") + "/v1/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        body = json.loads(resp.read().decode())
    return body["choices"][0]["message"]["content"]

def score_one(contract: dict, probe: dict, raw: str) -> dict:
    row = {"id": probe["id"], "pass": False, "parse_ok": False, "detail": ""}
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        row["detail"] = "not_json"
        return row
    row["parse_ok"] = True
    if set(contract["required"]) - set(data):
        row["detail"] = "missing_keys"
        return row
    if data.get("bump") != probe["expect"]["bump"]:
        row["detail"] = f"bump:{data.get('bump')}"
        return row
    reason = data.get("reason", "")
    if not isinstance(reason, str) or not (8 <= len(reason) <= 160):
        row["detail"] = "reason_len"
        return row
    row["pass"] = True
    row["detail"] = "ok"
    return row

def run(probes: dict, base_url: str) -> dict:
    rows = []
    for probe in probes["probes"]:
        raw = (
            complete_file(probe)
            if base_url.startswith("file:")
            else complete_http(base_url, probe)
        )
        rows.append(score_one(probes["contract"], probe, raw))
    passed = sum(1 for r in rows if r["pass"])
    parse_fail = sum(1 for r in rows if not r["parse_ok"])
    floor = probes["floor"]
    return {
        "task": probes["task"],
        "passed": passed,
        "parse_fail": parse_fail,
        "floor_ok": passed >= floor["min_pass"] and parse_fail <= floor["max_parse_fail"],
        "rows": rows,
    }

def diff_cards(a: dict, b: dict) -> int:
    print(f"local_floor_ok={a['floor_ok']} remote_floor_ok={b['floor_ok']}")
    ids = {r["id"]: r for r in a["rows"]}
    rc = 0
    for row in b["rows"]:
        prior = ids.get(row["id"], {})
        if prior.get("pass") and not row["pass"]:
            print(f"REGRESS {row['id']} {prior.get('detail')} -> {row['detail']}")
            rc = 1
        elif prior.get("pass") != row["pass"]:
            print(f"CHANGE  {row['id']} pass {prior.get('pass')} -> {row['pass']}")
            rc = 1
    return rc

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--probes")
    p.add_argument("--base-url")
    p.add_argument("--out")
    p.add_argument("--diff", nargs=2)
    args = p.parse_args()
    if args.diff:
        a = json.loads(Path(args.diff[0]).read_text())
        b = json.loads(Path(args.diff[1]).read_text())
        return diff_cards(a, b)
    pack = load_probes(Path(args.probes))
    card = run(pack, args.base_url)
    Path(args.out).write_text(json.dumps(card, indent=2) + "\n")
    print(json.dumps({"floor_ok": card["floor_ok"], "passed": card["passed"]}, indent=2))
    return 0 if card["floor_ok"] else 2

if __name__ == "__main__":
    sys.exit(main())

Expected fixture command for every pair, before anyone exports a remote URL:

python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json

The teaching fixture always returns the expected bump, so floor_ok must be true before PROBE_BASE_URL is set. That order is the lab, not a ceremony around the lab.

Exercise 1 — freeze the contract, not the essay (15 minutes)

Students often want a long rubric because it feels more serious than five enum checks. Stop that impulse and ask each pair to delete any probe whose expect field is a free-text essay. A probe that cannot fail in one sentence is not frozen yet, and it will be edited to match a nicer model next week.

Checklist for the teaching assistant:

Every probe has a stable id that will survive later wording edits.
bump is an enum, never a list of allowed synonyms in natural language.
reason is length-bounded so empty strings and novels both fail the same way.
min_pass equals the probe count on day one; lowering it requires a written note.

Exercise 2 — change the host, not the scorer (20 minutes)

Export one URL and keep probes.json byte-identical. A local scorer should not care which process sits behind a compatible HTTP path, only whether the floor still holds. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can sit behind PROBE_BASE_URL once the file backend already prints floor_ok: true.

export PROBE_BASE_URL="http://127.0.0.1:8080"
make remote
make diff

Interpret the diff with a table, not with a vibe from a single chat window in another tab.

Diff signal	Meaning in this lab	Next action
`floor_ok` stays true	Floor still holds on this host	Record the card; do not add probes yet
parse failures greater than zero	Contract broke into prose or invalid JSON	Fix the prompt or reject the host
one bump mismatch	Task policy drifted on a frozen case	Keep the probe; do not edit `expect`
all bumps become `minor`	Classifier collapsed to a constant	Fail the floor; do not average scores

Do not invent a latency number, a token budget, or a quality rank from this table. The only honest outputs are floor_ok, per-probe detail codes, and whether a previously passing id regressed.

Exercise 3 — inject one honest failure (15 minutes)

Change the system string so it asks for Markdown fences, then rerun make remote without touching expects. The ledger should show not_json and floor_ok false on that host. If the card stays green, the scorer is too loose, and the rest of the workshop is invalid until the detail codes are strict again.

Optional mutation list, one change at a time:

Truncate reason to three characters and expect reason_len.
Force bump to the string Minor with surrounding spaces.
Return two JSON objects concatenated in one message body.
Echo the user input with no bump key at all.

Each mutation should map to a single detail code already printed by the scorer. Students who add a new code must document it beside the probe pack before they change Python.

Limitations

A five-probe floor card will not tell you that a host is good enough for production traffic. It only tells you that a host is still able to clear a tiny, frozen bar you wrote down in advance. Exact enum match is brittle if the real task is stylistic writing, multi-file refactors, or tool loops with side effects. Shared hosts can pass the card in the morning and fail it in the afternoon because routing is not a pin you control. This outline also assumes JSON contracts; if you cannot shrink the task to fields, do not fake a floor with a subjective one-to-five score.

The HTTP helper is a lab default, not a specification of any product. Timeouts, auth headers, response envelopes, and route names must follow the host you actually run. Do not paste secrets, customer diffs, or licensed source into probes.json just to make the cases feel realistic.

Who should not use this approach

Teams that need contractual uptime, data-handling terms, or a named model revision they can cite.
Classrooms that skip the file fixture and start on a public URL during the first half hour.
Workloads with secrets, personal data, or licensed code inside the probe inputs.
Anyone hoping five cases will replace unit tests, code review, or a staging environment.

Recap for the last five minutes

Write three lines on the board and stop talking over them.

Freeze inputs and expects on disk before you change hosts or prompts.
Score fields, not vibes, and fail closed on parse errors every time.
Treat a green floor card as permission to continue testing, not as proof of quality.

The same pack should rerun next week without editing expects to match a nicer answer. If you must change a probe, bump its id and record why the old floor died, because silent edits are how drift becomes folklore.

Workshop: Pin a Dual-Backend Tool Receipt in 80 Minutes

Finley Zhu — Sat, 19 Sep 2026 10:04:27 +0000

A live agent demo is not a lab until both backends emit the same canonical tool receipt. Students often treat a successful chat reply as evidence, even when the tool JSON cannot be replayed. This workshop isolates that failure with a dual-run harness, a fixture server, and a sorted-key canonicalizer. The core result is a failing or passing receipt file, not a screenshot of a fluent answer.

Recent discussion around vibe coding versus engineering is useful here only as a measurement problem. A fluent model turn can hide argument drift, extra keys, and type coercion that later labs cannot rerun. The fix is not a longer prompt. The fix is a receipt that two backends must both satisfy before the exercise is marked complete.

What this lab measures

The unit of evidence is a tool call, not assistant prose. A student prompt may produce a correct English summary while the tool payload silently changes shape. That split is why classroom reruns collapse after a backend swap. This outline treats that collapse as the bug under study.

You will leave with four artifacts:

A local fixture server that returns a frozen tool payload.
A canonicalizer that sorts keys and strips non-contract fields.
A dual-run script that hits fixture and live backends with one prompt hash.
A pass or fail receipt that classmates can rerun without the original chat UI.

Timing box

Total scheduled time is eighty minutes, including a short debrief. Keep a visible timer so discussion does not eat the dual-run.

Minutes 0–10 — Frame the failure. Show two JSON payloads that look equal in a chat log and fail after canonicalization.
Minutes 10–25 — Exercise A: capture. Record one tool name plus raw arguments from a single agent turn.
Minutes 25–45 — Exercise B: canonicalize. Implement sorted-key JSON and a deny list of extra fields.
Minutes 45–70 — Exercise C: dual-run. Compare fixture output against one live backend using the same prompt hash.
Minutes 70–80 — Decision table and debrief. Classify misses as shape drift, value drift, or transport noise.

Lab layout students can rerun

Keep the repository to three files so the receipt stays reviewable in a pull request. Extra notebooks hide the contract. Label the snippets below as a teaching harness, not as production agent infrastructure.

lab_tool_receipt/
  fixture_server.py
  canonicalize.py
  dual_run.py
  receipts/
    prompt_a.fixture.json
    prompt_a.live.json

Exercise A — Capture a raw tool turn

Start from a single intended tool, not a multi-step agent loop. Multi-step traces mix planning noise with contract noise. The worked example uses create_ticket with three fields that classrooms actually grade: title, priority, and project_id.

# Label: teaching example, not a live production client.
RAW_TURN = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "high",
        "project_id": "ops-214",
        "confidence": 0.81,          # not in the contract
        "Priority": "high",          # duplicate key after lowercasing
    },
}

Write the raw turn to disk before you clean it. Students who canonicalize in their head cannot later prove which field the model invented. A receipt that starts after cleanup is not replayable.

Exercise B — Canonicalize before you compare

Canonical form is the only equality that survives pretty-printers and key reordering. The function below lowercases object keys, drops unknown fields, and encodes with sorted keys. That trio catches the three classroom failures that look like “the model changed its mind.”

# canonicalize.py — teaching harness
import json
from typing import Any

CONTRACT_FIELDS = {
    "create_ticket": ("title", "priority", "project_id"),
}

ALLOWED_PRIORITY = {"low", "medium", "high"}


def canonicalize_args(tool: str, args: dict[str, Any]) -> dict[str, Any]:
    lowered = {str(k).lower(): v for k, v in args.items()}
    allowed = CONTRACT_FIELDS[tool]
    clipped = {k: lowered[k] for k in allowed if k in lowered}
    missing = [k for k in allowed if k not in clipped]
    if missing:
        raise ValueError(f"missing contract fields: {missing}")
    if clipped["priority"] not in ALLOWED_PRIORITY:
        raise ValueError("priority outside contract enum")
    if not isinstance(clipped["title"], str) or not clipped["title"].strip():
        raise ValueError("title must be a non-empty string")
    if not isinstance(clipped["project_id"], str):
        raise ValueError("project_id must be a string")
    return {k: clipped[k] for k in allowed}


def receipt_bytes(tool: str, args: dict[str, Any], backend: str, prompt_sha: str) -> bytes:
    body = {
        "prompt_sha256": prompt_sha,
        "tool": tool,
        "backend": backend,
        "args": canonicalize_args(tool, args),
    }
    return json.dumps(body, sort_keys=True, separators=(",", ":")).encode("utf-8")

Run a local self-check before the dual-run so canonicalizer bugs are not blamed on the model.

python - <<'PY'
from canonicalize import receipt_bytes, RAW_TURN
raw = {
  "title": "Pager duty runbook is stale",
  "priority": "high",
  "project_id": "ops-214",
  "confidence": 0.81,
}
print(receipt_bytes("create_ticket", raw, "fixture", "abc123").decode())
PY

Expected canonical object, ignoring hash, is exactly three keys in sorted order. Extra model fields must not appear. If confidence survives, the deny list is incomplete and later diffs will be noisy.

Exercise C — Dual-run fixture versus live

The fixture server is the frozen half of the experiment. It should not call a model. It exists so students can fail the harness on purpose and still rerun the passing side after class.

# fixture_server.py — stdlib only, teaching example
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

FIXTURE = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "high",
        "project_id": "ops-214",
    },
}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(length)
        payload = json.dumps(FIXTURE).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Start the fixture in one terminal, then run the comparator in another. The live client is intentionally tiny so transport code does not become the lesson.

python fixture_server.py

# dual_run.py — teaching example; swap LIVE_URL for your classroom endpoint
import hashlib, json, urllib.request
from pathlib import Path
from canonicalize import receipt_bytes

PROMPT = "Create a high-priority ops ticket about the stale pager runbook in ops-214."
PROMPT_SHA = hashlib.sha256(PROMPT.encode()).hexdigest()
LIVE_URL = "http://127.0.0.1:8765"  # replace with the live classroom backend

def post_json(url: str) -> dict:
    req = urllib.request.Request(
        url,
        data=json.dumps({"prompt": PROMPT}).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode("utf-8"))

def write_receipt(backend: str, turn: dict) -> Path:
    blob = receipt_bytes(turn["tool"], turn["arguments"], backend, PROMPT_SHA)
    path = Path("receipts") / f"prompt_a.{backend}.json"
    path.parent.mkdir(exist_ok=True)
    path.write_bytes(blob)
    return path

fixture_turn = post_json("http://127.0.0.1:8765")
live_turn = post_json(LIVE_URL)
f_path = write_receipt("fixture", fixture_turn)
l_path = write_receipt("live", live_turn)
print("match" if f_path.read_bytes() == l_path.read_bytes() else "drift")
print(f_path.read_text())
print(l_path.read_text())

If you need a no-cost live backend for Exercise C, MonkeyCode’s free model access and free server option can occupy that slot without inventing a paid key workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The receipt still has to match the fixture; availability of a free path does not relax the contract.

Worked miss that students should rerun

Seed a deliberate live payload that would pass a casual eyeball check. Then watch the canonicalizer fail it. This is the teaching moment that screenshots never capture.

LIVE_DRIFT = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "High",      # enum case drift
        "project_id": 214,       # type drift
        "assignee": "oncall",    # extra field
    },
}

Expected classification after Exercise C:

Symptom	Canonicalizer result	Grade as
Key order changed, values identical	Pass after sort	Transport noise
Extra `confidence` or `assignee`	Dropped, then pass if core equal	Shape noise
`priority` case or unknown enum	`ValueError`	Contract miss
`project_id` became an integer	`ValueError`	Type drift
`title` paraphrased	Pass shape, fail bytes	Value drift
Tool name changed	Do not canonicalize	Wrong action

Value drift is the only row that should reopen prompt design. Shape and type misses belong to the contract, not to more adjectives in the system prompt. Students who respond to every miss by editing prose will not finish Exercise C inside the remaining twenty-five minutes.

Commands for a cold rerun

A classmate who was absent should reach the same pass or fail without the original chat transcript. That constraint is the point of the receipt.

python -m py_compile fixture_server.py canonicalize.py dual_run.py
mkdir -p receipts
python fixture_server.py &
python dual_run.py
sha256sum receipts/prompt_a.fixture.json receipts/prompt_a.live.json

If the hashes differ, print a field-level diff of the decoded objects. Do not paste both files into a chat model and ask which is better. The lab already defined equality.

import json
from pathlib import Path

a = json.loads(Path("receipts/prompt_a.fixture.json").read_text())
b = json.loads(Path("receipts/prompt_a.live.json").read_text())
for key in sorted(set(a) | set(b)):
    if a.get(key) != b.get(key):
        print(key, a.get(key), b.get(key))

Limitations

Canonical byte equality is stricter than human agreement and weaker than a full behavioral specification. It will fail useful paraphrases of title and it will pass a wrong ticket that still uses the three required keys. The fixture server also cannot represent rate limits, auth failures, or streaming token events. Treat those as out of scope for an eighty-minute session.

Free live backends can still change outputs between class meetings. The receipt documents that drift; it does not freeze a third-party endpoint. If the live path moves, keep the fixture receipt as the gradeable object and record the live miss as a dated lab note. Do not silently retune the fixture to match a new live payload, because that erases the teaching signal.

Who should not use this outline

Skip this workshop if your tools have no stable argument schema, or if you are grading writing quality rather than tool effects. Skip it for production incident work that needs authentication, PII controls, and real service-level objectives. Skip it when the assignment is open-ended generation with no tool layer, because there is then no receipt to pin. Teams that already freeze OpenAPI contracts and run consumer-driven pact tests will find this lab redundant except as a student on-ramp.

The conclusion stays narrow. A classroom can use a free live path, but only the dual-run receipt turns that path into engineering evidence. Fluent text remains optional commentary beside the JSON, not the thing you grade.

Workshop: Map Agent Claims to a Failing Command Ledger in 90 Minutes

Finley Zhu — Fri, 18 Sep 2026 08:32:22 +0000

An agent-assisted lab only counts as engineering when each claim maps to a command that fails first. Natural-language success reports from a coding model are not evidence, because they can pass by deleting tests. This workshop gives students a ledger format, a Makefile contract, and a mutation check they can rerun. The same method still works if the lab later moves onto another provider or endpoint.

Who this session is for

The outline targets instructors and self-taught developers who already run short coding labs with language models. Students should be comfortable with Python 3.11 or newer, pytest, and a local POSIX shell. Teams that need production SLAs, private data handling, or multi-service orchestration should pick another control plane. This session is not a model bake-off, and it does not rank vendors or quote unpublished benchmarks.

Use the ledger when the problem has a small oracle that students can encode as JSON fixtures. Skip it when the task has no deterministic expected output, or when claim commands would execute untrusted strings. The decision table below is a teaching filter, not a capacity plan.

Situation	Use this ledger	Do not use it
Classroom lab with a known oracle	Yes	—
Agent output is prose without a fixture	—	Yes
Commands in the ledger are student-authored and allowlisted	Yes	—
Commands could include untrusted shell fragments	—	Yes until argv is allowlisted
Production incident work with private payloads	—	Yes

Timing

Treat the schedule as a hard budget rather than a loose suggestion for discussion.

0–10 min — State the rule: no claim enters the ledger without a currently failing command.
10–25 min — Write three claims and fixtures for a small interval-merge problem.
25–50 min — Request an implementation from a free coding endpoint, then run make prove.
50–70 min — Inject a mutation that drops one assertion and confirm the prove target goes red.
70–85 min — Review which ledger fields students actually filled, and which they left blank.
85–90 min — Recap limitations and name the claims this ledger cannot prove.

The core artifact

Students keep one JSON ledger next to the code under test, and they treat every row as an untrusted claim. Each row binds a claim id, a shell command, an expected initial exit code, and a fixture path. The Makefile refuses prove until fail-first has been recorded in the same working tree. That stamp is session memory, not a gradebook.

{
  "problem": "merge_intervals",
  "endpoint": "free-lab",
  "claims": [
    {
      "id": "C1",
      "statement": "Overlapping closed intervals collapse into a single span.",
      "command": "pytest -q tests/test_c1_overlap.py",
      "fixture": "fixtures/c1_overlap.json",
      "fail_first_exit": 1
    },
    {
      "id": "C2",
      "statement": "Touching intervals that share an endpoint also merge.",
      "command": "pytest -q tests/test_c2_touch.py",
      "fixture": "fixtures/c2_touch.json",
      "fail_first_exit": 1
    },
    {
      "id": "C3",
      "statement": "An empty input list returns an empty list without raising.",
      "command": "pytest -q tests/test_c3_empty.py",
      "fixture": "fixtures/c3_empty.json",
      "fail_first_exit": 1
    }
  ]
}

Label the JSON above as a teaching fixture, not as production telemetry from a live classroom. Students copy it into claims.json and must not edit fail_first_exit after the first red run. If a stub accidentally returns a plausible list, the claim is too weak, and the fixture must be tightened before the generate step starts.

Makefile contract

PYTHON ?= python3
LEDGER := claims.json
STAMP  := .fail-first.ok

.PHONY: fail-first prove mutate clean

fail-first:
    $(PYTHON) tools/check_ledger.py --mode fail-first --ledger $(LEDGER)
    touch $(STAMP)

prove: $(STAMP)
    $(PYTHON) tools/check_ledger.py --mode prove --ledger $(LEDGER)

mutate:
    $(PYTHON) tools/mutate_drop_assert.py tests/test_c1_overlap.py
    -$(PYTHON) tools/check_ledger.py --mode prove --ledger $(LEDGER); \
        status=$$?; \
        $(PYTHON) tools/mutate_drop_assert.py --restore tests/test_c1_overlap.py; \
        exit $$status

clean:
    rm -f $(STAMP)

The stamp file is the only session memory the lab needs to keep honest. If a student skips fail-first, prove does not run, which blocks the shortcut of writing tests after a green implementation. Instructors should collect the stamp and the ledger, not a chat export, when they grade the lab.

Exercise 1 — Write claims that already fail (15 minutes)

Create src/intervals.py with a stub that raises NotImplementedError so every first run is red for a known reason. Then add one pytest module per claim, each loading JSON rather than inlined literals that drift during copy. Keep the assertion count at one behavioral check per file so later mutations stay obvious to a reader. The empty-input claim exists to catch models that return None instead of [].

# tests/test_c1_overlap.py
import json
from pathlib import Path
from src.intervals import merge_intervals

def test_overlapping_closed_intervals_collapse():
    payload = json.loads(Path("fixtures/c1_overlap.json").read_text())
    assert merge_intervals(payload["input"]) == payload["expected"]

{
  "input": [[1, 3], [2, 6], [8, 10], [15, 18]],
  "expected": [[1, 6], [8, 10], [15, 18]]
}

{
  "input": [[1, 4], [4, 5]],
  "expected": [[1, 5]]
}

{
  "input": [],
  "expected": []
}

Run make fail-first and keep the checker exit at zero while pytest exits non-zero for every claim. If C2 is omitted, many generated functions will treat touching ranges as disjoint and still look correct on C1. That gap is a ledger defect, not a model defect, and it should be fixed before anyone pastes a prompt.

Exercise 2 — Generate an implementation on a shared free endpoint (25 minutes)

This is the point where a shared free coding setup matters, because a classroom cannot grade work that only ran inside one paid chat window. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which is enough to host the same prompt and the same runner for every student laptop in the room.

Treat that product as a lab backend, not as the subject of the lesson or as a scored vendor. Students paste a prompt that forbids extra files and forbids editing tests, fixtures, or the ledger. The prompt below is a proposal for this outline, not a measured template with claimed pass rates.

Implement merge_intervals(intervals) in src/intervals.py only.
Do not edit tests/, fixtures/, claims.json, or the Makefile.
Intervals are inclusive integer pairs [start, end] with start <= end.
Return merged spans sorted by start. Empty input returns [].

After the model writes the function, students run make prove against the unchanged tests. The ledger checker should load each claim, execute the command, and require exit code 0. Any extra file the model created is a lab failure even if tests pass, because the contract was file-scoped rather than chat-scoped.

Ledger checker (teaching code)

# tools/check_ledger.py
from __future__ import annotations

import argparse, json, subprocess, sys
from pathlib import Path

def load_claims(path: Path) -> list[dict]:
    data = json.loads(path.read_text())
    claims = data.get("claims") or []
    if not claims:
        raise SystemExit("ledger has no claims")
    return claims

def run_cmd(command: str) -> int:
    completed = subprocess.run(command, shell=True)
    return int(completed.returncode)

def fail_first(claims: list[dict]) -> int:
    bad = 0
    for claim in claims:
        code = run_cmd(claim["command"])
        expected = int(claim["fail_first_exit"])
        if code == 0:
            print(f"{claim['id']} already passes; claim is not a test")
            bad += 1
        elif code != expected:
            print(f"{claim['id']} exited {code}, expected {expected}")
            bad += 1
        else:
            print(f"{claim['id']} fails as required")
    return 1 if bad else 0

def prove(claims: list[dict]) -> int:
    bad = 0
    for claim in claims:
        code = run_cmd(claim["command"])
        if code != 0:
            print(f"{claim['id']} still failing with exit {code}")
            bad += 1
        else:
            print(f"{claim['id']} proved")
    return 1 if bad else 0

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--mode", choices=("fail-first", "prove"), required=True)
    parser.add_argument("--ledger", required=True)
    args = parser.parse_args()
    claims = load_claims(Path(args.ledger))
    status = fail_first(claims) if args.mode == "fail-first" else prove(claims)
    sys.exit(status)

if __name__ == "__main__":
    main()

This checker is intentionally small so students can read it in one sitting without a framework tour. It does not sandbox the shell, so untrusted commands do not belong in claims.json during a first offering. Instructors grading a second cohort should replace shell=True with an allowlisted argv map before they collect stamps.

Exercise 3 — Mutation that drops an assertion (20 minutes)

Green tests after an agent edit are still weak if the agent can delete the assertion and keep the file importable. The mutate target comments out the first assert in tests/test_c1_overlap.py and reruns prove without touching src/intervals.py. The expected teaching result is a non-zero exit from the prove path. If prove stays green after the drop, the claim never tested behavior and must be rewritten.

# tools/mutate_drop_assert.py
from pathlib import Path
import sys

MARKER = "# mutated-assert: "

def drop(path: Path) -> None:
    lines = path.read_text().splitlines(True)
    for i, line in enumerate(lines):
        stripped = line.lstrip()
        if stripped.startswith("assert "):
            indent = line[: len(line) - len(stripped)]
            lines[i] = f"{indent}{MARKER}{stripped}"
            path.write_text("".join(lines))
            return
    raise SystemExit(f"no assert found in {path}")

def restore(path: Path) -> None:
    text = path.read_text().replace(MARKER, "")
    path.write_text(text)

if __name__ == "__main__":
    restore_mode = "--restore" in sys.argv
    target = Path(sys.argv[-1])
    restore(target) if restore_mode else drop(target)

Students should log three numbers only: fail-first checker status, prove exit, and mutate exit. Do not collect latency, token counts, or model quality scores in this lab, because those numbers change by endpoint and distract from the ledger rule. A useful board note is a three-column table with claim ids, not a screenshot of a chat transcript.

Worked example students can rerun

Use this sequence on a clean tree after the files above exist. The commands assume a POSIX shell and a virtualenv that already has pytest.

python3 -m venv .venv
. .venv/bin/activate
pip install pytest
mkdir -p src tests fixtures tools
# copy the ledger, tests, fixtures, Makefile, and tools from this article
rm -f .fail-first.ok
make fail-first    # pytest red per claim; checker status 0
# paste the prompt into the shared free lab endpoint, write src/intervals.py only
make prove         # expect 0
make mutate        # expect non-zero from prove, then restored tests

A reference implementation that satisfies the three fixtures is shown next. Label it as a teaching solution, not as the only correct merge, and do not paste it into the prompt.

# src/intervals.py
def merge_intervals(intervals):
    if not intervals:
        return []
    ordered = sorted(intervals, key=lambda pair: (pair[0], pair[1]))
    merged = [list(ordered[0])]
    for start, end in ordered[1:]:
        last = merged[-1]
        if start <= last[1]:
            last[1] = max(last[1], end)
        else:
            merged.append([start, end])
    return merged

If a student's model returns a version that drops touching intervals, claim C2 stays red and the ledger remains honest. That failure is the lesson, not a reason to loosen the fixture or to delete C2 from claims.json. Students who “fix” the lab by editing expected JSON have left the engineering rule, even when the chat window still looks successful.

What the ledger does not prove

The method does not prove algorithmic optimality, concurrency safety, or prompt robustness across vendors. It also does not prove that a free server will remain available, because availability is an operational choice rather than a test assertion. Students who need property-based coverage should add Hypothesis after this lab, not during the ninety-minute window, because shrinking counterexamples will blow the timing budget.

Do not use this workshop when the task has no oracle, when fixtures would leak private data, or when shelling out from check_ledger.py would execute untrusted strings. Do not use it as a substitute for code review on security-sensitive diffs, because a passing ledger only shows that named fixtures stayed green. Instructors running large classes should allowlist commands before the second session, then keep the fail-first stamp as the gradeable artifact.

Closing

The transferable output is the ledger plus the fail-first stamp, not a chat transcript and not a vendor comparison. If a class needs one shared free endpoint for the generate step, MonkeyCode's free model access and free server option can host that lab runner without changing the Makefile contract.

Workshop: Score AI Refactors With a Frozen Oracle and Diff Budget in 75 Minutes

Finley Zhu — Thu, 17 Sep 2026 06:43:47 +0000

An AI refactor is not an engineering change until a frozen oracle and a bounded diff both accept it. Chat tone, shorter functions, and confident commit messages do not constitute acceptance criteria for production code. This seventy-five minute workshop builds a rerunnable scoring harness that grades patches independently of which host produced them. Students leave with a golden test file, a diff-budget checker, and a command sequence they can replay after class.

Why this workshop exists

Public threads keep asking whether models already outcode typical developers, which measures the wrong unit of work. A patch can delete logging, widen types, or skip the failing test that motivated the original change. Engineering accepts a change when behavior stays inside a declared envelope, not when a diff looks tidy. Cleaner chat output is not evidence that reserved-stock math, error paths, or invariants still hold.

This lab treats the model as an untrusted patch generator, not as a reviewer of its own work. The oracle is frozen before any prompt is sent, and the same scorer runs against local stubs and remote completions. If the remote host is unavailable, the local cassette still grades known-good and known-bad patches on schedule. Classroom time then depends on git and pytest, rather than on vendor latency or prompt fashion.

Audience, timing, and materials

The intended audience is backend or tooling developers who already write pytest and need a classroom gate for model-authored diffs. Facilitators should have a terminal, Python 3.11 or newer, git, and a working pytest install before the session starts. Groups without tests, repos that hold production secrets, or workshops that need a guaranteed model SLA should stop after Exercise 1. Those groups can still treat the oracle as a local linter, without sending source to any remote host.

Timing (75 minutes)

Minutes 0–10 — Freeze the golden tests and record a baseline hash.
Minutes 10–25 — Encode a diff budget and a forbidden-path list.
Minutes 25–40 — Score a known-good patch and a vibe-coded patch.
Minutes 40–60 — Optional remote run against a free hosted model.
Minutes 60–75 — Debrief false accepts, false rejects, and homework.

The kit is a tiny inventory module, one pytest file, and a scorer script that exits non-zero on oracle failure. Students should not edit the oracle after minute ten unless the facilitator records a new hash in the lab log. Late edits to tests are protocol violations, because they make later scores incomparable across machines.

Learning objectives

By the end of the session, a student should be able to:

Explain why a passing chat demo is not an acceptance test for a production refactor.
Freeze pytest output and a tree hash so later runs remain comparable across student machines.
Reject patches that exceed a line budget or that touch a forbidden path in the tree.
Replay the same scorer against a local fixture and an optional remote host without changing rules.

The artifact students will rerun

Create a directory named oracle_lab and keep every file below under version control for the full session. The module under test is intentionally small so the class can read the entire diff during debrief. A small surface also keeps the twelve-line budget meaningful, because a rewrite cannot hide inside noise.

`inventory.py`

"""In-memory SKU ledger used as the system under test."""

from dataclasses import dataclass


@dataclass
class Line:
    sku: str
    qty: int
    reserved: int = 0

    def available(self) -> int:
        return self.qty  # bug: reserved stock is still counted as available


class Ledger:
    def __init__(self) -> None:
        self._lines: dict[str, Line] = {}

    def add(self, sku: str, qty: int) -> None:
        if qty <= 0:
            raise ValueError("qty must be positive")
        current = self._lines.get(sku)
        if current is None:
            self._lines[sku] = Line(sku=sku, qty=qty)
            return
        current.qty += qty

    def reserve(self, sku: str, qty: int) -> None:
        line = self._lines[sku]
        if qty > line.available():
            raise ValueError("insufficient stock")
        line.reserved += qty

    def available(self, sku: str) -> int:
        return self._lines[sku].available()

The planted defect lives on Line.available, where reserved units remain inside the sellable count. Students must not fix the file by hand until the scorer is in place and the baseline commit exists. A manual fix would skip the comparison the lab is measuring and collapse Exercise 3 into a git checkout.

`test_inventory.py`

import pytest

from inventory import Ledger


def test_add_and_available_without_reserve():
    ledger = Ledger()
    ledger.add("sku-a", 10)
    assert ledger.available("sku-a") == 10


def test_reserve_reduces_available():
    ledger = Ledger()
    ledger.add("sku-a", 10)
    ledger.reserve("sku-a", 4)
    assert ledger.available("sku-a") == 6


def test_over_reserve_raises():
    ledger = Ledger()
    ledger.add("sku-a", 2)
    with pytest.raises(ValueError):
        ledger.reserve("sku-a", 3)


def test_second_add_accumulates():
    ledger = Ledger()
    ledger.add("sku-a", 4)
    ledger.add("sku-a", 6)
    assert ledger.available("sku-a") == 10

These four tests form the frozen oracle and must stay byte-stable after the first hash is recorded. After minute ten, facilitators should treat any edit to this file as a protocol violation, not as cleanup. A model that rewrites tests can manufacture a green run without restoring reserved-stock behavior at all.

`score_patch.py`

#!/usr/bin/env python3
"""Score a git diff against a frozen test oracle and a line budget."""

from __future__ import annotations

import argparse
import hashlib
import subprocess
import sys
from pathlib import Path

ORACLE_SHA256 = "REPLACE_AFTER_EXERCISE_1"
MAX_CHANGED_LINES = 12
FORBIDDEN_PREFIXES = ("test_inventory.py", "score_patch.py", ".git/")


def run(cmd: list[str]) -> subprocess.CompletedProcess[str]:
    return subprocess.run(cmd, text=True, capture_output=True, check=False)


def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def changed_lines(diff_text: str) -> int:
    total = 0
    for line in diff_text.splitlines():
        if line.startswith("+++") or line.startswith("---"):
            continue
        if line.startswith("+") or line.startswith("-"):
            total += 1
    return total


def forbidden_paths(diff_text: str) -> list[str]:
    hits: list[str] = []
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            if any(path.startswith(prefix) for prefix in FORBIDDEN_PREFIXES):
                hits.append(path)
    return hits


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", default="HEAD")
    args = parser.parse_args()

    oracle = Path("test_inventory.py")
    digest = sha256_file(oracle)
    if ORACLE_SHA256 == "REPLACE_AFTER_EXERCISE_1":
        print(f"oracle sha256 (record this): {digest}")
    elif digest != ORACLE_SHA256:
        print("oracle drift: test_inventory.py no longer matches the freeze")
        return 2

    diff = run(["git", "diff", args.base, "--", "."])
    if diff.returncode != 0:
        print(diff.stderr)
        return 3

    budget = changed_lines(diff.stdout)
    blocked = forbidden_paths(diff.stdout)
    pytest_result = run([sys.executable, "-m", "pytest", "-q", "test_inventory.py"])

    print(f"changed_lines={budget} max={MAX_CHANGED_LINES}")
    print(f"forbidden={blocked or 'none'}")
    print(pytest_result.stdout)
    print(pytest_result.stderr)

    failures = []
    if budget > MAX_CHANGED_LINES:
        failures.append("diff budget exceeded")
    if blocked:
        failures.append("forbidden path edited")
    if pytest_result.returncode != 0:
        failures.append("oracle tests failed")

    if failures:
        print("SCORE: REJECT")
        print("reasons:", "; ".join(failures))
        return 1

    print("SCORE: ACCEPT")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The scorer never asks the model whether the patch is good, which keeps generation and judgment on separate sides. Exit code one means reject, exit code two means oracle drift, and exit code zero is the only accept path. Students should read the failure list before they retry a prompt, because retries without logs hide systematic errors.

Exercise 1 — Freeze the oracle (0–10 minutes)

Initialize git so later diffs have a stable base, then record the oracle digest that the scorer prints. The first pytest run should fail on test_reserve_reduces_available, because reserved stock is still counted as free. Paste the printed sha256 into ORACLE_SHA256 and commit that constant as a second freeze commit before continuing.

cd oracle_lab
python3 -m venv .venv
source .venv/bin/activate
pip install pytest
git init
git add inventory.py test_inventory.py score_patch.py
git commit -m "freeze baseline before any model run"
python score_patch.py
pytest -q test_inventory.py; echo pytest_exit:$?

From this minute forward, a student who simplifies the tests has broken the protocol, not improved the module. Facilitators should write the digest on a shared board so late arrivals can catch up without regenerating hashes. If two machines print different digests, stop and compare line endings before continuing, because that drift poisons later scores.

Exercise 2 — Encode the budget (10–25 minutes)

Keep MAX_CHANGED_LINES at twelve for this module, since the correct fix is a one-line correction plus a comment. Ask students to list files the model must not touch, then confirm those prefixes match FORBIDDEN_PREFIXES in the scorer. A useful discussion prompt is why test_inventory.py is forbidden, since edited tests can fake a green run. Optional extra rules should stay in a table so each rule remains a measurable predicate rather than taste.

Rule	Predicate	Fail code
Oracle hash	`sha256(test_inventory.py)` matches freeze	2
Diff budget	changed lines ≤ 12	1
Path lock	patch does not edit tests or scorer	1
Behavior	pytest on the frozen file exits 0	1

Students should explain each row without referring to any model vendor, product name, or chat screenshot. If a row cannot be checked by a command, it does not belong in the acceptance table for this lab. Taste-based rules such as formatter churn can wait until the behavioral oracle is stable on every machine.

Exercise 3 — Score two fixtures (25–40 minutes)

Do not call a hosted model yet, because the scorer must be proven against fixtures before the network is involved. Hand the class two patches as files, then apply each one on a clean tree and record the scorer exit code. The known-good patch only subtracts reserved quantity, while the vibe patch adds helpers and rewrites tests.

Known-good patch (good.patch):

--- a/inventory.py
+++ b/inventory.py
@@ -10,7 +10,7 @@ class Line:
     reserved: int = 0

     def available(self) -> int:
-        return self.qty  # bug: reserved stock is still counted as available
+        return self.qty - self.reserved

Vibe patch (vibe.patch) — this one looks more complete in a chat window and must still reject:

--- a/inventory.py
+++ b/inventory.py
@@ -1,5 +1,6 @@
 """In-memory SKU ledger used as the system under test."""
+
+import json
 from dataclasses import dataclass
@@ -14,6 +15,18 @@ class Line:
         return self.qty  # bug: reserved stock is still counted as available
+
+    def as_json(self) -> str:
+        return json.dumps({"sku": self.sku, "qty": self.qty})
+
+
def dump_ledger(ledger: "Ledger") -> str:
+    return json.dumps(sorted(ledger._lines))
--- a/test_inventory.py
+++ b/test_inventory.py
@@ -10,11 +10,6 @@ def test_add_and_available_without_reserve():
     assert ledger.available("sku-a") == 10
-
-
def test_reserve_reduces_available():
-    ledger = Ledger()
-    ledger.add("sku-a", 10)
-    ledger.reserve("sku-a", 4)
-    assert ledger.available("sku-a") == 6

git apply good.patch
python score_patch.py; echo score_exit:$?
git checkout -- inventory.py
git apply vibe.patch
python score_patch.py; echo score_exit:$?
git checkout -- .

Record both exit codes in a shared sheet so the room can see accept and reject as data rather than opinion. Extra features are not credit when they blow the budget or rewrite the oracle that was frozen in Exercise 1. If any group accepts the vibe patch, inspect whether they ran pytest without score_patch.py, which is a process failure.

Expected transcripts look like the following pair, with hashes replaced by the classroom freeze value.

# good.patch
changed_lines=2 max=12
forbidden=none
SCORE: ACCEPT

# vibe.patch
changed_lines=18 max=12
forbidden=['test_inventory.py']
SCORE: REJECT
reasons: diff budget exceeded; forbidden path edited; oracle tests failed

Exercise 4 — Optional remote generation (40–60 minutes)

Only start this exercise after both fixtures have been scored, so remote noise cannot mask a broken harness. The model remains a patch source, and the scorer remains the only judge that can print SCORE: ACCEPT. One convenient option is MonkeyCode, which currently offers free model access and a free server option. Students can point that host at the same prompt after the local oracle is frozen.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the prompt narrow so the remote run stays comparable to the known-good patch from Exercise 3. Students may tune wording but must not paste test_inventory.py into an editable region of the prompt. A starter prompt follows, and the class should treat extra commentary from the model as a defect in the patch.

Return a unified diff against inventory.py only.
Do not edit tests. Do not add files. Do not exceed twelve changed lines.
Line.available must subtract reserved from qty.
Do not reformat unrelated code.

The request helper below is labeled as an unexecuted proposal. Wire TLS, auth, and timeouts in the lab, using whatever HTTP client the room already maintains.

# labeled example: unexecuted request helper, fill in the classroom endpoint
import os
import urllib.request

def request_patch(prompt: str, endpoint: str) -> str:
    # Proposal only: do not hard-code secrets; read them from the environment.
    raise NotImplementedError("connect the classroom endpoint here")

Apply and score exactly as in Exercise 3:

git apply model.patch
python score_patch.py; echo score_exit:$?
git checkout -- inventory.py

Apply model.patch with the same git apply and score_patch commands used on the local fixtures. If the free server is busy or the model returns prose instead of a diff, log a reject and continue. The oracle does not pause for vendor availability, which is the behavior a merge gate needs in practice. This workshop does not publish latency numbers, token quotas, or model names, because those figures drift quickly.

Exercise 5 — Debrief (60–75 minutes)

Walk the board through four failure classes and map each class to a scorer branch or a written protocol rule. Keep the discussion forensic, with transcripts and exit codes, rather than with impressions of how helpful the chat felt. Homework is a second module of the student's choice, scored with a budget taken from a hand-written fix.

Oracle drift. Someone edited tests to match a messy patch, and the sha256 guard should have blocked the run.
Budget overflow. A rewrite that cleans the file changes thirty lines and hides the one-line semantic fix.
Unparseable output. The model returned markdown fences, a lecture, or a partial hunk, which is a reject.
False confidence. The patch passes tests but touches score_patch.py, which is part of the trusted computing base.

Students submit the scorer transcript, not a screenshot of a chat window, because screenshots are not replayable evidence. Replay after class by resetting to the freeze commit and applying good.patch again on a clean tree. The accept path should be stable on every machine if line endings and the oracle hash were recorded correctly.

Limitations

This harness does not prove concurrency correctness, API compatibility, or runtime performance under production traffic. Four pytest cases cannot stand in for a production suite, and a twelve-line budget is a classroom constant. Binary files, generated code, and license headers will inflate changed_lines unless the class extends the counter. Semantic edits that stay under twelve lines can still be wrong in ways the oracle never observed.

The remote exercise depends on a host that may rate-limit, change, or refuse the prompt shape without notice. Students should keep local fixtures so the class still completes when the network path fails or stalls. Nothing in the scorer inspects prompt injection, secret leakage, or the license of generated code. Those reviews stay human, and they should happen before any patch is merged outside this classroom repository.

Who should not use this approach

Repositories that cannot run tests offline, because the oracle would then depend on the same network as the model.
Compliance-bound codebases that forbid sending source to a third-party host, even when that host is free.
Facilitators who need a fixed model identity, quota, or uptime number in the syllabus, which this lab does not provide.
Teams that already gate merges with mature CI covering tests, ownership, and size, for whom the budget table is redundant.

After class

Replay the lab by resetting to the freeze commit and applying good.patch; the accept path should stay stable. If a machine rejects that fixture, compare the oracle hash and line endings before blaming the scorer logic. For an optional Exercise 4 backend, try MonkeyCode's free model access and free server after the oracle hash is recorded. Leave the local fixtures in the repo so later cohorts can grade patches even when that host is offline.

Workshop: Freeze Agent Output Contracts Before You Swap Endpoints in 80 Minutes

Finley Zhu — Wed, 16 Sep 2026 04:46:32 +0000

Swapping an inference endpoint without a frozen JSON contract is how agent demos pass and production loops fail. This workshop treats each model reply as a versioned API, not as free-form text that happens to look structured. Students pin required fields, enumerations, and forbidden keys, then hash a golden fixture so later runs fail closed on silent drift. The same harness works against any HTTP completion endpoint, including a local stub, so the lesson stays vendor-neutral.

The outline below is a teaching plan, not a production incident report. Commands and modules are workshop starters you can rerun; they are not claims about live traffic, latency, or model rankings. Keep personal API keys off the projector, and keep prompt files free of customer data while you practice.

Audience and prerequisites

This session is for instructors and students who already have a tiny agent loop that returns JSON. It also fits platform engineers who change completion URLs more often than they change unit tests. You need Python 3.11+, pytest, and the ability to POST a prompt to an HTTP endpoint that returns a JSON object.

Skip the session if your agent streams unstructured prose with no schema, or if your only goal is a qualitative vibe check. The contract ledger will not grade reasoning quality, retrieval freshness, or safety policy. It only answers whether the reply still matches the shape your downstream code already assumed.

Eighty-minute agenda

Use a visible timer. When a block overruns, cut discussion rather than skipping the fail-closed test, because that test is the artifact students take home.

0–8 min — Frame the miss. Show a passing test that only asserts tool == "triage" while the JSON body quietly changes type.
8–20 min — Define the ledger. Agree on required keys, enums, ranges, and keys that must never appear.
20–38 min — Exercise 1. Record one golden fixture and print its SHA-256.
38–55 min — Exercise 2. Encode fail-closed rules and watch pytest fail on extra keys.
55–72 min — Exercise 3. Point the same fixture at a second endpoint without editing assertions.
72–80 min — Decision table and limitations. Classify warn versus fail, then list who should not ship this as a safety gate.

Failure mode: tests that only watch tool names

Many agent suites assert that a tool was selected, then treat the remaining payload as an untyped blob. That pattern survives a model swap even when confidence flips from a float to a string, or when priority gains a new label such as urgent. Downstream routers then throw at runtime, while CI stays green because the tool name never changed.

A second miss is leakage. Completions sometimes echo raw_prompt, chain-of-thought, or a key-shaped string that a student pasted into a notebook. Field allowlists catch that class of accident faster than a human reading a transcript. The workshop therefore freezes three layers at once: required types, closed enumerations, and forbidden keys.

Current public debate about agents outgrowing their tests is useful as motivation, not as a scoring rubric. You do not need a new leaderboard. You need a replayable object that fails when the JSON contract moves.

Artifact: frozen contract ledger

Create a directory named agent_contracts/ and keep fixtures next to the checker. The golden file is the source of truth; the checker is only an interpreter of that file. Students should rerun the same commands after any endpoint change, including a move from a laptop stub to a shared classroom URL.

Golden fixture

{
  "schema_version": "triage.v1",
  "category": "billing",
  "priority": "p1",
  "confidence": 0.64,
  "rationale_id": "r-0142"
}

Save that object as agent_contracts/golden/triage_v1.json. Do not pretty-print it differently across machines if you want stable hashes; the checker below canonicalizes keys before hashing.

Checker module

# agent_contracts/ledger.py
from __future__ import annotations

import hashlib
import json
from typing import Any

REQUIRED = {
    "category": str,
    "priority": str,
    "confidence": float,
    "rationale_id": str,
}
ALLOWED_PRIORITY = {"p0", "p1", "p2", "p3"}
FORBIDDEN_KEYS = {"raw_prompt", "api_key", "internal_cot", "tool_dump"}
OPTIONAL = {"notes", "schema_version"}


def canonical_dumps(payload: dict[str, Any]) -> str:
    return json.dumps(payload, sort_keys=True, separators=(",", ":"))


def fixture_hash(payload: dict[str, Any]) -> str:
    body = canonical_dumps(payload).encode("utf-8")
    return hashlib.sha256(body).hexdigest()


def check_contract(payload: Any) -> list[str]:
    errors: list[str] = []
    if not isinstance(payload, dict):
        return ["payload_not_object"]

    for key, expected in REQUIRED.items():
        if key not in payload:
            errors.append(f"missing:{key}")
            continue
        value = payload[key]
        if expected is float and isinstance(value, bool):
            errors.append(f"type:{key}:bool")
        elif expected is float and isinstance(value, int) and not isinstance(value, bool):
            value = float(value)
        elif not isinstance(value, expected):
            errors.append(f"type:{key}:{type(payload[key]).__name__}")

    allowed = set(REQUIRED) | OPTIONAL
    for key in sorted(set(payload) - allowed):
        errors.append(f"unexpected:{key}")
    for key in FORBIDDEN_KEYS:
        if key in payload:
            errors.append(f"forbidden:{key}")

    priority = payload.get("priority")
    if isinstance(priority, str) and priority not in ALLOWED_PRIORITY:
        errors.append(f"enum:priority:{priority}")

    confidence = payload.get("confidence")
    if isinstance(confidence, bool):
        errors.append("range:confidence")
    elif isinstance(confidence, (int, float)) and not (0.0 <= float(confidence) <= 1.0):
        errors.append("range:confidence")
    return errors

Pytest gate

# tests/test_triage_contract.py
import json
from pathlib import Path

from agent_contracts.ledger import check_contract, fixture_hash

GOLDEN = Path("agent_contracts/golden/triage_v1.json")


def test_golden_file_parses_and_passes_contract():
    payload = json.loads(GOLDEN.read_text(encoding="utf-8"))
    assert check_contract(payload) == []
    digest = fixture_hash(payload)
    assert len(digest) == 64


def test_string_confidence_is_a_contract_break():
    payload = json.loads(GOLDEN.read_text(encoding="utf-8"))
    payload["confidence"] = "0.64"
    assert "type:confidence:str" in check_contract(payload)


def test_forbidden_key_fails_closed():
    payload = json.loads(GOLDEN.read_text(encoding="utf-8"))
    payload["internal_cot"] = "..."
    assert "forbidden:internal_cot" in check_contract(payload)

Run the local gate before any remote call:

python -m pip install pytest
python -m pytest tests/test_triage_contract.py -q

Exercise 1 — Record the golden path (18 minutes)

Students call their current loop once, write the JSON to disk, and refuse to hand-edit fields that the model did not emit. If the model omitted rationale_id, that is a contract failure in the generator, not a license to invent an identifier in the fixture. Label any repaired example as synthetic in notes so later readers do not treat it as a live trace.

Proposed recording client (unexecuted until you fill ENDPOINT):

# scripts/record_fixture.py
from __future__ import annotations

import json
import os
import urllib.request
from pathlib import Path

from agent_contracts.ledger import check_contract, fixture_hash

ENDPOINT = os.environ["AGENT_COMPLETE_URL"]
PROMPT = {
    "task": "triage",
    "ticket": "Invoice #441 was charged twice after the March workspace upgrade.",
    "schema_version": "triage.v1",
}


def complete(prompt: dict) -> dict:
    req = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(prompt).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode("utf-8"))


def main() -> None:
    payload = complete(PROMPT)
    errors = check_contract(payload)
    Path("agent_contracts/last_errors.json").write_text(
        json.dumps(errors, indent=2), encoding="utf-8"
    )
    if errors:
        raise SystemExit(f"contract_failed:{errors}")
    out = Path("agent_contracts/golden/triage_v1.json")
    out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
    print(fixture_hash(payload))


if __name__ == "__main__":
    main()

export AGENT_COMPLETE_URL="http://127.0.0.1:8080/complete"
python scripts/record_fixture.py

Print the hash on the board. Students should treat a hash change without a schema_version bump as an incident, even when the test count stays the same.

Exercise 2 — Encode fail-closed rules (17 minutes)

Add one mutation per pair of students. Useful mutations include stringified numbers, an extra debug_trace key, a priority of urgent, and a confidence of 1.7. Each mutation must produce a stable error token such as enum:priority:urgent, not a traceback from application code.

If your loop wraps the model object in { "ok": true, "data": { ... } }, contract the inner object, not the envelope. Mixing transport metadata into the domain schema is how later retries look like product changes. Keep schema_version on the domain object so two envelopes can carry the same contract.

Exercise 3 — Swap the endpoint, keep the fixture (17 minutes)

Point AGENT_COMPLETE_URL at a second completion URL without editing pytest. The golden file stays read-only. If the second endpoint cannot satisfy triage.v1, the workshop result is a documented incompatibility, not a rewritten fixture. That is the entire point of freezing the contract before the swap.

Classroom groups often need a shared completion URL so nobody pastes personal keys into a projector demo. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option, which can host this replay runner for a lab section. Confirm live availability on the project page before you schedule the session, because free capacity is not a contractual SLA and must not be described with invented quotas or hardware claims.

A local stub remains the correct fallback when the shared box is busy:

# scripts/stub_server.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

FIXTURE = {
    "schema_version": "triage.v1",
    "category": "billing",
    "priority": "p1",
    "confidence": 0.64,
    "rationale_id": "r-0142",
}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        length = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(length)
        body = json.dumps(FIXTURE).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

python scripts/stub_server.py

Worked example: ticket triage loop

The domain rule for this example is narrow on purpose. A ticket about duplicate charges must come back as category=billing, priority in {p0,p1,p2,p3}, and confidence in [0,1]. rationale_id is an opaque string your notes system already knows; the model may not invent a new citation format mid-workshop.

Proposed loop (label as a teaching stub, not a production agent):

# agent_contracts/loop.py
from __future__ import annotations

import json
from typing import Any, Callable

from agent_contracts.ledger import check_contract

CompleteFn = Callable[[dict[str, Any]], dict[str, Any]]


def triage_once(ticket: str, complete: CompleteFn) -> dict[str, Any]:
    prompt = {
        "task": "triage",
        "ticket": ticket,
        "schema_version": "triage.v1",
        "fields": ["category", "priority", "confidence", "rationale_id"],
    }
    payload = complete(prompt)
    errors = check_contract(payload)
    if errors:
        raise ValueError("contract_failed:" + ",".join(errors))
    return payload

A replay test that students can keep after class:

# tests/test_triage_loop.py
import json
from pathlib import Path

from agent_contracts.loop import triage_once


def test_loop_rejects_drifted_endpoint():
    golden = json.loads(
        Path("agent_contracts/golden/triage_v1.json").read_text(encoding="utf-8")
    )

    def drifted(_prompt):
        broken = dict(golden)
        broken["priority"] = "urgent"
        return broken

    try:
        triage_once("Invoice charged twice", drifted)
        raise AssertionError("expected contract failure")
    except ValueError as err:
        assert "enum:priority:urgent" in str(err)

python -m pytest tests/test_triage_loop.py tests/test_triage_contract.py -q

If both files pass, students have a rerunnable cassette of shape, not of prose. They can later attach a real completer without rewriting the assertions. That is the workshop’s take-home artifact.

Decision table for contract severity

Observation	Default action	Why
Missing required key	Fail closed	Downstream code will throw anyway
Type change (`float` → `str`)	Fail closed	Silent coercion hides ranking bugs
New optional key in `OPTIONAL`	Allow	Notes and version stamps are expected
New unknown key	Fail closed in class	Unknown keys are how leakage starts
Enum extension (`urgent`)	Fail closed until version bump	Routers cannot interpret a new label
Hash change with same schema_version	Fail closed	Someone edited the golden file by hand
Semantic nonsense with valid shape	Do not catch here	Needs a separate eval set

Bump schema_version when you intentionally widen the contract. Do not widen it to make a weaker model look compatible. Compatibility theater is how this workshop turns into theater.

Limitations

JSON contracts do not prove that billing was the right category, only that the field existed as a string. They also do not bound token spend, tool fan-out, or retry storms; those are different workshops with different meters. Free or shared endpoints can change behavior without notice, so a passing lab on Monday is not evidence for a capacity plan on Friday.

Canonical hashing is sensitive to key order only after canonicalization; it is still sensitive to semantically equal numbers encoded as 0.64 versus 0.640. If you need numeric tolerance, add an explicit rounding step before the hash, and document it in the fixture README. Do not hide rounding inside the completer.

Who should not run this workshop

Teams that need a safety certifier for medical, legal, or financial decisions should not treat this ledger as that certifier.
Groups without permission to send tickets to a shared server should stay on the local stub.
Courses that only want a chat demo will find the fail-closed posture frustrating, and they should pick a different lab.
Operators who cannot state a schema_version should fix naming first; the checker cannot invent a product contract.

Close-out checklist

Students leave with four files: the golden JSON, ledger.py, two pytest modules, and a one-row note of the endpoint URL they last used. The URL is configuration, not curriculum. If the next lab swaps endpoints, the curriculum stays still, and the contract either holds or it does not.

Failure Injection for Agent Loops: A Replayable Cassette Workshop

Finley Zhu — Mon, 14 Sep 2026 22:00:25 +0000

Failure Injection for Agent Loops: A Replayable Cassette Workshop

Most agent loops do not have a degradation plan; they have a retry wrapper and an optimistic assumption. This workshop replaces that assumption with rehearsal: students build a cassette-backed fault injector that replays quota exhaustion, timeouts, and malformed tool calls entirely offline. The whole 90-minute session runs on a laptop with no network access after the first recording pass. Below is the schedule, the reference harness, and the assertions that define a passing submission.

Why rehearsal beats monitoring for quota-limited endpoints

Monitoring tells you that a loop failed at 03:00; a cassette harness tells you what your loop does before it ever runs in production. When many exercises share one rate-limited endpoint, failures stop being rare events and become ordinary traffic. Three patterns dominate in that setting: a hard quota rejection, a transient timeout, and a schema-invalid tool call that arrives as a successful HTTP response.

This is where the free-tier framing becomes practical rather than promotional. MonkeyCode's operator describes a free tier that includes model access (the outreach brief cites 10 million free tokens) and a free server option, and the project is described as open source. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat every availability number as operator-supplied and re-check current terms before you depend on them, because the workshop method below does not change if the quota changes.

The free server option matters here for exactly one narrow reason. If your runner process can be recycled between exercises, then process lifetime is a fault you must model, not an assumption you may keep. That single reframing is the difference between a harness that survives a mid-session restart and one that loses partial work silently.

The artifact: a clock-injected harness in three files

The starter repository has three files. Students receive the first two, and they write the third themselves, because the tests are the deliverable.

1. `harness.py` — the loop under test

# harness.py
import time

class QuotaExhausted(RuntimeError):
    """Provider said stop. Carries Retry-After when the header was present."""
    def __init__(self, retry_after=None):
        super().__init__("quota_exhausted")
        self.retry_after = retry_after

class TransientError(RuntimeError):
    """Timeouts and 5xx responses: retryable by default."""

class BadToolCall(RuntimeError):
    """Response arrived, but the tool-call JSON did not fit the schema."""

def fault_transport(cassette, faults):
    """cassette: dict op -> response. faults: consumed one item per call."""
    plan = list(faults)
    calls = {"n": 0}

    def transport(payload):
        calls["n"] += 1
        fault = plan.pop(0) if plan else None
        if fault is not None:
            raise fault
        return cassette[payload["op"]]

    transport.calls = calls
    return transport

def run_steps(transport, ops, policy, sleep=time.sleep):
    outcomes = []
    for op in ops:
        attempt = 0
        while True:
            attempt += 1
            try:
                outcomes.append({"op": op, "status": "ok",
                                 "value": transport({"op": op}),
                                 "attempts": attempt})
                break
            except BadToolCall:
                outcomes.append({"op": op, "status": "degraded",
                                 "reason": "schema", "attempts": attempt})
                break
            except TransientError:
                if attempt >= policy["max_attempts"]:
                    outcomes.append({"op": op, "status": "degraded",
                                     "reason": "timeout", "attempts": attempt})
                    break
                sleep(policy["base_delay"] * 2 ** (attempt - 1))
            except QuotaExhausted as exc:
                outcomes.append({"op": op, "status": "stopped",
                                 "reason": "quota", "retry_after": exc.retry_after})
                return outcomes
    return outcomes

2. `cassettes.py` — record once, replay forever

# cassettes.py
import hashlib, json, pathlib

class CassetteMiss(Exception):
    pass

def fingerprint(payload: dict) -> str:
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16]

class CassetteStore:
    def __init__(self, root="cassettes", mode="replay"):
        assert mode in {"record", "replay"}
        self.root = pathlib.Path(root)
        self.root.mkdir(exist_ok=True)
        self.mode = mode

    def call(self, transport, payload: dict) -> dict:
        path = self.root / f"{fingerprint(payload)}.json"
        if self.mode == "replay":
            if not path.exists():
                raise CassetteMiss(f"no cassette for {payload.get('op')}")
            return json.loads(path.read_text())["response"]
        response = transport(payload)
        path.write_text(json.dumps({"request": payload, "response": response}, indent=2))
        return response

3. `test_policy.py` — the graded deliverable

# test_policy.py
from harness import (BadToolCall, QuotaExhausted, TransientError,
                     fault_transport, run_steps)

POLICY = {"max_attempts": 3, "base_delay": 0.01}
CASSETTE = {"plan": {"steps": ["read", "patch"]}, "patch": {"ok": True}}

def test_transient_is_retried_once_then_succeeds():
    transport = fault_transport(CASSETTE, [TransientError, None])
    out = run_steps(transport, ["plan"], POLICY, sleep=lambda _: None)
    assert [o["status"] for o in out] == ["ok"]
    assert out[0]["attempts"] == 2

def test_quota_stops_the_loop_and_keeps_partial_work():
    transport = fault_transport(CASSETTE, [None, QuotaExhausted(retry_after="2")])
    out = run_steps(transport, ["plan", "patch"], POLICY, sleep=lambda _: None)
    assert [o["status"] for o in out] == ["ok", "stopped"]
    assert out[1]["retry_after"] == "2"

def test_bad_tool_call_degrades_without_retrying():
    transport = fault_transport(CASSETTE, [BadToolCall])
    out = run_steps(transport, ["plan"], POLICY, sleep=lambda _: None)
    assert out[0]["status"] == "degraded"
    assert transport.calls["n"] == 1

The full suite finishes in well under a second because the sleep function is injected, which is the point of the exercise.

The 90-minute schedule

Minutes	Segment	Student output
0-10	Framing: three failure classes and why monitoring arrives late	Written list of their own loop's failure modes
10-25	Exercise 1: record one golden transcript to `cassettes/`	Five cassette files committed
25-40	Exercise 2: replay offline and diff two runs	Proof that replay is byte-identical after stripping volatile fields
40-65	Exercise 3: inject quota, timeout, and bad schema	Three failing tests, then three passing ones
65-80	Exercise 4: write the degradation policy and encode it	The policy table below, expressed as assertions
80-90	Debrief: red-team a neighbour's policy	One adversarial case that breaks their loop

Exercise notes that keep the room moving

In Exercise 1, strip timestamps and request IDs before committing, or replay determinism fails for reasons unrelated to your loop.
In Exercise 2, assert equality on the outcome list, not on raw response bodies, because provider payloads drift more often than your control flow.
In Exercise 3, keep one fault list per test; mixing a quota fault with a timeout fault in one list teaches the wrong lesson about causality.
In Exercise 4, require a numeric bound on total wait time, otherwise a student can pass every assertion with an unbounded backoff.

The degradation policy, before and after

Failure	Detection	Policy	Budget effect	Required log field
Quota rejection with `Retry-After`	Header present	One retry after the stated delay, then stop the run	At most one extra call	`retry_after`
Quota rejection without a header	Header absent	Stop immediately, no retry	Zero extra calls	`reason=quota`
Timeout	Transport exception	Up to `max_attempts`, then degrade	Bounded by policy	`attempts`
Schema-invalid tool call	Validator rejects JSON	Repair once with a stricter instruction, then mark degraded	One repair call	`reason=schema`
Slow first token	Elapsed time over threshold	No retry; reduce concurrency instead	Unchanged call count	`elapsed_ms`

Four invariants every submission must hold

Call budget: for a scripted op list, the transport is invoked no more times than the policy allows.
Replay determinism: the same cassette and policy produce an identical outcome list, ignoring declared volatile fields.
Partial work survival: a quota stop returns all completed steps, not an empty list.
Bounded waiting: total injected clock time stays under the policy's declared maximum.

Limitations, and who should skip this workshop

The method has real edges, and pretending otherwise would waste your time. A cassette records responses, not latency distributions, so throughput questions stay unanswered unless you record timings separately. Replay also cannot detect that a provider changed its behaviour last week, because the whole point is to serve the old bytes. The free-tier figures quoted above are operator-supplied and can change without notice, so treat them as the workshop's starting conditions rather than a permanent guarantee.

If your endpoint has a paid SLA, existing chaos tooling, and cheap idempotent calls, this harness is ceremony without benefit. If your correctness depends on live streaming semantics such as token timing, prompt caching behaviour, or log probabilities, replay-only testing is insufficient by construction. Finally, never commit real prompts, customer data, or credentials into cassettes/; redact first, and assume the directory will be published someday.

What to take back to your own repository

The transferable lesson is not the file layout, but the habit of treating availability as a tested input rather than a background hope. Teams that rehearse quota exhaustion tend to discover two things quickly: their retry loop spends budget it never needed, and their partial results are lost in a code path nobody had read. Both are cheap to fix once a cassette makes them reproducible in under a second.

If you want somewhere low-volume to record that first golden transcript before you buy a key, the free tier described earlier is designed for exactly this kind of rehearsal; record once, replay forever, and bring assertions sharp enough to break your own loop.

MonkeyCode provides free models that can run this workflow.

Workshop: Catch No-Op Agent Turns With a State Digest in 80 Minutes

Finley Zhu — Sun, 13 Sep 2026 20:55:32 +0000

Agent loops should terminate when tool results stop changing application state, not when the model still wants another call. This workshop treats an observed stall as a first-class failure mode rather than a retryable timeout. Students will instrument call fingerprints, count consecutive no-op turns, and exit with a structured reason code. The full sequence fits in eighty minutes with a rerunnable fixture and a failing control case.

A practical reading of this week's agent debate

Public developer threads this week keep asking whether typical agents are planners or retrying conditionals with extra ceremony. This workshop does not try to settle that argument, because the stall test does not depend on the label. If a loop can spend tokens without moving state, the wrapper around the tools is a secondary concern. A fail-closed stall bound remains a testable control whether the planner is a model or a scripted list.

Why stalls hide inside successful tool calls

Many classroom agents report success because each tool wrapper returned JSON without raising an exception. That signal is too weak for a planner that slightly mutates arguments and repeats the same tool. Instructors usually notice the waste only after a shared rate limit starts failing during a live lab. The missing check is whether application state actually changed between consecutive planner turns in the store.

A progress bound is not a latency timeout and is not a maximum step cap copied from tutorials. Timeouts fire when the network is slow, and step caps fire even when useful work continues. A stall counter fires only when tool fingerprints and a state digest repeat without a write. That distinction keeps long retrieval jobs alive while still stopping silent argument churn on no-op writes.

Workshop timing and outcomes

The intended audience is backend and ML-adjacent developers who already maintain a tool-calling agent loop. Prior labs on this account treated exit events, router scoring, source tags, and fan-out caps separately. This session isolates state-change detection so those other controls are not mixed into one patch.

Clock

0–10 min: Freeze a tiny ticket store and a planner fixture that repeats equivalent updates on purpose.
10–30 min: Canonicalize tool arguments, hash a domain snapshot, and print both fingerprints after every turn.
30–50 min: Increment a stall counter on repeated no-op digests, then fail closed with STALL_NO_STATE_CHANGE.
50–70 min: Rerun the looping control case and the progressing case, then fill the four-row decision table.
70–80 min: Review snapshot pitfalls, list who should skip this bound, and decide whether a hosted planner swap is even useful.

Exit criteria

A rerunnable Python module named stall_lab.py that loops on purpose before the bound is applied.
Working call_fp and state_fp values that stay stable across JSON key reordering on the same arguments.
A fail-closed policy that returns STALL_NO_STATE_CHANGE before eight identical updates can finish.
A decision table that classmates can reuse in review without reading an entire tool-call transcript.

Prerequisites

Each student needs Python 3.11 or newer, a local shell, and permission to run scripts. No production credentials belong in this lab, because the planner is a fixture rather than a live client. If the group later swaps the fixture planner for a hosted model, keep the stall policy outside the adapter. The bound must not depend on prompt wording if the exercises are going to stay testable.

Exercise 1 (10 min): freeze a looping fixture

The fixture below is labeled workshop material and is not a captured production trace from any vendor. It models a planner that keeps calling update_ticket with equivalent payloads while the store stays unchanged. Students should save the file as stall_lab.py and run it before adding any fail-closed bound.

# Workshop fixture: local only. Not a vendor client and not production telemetry.
from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass, field
from typing import Any, Callable


def canonical_json(value: Any) -> str:
    return json.dumps(value, sort_keys=True, separators=(",", ":"))


def sha256_hex(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


@dataclass
class TicketStore:
    status: str = "open"
    assignee: str = "unassigned"
    writes: int = 0

    def snapshot(self) -> dict[str, Any]:
        return {"status": self.status, "assignee": self.assignee}

    def update_ticket(self, status: str, assignee: str) -> dict[str, Any]:
        if self.status != status or self.assignee != assignee:
            self.status = status
            self.assignee = assignee
            self.writes += 1
        return {"ok": True, "status": self.status, "assignee": self.assignee}


TOOLS: dict[str, Callable[..., dict[str, Any]]] = {}


def looping_plan() -> list[dict[str, Any]]:
    same = {"status": "open", "assignee": "unassigned"}
    return [{"tool": "update_ticket", "args": same} for _ in range(8)]


def progress_plan() -> list[dict[str, Any]]:
    return [
        {"tool": "update_ticket", "args": {"status": "open", "assignee": "ada"}},
        {"tool": "update_ticket", "args": {"status": "pending", "assignee": "ada"}},
    ]


def run_naive_loop(store: TicketStore, plan: list[dict[str, Any]]) -> dict[str, Any]:
    last = None
    for step in plan:
        last = TOOLS[step["tool"]](store, **step["args"])
    return {"ok": True, "steps": len(plan), "last": last, "writes": store.writes}

Smoke command

TOOLS["update_ticket"] = lambda store, **kwargs: store.update_ticket(**kwargs)

if __name__ == "__main__":
    report = run_naive_loop(TicketStore(), looping_plan())
    print(canonical_json(report))

python stall_lab.py

The smoke run should report eight steps, a true ok flag, and zero writes against the ticket store. That combination is the failure mode this workshop is designed to catch in the later exercises. If students already mutate the store on every call, they should restore the no-op path first.

Exercise 2 (20 min): fingerprint the call and the store

A fingerprint has two halves that must be hashed separately if debugging is going to stay honest. The call half covers tool name plus canonical arguments, which catches retries that only shuffle JSON keys. The state half covers a domain snapshot, which catches tools that wrap unchanged records in new JSON. Combining those halves into one digest too early hides which side of the turn was actually stable.

@dataclass
class TurnTrace:
    index: int
    call_fp: str
    state_fp: str
    combined: str
    wrote: bool


def fingerprint_turn(
    index: int,
    tool: str,
    args: dict[str, Any],
    store: TicketStore,
    writes_before: int,
) -> TurnTrace:
    call_fp = sha256_hex(canonical_json({"tool": tool, "args": args}))
    state_fp = sha256_hex(canonical_json(store.snapshot()))
    combined = sha256_hex(f"{call_fp}:{state_fp}")
    wrote = store.writes > writes_before
    return TurnTrace(index, call_fp, state_fp, combined, wrote)

Students should print both hashes after each turn rather than logging only the combined digest value. Classroom debugging goes faster when the group can say the call matched and the state matched. Keep the snapshot function small and explicit, and do not serialize entire Python objects into the digest.

Gate before the next block

Reordered keys in args must produce the same call_fp after canonical_json runs.
A real assignee change must produce a new state_fp and a true wrote flag.
A no-op update must keep state_fp stable and leave wrote false for the stall counter.

Exercise 3 (20 min): fail closed on consecutive stalls

The policy is a small integer threshold, not a classifier score and not a model-generated confidence value. After each turn, increment stalls when the write flag is false and the combined fingerprint repeats. Reset the counter to zero whenever a turn records a real write against the local ticket store. When stalls reach max_stalls, stop immediately and return a structured reason instead of another planner step.

@dataclass
class LoopReport:
    ok: bool
    reason: str
    steps: int
    stalls: int
    writes: int
    traces: list[TurnTrace] = field(default_factory=list)


def run_bounded_loop(
    store: TicketStore,
    plan: list[dict[str, Any]],
    max_stalls: int = 2,
) -> LoopReport:
    traces: list[TurnTrace] = []
    stalls = 0
    prev_combined: str | None = None

    for index, step in enumerate(plan, start=1):
        writes_before = store.writes
        result = TOOLS[step["tool"]](store, **step["args"])
        trace = fingerprint_turn(index, step["tool"], step["args"], store, writes_before)
        traces.append(trace)

        stalled = (not trace.wrote) and (prev_combined == trace.combined)
        stalls = stalls + 1 if stalled else 0
        prev_combined = trace.combined

        if stalls >= max_stalls:
            return LoopReport(
                ok=False,
                reason="STALL_NO_STATE_CHANGE",
                steps=index,
                stalls=stalls,
                writes=store.writes,
                traces=traces,
            )
        _ = result

    return LoopReport(
        ok=True,
        reason="PLAN_EXHAUSTED",
        steps=len(plan),
        stalls=stalls,
        writes=store.writes,
        traces=traces,
    )

Replace the naive main block with two runs: the looping plan and a plan that assigns a ticket. Students should treat max_stalls equal to two as a lab default rather than a portable production constant. Reviewers can later raise the threshold for systems that acknowledge writes only after a short delay.

Worked example students can rerun

The assertions below are lab checks against the fixture, not measurements from a hosted model. Append them to stall_lab.py after TOOLS is wired, then run the file again from a clean shell. The looping case should die early; the progress case should exhaust its two mutating steps.

def report_to_dict(report: LoopReport) -> dict[str, Any]:
    return {
        "ok": report.ok,
        "reason": report.reason,
        "steps": report.steps,
        "stalls": report.stalls,
        "writes": report.writes,
        "trace_count": len(report.traces),
    }


def run_lab_assertions() -> dict[str, Any]:
    TOOLS["update_ticket"] = lambda store, **kwargs: store.update_ticket(**kwargs)

    looping = run_bounded_loop(TicketStore(), looping_plan(), max_stalls=2)
    progress = run_bounded_loop(TicketStore(), progress_plan(), max_stalls=2)

    reordered = [
        {"tool": "update_ticket", "args": {"assignee": "unassigned", "status": "open"}},
        {"tool": "update_ticket", "args": {"status": "open", "assignee": "unassigned"}},
        {"tool": "update_ticket", "args": {"status": "open", "assignee": "unassigned"}},
    ]
    shuffled = run_bounded_loop(TicketStore(), reordered, max_stalls=2)

    assert looping.ok is False and looping.reason == "STALL_NO_STATE_CHANGE"
    assert looping.writes == 0 and looping.steps < 8
    assert progress.ok is True and progress.reason == "PLAN_EXHAUSTED"
    assert progress.writes == 2
    assert shuffled.ok is False and shuffled.reason == "STALL_NO_STATE_CHANGE"

    return {
        "looping": report_to_dict(looping),
        "progress": report_to_dict(progress),
        "shuffled_keys": report_to_dict(shuffled),
    }


if __name__ == "__main__":
    print(canonical_json(run_lab_assertions()))

python stall_lab.py

The looping case should end with ok false and reason STALL_NO_STATE_CHANGE before all eight steps run. The progress case should exhaust the short plan and report two writes against a fresh ticket store. If both cases return PLAN_EXHAUSTED, the stall predicate is not wired to the fingerprints yet.

Exercise 4 (20 min): decision table for review

Use the table during code review rather than arguing from unstructured model logs or chat transcripts. Each row is a lab observation from the fixture, not a vendor benchmark and not a latency study. Rerun stall_lab.py whenever someone changes snapshot so the hashes stay aligned with the domain fields.

Observation	`wrote`	Fingerprints	Policy result	Lab meaning
Eight identical `update_ticket` calls	false	`combined` repeats	`STALL_NO_STATE_CHANGE`	planner churn with no store movement
Status `open` to `pending` with a new assignee	true	`state_fp` changes	stalls reset	real progress, keep planning
Same arguments with reordered keys	false	`call_fp` stable	counts as stall	canonicalization is doing the job
Two mutating steps under a step cap of eight	true	never repeats	`PLAN_EXHAUSTED`	do not confuse a cap with a stall

Instructors can add a third plan that alternates two equivalent argument dicts with different key insertion order. That plan should still fail closed if canonical_json is used instead of hashing a raw dictionary string. If it does not fail, students probably hashed str(args) and should repair the canonicalization helper.

python -c "import stall_lab as s; s.TOOLS['update_ticket'] = lambda store, **kwargs: store.update_ticket(**kwargs); print(s.run_bounded_loop(s.TicketStore(), s.looping_plan()).reason)"

Optional hosted planner after local green

Some groups replace the list-based planner with a hosted tool-calling model after the fixture tests are green. That swap is optional and should happen only after the stall assertions pass without any network calls. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can host this same classroom harness.

Keep the stall policy in process, next to the store, rather than burying it inside a prompt template. Prompts cannot be unit-tested against key order and no-op writes with the same local determinism. If a hosted model is used, log reason, steps, stalls, and writes exactly as the fixture already does. Do not write model names or quota numbers into the lab notes, because those claims drift and break assertions.

If the class needs a shared place to rerun the green fixture, that free server option is enough. The assignment remains the stall bound; hosting is optional and should not replace the local tests. Skip the hosted path entirely when students can already reproduce both rows of the decision table locally.

Limitations

This bound assumes the snapshot function sees every field the business actually cares about during the loop. If students omit assignee from snapshot, a real assignment looks like a stall and the loop dies early. The opposite error is worse, because volatile timestamps make every no-op look like genuine forward progress. Snapshot design is therefore a domain decision, not a hashing trick that can be copied between products.

The stall counter also assumes tools are synchronous and that writes is a trustworthy local signal. External queues, eventually consistent stores, and human approval steps can delay a visible state change past the threshold. In those systems, pair this lab with an explicit pending field in the snapshot instead of guessing. Raising max_stalls without evidence just hides the same no-op pattern behind a larger integer threshold.

Fingerprint equality is not a security boundary and does not prove that a tool call was authorized. It also does not replace result schema validation or memory gating covered in earlier independent workshops on this account. The only question it answers is whether the loop is still moving the store between turns.

Who should not use this approach

Do not install this fail-closed policy on agents whose useful work is side-effect free by design. Read-only research loops that reread the same document will look stalled even when the user is satisfied. Do not use the bound as a substitute for timeouts when the failure mode is a hung socket. Do not ship the fixture planner to production, because it exists only to make stalls cheap to demonstrate.

Teams without a defined state snapshot should not hash whatever JSON the tool happened to return. Returning a new UUID wrapper around the same ticket will reset stalls forever and hide churn. If the domain model is still in flux, keep a hard step cap and postpone this particular lab.

Close

The core conclusion stays the same after the exercises: stop the loop when observed state stops changing. Students who rerun the two plans should see STALL_NO_STATE_CHANGE on churn and PLAN_EXHAUSTED on real writes. That pair is enough to review a classmate's agent without reading an entire tool-call transcript by hand. Record the reason code beside the traces so later labs can consume stalls as data rather than folklore.

Workshop: Record Agent Loop Exits as Structured Events in 80 Minutes

Finley Zhu — Sat, 12 Sep 2026 18:55:20 +0000

Agent labs collapse at grading time when the only stop signal lives inside a free-form chat transcript. A structured exit event makes retry policy, tool failure, and completion comparable across students on the same fixture. This workshop records those exits in eighty minutes with a JSONL ledger students can rerun without rereading dialogue. The method is small, local, and deliberately boring, which is the point of a lab that must grade overnight.

Public discussion of agents this week still circles loop control, yet most classroom runners still treat "the model stopped talking" as a finished job. That heuristic hides timeout retries, schema mismatches, and silent max-turn cuts behind the same assistant message. Instructors then compare transcripts instead of comparing reasons, which makes two identical failures look like two different students. The ledger below turns each stop into a typed record before any narrative summary is written.

What this lab actually grades

Students are not asked to invent a better planner. They are asked to make every loop termination observable, replayable, and cheap to assert in CI. The worked example uses a toy weather tool, one retry, and a hard turn cap so the exit taxonomy stays finite. If a runner cannot emit completed, retry_exhausted, schema_mismatch, turn_cap, or tool_error, the lab is incomplete even when the chat looks fluent.

The core conclusion is operational rather than architectural. Loop quality is not the number of tools you registered. It is whether two reruns of the same fixture produce the same exit reason, the same turn count, and the same tool-name sequence.

Timing box (80 minutes)

Use a wall clock. Do not expand the taxonomy mid-lab when a student finds a poetic new failure name.

0–10 min — Failure review. Open two transcripts that both "just stopped" and list what you cannot prove.
10–25 min — Schema lock. Copy the LoopExit record below and refuse extra fields until the tests pass.
25–50 min — Worked runner. Implement run_loop() against the fixture in weather_timeout.json.
50–70 min — Assertions. Add tests for retry exhaustion and turn-cap exits on the same fixture.
70–80 min — Diff drill. Rerun twice, then diff JSONL lines rather than chat Markdown.

Keep phones away from the taxonomy debate. Extra exit reasons are homework, not a live edit to the shared schema.

Artifact: a loop-exit ledger

Treat the ledger as the product of the loop, not as debug residue. Each line is one termination. Partial turns do not get their own files because graders will otherwise sort noise. The schema below is intentionally narrow so a ninety-minute lab cannot invent telemetry religion.

# loop_exit.py
from __future__ import annotations

from dataclasses import asdict, dataclass
from typing import Literal, Optional
import json
from pathlib import Path

ExitReason = Literal[
    "completed",
    "retry_exhausted",
    "schema_mismatch",
    "turn_cap",
    "tool_error",
]

@dataclass(frozen=True)
class LoopExit:
    run_id: str
    fixture_id: str
    reason: ExitReason
    turns: int
    tool_names: tuple[str, ...]
    last_error: Optional[str]
    stopped_on_turn: int

    def to_jsonl(self) -> str:
        payload = asdict(self)
        payload["tool_names"] = list(self.tool_names)
        return json.dumps(payload, sort_keys=True)


def append_exit(path: Path, event: LoopExit) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("a", encoding="utf-8") as handle:
        handle.write(event.to_jsonl() + "\n")

Students should print the JSONL path in the lab log and nowhere else. Console chatter about "the model seemed done" is not an exit record. If last_error is set, reason must not be completed, which is enforced in the tests later rather than in a custom exception hierarchy.

Minimal loop that is allowed to stop

The runner below is labeled as lab pseudocode with a real control flow. It does not claim production scheduling, distributed locks, or model-specific behavior. Tool calls are injected by a fixture so the exit path does not depend on live weather.

# run_loop.py
from __future__ import annotations

from dataclasses import dataclass
from typing import Any
from loop_exit import LoopExit, append_exit
from pathlib import Path

MAX_TURNS = 4
MAX_RETRIES = 1

@dataclass
class ToolResult:
    ok: bool
    name: str
    payload: dict[str, Any] | None
    error: str | None


def run_loop(run_id: str, fixture: dict[str, Any], ledger: Path) -> LoopExit:
    tool_names: list[str] = []
    retries = 0
    last_error = None

    planned_calls: list[dict[str, Any]] = fixture["planned_calls"]
    for turn, call in enumerate(planned_calls, start=1):
        if turn > MAX_TURNS:
            event = LoopExit(
                run_id=run_id,
                fixture_id=fixture["id"],
                reason="turn_cap",
                turns=turn - 1,
                tool_names=tuple(tool_names),
                last_error=last_error,
                stopped_on_turn=turn - 1,
            )
            append_exit(ledger, event)
            return event

        name = call.get("name")
        if not isinstance(name, str) or not name:
            event = LoopExit(
                run_id=run_id,
                fixture_id=fixture["id"],
                reason="schema_mismatch",
                turns=turn,
                tool_names=tuple(tool_names),
                last_error="missing tool name",
                stopped_on_turn=turn,
            )
            append_exit(ledger, event)
            return event

        result = ToolResult(
            ok=bool(call.get("ok")),
            name=name,
            payload=call.get("payload"),
            error=call.get("error"),
        )
        tool_names.append(result.name)

        if result.ok and call.get("final"):
            event = LoopExit(
                run_id=run_id,
                fixture_id=fixture["id"],
                reason="completed",
                turns=turn,
                tool_names=tuple(tool_names),
                last_error=None,
                stopped_on_turn=turn,
            )
            append_exit(ledger, event)
            return event

        if not result.ok:
            last_error = result.error or "tool_error"
            if retries >= MAX_RETRIES:
                event = LoopExit(
                    run_id=run_id,
                    fixture_id=fixture["id"],
                    reason="retry_exhausted",
                    turns=turn,
                    tool_names=tuple(tool_names),
                    last_error=last_error,
                    stopped_on_turn=turn,
                )
                append_exit(ledger, event)
                return event
            retries += 1

    event = LoopExit(
        run_id=run_id,
        fixture_id=fixture["id"],
        reason="tool_error" if last_error else "turn_cap",
        turns=len(planned_calls),
        tool_names=tuple(tool_names),
        last_error=last_error,
        stopped_on_turn=len(planned_calls),
    )
    append_exit(ledger, event)
    return event

The fixture, not the model, decides whether the second weather call succeeds. That split is the teaching move. Students can later swap the fixture for a live tool without changing the exit schema, which keeps grading stable when inference is shared and slow.

Worked example students can rerun

Save the fixture as fixtures/weather_timeout.json. The first call times out. The second call returns a city temperature and marks the loop final. A third planned call exists only to prove the runner never reaches it after completion.

{
  "id": "weather_timeout_v1",
  "planned_calls": [
    {
      "name": "get_weather",
      "ok": false,
      "error": "timeout",
      "final": false
    },
    {
      "name": "get_weather",
      "ok": true,
      "payload": {"city": "Lisbon", "celsius": 22},
      "final": true
    },
    {
      "name": "get_weather",
      "ok": true,
      "payload": {"city": "Lisbon", "celsius": 22},
      "final": true
    }
  ]
}

Commands for a clean rerun on a laptop. No extra services are required for the fixture path.

python - <<'PY'
import json
from pathlib import Path
from run_loop import run_loop

fixture = json.loads(Path("fixtures/weather_timeout.json").read_text())
ledger = Path("artifacts/exits.jsonl")
if ledger.exists():
    ledger.unlink()
event = run_loop("run-001", fixture, ledger)
print(event.reason, event.turns, event.tool_names)
print(ledger.read_text())
PY

Expected line of reasoning for graders, not a live benchmark: reason is completed, turns is 2, and tool_names is ("get_weather", "get_weather"). If a student implementation emits retry_exhausted here, they incremented retries after success or treated timeout as terminal without applying MAX_RETRIES. If they emit three tool names, the loop ignored final and walked the leftover fixture row.

Tests that pin the taxonomy

These tests are the lab contract. They do not measure tokens, latency, or model quality. They measure whether exit reasons stay stable when the fixture is replayed.

# test_loop_exit.py
import json
from pathlib import Path
from run_loop import run_loop

FIX = {
    "id": "weather_timeout_v1",
    "planned_calls": [
        {"name": "get_weather", "ok": False, "error": "timeout", "final": False},
        {
            "name": "get_weather",
            "ok": True,
            "payload": {"city": "Lisbon", "celsius": 22},
            "final": True,
        },
    ],
}


def test_timeout_then_success_completes(tmp_path: Path) -> None:
    ledger = tmp_path / "exits.jsonl"
    event = run_loop("run-001", FIX, ledger)
    assert event.reason == "completed"
    assert event.turns == 2
    assert event.tool_names == ("get_weather", "get_weather")
    lines = ledger.read_text().strip().splitlines()
    assert len(lines) == 1
    body = json.loads(lines[0])
    assert body["reason"] == "completed"
    assert body["last_error"] is None


def test_retry_exhausted_on_double_timeout(tmp_path: Path) -> None:
    fixture = {
        "id": "weather_double_timeout_v1",
        "planned_calls": [
            {"name": "get_weather", "ok": False, "error": "timeout", "final": False},
            {"name": "get_weather", "ok": False, "error": "timeout", "final": False},
        ],
    }
    event = run_loop("run-002", fixture, tmp_path / "exits.jsonl")
    assert event.reason == "retry_exhausted"
    assert event.last_error == "timeout"


def test_missing_name_is_schema_mismatch(tmp_path: Path) -> None:
    fixture = {
        "id": "bad_schema_v1",
        "planned_calls": [{"ok": True, "final": True}],
    }
    event = run_loop("run-003", fixture, tmp_path / "exits.jsonl")
    assert event.reason == "schema_mismatch"

Run the suite with a single command so CI and laptops share the same gate.

pytest -q test_loop_exit.py

Exercise set after the worked example

Assign these in order. Each exercise must produce a new JSONL line, not a longer chat log.

Exercise A — Turn cap. Extend the fixture to five failing calls and assert turn_cap with MAX_TURNS = 4.
Exercise B — Schema mismatch. Drop name from the second call only, then prove the first tool name is still recorded.
Exercise C — Error text hygiene. Replace a stack trace in last_error with a short code such as timeout and reject strings over 80 characters.
Exercise D — Double run. Execute run-001 twice into the same file and assert two JSONL lines with identical reasons and tool-name tuples.

Exercise D is the grading trick. Instructors can hash the reason plus tool-name tuple and ignore run_id. Students who embed wall-clock timestamps inside last_error will fail the hash even when the control flow is correct, which is a useful lab bruise.

Where a shared free inference queue fits

The fixture path needs no network. The next teaching hour often does, because students want the planner to propose the planned_calls list instead of reading it from disk. That is the moment a classroom burns a pile of personal keys and then cannot reproduce Friday's exits on Monday. A shared endpoint keeps the ledger comparable if every student points at the same runner and the same fixture identifier.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that shared runner so the lab does not depend on each student pasting a billed key into a notebook. The exit schema does not change when the planner is remote. Only the producer of planned_calls changes, which is why the tests above stay fixture-first.

Do not treat a shared queue as unlimited capacity or as a promise about any named model. If the class must pause when the queue is busy, freeze planned_calls to disk and keep grading on the ledger. The educational artifact is the exit record, not the freshness of the planner.

Decision table for stop reasons

Use this table on the projector during the last ten minutes. It prevents students from encoding policy in adjectives.

Condition observed	Required `reason`	`last_error`	Typical student mistake
Tool payload accepted and `final` is true	`completed`	`null`	Writing `completed` after a timeout that later recovered
Failures exceed `MAX_RETRIES`	`retry_exhausted`	short code	Counting the successful retry as a failure
Tool object lacks a string `name`	`schema_mismatch`	`missing tool name`	Swallowing the row and calling the next tool
Turn index exceeds `MAX_TURNS`	`turn_cap`	prior error or `null`	Off-by-one when enumerating from zero
Tool returns `ok=false` with no retry left	`retry_exhausted` or `tool_error`	short code	Dumping a full exception into the ledger

If two rows could apply, pick the earlier row in the runner, not the more dramatic row in the transcript. Stability beats expressive failure poetry.

Limitations

This ledger is not a trace backend, not a safety interlock, and not a substitute for tool-result validation before memory writes. JSONL append is not atomic across networked filesystems, so a shared lab directory can interleave lines if two processes use one file without a run-id partition. The taxonomy is incomplete on purpose. It will not encode human escalation, policy refusals, or partial credit for almost-correct tool arguments.

Live planners introduce variance that the fixture path removed. If the remote planner emits a different tool order between reruns, the ledger will correctly show instability, which students may misread as a broken test. Teach that difference explicitly. A changing tool_names tuple is a planner bug or a prompt bug, not a pytest bug.

Who should not use this approach

Skip this lab if you already emit OpenTelemetry span status for every tool call and already grade those spans. Skip it if the agent may log personal data into last_error, because JSONL in a homework folder is not an access-controlled store. Skip it for production stop conditions that must halt dangerous tools in process, since this record is written after the call returns. Skip it for courses that cannot freeze a fixture, because a live-only loop will train students to argue with chat logs again.

Teams shipping multi-tenant agents need request isolation, quotas, and audit storage this workshop does not provide. The eighty-minute box is for instructors who need comparable exits across a class, not for operators who need compliance evidence.

Close the loop on Monday

Rerun the weather fixture, keep the five-reason taxonomy frozen, and grade hashes of reason plus tool_names before anyone opens a transcript. If the next session needs a shared planner instead of a disk fixture, point the runner at one free queue rather than collecting keys, and leave the ledger format untouched. Instructors who want that shared queue without building a billing workshop first can use MonkeyCode's free model access and free server option as the lab endpoint, then keep every assertion on the JSONL file.

Workshop: Score Router Agents Against LLM Tool Choice in 85 Minutes

Finley Zhu — Fri, 11 Sep 2026 16:42:43 +0000

Classroom agent demos often look autonomous while a hidden branch table still selects every tool. This workshop makes that split measurable with two implementations, one fixture pack, and a turn ledger students can rerun. The core conclusion is simple and testable: keep a deterministic router until phrasing variance actually beats a rule table on the same traces. Free inference capacity is useful here only as a lab endpoint, not as evidence that a chat loop is an agent.

The outline below is a teaching plan, not a production framework and not a personal benchmark report. Every numeric target is a classroom stopwatch budget, not a published SLA. Code samples are labeled workshop sketches that students should run locally against fixtures they control.

Why this comparison belongs in the lab

Recent community debate keeps collapsing two designs into one word: agent. A deterministic router maps verbs and entities onto a closed tool list, then stops. An LLM chooser proposes a tool name from natural language, then the same tools still execute. If those paths are not scored on identical fixtures, a class cannot tell whether extra model calls buy recall, noise, or only longer traces.

Shared inference makes the gap more expensive than it looks on a laptop demo. Each extra turn consumes tokens, waits on a queue, and multiplies retry policy. Students should see that cost on a ledger before they add a third tool or a memory write. The comparison also resists a common demo failure: a scripted if-chain dressed as tool-calling.

Workshop clock

Use a single 85-minute block with a visible timer and no open-ended homework during class.

0–10 min — Frame the claim. Define router, chooser, fixture, and illegal side effect.
10–25 min — Exercise 1. Freeze two tools and a ten-prompt fixture pack.
25–45 min — Exercise 2. Implement and test the deterministic router.
45–70 min — Exercise 3. Wrap an LLM chooser with schema checks and a turn ledger.
70–85 min — Exercise 4. Dual-run the fixtures, fill the decision table, debrief misses.

If the room is slower, drop the chooser retries and keep one model call per prompt. Do not expand the tool set during the same sitting. Extra tools hide scoring errors behind incomplete coverage.

Prerequisites

Students need Python 3.11+, pytest, and permission to make outbound HTTP calls from the lab network. They also need a JSONL fixture file they can commit, plus a dry-run flag on every write tool. No paid cloud account is required if the chooser step can target a free lab endpoint the instructor already configured.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can stand in as that shared lab endpoint so the chooser path does not depend on a personal API invoice. Do not treat that availability as a quota, hardware spec, latency claim, or permanence guarantee; pin whatever URL the instructor actually issued.

Exercise 1 — Freeze two tools and one fixture pack

Give every student the same closed world. One read tool returns inventory rows. One write tool opens a ticket only when dry_run is false. Ten prompts must include clear verbs, paraphrases, mixed intents, and two prompts that should refuse both tools.

Suggested fixture fields, committed as fixtures/tool_choice.jsonl:

id — stable string, never reused after an edit.
prompt — the user text, including typos the router must survive or fail.
allowed_tools — a list that may be empty when the correct action is refusal.
write_expected — boolean; true only when a ticket would be legitimate.

Label this rule on the board: write tools never execute during scoring. Students who skip that rule are not measuring tool choice; they are creating tickets. Keep the inventory table tiny, for example four SKUs, so expected read arguments stay obvious in review.

Exercise 2 — Ship a deterministic router

The router may use keyword maps, regexes, or a tiny intent table. It may not call a model. It must return a structured decision the tests can diff.

# workshop sketch — deterministic router, not a production NLU stack
from dataclasses import dataclass
import re
from typing import Literal

ToolName = Literal["read_inventory", "open_ticket", "refuse"]

@dataclass(frozen=True)
class Decision:
    tool: ToolName
    args: dict
    reason: str

SKU = re.compile(r"\b(SKU-\d{3})\b", re.I)
TICKET = re.compile(r"\b(open|file|create)\s+(a\s+)?ticket\b", re.I)
READ = re.compile(r"\b(stock|inventory|qty|quantity|on hand)\b", re.I)

def route(prompt: str) -> Decision:
    sku_match = SKU.search(prompt or "")
    sku = sku_match.group(1).upper() if sku_match else None
    wants_ticket = bool(TICKET.search(prompt or ""))
    wants_read = bool(READ.search(prompt or ""))
    if wants_ticket and wants_read:
        return Decision("refuse", {}, "mixed_intent")
    if wants_ticket:
        return Decision("open_ticket", {"sku": sku, "dry_run": True}, "ticket_verb")
    if wants_read and sku:
        return Decision("read_inventory", {"sku": sku}, "inventory_verb")
    return Decision("refuse", {}, "no_closed_tool")

Tests should lock three properties before anyone wires HTTP. First, mixed intent refuses instead of guessing. Second, ticket decisions keep dry_run true. Third, unknown SKUs still refuse rather than inventing a catalog row. Run the suite with a single command students can paste:

python -m pytest tests/test_router.py -q

If the router accuracy on the ten fixtures is already 9/10 or 10/10, say that out loud. That number is the baseline the chooser must beat, not a vibe.

Exercise 3 — Add an LLM chooser with a turn ledger

The chooser sends the prompt plus the tool catalog and expects a JSON object with tool, args, and reason. Reject any response that is not valid JSON, names an unknown tool, or omits dry_run on writes. Count every HTTP attempt as a turn, including parse failures and instructor-approved retries.

# workshop sketch — unexecuted against your lab URL until env is set
import json
import os
import time
import urllib.request

LAB_URL = os.environ.get("LAB_INFERENCE_URL", "")

SYSTEM = """Return JSON only with keys tool, args, reason.
Tools: read_inventory, open_ticket, refuse.
Writes must include dry_run true. Unknown work must refuse.
"""

def estimate_tokens(text: str) -> int:
    # labeled heuristic: four characters per token, not a vendor tokenizer
    return max(1, len(text) // 4)

def choose(prompt: str, timeout_s: float = 20.0) -> tuple[dict, dict]:
    if not LAB_URL:
        raise RuntimeError("LAB_INFERENCE_URL is unset; skip chooser or set the lab endpoint")
    payload = json.dumps({
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ]
    }).encode()
    req = urllib.request.Request(
        LAB_URL, data=payload, headers={"Content-Type": "application/json"}
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout_s) as resp:
        body = resp.read().decode()
    elapsed_ms = int((time.monotonic() - started) * 1000)
    parsed = json.loads(body)
    content = parsed["choices"][0]["message"]["content"]
    decision = json.loads(content)
    ledger = {
        "turns": 1,
        "elapsed_ms": elapsed_ms,
        "est_prompt_tokens": estimate_tokens(SYSTEM + prompt),
        "est_completion_tokens": estimate_tokens(content),
        "parse_ok": True,
    }
    return decision, ledger

Append one JSONL row per fixture to traces/chooser.jsonl. Required columns are id, tool, parse_ok, turns, elapsed_ms, and estimated tokens. Students who cannot reach the lab URL should still complete the router path and mark chooser rows as skipped, not as zeros. Zeros quietly win fake cost contests.

A cheap validator belongs in the same exercise, not as homework:

WRITE_TOOLS = {"open_ticket"}
KNOWN = {"read_inventory", "open_ticket", "refuse"}

def validate_decision(decision: dict) -> list[str]:
    errors = []
    tool = decision.get("tool")
    args = decision.get("args") if isinstance(decision.get("args"), dict) else None
    if tool not in KNOWN:
        errors.append("unknown_tool")
    if args is None:
        errors.append("args_not_object")
    if tool in WRITE_TOOLS and not (args or {}).get("dry_run") is True:
        errors.append("write_without_dry_run")
    return errors

Exercise 4 — Dual-run and score

Score both systems on the same ten ids. Do not grade prose quality. Grade tool identity first, then argument equality, then ledger totals.

id | gold_tool | router_tool | chooser_tool | router_ok | chooser_ok | chooser_turns | est_tokens

Publish three classroom numbers on the whiteboard before discussion starts:

Router exact-match rate on gold tools.
Chooser exact-match rate after schema validation, excluding skipped rows.
Chooser extra turns versus the router, which is always one local call and zero model tokens.

If the chooser ties the router but spends several thousand estimated tokens, the class should keep the router for this catalog. If the chooser recovers paraphrases the regex missed, keep the chooser only for those intents and route the rest. Mixing both without a recorded rule recreates the trench-coat agent the fixtures were meant to expose.

Decision table students fill in class

Observation on the ten fixtures	Keep the router	Promote the chooser	Do neither
Exact-match within one of gold, paraphrases still fail	Yes	No	—
Chooser recovers two or more paraphrases router missed	No	Yes, for those intents only	—
Chooser invents tools or drops `dry_run`	Yes	No	Add schema tests first
Lab endpoint timeouts exceed 20 percent of rows	Yes	No	Shrink prompts, forbid retries
Mixed-intent prompts trigger a write	—	—	Stop; the write gate is broken
Gold set itself disagrees after a fixture edit	—	—	Freeze ids and rescore

The table is the artifact that leaves the room. Code without the table usually collapses back into a single chat loop by the next lab.

Limitations and who should skip this

This plan assumes a closed tool list, synthetic inventory, and write tools that honor dry-run. It does not measure multi-hop planning, retrieval quality, or production safety. Character-chunk token estimates will disagree with vendor tokenizers; treat them as ordering, not billing. HTTP envelopes differ across labs, so instructors must adapt the chooser parser rather than copy the sketch as a client library.

Skip this workshop when students would hit live customer systems, when the network cannot inspect responses, or when the catalog is unbounded. Also skip it when the teaching goal is prompt style rather than tool identity. A free shared endpoint is the wrong backplane for secrets, for graded exams that need identical latency, and for overnight batch jobs that assume capacity will still be there tomorrow.

What to keep after class

Commit the fixture pack, the router tests, the chooser validator, and the filled decision table. Delete any trace file that captured raw prompts mixed with student names. If the chooser never beat the router on exact match, leave the model call out of the default demo path and document that result as a passing outcome. The useful habit is not “add an agent.” The useful habit is scoring both control flows before the next tool lands.

Instructors who want the chooser step on a no-invoice lab host can point LAB_INFERENCE_URL at the free server option already described, then rerun the same ten ids without changing fixtures. That is the only product substitution this outline needs; the scoring rules stay local.

Workshop: Stamp Agent Assumptions With Source Tags in 65 Minutes

Finley Zhu — Wed, 09 Sep 2026 16:42:03 +0000

Agents fail less from missing tools than from silent guesses that later look like retrieved facts. This workshop stamps every pre-tool claim with a source tag, then blocks calls when required fields remain assumed. Students leave with a rerunnable ledger, a pytest gate, and a timing plan that fits a single lab block. The tests stay deterministic even when a later extraction pass uses free inference.

The failure mode this lab isolates

Tool-calling agents often invent hostnames, ticket identifiers, file paths, and time ranges because the prompt asked them to be helpful. Those invented values then enter JSON arguments, after which logs treat them as ordinary parameters. Downstream tools succeed against the wrong object, or they fail with an error that never names the invented field. The lab treats that gap as a contract problem, not as a model-quality complaint.

A source tag is a small enum attached to each claim the agent wants to use. Allowed values in this lab are observed, retrieved, user, derived, and assumed. Tool schemas then declare which argument names must not arrive with assumed. That rule is boring on purpose, because boring rules survive a noisy classroom and a later code review.

Trend pieces about agents that “just assume things” are useful as a symptom list, not as a design spec. This workshop does not classify agent architectures or catalog vendor terms. It only asks whether a required argument still carries an assumed tag when the process is about to execute a side effect.

Lab goals and timing

By minute 65, each pair should be able to rerun the fixture and explain a rejected call in one sentence. The session is split so instructors can cut the optional extraction pass if the room has no network.

0–8 min — Frame the bug. Show one transcript where a missing ticket id becomes a plausible id, then a write tool runs.
8–18 min — Ledger schema. Students copy the typed ledger below and label three claims by hand.
18–38 min — Gate and tests. They implement assert_tool_args_sourced and run the pytest file twice.
38–52 min — Exercise A and B. One broken fixture, one derived-field fixture, both checked into the lab repo.
52–65 min — Debrief. Compare which fields were allowed to stay assumed, and name two fields that must never be.

If the room is large, keep the debrief on a whiteboard table with four columns: field, tag, source pointer, allowed for writes. Do not spend the remaining minutes adding another tool. Extra tools hide the contract instead of proving it.

Prerequisites

Students need Python 3.11+, pytest, and a text editor. No GPU is required for the core path, because the ledger is ordinary JSON. Optional extraction later can call a remote model, but the grade depends on the tests, not on the model.

Clone or copy the files into an empty directory named assumption-lab. The commands below assume a Unix shell. Windows students can use the same pytest invocation inside the project folder.

python -m venv .venv
source .venv/bin/activate
pip install pytest
mkdir -p lab tests fixtures

Worked example: a ledger the tests can own

The ledger is the original artifact for this session. It is not a prompt. It is a document the gate reads before any tool process starts. Students should treat a missing source_pointer on retrieved or observed tags as a test failure, because a tag without a pointer is another form of guessing.

{
  "run_id": "lab-65-ticket-reply",
  "claims": [
    {
      "name": "ticket_id",
      "value": "TCK-10419",
      "tag": "user",
      "source_pointer": "user_message:0",
      "required_for": ["ticket.update"]
    },
    {
      "name": "assignee_email",
      "value": "alex@example.com",
      "tag": "assumed",
      "source_pointer": null,
      "required_for": ["ticket.update"]
    },
    {
      "name": "last_status",
      "value": "waiting_on_customer",
      "tag": "retrieved",
      "source_pointer": "tools.ticket.get:result.status",
      "required_for": ["ticket.update"]
    }
  ],
  "pending_tool": {
    "name": "ticket.update",
    "args": {
      "ticket_id": "TCK-10419",
      "assignee_email": "alex@example.com",
      "last_status": "waiting_on_customer"
    }
  }
}

Save that document as fixtures/ticket_update_assumed_assignee.json. The second claim is the intended failure. The first and third claims show tags the gate should accept for a write.

Gate module

The gate is deliberately small. It does not call models, networks, or ticket APIs. It only joins pending tool arguments to claims by name, then applies the tag policy. Label this as a teaching implementation, not as production middleware.

# lab/assumption_gate.py
from __future__ import annotations

from typing import Any

WRITE_TOOLS = {"ticket.update", "ticket.close", "fs.write", "shell.run"}
FORBIDDEN_TAGS_FOR_WRITES = {"assumed"}
POINTER_REQUIRED = {"observed", "retrieved", "user", "derived"}


class AssumptionGateError(ValueError):
    pass


def _claim_map(ledger: dict[str, Any]) -> dict[str, dict[str, Any]]:
    claims = ledger.get("claims") or []
    out: dict[str, dict[str, Any]] = {}
    for claim in claims:
        name = claim.get("name")
        if not name:
            raise AssumptionGateError("claim missing name")
        if name in out:
            raise AssumptionGateError(f"duplicate claim name: {name}")
        out[name] = claim
    return out


def assert_tool_args_sourced(ledger: dict[str, Any]) -> None:
    pending = ledger.get("pending_tool") or {}
    tool_name = pending.get("name")
    args = pending.get("args") or {}
    if not tool_name:
        raise AssumptionGateError("pending_tool.name is required")

    claims = _claim_map(ledger)
    for arg_name, arg_value in args.items():
        claim = claims.get(arg_name)
        if claim is None:
            raise AssumptionGateError(
                f"{tool_name} arg {arg_name!r} has no claim in the ledger"
            )
        if claim.get("value") != arg_value:
            raise AssumptionGateError(
                f"{arg_name} value does not match ledger claim"
            )
        tag = claim.get("tag")
        pointer = claim.get("source_pointer")
        if tag in POINTER_REQUIRED and not pointer:
            raise AssumptionGateError(
                f"{arg_name} tag {tag!r} requires source_pointer"
            )
        if tool_name in WRITE_TOOLS and tag in FORBIDDEN_TAGS_FOR_WRITES:
            raise AssumptionGateError(
                f"refusing {tool_name}: {arg_name} is tagged {tag!r}"
            )
        required_for = set(claim.get("required_for") or [])
        if required_for and tool_name not in required_for:
            raise AssumptionGateError(
                f"{arg_name} is not listed for {tool_name}"
            )

Tests students rerun

The first test encodes the worked example. The second test shows a legal derived field, which is the usual student question after the first failure. Derived values are allowed when the pointer names the inputs, not when the model shrugs.

# tests/test_assumption_gate.py
import json
from pathlib import Path

import pytest

from lab.assumption_gate import AssumptionGateError, assert_tool_args_sourced

FIXTURES = Path(__file__).resolve().parents[1] / "fixtures"


def load(name: str) -> dict:
    return json.loads((FIXTURES / name).read_text(encoding="utf-8"))


def test_write_rejected_when_required_arg_is_assumed() -> None:
    ledger = load("ticket_update_assumed_assignee.json")
    with pytest.raises(AssumptionGateError, match="assignee_email"):
        assert_tool_args_sourced(ledger)


def test_derived_sla_bucket_is_allowed_with_pointer() -> None:
    ledger = {
        "claims": [
            {
                "name": "ticket_id",
                "value": "TCK-10419",
                "tag": "user",
                "source_pointer": "user_message:0",
                "required_for": ["ticket.update"],
            },
            {
                "name": "sla_bucket",
                "value": "t24h",
                "tag": "derived",
                "source_pointer": "derive.sla_bucket(last_status, opened_at)",
                "required_for": ["ticket.update"],
            },
        ],
        "pending_tool": {
            "name": "ticket.update",
            "args": {"ticket_id": "TCK-10419", "sla_bucket": "t24h"},
        },
    }
    assert_tool_args_sourced(ledger)  # no exception

Rerun from the project root. The first command should fail until students understand the fixture, then pass after they change assignee_email or drop it from the write. Keep both outcomes in git so later groups can diff them.

export PYTHONPATH=.
pytest -q tests/test_assumption_gate.py

Expected teaching sequence: first run fails on assignee_email, students edit the fixture, second run passes, then they add a third test that omits source_pointer on a retrieved field. That third test is Exercise A.

Exercise A — missing pointers are guesses

Duplicate the retrieved last_status claim, set source_pointer to null, and keep the tag as retrieved. The gate must raise. If a pair makes the test pass by weakening POINTER_REQUIRED, stop them and restore the set. The point is not coverage count. The point is that a tag without a pointer cannot be audited after class.

Suggested assertion text: last_status tag 'retrieved' requires source_pointer. Students may write a more specific matcher. Instructors should reject matchers that only look for Error, because those pass for the wrong reason.

Exercise B — read tools may keep unknowns, writes may not

Add a second pending tool named ticket.get with argument ticket_id tagged user. Reads can proceed with unknowns in other fields, as long as those fields are not in args. Writes cannot. Have students encode that distinction as a table, then as one extra test, not as a paragraph in the prompt.

Field	Tag	In write args?	Gate result
ticket_id	user	yes	allow
last_status	retrieved	yes	allow if pointer present
assignee_email	assumed	yes	reject
customer_region	assumed	no	allow, remains an open question
sla_bucket	derived	yes	allow if pointer names inputs

The table is the decision artifact. If a pair cannot fill the last column without looking at the code, the schema is still too vague for the room.

Optional extraction pass on free inference

The core lab does not need a model. A later station can feed a messy transcript into a constrained extractor that only emits claims, never tool calls. That split keeps pytest stable while still giving students a realistic upstream.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that extractor when the classroom has no paid inference budget. The product is optional here: any endpoint that returns JSON can fill the same role, and the gate still refuses assumed write args.

Keep the extractor prompt tiny. Ask only for claims with name, value, tag, and source_pointer. Discard any extra keys. If the model omits a pointer, store tag as assumed rather than inventing retrieved. Students should commit the extractor output as a fixture, then run the same tests they already have.

# lab/extract_claims.py  — teaching sketch, not a vendor SDK
EXTRACTOR_INSTRUCTIONS = """
Return JSON with key claims. Each claim needs name, value, tag,
and source_pointer. Tags: observed, retrieved, user, derived, assumed.
If the transcript does not ground a value, tag it assumed and set
source_pointer to null. Do not emit tool calls.
"""

Do not grade model fluency. Grade whether the written fixture still fails when assignee_email is assumed. If the extractor marks a user-supplied ticket id as assumed, that is a prompt bug, and Exercise A already taught the fix: correct the tag and pointer, then rerun pytest.

Limitations and who should skip this lab

The gate does not prove that retrieved values are true. It only proves that the run recorded a pointer. Poisoned retrieval, stale tickets, and lying tools remain out of scope. The enum also collapses several real-world source types, including human operators and batch jobs, into user or observed.

Skip this design when the agent cannot name arguments, when tools accept free-form blobs, or when a human already confirms every write. Also skip it as a production authorization layer. Source tags are an audit aid sitting in front of existing authz, not a replacement for it. Teams that need cryptographic provenance want signed tool results, which this 65-minute lab does not build.

Classroom risk is over-fitting the ticket example. If every fixture uses ticket.update, students will treat the policy as product-specific. Add one fs.write fixture before the debrief so the same tag rules apply to paths. If time is short, discuss that fixture instead of coding it.

What to keep after the session

Keep the ledger schema, the write-tool set, and the pytest file. Delete the extractor if it started to dominate discussion. The durable lesson is that assumptions need a field, a tag, and a pointer before they can approach a side effect. Models can propose those fields. Tests still decide whether the process may continue.

Pairs that finish early can add a JSON Schema file for the ledger and validate fixtures in CI. That extra file is optional homework, not a requirement for the 65-minute block. The minimum passing artifact is one rejected write, one accepted derived field, and a table the next lab group can read without the instructor.

Workshop: Cap Agent Tool Fan-Out on Shared Inference in 90 Minutes

Finley Zhu — Mon, 07 Sep 2026 15:06:17 +0000

Unbounded tool fan-out, not weak prompting, is what usually knocks a shared classroom inference endpoint offline. A ninety-minute lab makes that failure visible, then contains it with a permit file, a call budget, and a replay harness. The method stays useful if the remote model changes, because admission lives on the student machine. Instructors should grade the admission log rather than the fluency of whatever the planner happened to emit.

This outline is a teaching proposal, not a production runbook, and it assumes a toy agent with two or three tools. Times below fit a cohort that already serializes tool calls as JSON and can run pytest locally. Nothing in the lab requires a GPU on the laptop or a paid inference contract. The live model stays optional until the stub exercise, and even then it remains behind the budget.

Who this lab is for

Use this workshop when students share one inference endpoint and their agents may call tools in a loop. The usual damage pattern is a recursive self-repair step that fans out search, HTTP, and retry tools until the shared queue stalls. Instructors get a fair-share rule they can mark in minutes. Students get a deterministic fixture instead of an outage that depends on whoever ran first.

Skip the session if agents never call tools, or if every laptop already talks to an isolated endpoint with hard vendor caps. Also skip it when the assignment executes arbitrary student code on the same host as the model process. Those cases need isolation and quotas at the runtime, not a JSON permit beside the client.

Timing board (90 minutes)

0:00–0:12 — Reproduce a fan-out storm from a fixture, not from a live completion.
0:12–0:28 — Read the permit schema and fail closed when required keys are missing.
0:28–0:50 — Implement ToolBudget.admit() and unit-test three distinct denial reasons.
0:50–1:10 — Wire the budget in front of a stub planner and a stub tool runner.
1:10–1:25 — Replay the same transcript twice and diff the decision lists.
1:25–1:30 — Debrief what the budget cannot see, including raw HTTP bypasses.

Materials. Python 3.11 or newer, pytest, one JSON permit, one JSON trace, and any OpenAI-compatible chat client. Students may point that client at a shared free server if the instructor supplies a base URL. No laptop GPU is required, and wall-clock retries should stay disabled during the stub segment.

The failure you will reproduce

A naive agent treats every tool error as a reason to attempt more tools in the same turn. One failed search becomes three retries, each retry plans extra http_get calls, and a cohort of twenty laptops multiplies that tree together. Model quality is irrelevant once the endpoint queue is full of duplicate planning prompts. Starting from a recorded plan keeps the outage reproducible on a quiet machine.

The lab therefore injects a fixture that already contains an unbounded repair loop. Students prove the budget would have stopped it before anyone touches the shared endpoint. The fixture below is synthetic teaching data, not a capture from a production agent, and it should be checked into the lab repository beside the permit.

{
  "trace_id": "lab-fanout-01",
  "steps": [
    {"tool": "search", "args": {"q": "retry"}},
    {"tool": "search", "args": {"q": "retry site:example"}},
    {"tool": "http_get", "args": {"url": "https://example.invalid/a"}},
    {"tool": "http_get", "args": {"url": "https://example.invalid/b"}},
    {"tool": "http_get", "args": {"url": "https://example.invalid/c"}}
  ]
}

Count the planned calls on paper before writing code: two searches and three fetches. Under the permit in the next section, the third fetch must deny. If a student implementation admits all five steps, the later stub exercise will still be able to stampede the shared endpoint.

Permit file: fail closed

The permit is a small JSON document in the lab repo, not a comment in a prompt. Missing keys deny the call. Unknown tools deny the call. Depth and parallelism are first-class fields because retries hide inside both of those axes. Host allowlists belong here as well, because a shared planner may still emit absolute URLs outside the assignment.

{
  "max_total_calls": 6,
  "max_depth": 2,
  "max_parallel": 1,
  "tools": {
    "search": {"max_calls": 2, "max_args_bytes": 200},
    "http_get": {"max_calls": 2, "allowed_hosts": ["example.edu"]},
    "shell": {"max_calls": 0}
  }
}

shell is listed with max_calls set to zero so a hallucinated shell tool fails closed instead of falling through. That zero is a policy bit, not a seccomp jail, and the debrief should say so out loud. Students who delete the shell entry should watch the unknown-tool path deny the same call, which is the second fail-closed check.

Worked example students can rerun

The skeleton below is meant to be typed, tested, and extended during the lab. It is not a framework, and several branches are deliberately harsh. Parallelism other than one call is denied so the course does not pretend to schedule concurrent tool I/O. Label any extra fields you add as local extensions, then keep the four required permit keys stable so fixtures stay replayable.

from __future__ import annotations

import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from urllib.parse import urlparse


class BudgetDenied(Exception):
    """Raised when a tool call would exceed the local permit."""


@dataclass
class ToolBudget:
    permit: dict[str, Any]
    total_calls: int = 0
    depth: int = 0
    per_tool: dict[str, int] = field(default_factory=dict)
    decisions: list[dict[str, Any]] = field(default_factory=list)

    @classmethod
    def from_path(cls, path: Path) -> "ToolBudget":
        permit = json.loads(path.read_text())
        required = {"max_total_calls", "max_depth", "max_parallel", "tools"}
        missing = required - set(permit)
        if missing:
            raise BudgetDenied(f"permit missing keys: {sorted(missing)}")
        return cls(permit=permit)

    def admit(self, tool: str, args: dict[str, Any], *, depth: int) -> None:
        tools = self.permit["tools"]
        if tool not in tools:
            self._deny(tool, "unknown_tool")
        spec = tools[tool]
        used = self.per_tool.get(tool, 0)
        if used >= spec["max_calls"]:
            self._deny(tool, "per_tool_cap")
        if self.total_calls >= self.permit["max_total_calls"]:
            self._deny(tool, "total_cap")
        if depth > self.permit["max_depth"]:
            self._deny(tool, "depth_cap")
        if self.permit["max_parallel"] != 1:
            self._deny(tool, "parallel_not_implemented")
        encoded = json.dumps(args, sort_keys=True)
        max_bytes = spec.get("max_args_bytes", 4096)
        if len(encoded.encode("utf-8")) > max_bytes:
            self._deny(tool, "args_too_large")
        if tool == "http_get":
            host = urlparse(str(args.get("url", ""))).hostname or ""
            allowed = spec.get("allowed_hosts", [])
            if host not in allowed:
                self._deny(tool, "host_not_allowed")
        self.total_calls += 1
        self.per_tool[tool] = used + 1
        self.depth = max(self.depth, depth)
        self.decisions.append({"tool": tool, "ok": True, "reason": "admit"})

    def _deny(self, tool: str, reason: str) -> None:
        self.decisions.append({"tool": tool, "ok": False, "reason": reason})
        raise BudgetDenied(reason)


def replay(trace_path: Path, permit_path: Path) -> list[dict[str, Any]]:
    budget = ToolBudget.from_path(permit_path)
    trace = json.loads(trace_path.read_text())
    for depth, step in enumerate(trace["steps"], start=1):
        try:
            budget.admit(step["tool"], step.get("args", {}), depth=depth)
        except BudgetDenied:
            break
    return budget.decisions

Students replay lab-fanout-01 against the permit and stop at the first denial. The intended teaching result is a denial on the third http_get with reason per_tool_cap, after search has already used its cap of two. Host denial may fire first if a student leaves example.invalid in the fixture while the allowlist only contains example.edu; that ordering is worth a two-minute discussion, not a silent patch.

# test_budget.py
from pathlib import Path

from budget import BudgetDenied, ToolBudget, replay


def test_missing_depth_fails_closed(tmp_path: Path) -> None:
    payload = {
        "max_total_calls": 6,
        "max_parallel": 1,
        "tools": {"search": {"max_calls": 1}},
    }
    path = tmp_path / "permit.json"
    path.write_text(json_dumps(payload))
    try:
        ToolBudget.from_path(path)
        raise AssertionError("missing max_depth must deny")
    except BudgetDenied as exc:
        assert "max_depth" in str(exc)


def test_fixture_stops_on_http_or_host_cap() -> None:
    decisions = replay(Path("fixtures/lab-fanout-01.json"), Path("permit.json"))
    denied = [row for row in decisions if not row["ok"]]
    assert denied, "unbounded fixture must produce at least one denial"
    assert denied[0]["reason"] in {"per_tool_cap", "total_cap", "host_not_allowed"}


def json_dumps(payload: dict) -> str:
    import json

    return json.dumps(payload)

Commands the cohort should run without hitting a remote model:

python -m pytest test_budget.py -q
python -c "from pathlib import Path; from budget import replay; print(replay(Path('fixtures/lab-fanout-01.json'), Path('permit.json')))"

Exercise sequence

Reproduce (12 minutes). Import the fixture and do not call a live model. Confirm five planned tool calls exist. Write a pytest that still fails while admit() returns unconditionally, so the later caps have a red bar to turn green.
Fail closed (16 minutes). Load the permit and assert that missing max_depth raises BudgetDenied. Assert shell with max_calls of zero never admits. Assert an unknown tool name is denied even when total capacity remains.
Cap fan-out (22 minutes). Implement per-tool and total caps, then replay lab-fanout-01. Record the first denial reason in a markdown table on the lab sheet, including whether host policy fired before count policy.
Stub the planner (20 minutes). Wrap a chat client so each model-proposed tool call hits admit() before any I/O. If the instructor points students at a shared endpoint, keep the client timeout short and the retry count at zero. The budget, not the HTTP adapter, is the backpressure mechanism.
Replay twice (15 minutes). Run the same fixture through the stub twice and diff the decisions lists. They must match byte-for-byte on tool, ok, and reason. A mismatch usually means a wall-clock retry is still hiding in the client.

Proposed stub shape, unlabeled as production code:

import json
import os
import urllib.request

LAB_BASE_URL = os.environ.get("LAB_BASE_URL", "http://127.0.0.1:8080/v1")
LAB_TIMEOUT_SEC = float(os.environ.get("LAB_TIMEOUT_SEC", "20"))


def plan_once(messages: list[dict], budget) -> dict:
    req = urllib.request.Request(
        f"{LAB_BASE_URL}/chat/completions",
        data=json.dumps({
            "messages": messages,
            "tools": [{"type": "function", "function": {"name": "search"}}],
        }).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=LAB_TIMEOUT_SEC) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    tool = payload["choices"][0]["message"].get("tool_call", {})
    if tool:
        budget.admit(tool["name"], json.loads(tool.get("arguments") or "{}"), depth=1)
    return payload

Set LAB_BASE_URL only after local replay is green. Leave retries at zero so a slow shared server cannot multiply fan-out through the HTTP layer. If the stub cannot parse a tool call, deny by default rather than executing a guessed name.

Instructor decision table

Symptom in the room	First check	Permit field to tighten
Endpoint returns 429 or queue timeouts	Concurrent student jobs and retry settings	`max_total_calls`
Duplicate web fetches in traces	Repeated `http_get` argument objects	`tools.http_get.max_calls`
Recursive “try again” plans	Depth of tool-calling messages	`max_depth`
Surprise shell execution	Tool name allowlist	`tools.shell.max_calls = 0`
Prompt injection through huge arguments	Byte length of serialized args	`max_args_bytes`
Hosts outside the assignment	URL hostname after `urlparse`	`allowed_hosts`

Where a free shared server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The budget object does not require a particular vendor, and it only requires that tool calls pass through admit() before I/O. If the course needs a shared chat endpoint without each student provisioning hardware, MonkeyCode's free model access and free server option can sit behind that same client stub. Treat the remote model as an untrusted planner. Treat the permit file as the trusted policy, and grade the decision log rather than the prose.

Do not send secrets in tool arguments or prompts to any shared endpoint, including a classroom server. Redact tokens before writing decisions to disk. The permit is not an isolation boundary for untrusted code execution, and a free shared planner does not change that rule. If you want that shared endpoint for the stub segment, point LAB_BASE_URL at the server the course already approved, then collect the same fixtures you collected on localhost.

Limitations

This lab does not measure model quality, latency, or token cost, and it should not be cited as a benchmark. It does not implement true parallelism; max_parallel other than 1 is denied on purpose so students cannot hide fan-out in threads. It does not sandbox shell, so a zero cap is policy, not a jail. It cannot stop a student from bypassing admit() with a raw HTTP client, which is why the debrief asks for the decision file rather than a screenshot of model text.

The approach also assumes JSON tool calls that are complete before admission. Streaming function-call deltas need a join step this skeleton omits. Multi-agent graphs need one budget per agent plus a course-level job queue, which is a different workshop. A proposed extension, not implemented here, is a fair-share token on the server that rejects a student id after N admitted calls per hour; without that, a determined client can still starve peers.

Who should not use this approach

Teams that already enforce tool policy inside a managed agent runtime with audited caps.
Assignments that run arbitrary student code on the same host as the model process.
Production incident response, where you need distributed tracing rather than a JSON permit.
Courses that cannot fail closed, because a denied tool call would strand a graded live demo.

If those constraints apply, teach vendor quotas and request tracing instead of this local budget. The permit pattern is for shared teaching endpoints where the cheapest control plane is a file students can diff.

What to collect before dismissing the cohort

Ask each pair for three artifacts: the permit they used, the decisions list from lab-fanout-01, and a one-line statement of the first denial reason. Grade those artifacts, not the fluency of the planner output, and keep the live endpoint off until replay is stable. Shared inference survives a tool-using agent lab only when fan-out is admitted locally, with fixtures students can rerun without touching the server.

Workshop: Dual-Run Agent Fixtures Until Tool-Call JSON Stays Stable in 90 Minutes

Finley Zhu — Mon, 07 Sep 2026 14:05:57 +0000

Silent tool-argument drift is a more common agent failure than a hard crash, because the run still returns a fluent answer. This workshop treats that drift as a testable contract rather than a prompt-writing exercise. Two independent runs of the same fixture must emit JSON that matches a frozen schema and a frozen set of assumption keys. Students leave with a rerunnable checker, a timed exercise sequence, and a stop rule for labs that should not dual-run at all.

Recent developer threads often dwell on agent vocabulary and on agents that invent missing facts. Those threads are useful as topic signals, not as a glossary to reproduce in class. The failure that actually burns a lab hour is narrower and measurable: a tool call that adds a field, drops a required id, or fills a default the user never supplied. Fluent prose after that call is not evidence of stability.

Who should take this workshop

This session is for people who already have a small agent loop with tools, fixtures, and logs. It is not an introduction to agents, tool protocols, or prompt style. Instructors can run it as one 90-minute studio or as three 30-minute blocks on consecutive days.

Bring the following materials to the session, and do not start Lab 0 until every student has them:

One frozen user task in a JSON file, never in an unsaved chat window
One tool schema with required fields and explicit additionalProperties: false
Two runnable executions that accept the same request body
Python 3.11+ with pytest and jsonschema installed in a clean virtualenv

If a student only has one network endpoint, they can still complete Lab 0 and Lab 1 on captured traces. Dual-run comparison in Lab 2 needs two isolated executions, which may be two local processes, two sequential calls after a memory reset, or two hosted endpoints with identical request bodies.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Students who need a no-key host for the second execution can park the fixture runner on MonkeyCode's free model access and free server option, then keep every assertion on their own machine so grading never depends on a vendor UI.

Timing board

Block	Minutes	Outcome students must show
Briefing	10	Shared definition of drift versus crash
Lab 0	15	Frozen tool-argument schema that rejects one extra field
Lab 1	20	Assumption-key extractor with a short allow list
Lab 2	30	Dual-run JSON fingerprint that fails on a field change
Lab 3	15	Fail-closed `pytest` log from a deliberately broken fixture
Debrief	10	Decision table filled with one real trace from class

Instructors should refuse to extend the clock for extra prompt tuning. The graded artifact is the checker and the failing test, not a prettier final answer.

The contract students will enforce

Define three artifacts before any model call leaves the laptop:

task.json — the user request, known ids, and hard budgets
tool_args.schema.json — allowed arguments for every tool the fixture may call
assumptions.allowed.txt — keys the agent may infer, one per line, eight or fewer

A run counts as stable only when both executions produce tool-argument JSON that:

Validates against the frozen schema with additionalProperties disabled
Contains no assumption key outside the allow list
Differs from the sibling run only in fields marked volatile, such as timestamps or request ids

Everything else is drift, even when the natural-language summary looks careful and complete.

Lab 0: freeze the tool-argument schema (15 minutes)

Start from a tool the class already uses in a prior lab, and do not invent a second product surface for this exercise. The example below is a ticketing lookup that students commonly over-parameterize when a model tries to be helpful.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "LookupTicketArgs",
  "type": "object",
  "additionalProperties": false,
  "required": ["ticket_id", "fields"],
  "properties": {
    "ticket_id": { "type": "string", "pattern": "^TCK-[0-9]{6}$" },
    "fields": {
      "type": "array",
      "minItems": 1,
      "maxItems": 8,
      "items": { "enum": ["status", "assignee", "priority", "updated_at"] }
    },
    "volatile": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "request_id": { "type": "string" }
      }
    }
  }
}

Exercise 0a: take one captured tool call from last week's lab and validate it against this schema. Exercise 0b: add one extra field that models commonly invent, such as include_comments, and confirm the validator rejects it. Students who cannot produce a failing extra field have not frozen the schema; they have only restated the happy path.

python -m venv .venv
. .venv/bin/activate
pip install jsonschema pytest
python -m jsonschema -i captures/run_a.args.json schemas/tool_args.schema.json

Lab 1: extract assumption keys (20 minutes)

Agents fill gaps with polite defaults: timezone, page size, current user, and priority. Those defaults are assumptions, not facts, and they belong in an allow list or they belong in task.json. The extractor below is labeled lab code; it reads a local trace and does not call a network.

# lab_assumption_keys.py — workshop extractor, not production telemetry
import json
import re
from pathlib import Path

FACT_KEYS = {"ticket_id", "fields", "user_id", "org_id"}
ASSUMPTION_PATTERNS = (
    re.compile(r"assume[d]?", re.I),
    re.compile(r"default(?:ing)? to", re.I),
    re.compile(r"current user", re.I),
    re.compile(r"probably", re.I),
)

def extract_keys(trace_path: Path) -> set[str]:
    blob = json.loads(trace_path.read_text())
    found: set[str] = set()
    for event in blob.get("events", []):
        text = str(event.get("content", ""))
        for pat in ASSUMPTION_PATTERNS:
            if pat.search(text):
                found.add(pat.pattern)
        for key in event.get("tool_args", {}):
            if key not in FACT_KEYS:
                found.add(key)
    return found

def assert_allowed(found: set[str], allow_path: Path) -> None:
    allowed = {
        line.strip()
        for line in allow_path.read_text().splitlines()
        if line.strip() and not line.startswith("#")
    }
    extra = found - allowed
    if extra:
        raise AssertionError(f"undeclared assumptions: {sorted(extra)}")

Exercise 1a: run the extractor on a fixture that never mentions defaults and record the empty set. Exercise 1b: inject the sentence “defaulting to page size 50” into the trace and watch the assertion fire. Keep the allow list shorter than eight keys; a longer list usually means the task file is missing facts the agent should not have to guess.

Lab 2: dual-run and diff (30 minutes)

Dual-run means two executions of the same task.json with isolated memory and no shared scratchpad. Do not compare prose, because two different argument lists can still produce similar sentences. Compare tool-argument JSON after stripping volatile fields, then fingerprint the remainder.

# lab_dual_run.py — rerunnable checker for the 90-minute workshop
import hashlib
import json
from pathlib import Path

from jsonschema import Draft202012Validator

VOLATILE_KEYS = {"request_id", "ts", "latency_ms"}

def strip_volatile(obj):
    if isinstance(obj, dict):
        return {
            k: strip_volatile(v)
            for k, v in obj.items()
            if k not in VOLATILE_KEYS
        }
    if isinstance(obj, list):
        return [strip_volatile(x) for x in obj]
    return obj

def load_tool_calls(path: Path) -> list:
    payload = json.loads(path.read_text())
    return payload["tool_calls"]

def schema_ok(args: dict, schema: dict) -> None:
    Draft202012Validator(schema).validate(args)

def fingerprint(tool_calls: list) -> str:
    stable = strip_volatile(tool_calls)
    blob = json.dumps(stable, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()

def assert_stable(run_a: Path, run_b: Path, schema_path: Path) -> None:
    schema = json.loads(schema_path.read_text())
    a = load_tool_calls(run_a)
    b = load_tool_calls(run_b)
    for call in a + b:
        schema_ok(call["args"], schema)
    fa, fb = fingerprint(a), fingerprint(b)
    if fa != fb:
        raise AssertionError(f"tool-call drift: {fa} != {fb}")

Exercise 2a: duplicate a fixture, change only request_id, and confirm the checker passes. Exercise 2b: change fields from ["status"] to ["status", "priority"] and confirm it fails. Exercise 2c: send the same body to two isolated runners and store traces as run_a.json and run_b.json without editing either file by hand.

A local loop that students can rerun without a dashboard looks like this:

python agent_runner.py --task fixtures/task.json --out traces/run_a.json
python agent_runner.py --task fixtures/task.json --out traces/run_b.json
python -c "from pathlib import Path; from lab_dual_run import assert_stable; assert_stable(Path('traces/run_a.json'), Path('traces/run_b.json'), Path('schemas/tool_args.schema.json')); print('stable')"

If a hosted second endpoint is required, point agent_runner.py at the class host chosen in the briefing and keep the same task.json. The grade still comes from assert_stable, not from a screenshot of a chat transcript.

Lab 3: fail closed in pytest (15 minutes)

Wrap the checker so continuous integration, not a teaching assistant, rejects drift. The broken-fixture trial is mandatory; a green run with no negative control does not count as completion.

# test_dual_run_stability.py
import json
from pathlib import Path

import pytest
from jsonschema import Draft202012Validator, ValidationError

from lab_assumption_keys import assert_allowed, extract_keys
from lab_dual_run import assert_stable

FIXTURES = Path("fixtures")
TRACES = Path("traces")
SCHEMAS = Path("schemas")

def test_schema_rejects_extra_field():
    schema = json.loads((SCHEMAS / "tool_args.schema.json").read_text())
    bad = {
        "ticket_id": "TCK-000042",
        "fields": ["status"],
        "include_comments": True,
    }
    with pytest.raises(ValidationError):
        Draft202012Validator(schema).validate(bad)

def test_dual_run_fingerprints_match():
    assert_stable(
        TRACES / "run_a.json",
        TRACES / "run_b.json",
        SCHEMAS / "tool_args.schema.json",
    )

def test_assumption_allow_list():
    found = extract_keys(TRACES / "run_a.json") | extract_keys(TRACES / "run_b.json")
    assert_allowed(found, FIXTURES / "assumptions.allowed.txt")

Exercise 3: break one fixture on purpose, run pytest -q, and paste the assertion message into the lab notes. If the class cannot show a failing log, the schema is not yet a gate.

Worked example students can rerun

Use this minimal task.json as the shared fixture so dual-run cost stays visible and comparable across laptops. The budgets are part of the contract, not comments for humans.

{
  "task_id": "lab-dual-run-001",
  "user": "Return status and assignee for ticket TCK-000042. Do not add fields.",
  "facts": {
    "ticket_id": "TCK-000042",
    "fields": ["status", "assignee"]
  },
  "budgets": {
    "max_tool_calls": 2,
    "max_retries": 0
  }
}

Expected stable tool call after both runs:

{
  "name": "lookup_ticket",
  "args": {
    "ticket_id": "TCK-000042",
    "fields": ["status", "assignee"]
  }
}

Unstable variants the checker must reject, even when the final sentence still names the ticket:

fields expanded to include priority after a reorder that looks harmless
ticket_id rewritten as an integer or as a bare six-digit string
a second tool call such as list_comments that the task never requested
prose that says “I assumed the current user is the assignee” while emitting extra arguments

Students rerun by deleting traces/, executing the two runner commands, and calling pytest. If results change after a prompt edit, the schema was not the source of truth and the lab is incomplete.

Decision table

Observation in the traces	Treat as	Next action in the remaining minutes
Schema fail on an extra key	Drift	Shrink the prompt; do not widen the schema
Fingerprints differ only in volatile keys	Stable	Keep the strip list explicit in code
Assumption key missing from the allow list	Hidden default	Move the fact into `task.json` or fail the run
Answers match while tool args differ	Silent drift	Fail the lab; do not grade the prose
One endpoint unavailable	Incomplete dual-run	Use two local processes; do not skip Lab 2
Two identical wrong tool calls	False stable	Add a domain assertion outside this workshop

Limitations and who should not use this approach

Dual-run comparison is a cheap consistency gate, not a correctness proof, and two wrong tool calls with the same JSON will pass. Non-deterministic tools such as clocks, live queues, or sampling with temperature above zero will fail the fingerprint even when operators consider the behavior acceptable. Teams that need creative variation should mark those fields volatile, lower temperature for the lab, or skip this workshop entirely.

Do not use this method when any of the following constraints are true:

The tool has required side effects that cannot be sandboxed for a second run
The class cannot freeze a schema because the upstream API is unbounded
The latency or token cost of a second run exceeds the published lab budget
Graders want to score writing quality rather than tool-call contracts

The checker also ignores natural-language caveats on purpose. If the agent writes “I might be wrong” but still emits extra arguments, the test fails, and that severity is the intended teaching signal.

What instructors should collect

Before dismissing the class, collect four files per student: task.json, tool_args.schema.json, assumptions.allowed.txt, and the pytest log. Those files are the evidence that the contract exists outside a chat window. Screenshots of a vendor UI are not substitutes, because they cannot be rerun after the session ends.

This workshop stays useful if any hosted model is removed, because the schema, extractor, and fingerprint live in the repository. Instructors should treat dual-run stability as a gate that happens before memory writes or side effects, not as a substitute for domain tests that check ticket state in a real system.

DEV Community: Finley Zhu

Workshop: Catch Shared-Host Drift With a Five-Probe Floor Card in 85 Minutes

Why a floor card beats a chat anecdote

What this workshop is not

Workshop clock

Artifact layout

Probe pack schema

Worked example students can rerun

Exercise 1 — freeze the contract, not the essay (15 minutes)

Exercise 2 — change the host, not the scorer (20 minutes)

Exercise 3 — inject one honest failure (15 minutes)

Limitations

Who should not use this approach

Recap for the last five minutes

Workshop: Pin a Dual-Backend Tool Receipt in 80 Minutes

What this lab measures

Timing box

Lab layout students can rerun

Exercise A — Capture a raw tool turn

Exercise B — Canonicalize before you compare

Exercise C — Dual-run fixture versus live

Worked miss that students should rerun

Commands for a cold rerun

Limitations

Who should not use this outline

Workshop: Map Agent Claims to a Failing Command Ledger in 90 Minutes

Who this session is for

Timing

The core artifact

Makefile contract

Exercise 1 — Write claims that already fail (15 minutes)

Exercise 2 — Generate an implementation on a shared free endpoint (25 minutes)

Ledger checker (teaching code)

Exercise 3 — Mutation that drops an assertion (20 minutes)

Worked example students can rerun

What the ledger does not prove

Closing

Workshop: Score AI Refactors With a Frozen Oracle and Diff Budget in 75 Minutes

Why this workshop exists

Audience, timing, and materials

Learning objectives

The artifact students will rerun

inventory.py

test_inventory.py

score_patch.py

Exercise 1 — Freeze the oracle (0–10 minutes)

Exercise 2 — Encode the budget (10–25 minutes)

Exercise 3 — Score two fixtures (25–40 minutes)

Exercise 4 — Optional remote generation (40–60 minutes)

Exercise 5 — Debrief (60–75 minutes)

Limitations

Who should not use this approach

After class

Workshop: Freeze Agent Output Contracts Before You Swap Endpoints in 80 Minutes

Audience and prerequisites

Eighty-minute agenda

Failure mode: tests that only watch tool names

Artifact: frozen contract ledger

Golden fixture

Checker module

Pytest gate

Exercise 1 — Record the golden path (18 minutes)

Exercise 2 — Encode fail-closed rules (17 minutes)

Exercise 3 — Swap the endpoint, keep the fixture (17 minutes)

Worked example: ticket triage loop

Decision table for contract severity

Limitations

Who should not run this workshop

Close-out checklist

Failure Injection for Agent Loops: A Replayable Cassette Workshop

Failure Injection for Agent Loops: A Replayable Cassette Workshop

Why rehearsal beats monitoring for quota-limited endpoints

The artifact: a clock-injected harness in three files

1. harness.py — the loop under test

2. cassettes.py — record once, replay forever

3. test_policy.py — the graded deliverable

The 90-minute schedule

Exercise notes that keep the room moving

The degradation policy, before and after

Four invariants every submission must hold

`inventory.py`

`test_inventory.py`

`score_patch.py`

1. `harness.py` — the loop under test

2. `cassettes.py` — record once, replay forever

3. `test_policy.py` — the graded deliverable