DEV Community: Finley Zhu

Workshop: Classify Endpoint Failures Before a Second Call in 70 Minutes

Finley Zhu — Sun, 11 Oct 2026 22:23:11 +0000

A second model call is a budget decision, not a reflex, and this seventy-minute workshop teaches a local classifier that blocks the reflex. Students replay seven recorded failures, assign retry once, stop, or inspect, and leave with a standard-library script that runs without a network. The exercise fits a shared lab that already calls a free model endpoint and may attach a free server for class time. It does not prove that any host will keep those options, publish quotas, or name a stable model.

Why the label comes before the dashboard

The useful question is narrower than a general retry slogan, and it shows up whenever a lab client treats every error as temporary. A free endpoint still spends shared capacity and attention when the client repeats a request that the contract already rejected. A transport timeout and a schema rejection both print as failures, yet only the timeout can justify one more attempt. The workshop therefore grades the failure label, not the prose quality of whatever answer a model might have returned.

Outcomes students must be able to defend

By minute seventy, each student should be able to defend three actions with a record, a reason, and a second-call boolean. Retry once means a single extra attempt is allowed, and the next identical failure must stop. Stop means the client must not call again until a human changes the request, the credential, or the contract. Inspect means the row is incomplete, so automation should surface it instead of guessing a retry.

Clock for a prepared room

The clock assumes a prepared room, so instructors should paste the script into the workspace before students arrive. Each block has an exit check, and the class should not advance while that check is still red. Ten minutes of slack sits inside the middle blocks, which is enough to debug a path error or a Python version mismatch. The final ten minutes are for limitations, not for a live demo that the room has not measured.

Minutes 0-10: define retry once, stop, and inspect without opening the script.
Minutes 10-25: run the module and match the designed seven-line summary.
Minutes 25-45: write a defense for the schema row and the rate-limit row.
Minutes 45-60: trace the dry-run guard and add one unclassified status.
Minutes 60-70: submit a memo listing three cases the script must not decide.

Materials and honesty rules

The graded path uses Python 3.11 or newer and no third-party packages, which keeps the room independent of package mirrors. Students need a terminal, a text editor, and the fixture list inside the module they are about to save. A live key is out of scope, because the point is to classify records the instructor already captured or composed. Treat the script as a teaching artifact with a designed exit check, not as a measured benchmark from a production host.

Decision table students implement

The matrix below is the contract students implement, and the script is only a runnable copy of that contract. Status numbers are a teaching subset, so an unlisted code must land in inspect rather than in a silent default retry. A 429 is inspect on purpose, because a free shared pool may be enforcing a limit rather than reporting a short blip. Connection failures stay eligible for one retry, and the already-retried flag then forces stop on the following row.

Signal	Action	Second call	Note
error is timeout, connection_reset, or dns_temporary	retry_once	yes, once	Transport may clear without a contract edit
status 408, 500, 502, 503, or 504	retry_once	yes, once	Stop instead if already_retried is true
already_retried is true	stop	no	The one-attempt teaching budget is spent
status 400, 401, 403, 404, 413, or 422	stop	no	The same call will not repair the contract
status 429	inspect	no	Do not inherit the transient retry path
status 200 with an empty body	inspect	no	A success status is not a complete answer
missing or unlisted status	inspect	no	Guessing a retry is a failed exit check

The classifier checks already_retried before status, so a repeated 503 stops because the one-attempt budget is already spent.
The classifier checks coarse transport errors before status, so a timeout with a missing status still qualifies for one retry.
The classifier checks 429 before the transient set, so a rate-limit row cannot slip into an automatic second call.

Worked module students rerun

Save the following module as retry_gate.py and run it with Python 3 before the discussion period begins. The example is local and unexecuted against any vendor, and the printed lines are the designed checks students should reproduce. If a row fails, fix the classifier rather than editing the expected label to match a convenient result. That rule keeps the lab honest when a later cohort wants to add records from a real log.

#!/usr/bin/env python3
'''Teaching fixture: classify a failure before a second model call.

Local and unexecuted against any vendor. Designed checks are not a benchmark.
'''

from __future__ import annotations

import sys
from dataclasses import dataclass

RETRY_ONCE = 'retry_once'
STOP = 'stop'
INSPECT = 'inspect'

STOP_STATUS = {400, 401, 403, 404, 413, 422}
RETRY_STATUS = {408, 500, 502, 503, 504}
TRANSPORT_ERRORS = {'timeout', 'connection_reset', 'dns_temporary'}


@dataclass(frozen=True)
class Decision:
    action: str
    reason: str


def classify(record: dict) -> Decision:
    status = record.get('status')
    error = (record.get('error') or '').lower()
    body = record.get('body')
    if record.get('already_retried'):
        return Decision(STOP, 'retry budget already spent')
    if error in TRANSPORT_ERRORS:
        return Decision(RETRY_ONCE, 'transport error: %s' % error)
    if status in STOP_STATUS:
        return Decision(STOP, 'client or auth status %s' % status)
    if status == 429:
        return Decision(INSPECT, 'rate limit needs a human rule')
    if status in RETRY_STATUS:
        return Decision(RETRY_ONCE, 'transient status %s' % status)
    if status == 200 and not body:
        return Decision(INSPECT, 'empty success body')
    if status is None:
        return Decision(INSPECT, 'missing status')
    return Decision(INSPECT, 'unclassified status %s' % status)


def allow_second_call(record: dict) -> bool:
    decision = classify(record)
    return decision.action == RETRY_ONCE and not record.get('already_retried')


def plan_next(record: dict) -> dict:
    decision = classify(record)
    if allow_second_call(record):
        return {
            'id': record['id'],
            'plan': 'one_retry',
            'reason': decision.reason,
        }
    return {
        'id': record['id'],
        'plan': 'skip',
        'reason': decision.reason,
    }


FIXTURES = [
    {
        'id': 't1',
        'status': 503,
        'error': '',
        'body': None,
        'already_retried': False,
        'expect': RETRY_ONCE,
        'expect_plan': 'one_retry',
    },
    {
        'id': 't2',
        'status': 401,
        'error': '',
        'body': {'message': 'unauthorized'},
        'already_retried': False,
        'expect': STOP,
        'expect_plan': 'skip',
    },
    {
        'id': 't3',
        'status': None,
        'error': 'timeout',
        'body': None,
        'already_retried': False,
        'expect': RETRY_ONCE,
        'expect_plan': 'one_retry',
    },
    {
        'id': 't4',
        'status': 429,
        'error': '',
        'body': None,
        'already_retried': False,
        'expect': INSPECT,
        'expect_plan': 'skip',
    },
    {
        'id': 't5',
        'status': 503,
        'error': '',
        'body': None,
        'already_retried': True,
        'expect': STOP,
        'expect_plan': 'skip',
    },
    {
        'id': 't6',
        'status': 200,
        'error': '',
        'body': None,
        'already_retried': False,
        'expect': INSPECT,
        'expect_plan': 'skip',
    },
    {
        'id': 't7',
        'status': 422,
        'error': '',
        'body': {'message': 'schema'},
        'already_retried': False,
        'expect': STOP,
        'expect_plan': 'skip',
    },
]


def run() -> int:
    failures = 0
    for row in FIXTURES:
        decision = classify(row)
        plan = plan_next(row)
        action_ok = decision.action == row['expect']
        plan_ok = plan['plan'] == row['expect_plan']
        if not (action_ok and plan_ok):
            failures += 1
            print(
                'mismatch id=%s expected_action=%s expected_plan=%s'
                % (row['id'], row['expect'], row['expect_plan']),
                file=sys.stderr,
            )
        second = allow_second_call(row)
        print(
            '%s: action=%s second_call=%s plan=%s reason=%s'
            % (row['id'], decision.action, second, plan['plan'], decision.reason)
        )
    print('checked=%s failed=%s' % (len(FIXTURES), failures))
    return 1 if failures else 0


if __name__ == '__main__':
    sys.exit(run())

python3 retry_gate.py
echo exit:$?

How to read the designed output

Read the designed output as a contract for the checker, not as a log captured from a running server. True on second_call appears only for t1 and t3, and every other row must print False and plan skip. Row t5 is the budget lesson, because the status is still 503 while already_retried forces an immediate stop. If your local run differs, treat the difference as a bug in the copy, not as a new research finding about endpoints.

t1: action=retry_once second_call=True plan=one_retry reason=transient status 503
t2: action=stop second_call=False plan=skip reason=client or auth status 401
t3: action=retry_once second_call=True plan=one_retry reason=transport error: timeout
t4: action=inspect second_call=False plan=skip reason=rate limit needs a human rule
t5: action=stop second_call=False plan=skip reason=retry budget already spent
t6: action=inspect second_call=False plan=skip reason=empty success body
t7: action=stop second_call=False plan=skip reason=client or auth status 422
checked=7 failed=0

The designed process exit is 0. That number only means the local expectations matched the classifier, and it says nothing about latency, token use, or host capacity.

Guided exercises

Minutes 0 to 10: name the actions

Ask each pair to rewrite the three definitions in their own words without looking at the script. Collect one example of a stop case from ordinary HTTP practice, such as a rejected schema or a missing credential. Do not accept a complaint about model quality as a class, because that sentence does not tell the client what to do next. End this block only when every pair can point to the boolean that would allow a second call.

Minutes 10 to 25: run the deck

Students run the module and compare seven lines with the decision table, including the second-call column. The designed summary is checked equals 7 and failed equals 0, which is a specification rather than a published performance number. If the room has no Python, an instructor can still walk the table row by row and mark the same booleans on paper. That paper path is slower, but it preserves the lesson when install rights are locked down by the lab image.

Minutes 25 to 45: defend stop and inspect

Each student picks the 422 row and writes why a second identical call cannot repair a schema mismatch. Each student also picks the 429 row and writes why an automatic retry could amplify load on a shared free pool. The discussion should stay on client behavior, not on blaming a model for a status code the client already understands. A useful note names the field that carried the evidence, such as status, error, or already_retried.

Minutes 45 to 60: trace the guard and add t8

Students trace plan_next and confirm that skip is the plan for every row except the first 503 and the timeout. They then add a t8 fixture with status 409, an expect action of inspect, and an expect plan of skip. The existing fall-through rule should accept that row without a new branch, which is the point of the unclassified path.

After that row is appended correctly, the designed summary becomes checked equals 8 and failed equals 0. No socket opens in this block, so a mistaken label cannot trigger a live call while the class is still learning. Keep the new fixture in the same file so the next cohort can rerun the extended deck without a separate handout.

{
    'id': 't8',
    'status': 409,
    'error': '',
    'body': {'message': 'conflict'},
    'already_retried': False,
    'expect': 'inspect',
    'expect_plan': 'skip',
}

Minutes 60 to 70: write the limitation memo

Each student writes three cases the script must not decide, using complete sentences and naming the missing field. Acceptable cases include a non-idempotent tool call, a 429 that includes Retry-After, and a vendor code outside the teaching set. Unacceptable cases are vague complaints about model quality, cost, or speed, because this checker never saw those signals. Collect the memos before anyone opens a host dashboard, so the product discussion cannot replace the exit check.

Map one synthetic line

Use a synthetic line, not a production export, when you demonstrate the mapping on a projector. A synthetic line with id lab-17, status 503, an empty error, and retried no becomes the same shape as fixture t1. Students can append that mapped object to FIXTURES only after they also write the expected action and expected plan. If they forget the expected fields, the checker should fail closed rather than treat a missing expectation as success.

Where a host with free access fits

MonkeyCode enters the lesson only after the labels are stable, and only as one possible lab host. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This article relies on two availability claims supplied for the draft: free model access and a free server option. This draft does not state quotas, hardware, duration, uptime, or benchmark scores, because those facts were not supplied for verification.

Students should read the host's current documentation before class, since availability claims can change and this page is not that documentation. A free server option may help a classroom share one practice host, but it does not change the failure labels or the one-retry budget. If documentation contradicts this workshop, follow the documentation and keep the local fixtures as the graded artifact.

Export rules if logs come later

If the instructor later exports logs, keep a local id, status, a coarse error tag, and an empty-body flag. Drop raw prompts, completions, authorization headers, and user identifiers before the file reaches any student machine. Map vendor error strings to a small tag set such as timeout, connection_reset, or dns_temporary, and leave unknown strings empty. An unknown string must not become a retry, because the classifier treats an empty error plus an unlisted status as inspect.

Limitations of the teaching contract

The status sets are incomplete, and a 409, 425, or vendor-specific code will fall through to inspect until an instructor adds a reviewed rule. One retry can still be the wrong policy for 503 storms, so the flag is a teaching brake rather than a production backoff schedule. The script does not measure tokens, latency, or queue depth, and it should not be cited as evidence about cost or speed. Empty body detection is naive, because a JSON null, a whitespace string, and a missing key are different defects in a real client.

A 429 with a Retry-After hint might be safe to defer, but these fixtures carry no such hint, so the honest label is inspect. Idempotency is assumed for the one retry, and a non-idempotent tool call should be stop even when the status looks transient. That case is listed in the closing memo because the script cannot see side effects that were not recorded in the row.

Who should not use this lab

Skip the lab if the course outcome is a tuned retry library with jitter, circuit breakers, and idempotency keys. Skip it if students must handle protected data, because the safe path here assumes fixtures that contain no personal content. Skip it if the goal is to compare model quality, since none of the rows include a completion to score. On-call playbooks also need vendor documents and ownership, which this seventy-minute room deliberately does not provide.

Facilitation notes

Instructors should resist turning the closing memo into a tool bake-off, because the room has no measured vendor results to compare. Grade the written reasons, the passing local checker, and the limitation list, not the number of live calls a student managed to send. Pairs are enough; a larger group hides whether each person can name the field that justified stop. If Python is unavailable, the paper walk still counts, provided the booleans match the table exactly.

Closing check

The core result is testable: seven local rows, three actions, and a guard that refuses an unearned second call. Run the module, confirm the designed summary, and only then consider pointing the same field map at a host log. If a lab already uses MonkeyCode's free model access and free server option, classify redacted error rows with these labels first. That pass is a review step, not a claim that the host produced the sample rows in this article.

Workshop: Grade a Free Model Swap With a Three-Row Fixture Deck in 65 Minutes

Finley Zhu — Fri, 09 Oct 2026 15:35:34 +0000

A teaching lab can trust a free model swap only after three frozen prompts still score the same way. Students should be able to rerun those prompts next week and separate a wrong answer from a dead host. This session builds that separation as a fixture deck, a local scorer, and a short decision table. The deck remains useful when the remote seat is down, because stub mode never leaves the student laptop.

Recent teaching discussions keep circling the same practical question: a smaller or free host may be enough if a narrow check still passes. This workshop takes that question into a timed classroom clock instead of turning it into a product review. The check here is three frozen JSON rows, not a leaderboard and not a claim about any named model family. Students leave with a rerunnable gate, which is more useful for next week's lab than a headline about model choice.

This fixture deck grades structured answers rather than token spend, request latency, or separate tool-permission stamps. Each row names one prompt, the required JSON keys, a list of forbidden phrases, and a character ceiling. A later cohort can replace those three prompts without rewriting the scorer or the shared decision table. Transport errors stay in their own bucket so a timeout is never recorded as a content failure.

What a 65-minute pass actually proves

The lab proves that a small contract can catch obvious drift before a class depends on a new host. It does not prove that any model is generally correct, inexpensive, or stable across every course topic. Three rows are a teaching gate, not a benchmark suite and not a statement about vendor capacity. Instructors should treat every green pass as evidence about these fixtures alone, not about the wider course.

Constraints to state before minute eight

Say aloud that no product name, seat length, hardware shape, or quota number is assumed by these exercises. Students will author fixtures, run a stub, inject one deliberate drift, and only then consider a remote host. Prompts in the deck must use public sample text, never passwords, student records, or private repository contents. If the classroom cannot review a third-party terms page that morning, stop the plan after the local stub.

Session clock

Use the blocks below so the worked drift loop is not squeezed out by setup chatter. Each timed block has one visible output that the instructor can check on a nearby student screen. Do not add a fourth prompt during this sitting, because the constraint is part of the lesson. A cohort that wants more coverage should schedule a second sitting rather than stretching this clock.

Minutes 0–8: name the failure

Describe a lab where yesterday's JSON summary gained an extra field and then broke a downstream checker. Ask students which failure is content drift, which failure is transport loss, and which failure is a parser error. Write those three labels on the board before anyone opens an editor or calls a remote host. Those three labels become the only status values the scorer is allowed to emit during this session.

Minutes 8–22: author the deck

Students create a file named fixtures/deck.json that contains exactly three rows and no hidden fourth case. Each row carries an identifier, one prompt, the required keys, forbidden substrings, and a maximum character count. One row should demand a refusal to invent a version number that the source text never stated. That refusal row teaches the class that a fluent answer can still fail the written contract on purpose.

Minutes 22–40: prove the harness

Run the scorer in stub mode so the first green result does not depend on any network. The stub returns canned JSON that already satisfies every required key and avoids every forbidden phrase listed. Students should read the exit code, the status column, and the reason string before they edit anything. A green stub result only proves the harness, not the quality of any live model response.

Minutes 40–52: inject one drift

Change the stub for the endpoints row so the path field disappears while the method field remains. Rerun the same command and confirm that only that row becomes content drift while the other rows stay passing. Restore the stub and rerun the command once more so the deck returns to a clean baseline score. This short loop is the worked example that students can repeat later without a teacher in the room.

Minutes 52–65: record the swap rule

Fill the decision table before anyone discusses a remote host, a product seat, or a model name. Content drift blocks the planned swap until the fixture or the prompt is intentionally revised by the instructor. A transport failure blocks the swap for that class day, while a parser error blocks it until the expected shape is clarified. Only a full pass on these three rows allows an optional retarget of the same unchanged deck.

Fixture students commit

Save the following document as the only fixture input that this sitting is allowed to grade today. The prompts are public samples, and the ceilings are teaching limits rather than any product measurements. Keep the row identifiers stable so later diffs show prompt edits instead of merely renamed rows. If a student wants a different sample, replace the source sentence and leave the required keys in place.

{
  "rows": [
    {
      "id": "summary",
      "prompt": "Source: the patch note says retries now wait 200ms. Return a JSON object with keys summary and risk. Do not add fields.",
      "required_keys": ["summary", "risk"],
      "forbidden": ["password", "api_key"],
      "max_chars": 240
    },
    {
      "id": "endpoints",
      "prompt": "Source: clients call GET /health. Return a JSON object with keys method and path. Do not invent other routes.",
      "required_keys": ["method", "path"],
      "forbidden": ["delete /", "drop table"],
      "max_chars": 180
    },
    {
      "id": "version",
      "prompt": "Source text has no version number. Return a JSON object with keys status and reason. Refuse to invent a version.",
      "required_keys": ["status", "reason"],
      "forbidden": ["v1.2.3", "version 2"],
      "max_chars": 200
    }
  ]
}

Scorer students rerun

The script below is an unexecuted teaching artifact for the lab, not a measured client for any hosted product. Stub results are defined by the canned objects in this file, so a green run does not describe a live model. HTTP mode is a class-defined adapter with a placeholder path, and instructors must replace that path only after reading current host documentation. Do not treat the sample URL shape as a vendor interface or as evidence of uptime.

#!/usr/bin/env python3
"""Three-row fixture scorer for a classroom model-swap gate.

Label: proposed lab code. Status lines from stub mode are defined here.
They are not observations from a hosted model run.
"""

import argparse
import json
import sys
import urllib.error
import urllib.request


def load_deck(path):
    with open(path, encoding="utf-8") as handle:
        deck = json.load(handle)
    rows = deck.get("rows")
    if not isinstance(rows, list) or len(rows) != 3:
        raise SystemExit("deck must contain exactly three rows")
    return rows


def stub_body(row, drift_id):
    canned = {
        "summary": {"summary": "Patch notes add a retry hint.", "risk": "low"},
        "endpoints": {"method": "GET", "path": "/health"},
        "version": {"status": "refused", "reason": "source omits a version"},
    }
    if row["id"] not in canned:
        raise SystemExit("stub has no canned body for " + row["id"])
    body = dict(canned[row["id"]])
    if drift_id == row["id"] and "path" in body:
        body.pop("path")
    return json.dumps(body)


def fetch_remote(base_url, prompt, timeout):
    """Placeholder adapter. Replace the path after reading current host docs."""
    payload = json.dumps({"prompt": prompt}).encode("utf-8")
    request = urllib.request.Request(
        base_url.rstrip("/") + "/lab-complete",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            raw = response.read().decode("utf-8")
            code = response.status
    except (urllib.error.URLError, TimeoutError, OSError) as exc:
        return None, "transport_failure: " + exc.__class__.__name__
    if code != 200 or not raw.strip():
        return None, "transport_failure: bad response"
    return raw, None


def grade(row, raw):
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return "parser_error", "body is not json"
    if not isinstance(parsed, dict):
        return "parser_error", "body is not an object"
    missing = [key for key in row["required_keys"] if key not in parsed]
    if missing:
        return "content_drift", "missing " + ",".join(missing)
    blob = json.dumps(parsed).lower()
    for phrase in row["forbidden"]:
        if phrase.lower() in blob:
            return "content_drift", "forbidden phrase present"
    if len(blob) > int(row["max_chars"]):
        return "content_drift", "over character ceiling"
    return "content_pass", "keys present and ceiling held"


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--deck", required=True)
    parser.add_argument("--mode", choices=["stub", "http"], default="stub")
    parser.add_argument("--base-url", default="")
    parser.add_argument("--timeout", type=float, default=20.0)
    parser.add_argument("--drift", default="")
    args = parser.parse_args()
    failures = 0
    for row in load_deck(args.deck):
        if args.mode == "stub":
            raw, err = stub_body(row, args.drift), None
        else:
            if not args.base_url:
                raise SystemExit("http mode requires --base-url")
            raw, err = fetch_remote(args.base_url, row["prompt"], args.timeout)
        if err:
            status, reason = "transport_failure", err
        else:
            status, reason = grade(row, raw)
        if status != "content_pass":
            failures += 1
        print(f"{row['id']}|{status}|{reason}")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

Expected lines from the worked example

The clean stub is defined to print three content_pass lines and to exit with code zero. The drifted stub is defined to print content_drift only for the endpoints row and to exit with code one. Students should compare their terminal text with those two patterns before they change any prompt wording. If the drifted run still exits zero, the harness is wrong and the remote step stays closed.

summary|content_pass|keys present and ceiling held
endpoints|content_pass|keys present and ceiling held
version|content_pass|keys present and ceiling held

summary|content_pass|keys present and ceiling held
endpoints|content_drift|missing path
version|content_pass|keys present and ceiling held

Commands for the two local passes

Run the clean command first and save the text beside the cohort date in the lab notes. Run the drift command second and confirm that the exit status changed from zero to one. Do not point HTTP mode at a host until both local patterns match the expected lines above. Replace the placeholder path in the adapter before any remote command is shared with the class.

python3 score_deck.py --deck fixtures/deck.json --mode stub
python3 score_deck.py --deck fixtures/deck.json --mode stub --drift endpoints
python3 score_deck.py --deck fixtures/deck.json --mode http --base-url "$LAB_BASE" --timeout 20

Decision table for the swap

Observed status	What the row showed	Decision for this class day
content_pass	Required keys exist, forbidden phrases are absent, and the ceiling holds	Row may stay in the swap set
content_drift	A key is missing, a forbidden phrase appears, or the ceiling breaks	Block the swap until the instructor revises the row
parser_error	The body is missing, not JSON, or not an object	Block the swap until the expected shape is clarified
transport_failure	The call times out, returns a non-200, or comes back empty	Block the swap today and keep grading on the stub

Read the table from top to bottom and stop at the first failing row rather than averaging the three results. A single drifted row is enough reason to keep the previous host in place for that class day. Parser errors should be fixed in the prompt notes before anyone retries a remote call again. Transport failures should be logged with the clock time and then left out of any accuracy discussion.

Where an optional free host fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as one optional place to obtain free model access and a free server option for the retarget step. This plan names no models, quotas, hardware, duration, or permanence, and those facts must be read from current terms that morning. If those terms are missing, unclear, or incompatible with classroom policy, keep the lab on stub mode and still grade the deck.

After local statuses are green, a class that already has the free server option enabled can retarget this same unchanged deck. That retarget is optional, and a missing seat does not lower the local grade for the stub exercise. Students should paste only the public prompts from the deck, then compare status lines against the table above. Archive the status lines with the cohort date so the next section can see whether the host still matches the frozen rows.

Outputs the instructor collects

Collect four artifacts before the room is dismissed, and do not accept a screenshot of a chat window as a substitute. The deck file shows what was frozen, while the two transcripts show that drift is detectable. The written decision line shows whether a remote retarget is even eligible for that particular class day. Store the four artifacts together so a later cohort can rerun the same checks without reconstructing the lesson from memory.

Students submit a three-row deck file that keeps stable identifiers and uses only public source sentences.
Students submit a clean stub transcript that exits zero and shows three content_pass lines for every row.
Students submit a drifted stub transcript that exits one and isolates the failure on the endpoints row.
Students submit one decision line that either blocks the swap or records an optional retarget after a full pass.

Limitations and who should skip the lab

Exact key checks will miss a correct paraphrase that uses different field names or a nested object. Three public prompts cannot support a claim about accuracy, cost, latency, or uptime for a whole course. Do not send secrets, grades, or unpublished student work to any remote host, whether the seat is free or paid. Skip this workshop when you need a statistical evaluation, regulated medical or legal wording, or a graded exam that cannot tolerate a missing seat.

A team that already runs a large offline eval suite will gain little new procedure from a three-row gate. A classroom with no time to review third-party terms should not add the remote step at all. The scorer also ignores semantic similarity, so two answers can differ in wording and still need a human note. Those limits are intentional, because the sitting is about catching obvious contract breaks before a swap.

What to archive before the next cohort

Store the deck file, the scorer version, and the status lines from both the clean stub and the drifted stub. Add one sentence that names who may edit prompts and who may only rerun the commands. Do not archive credentials, host tokens, or private base URLs anywhere inside the shared class repository. The next cohort should diff the deck first, then decide whether a new row belongs in a later sitting.

Workshop: Ledger Prompt, Host, and Route in an 85-Minute Lab

Finley Zhu — Thu, 08 Oct 2026 10:53:54 +0000

A free model lab turns noisy when the prompt, the host, and the route all change inside one hour. This eighty-five minute workshop pins those three lanes before the room starts debating answer quality. Students leave with a ledger they can rerun on the next day, even if the public terms page changed overnight. The ledger shows which lane moved, and it refuses to treat a remembered allowance as if it were evidence.

Why a class needs the ledger first

Hosted model classes often mix three unstable inputs and then grade the resulting text as one event. A prompt edit, a server image change, and a route swap can each alter the output on their own. When those events share one timestamp, the class cannot tell which event deserves the next repair. The ledger separates those events so the following edit has one named owner instead of three suspects.

The workshop assumes one small group, one frozen prompt, and one candidate open-source host for the hour. It does not assume a private cluster, a paid API key, or a published scoreboard of model names. It also does not assume that yesterday's quota sentence is still printed on the project page. Every numeric claim about access must be copied from the page the facilitator opens during class.

What the facilitator may claim

Two availability claims are operator-supplied for this session: free model access, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode enters the lab as the open-source candidate the operator asked the class to inspect, not as a measured winner. The inspection is a documentation check plus a rerunnable row, not a bake-off against unnamed rivals.

Write no token count, hardware shape, or expiry date into the slides from an old memory. In the first ten minutes, open the current project documentation and paste only the sentences that page shows today. If the page is silent on a quota, leave the quota field blank rather than filling it from a blog post. A blank field is a valid result, and it keeps the later comparison from leaning on a stale number.

How the eighty-five minutes are split

Minutes 0 to 10 cover the live terms paste and a hard ban on recalled quotas.
Minutes 10 to 25 freeze the prompt fixture, hash the file, and store that hash.
Minutes 25 to 40 stamp the server fingerprint and place it beside the prompt hash.
Minutes 40 to 60 send one route call, or record an explicit skip when no key exists.
Minutes 60 to 75 compare two ledger rows and name the single lane that moved.
Minutes 75 to 85 write the limitation note and decide whether a second route is allowed.

Lane rules the room must keep

Lane A holds the prompt only

Lane A stores the task text, the input text, and a content hash of both fields together. Students may not improve the wording after that hash has already been written into the ledger. A clearer prompt becomes a new fixture id, not a silent edit of the first file. This rule stops the class from hiding a prompt change inside a host comparison.

Lane B holds the server only

Lane B stores the operating system name, the Python version, and a short hash of that pair. It does not store a story about how fast the machine felt during the exercise. If two students produce different fingerprints, they must not share one result row. A free server option, when the live page still offers one, is only another fingerprint source.

Lane C holds the route only

Lane C stores the route name, the call time, and the hash of the returned text. It does not store a quality score, a preference vote, or a claimed cost winner. If the call never left the laptop, the route name stays unset and the note explains why. An empty call is valid evidence that the lab stopped before anyone invented an output.

A worked skeleton students can rerun

The script below is a proposed lab skeleton, not an executed benchmark and not a product trial. It prints a schema row before any network call, so the class can review fields while the terms page is open. Replace each placeholder with a sentence you copied, and do not invent a quota when the page is quiet. The sample row uses an empty output on purpose, which keeps the first run offline.

# Proposed lab skeleton. Not an executed benchmark.
# Fill LIVE_TERMS from the documentation page opened in class.

from datetime import datetime, timezone
import hashlib
import json
import platform

LIVE_TERMS = {
    "source": "paste the documentation URL opened in class",
    "free_model_access": None,
    "free_server_option": None,
    "quota_text": "",
    "captured_at": datetime.now(timezone.utc).isoformat(),
}

FIXTURE = {
    "id": "fx-ledger-01",
    "task": "Rewrite the function comment in one sentence. Do not add features.",
    "input": "def add(a, b):\n    return a + b\n",
}

def fingerprint():
    raw = "|".join(
        [platform.system(), platform.release(), platform.python_version()]
    )
    digest = hashlib.sha256(raw.encode()).hexdigest()[:12]
    return {"host": raw, "sha256_12": digest}

def ledger_row(route_name, output_text, note):
    fixture_bytes = json.dumps(FIXTURE, sort_keys=True).encode()
    return {
        "fixture_id": FIXTURE["id"],
        "fixture_sha256": hashlib.sha256(fixture_bytes).hexdigest(),
        "route": route_name,
        "terms": LIVE_TERMS,
        "server": fingerprint(),
        "output_sha256": hashlib.sha256(output_text.encode()).hexdigest(),
        "output_chars": len(output_text),
        "note": note,
    }

if __name__ == "__main__":
    sample = ledger_row(
        "unset",
        "",
        "no call yet; row exists so the schema can be reviewed",
    )
    print(json.dumps(sample, indent=2))

Save the skeleton as ledger.py, then run the three review commands before any live call. The first command proves that the schema prints without a network call. The second command checks that the printed JSON is well formed. The third command hashes the script so a later edit is visible in the notes.

python3 ledger.py > row0.json
python3 -m json.tool row0.json >/dev/null && echo "json ok"
python3 - <<'PY'
import hashlib
from pathlib import Path
print(hashlib.sha256(Path("ledger.py").read_bytes()).hexdigest())
PY

Read two rows without building a scoreboard

Use the table when two students disagree about a result they both just produced. The table names one lane, and it names the only edit that lane allows. It does not name a better model, because this packet never collected a fair paired sample. Treat a void row as data about the lab design, not as a failure of the host.

What changed between the rows	Lane that owns the change	Edit that is allowed next
Prompt hash differs	Lane A	Keep the new fixture id and drop the comparison
Server hash differs	Lane B	Rerun both rows on one pinned host
Route name differs	Lane C	Record both names and do not retune the prompt
Quota sentence differs	Terms paste	Keep both pastes and do not average them
Terms fields are empty	Evidence gap	Stop scoring until the live paste exists
Output hash matches	No quality claim	Log stability only, not correctness

A matching output hash means the text stayed stable under that fixture and that host. It does not mean the comment was good, short, or safe enough to merge. Correctness still needs a human check, or a separate test this workshop does not replace. Say that limit every time a student treats a stable hash as a grade.

Four exercises with clocks attached

Exercise 1 freezes Lane A

Give the group fifteen minutes and one function that already returns the right value. The task line must forbid new features, so the model is not invited to refactor the file. Students hash the fixture and write that hash into the ledger before they discuss wording. If someone wants a clearer task, they create fx-ledger-02 and leave the first hash untouched.

Exercise 2 stamps Lane B

Give the group ten minutes to run the fingerprint function on the machine they will use. If the live documentation still shows a free server option, one pair may stamp that host instead. They still paste the terms sentence beside the fingerprint before they call anything. A fingerprint without a terms paste is incomplete, even when the process list looks familiar.

Exercise 3 attempts Lane C once

Give the group twenty minutes for a single call, or for a written skip with a reason. The skip is the right outcome when the page no longer shows free model access, or when no key exists. Students store the output hash, the character count, and a note of twenty words or fewer. They must not retry until the output looks smarter, because unmarked retries destroy the ledger.

Exercise 4 names the lane that moved

Give the group fifteen minutes to compare one row with a neighbor using the table above. They must name the single lane that moved, and they must name the edit that lane allows. If more than one lane moved, the comparison is void and the pair schedules a cleaner rerun. The facilitator counts void comparisons, because that count is the only class metric this workshop defines.

Limits that belong in the closing note

This ledger cannot price a vendor, size a fleet, or predict an allowance for next month. It cannot prove that an open-source host is faster, cheaper, or safer than some other host. It cannot repair a bad task, and it cannot bless output that merely stayed stable across two runs. Facilitators should read those limits aloud near the end, so notes do not grow a claim the rows cannot carry.

The class metric is the count of void comparisons, not a satisfaction score and not a latency trophy. A high void count means the room changed too many lanes, which is a teaching result rather than a product result. A low void count means the pins held for this hour, which is also not a quality prize. Either number is local to this room, this fixture, and this calendar date.

Who should skip this workshop

Procurement teams that need a signed benchmark should not file these rows as a vendor bid.
Anyone holding production secrets should not place those secrets inside the fixture text.
A class that cannot open the current documentation should wait, rather than teach from an old slide.
Readers who want a ranked list of model names will not find one, because no list was verified for this date.
Groups that need medical, legal, or financial advice from a model should choose a different exercise.

After the clock stops

If the page you opened still shows free model access and a free server option, point Lane C there. Keep the same fixture id, and do not retune the task while you try that documented project. If that page does not show those options, leave the product name out of the results file. Finish the ledger on a host you already control, and keep the rerunnable row as the souvenir.

Workshop: Cap a Shared Lab Hour With a Session Envelope in 75 Minutes

Finley Zhu — Thu, 08 Oct 2026 08:34:39 +0000

Workshop: Cap a Shared Lab Hour With a Session Envelope in 75 Minutes

A shared lab hour stays usable only when each student run carries a written session envelope before the first model call. The envelope records a turn cap, a local token budget, a timeout, and a concurrency limit that the harness can refuse. This workshop teaches that preflight in seventy-five minutes, with a checker students can rerun on their own laptops. It does not measure a vendor, and it does not treat any live quota as a permanent classroom constant.

The failure this hour is built to catch

Early October rooms often mix a classroom exercise with a contribution sprint, and both patterns share one scarce resource. Several notebooks then retry the same prompt until a demo looks finished for the person who started first. One laptop can open parallel loops while another student is still reading the worksheet and has not called yet. Shared capacity then looks slow, even though the defect is an unbounded client rather than a mysterious network.

How this differs from after-the-fact metering

Spend ledgers and latency charts explain a run that has already finished and left a log. This session starts earlier, and the harness blocks the call when the envelope is missing or internally inconsistent. Students leave with a file they can diff in git, not with a dashboard crop they cannot reproduce next week. The same checker runs against a mock client, so the lesson does not depend on a live allowance.

Seventy-five minute timetable

Keep the order below, and do not skip the refusal exercise near the middle of the hour. Acceptance-only demos hide the bug this workshop exists to teach, which is a client that never refuses. Pairs should rotate the keyboard every time the timetable moves into a new section of the hour. The instructor collects reason codes rather than model prose, because the gradeable artifact is the gate itself.

From minute 0 to minute 10, state the shared-hour constraint and project one unbounded retry trace on the board.
From minute 10 to minute 25, write the envelope fields and agree on class-local numbers for today's worksheet only.
From minute 25 to minute 45, run the checker on three fixtures, including two files that must fail closed.
From minute 45 to minute 60, place the checker in front of a stub client, then optionally attempt one live call.
From minute 60 to minute 75, compare two refused runs and write a three-line lab note in the shared repository.

Envelope fields the room must fill

Keep the envelope small enough that a pair can review every field in under two minutes. Four numeric limits plus an exercise id are enough for a first shared hour in this room. Extra fields tend to become unread comments that nobody checks again before the call leaves the laptop. Treat every number as class policy that the next cohort can revise in the lab note.

The max_turns field limits model round trips inside one exercise, and that count includes every automatic retry.
The local_token_budget field is the student's own ceiling, not a statement about any provider allowance or grant.
The timeout_seconds field bounds a single call so one hung socket cannot own the rest of the hour.
The max_in_flight field stops one laptop from opening several loops while the rest of the room waits.
The exercise_id field ties the envelope to today's worksheet so yesterday's accepted file cannot be reused by accident.

Class-local starting numbers for this seventy-five minute room are only a proposal, not a measured optimum. Use eight turns, two thousand local tokens, twenty seconds, and one in-flight call as the starting envelope. Shorten the token band if the worksheet asks for a single function, and record that reason in the note. Do not copy these numbers onto a slide as if they were a published vendor limit.

Trace shape students should recognize

Before writing envelopes, show a six-line trace from a stub log so the room can see fan-out without a live model. Each line carries a timestamp, a student id, an exercise id, and a turn index for the board. The sample below is synthetic teaching data, and it is not a capture from any production host. Ask pairs to count overlapping starts rather than to judge the prompt text inside the trace.

t=0  student=s1 exercise=lab-hour-03 turn=1 state=start
t=1  student=s1 exercise=lab-hour-03 turn=1 state=start
t=1  student=s1 exercise=lab-hour-03 turn=2 state=start
t=2  student=s2 exercise=lab-hour-03 turn=1 state=waiting
t=9  student=s1 exercise=lab-hour-03 turn=1 state=retry
t=9  student=s1 exercise=lab-hour-03 turn=3 state=start

In this fixture, student s1 has three starts overlapping while student s2 is still waiting on the first turn. That pattern is the defect the max_in_flight field is meant to refuse before any live call starts. Do not treat the timestamps as latency measurements, because the file was written by hand for the board.

Worked checker students can rerun

The script below is a teaching artifact meant for local fixtures on student laptops during the hour. It has not been executed against a production fleet for this article, and it does not call a model by itself. Run it on the pass file and the two fail files before anyone in the room discusses providers. If the output disagrees with the comments beside the commands, fix the fixture before you change the checker.

#!/usr/bin/env python3
"""Session envelope checker. Local teaching artifact, not a vendor benchmark."""

import json
import sys

REQUIRED = (
    "exercise_id",
    "max_turns",
    "local_token_budget",
    "timeout_seconds",
    "max_in_flight",
)

def load_envelope(path):
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)

def check(card, expected_exercise):
    errors = []
    for key in REQUIRED:
        if key not in card:
            errors.append(f"missing:{key}")
    if card.get("exercise_id") != expected_exercise:
        errors.append("exercise_id_mismatch")
    turns = card.get("max_turns")
    if not isinstance(turns, int) or turns < 1 or turns > 12:
        errors.append("max_turns_outside_class_band")
    budget = card.get("local_token_budget")
    if not isinstance(budget, int) or budget < 100 or budget > 4000:
        errors.append("local_token_budget_outside_class_band")
    timeout = card.get("timeout_seconds")
    if not isinstance(timeout, int) or timeout < 5 or timeout > 30:
        errors.append("timeout_seconds_outside_class_band")
    if card.get("max_in_flight") != 1:
        errors.append("max_in_flight_must_be_one")
    return errors

def main():
    if len(sys.argv) != 3:
        print("usage: check_envelope.py ENVELOPE.json EXERCISE_ID", file=sys.stderr)
        return 2
    errors = check(load_envelope(sys.argv[1]), sys.argv[2])
    if errors:
        print("REFUSE")
        for item in errors:
            print(item)
        return 1
    print("ACCEPT")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Passing fixture, saved as pass_envelope.json:

{
  "exercise_id": "lab-hour-03",
  "max_turns": 8,
  "local_token_budget": 2000,
  "timeout_seconds": 20,
  "max_in_flight": 1
}

Broken fixture, saved as missing_turns.json:

{
  "exercise_id": "lab-hour-03",
  "local_token_budget": 2000,
  "timeout_seconds": 20,
  "max_in_flight": 1
}

Commands for the room:

python3 check_envelope.py pass_envelope.json lab-hour-03
python3 check_envelope.py missing_turns.json lab-hour-03
python3 check_envelope.py pass_envelope.json lab-hour-04

The expected teaching outcomes are classroom labels only, and they are not measured metrics from a live service. The first command should print ACCEPT, and the other two should print REFUSE plus a reason code. A missing numeric field can emit two codes, one for absence and one for the band, and that pairing is expected. A different printout means the fixture or the Python version needs a look, not that a model misbehaved.

Exercise sequence

Exercise A, minutes 10–25

Each pair writes one valid envelope for lab-hour-03 and one intentionally broken envelope for the same checker. Broken envelopes should omit a required field, reuse yesterday's exercise id, or set max_in_flight above one. The useful result is a refuse code the other pair can predict before the command even runs. Prompt wording is out of scope for this segment, so leave the model client closed on every laptop.

Exercise B, minutes 25–45

Run the three commands and paste reason codes into the shared note without pasting secrets or prompts. Add one fixture that sets local_token_budget to 50000 and confirm the class band rejects that envelope. That rejection is local policy for this room, and it is not evidence about any provider allowance. If two pairs disagree on a reason code, diff the envelope files before anyone edits the checker.

Exercise C, minutes 45–60

Put the checker in front of a stub that sleeps only after ACCEPT and never opens a socket on REFUSE. Only after that gate works should one volunteer try a live call, and only inside the accepted timeout. Everyone else watches the reason code on the terminal, not the generated text from the optional call. If the live page cannot be confirmed that morning, stop at the stub and mark the live step skipped.

def gated_call(card, exercise_id, client):
    errors = check(card, exercise_id)
    if errors:
        return {"status": "refused", "errors": errors}
    return client.complete(
        max_turns=card["max_turns"],
        timeout_seconds=card["timeout_seconds"],
    )

The function above is whiteboard pseudocode, and students should implement the sleep-or-refuse stub themselves in class. Copying it without a failing test recreates the unread-comment problem this hour is trying to remove from the lab. A three-line test that expects refused for a stale exercise id is enough to lock the gate. Run that small test before any volunteer is allowed to open a live client from the room.

Where a free model path fits this hour

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Operator notes for this piece describe MonkeyCode as an open-source project with free model access and a free server option. A class may point the optional live step at those paths only after reading the current project page that morning. Students who will read or fork the code should confirm the license on that same page before they clone anything.

This article does not name models, freeze a token grant, describe hardware, or claim that free access will stay unchanged. If the live page disagrees with a slide, the live page wins and the stub exercise remains the gradeable path. Use a free server only as a shared place to run the checker when local Python installs are uneven. It is not a private production host, and it is not a reason to skip the refuse fixtures in the timetable.

Refusal decision table

Condition	Harness result	Class action
Required field missing	REFUSE	Fix the envelope, and do not retry a model
Exercise id is stale	REFUSE	Mint today's id and discard yesterday's envelope
In-flight value above one	REFUSE	Close extra notebooks before any live step
Live access terms unknown	Skip live step	Finish on the stub and record the gap
Envelope accepted and stub passes	Optional live call	One volunteer, inside the written timeout

Read the table from top to bottom, and stop at the first matching row before you take any other action. Do not average the rows into a score, because a single missing field is already a failed preflight. The last row permits one call from one volunteer, not a fan-out from the rest of the room. Record a skipped live step as a completed outcome when the access page cannot be confirmed.

Lab note template

Keep the note to six lines so review stays faster than the exercise itself for the instructor. Ask each pair to commit the note beside the fixtures before they leave the room today. The change_next_week line should name a fixture gap, not a feeling about model quality or speed. An acceptable example is to add an envelope whose timeout is four seconds so the lower band is tested.

exercise_id: lab-hour-03
class_band: turns<=12 tokens 100-4000 timeout 5-30 inflight=1
refuse_codes_seen: missing:max_turns, exercise_id_mismatch
stub_gate: pass
live_step: skipped|one_volunteer
change_next_week: Add an envelope whose timeout is four seconds.

Limitations

The two-thousand-token class band is a teaching cap, not a cost model and not a provider quota. The checker does not inspect prompt content, tool side effects, or whether data leaves the room. A student can still paste secrets into an accepted call, so the envelope is not a privacy control. Shared free capacity can still saturate for reasons this script cannot see, including other tenants on the same host.

Do not publish timings from one classroom as a benchmark, and do not compare models with this harness. The synthetic trace is hand-written, so its gaps are illustrations rather than observations from a running host. A missing fixture file should fail the command with a visible error, which the script currently leaves to the interpreter. Teach that traceback as an environment failure, not as a refused envelope, so students do not confuse the two.

Who should not use this hour

Skip this workshop if the room needs authenticated production traffic, regulated data, or a graded accuracy comparison between models. Skip it if students cannot install Python 3 and you also lack a shared server you are allowed to use. Skip the live segment when you cannot confirm today's access terms in writing before the session starts. A team that already enforces admission control at a gateway should not replace that gateway with this laptop script.

Close

Leave the three fixtures, the synthetic trace, and the checker in the class repository beside the timetable. Next week, change only the exercise id and one class-band edge, then rerun the same commands before any live client opens. If a shared optional client would help the last segment, read the current MonkeyCode access notes before you rely on them. Keep the stub path as the graded path even when that optional client is available to the room.

Workshop: Pack a Replay Packet Before a Free Lab Call in 75 Minutes

Finley Zhu — Thu, 24 Sep 2026 22:26:15 +0000

A shared free model path can support a classroom exercise, yet it should never become the grading oracle. Students should freeze a replay packet that stores the task, the fixture output, and the assertions before any live call. Instructors can then mark the same submission when that free path is slow, capped, or temporarily unreachable. This 75-minute workshop builds that packet and places a free server in an execution lane rather than in the score.

Why the packet comes before the call

Generated output is useful as a draft, and it is a poor source of truth for a grade. A later run can differ because the endpoint is busy, the prompt is underspecified, or the service returns a shorter completion. If the class waits for a live call before writing assertions, every outage becomes an incomplete lab. The packet reverses that order so the expected evidence already exists on disk before the first request leaves the room.

Recent public writing has spent more attention on agent demos than on how a reviewer repeats a result the next morning. That gap matters in a teaching lab, where a full room may share one free path and still need a comparable score. This workshop does not claim a benchmark, a token quota, a hardware profile, or a permanent free offer. It claims only that a frozen fixture plus three explicit lanes makes the same exercise rerunnable on a later date.

What students leave with

By the end of the session, each student can regenerate a local packet and pass three assertions without a network. A second lane may call a free model endpoint only after the packet already contains a frozen fixture. A third lane may run a small command on a free server, then compare stdout with the fixture hash. The live lanes stay optional, so a missing key does not block grading or a later replay.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as one place an instructor may obtain free model access and a free server option for those optional lanes. This draft does not name models, quotas, or hardware, because those details change and were not verified for this class date. Students should read the current offer on the class day before they export an endpoint or share a host.

75-minute timing

Use the blocks below as a wall clock, not as a suggestion to skip the fixture lane. Each block names an output so a facilitator can see whether the room is still on the fixture. These minutes are planning targets for a single 75-minute room, not timings measured from any previous cohort. If a block overruns, cut the optional live lanes first and keep the deliberate assertion-failure exercise intact.

Minutes 0–10: State the rule that the fixture lane is the only required score, then list the three fields every packet must contain.
Minutes 10–22: Students write a three-line fixture and three assertions for a status note that must contain the word READY.
Minutes 22–40: Students run the packing script, inspect the printed hash, and repair one deliberate assertion failure before moving on.
Minutes 40–52: Pairs review a neighbor packet and reject any file whose stored hash does not match the normalized fixture text.
Minutes 52–65: Optional lanes run only after the instructor confirms current terms for free model access and the free server option.
Minutes 65–75: Each student records one limitation, then reruns the local script with the network disabled and keeps the passing JSON.

Packet fields

Keep the schema small enough that a grader can read every field without asking a model for help. Version 1 stores the human task, the normalized fixture, its digest, and a short list of assertions. Optional lanes default to skipped, so a packet remains valid when no endpoint has been configured yet. Unknown assertion kinds fail closed, which stops a student from smuggling an unchecked rule into the file.

version is an integer, and this workshop uses version 1 only.
task stores the human instruction so a replay does not depend on chat history.
fixture_text is the expected body, normalized to one trailing newline.
fixture_sha256 is the hex digest of that normalized body.
assertions holds contains, max_lines, or exact_hash checks, and no other kinds.
lanes.fixture is required, while lanes.free_model and lanes.free_server default to skipped.

Worked example students can rerun

The script below is a teaching artifact that writes replay-packet.json and exits zero when the fixture lane passes. The optional lanes are not called in this published example, and they should stay skipped on the first run. Treat any live client as a later exercise that must follow provider documentation you verified during that same week. Do not add network code until exercise 4, because an early call would hide whether the local packet is sound.

#!/usr/bin/env python3
"""Pack a replay packet. Live lanes stay skipped unless you add them later."""

import hashlib
import json
import sys
from pathlib import Path

def normalize(text):
    return text.strip() + "\n"

def digest(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def build_packet(task, fixture, assertions):
    body = normalize(fixture)
    return {
        "version": 1,
        "task": task,
        "fixture_text": body,
        "fixture_sha256": digest(body),
        "assertions": assertions,
        "lanes": {
            "fixture": {"required": True, "status": "pending"},
            "free_model": {"required": False, "status": "skipped"},
            "free_server": {"required": False, "status": "skipped"},
        },
    }

def check(packet):
    errors = []
    text = packet["fixture_text"]
    if digest(text) != packet["fixture_sha256"]:
        errors.append("fixture hash mismatch")
    for item in packet["assertions"]:
        kind = item["kind"]
        if kind == "contains" and item["value"] not in text:
            errors.append("missing substring: " + item["value"])
        elif kind == "max_lines" and len(text.splitlines()) > int(item["value"]):
            errors.append("line count above cap")
        elif kind == "exact_hash" and digest(text) != item["value"]:
            errors.append("exact hash assertion failed")
        elif kind not in ("contains", "max_lines", "exact_hash"):
            errors.append("unknown assertion kind")
    return errors

def main():
    task = "Emit a three-line status note that includes the word READY."
    fixture = "status: READY\nlane: fixture\nexit: 0"
    packet = build_packet(task, fixture, [])
    packet["assertions"] = [
        {"kind": "contains", "value": "READY"},
        {"kind": "max_lines", "value": 3},
        {"kind": "exact_hash", "value": packet["fixture_sha256"]},
    ]
    errors = check(packet)
    packet["lanes"]["fixture"]["status"] = "pass" if not errors else "fail"
    Path("replay-packet.json").write_text(
        json.dumps(packet, indent=2) + "\n", encoding="utf-8"
    )
    print("errors", len(errors), "hash", packet["fixture_sha256"][:12])
    return 0 if not errors else 1

if __name__ == "__main__":
    sys.exit(main())

Local commands

Run the three commands from a clean directory, and disable the network if you want a stricter proof. The expected local result is a zero exit, a printed error count of zero, and two lanes left as skipped. If you change the fixture text after the hash is stored, the check fails and the process exits one. Students should trigger that failure once, restore the matching text, and keep both JSON files for the pair review.

python3 pack_replay.py
python3 -m json.tool replay-packet.json > /dev/null
python3 -c "import json; p=json.load(open('replay-packet.json')); assert p['lanes']['fixture']['status']=='pass'"

Failure you should force

After the first successful run, edit replay-packet.json and change one character inside fixture_text without updating the digest. Recompute the check in a short shell snippet, and confirm that the hash mismatch is reported before any lane is trusted. Students should restore the file by rerunning the packing script, rather than editing the digest to bless the broken text. That sequence shows why the digest is a guard against silent edits, not a decorative field copied from a tutorial.

python3 -c "import json,hashlib; p=json.load(open('replay-packet.json')); t=p['fixture_text']; print(hashlib.sha256(t.encode()).hexdigest()==p['fixture_sha256'])"

Exercise sequence

Work these steps in the listed order, and do not open a model client until the fourth exercise. Skip the fourth and fifth exercises when the instructor has not confirmed the current free-path terms. The first three exercises are the grade, and the last exercise is a peer check rather than a new feature. A student who finishes only the local steps has still completed the workshop outcome described above.

Write the fixture by hand, and do not ask a model for the expected text, because that would hide the oracle inside the tool you are trying to constrain.
Add one contains check, one max_lines check, and one exact_hash check, then run the script and keep the JSON.
Break the stored fixture on purpose, confirm a failed comparison, and write one sentence naming which guard fired.
Optional: if free model access is available, send only the task string, store the completion beside the fixture, and never overwrite fixture_text.
Optional: if a free server is available, print the fixture there and mark pass only when the stdout hash matches.
Exchange packets, and reject any file that lacks a hash, uses an unknown kind, or marks a live pass without a digest.

Decision table

Use the table when the room must choose a lane without turning a missing host into a failed grade. The entries are class rules written for this workshop, not measured limits taken from any provider. A match means the compared digest equals fixture_sha256, and a diverge means the side file differs. Never relax an assertion after reading a live completion, because that move lets the model rewrite the test.

Situation	Lane to score	What to record	Do not do
Network off or key missing	fixture	both live lanes skipped	Invent a completion
Free model returns text	fixture, plus a side file	match or diverge against the fixture digest	Replace the fixture with the completion
Free server prints stdout	fixture, then server	pass only on a hash match	Treat server uptime as the grade
Completion exceeds three lines	fixture	keep the original max_lines cap	Relax the cap after seeing model text
Offer terms unread today	fixture only	note that live lanes were not authorized	Copy a quota from an older article

How the free options fit, and where they stop

Free model access is a reasonable way to let students inspect a live completion after the oracle already exists. A free server is a reasonable place to run the print command when local machines differ and logs have been reviewed. Neither option should set the score, because availability and limits are not treated as fixed facts in this article. If either path is down on class day, the fixture lane still finishes the lab and preserves a comparable result.

Re-read the current terms before minute 52, since a saved screenshot is not evidence for a later cohort. Do not send credentials, personal data, or unpublished student work to a shared host until retention and access are checked. The packing script does not open a socket, which keeps the required lane independent of whichever host you add later. That separation is the lesson, and a branded endpoint does not change the order of fixture first and call second.

Limitations

Hash equality is strict, so a valid paraphrase looks like a failure if you grade the live completion instead of the fixture. The schema does not estimate tokens, latency, or cost, and it should not be reused as a billing or capacity report. Version 1 also ignores tool calls, file edits, and multi-turn chats, so it will not catch an agent that mutates a repository. The example was not executed against a live endpoint in this draft, so a local pass proves only the packing script.

Who should not use this approach

Skip this workshop if the learning goal is open-ended writing, where many different answers remain equally acceptable. Skip it when the lab requires production credentials, private customer data, or a shared host that nobody can audit. Skip it if students need a statistical quality score, because three string checks will not measure taste or coverage. Also skip it when the instructor cannot commit to a fixture before class, since the method fails if the expected text is still moving.

Close

Pack the fixture, hash it, and rerun the three local commands before the room discusses any endpoint. Keep the free model lane and the free server lane as optional evidence that you compare, not as text you trust. If a cohort already has MonkeyCode free model access and the free server option, use both only after a same-day terms check. Leave the fixture status as the grade either way, including when those optional lanes are skipped or unavailable.

Workshop: Teach a Write-Once Stamp Card for Agent Tools in 80 Minutes

Finley Zhu — Wed, 23 Sep 2026 17:34:18 +0000

Agent write tools fail quietly when retries replay the same mutation without a stable idempotency key. This workshop freezes that failure into an 80-minute lab with a stamp card, a local stub, and a replay assertion students can rerun. The method stays useful for teaching even if you never touch a hosted model product during class.

What this lab actually proves

Most public tool-calling demos stop after the model emits a function name plus JSON arguments for the host. That stopping point is not enough for POST, PATCH, PUT, or delete-style tools under host retry. The host may retry after a timeout while the first mutation already landed on the stub. Students then debug a double-applied fixture instead of learning a write-once contract they can actually test.

The core conclusion is simple enough to put on a whiteboard before anyone opens an editor. A write tool is not complete until the host can prove that identical stamps produce one side effect. Identical stamps must include the tool name, a canonical argument digest, and an idempotency key the model cannot silently omit. If any of those three fields is missing, the stub must fail closed instead of applying a mutation.

This article is a teaching outline, not a production payments design, and the sample server is a lab fixture. Treat every command below as a classroom rerun recipe unless you later replace the in-memory map with a real store. No metric in this outline is a product benchmark, and no timing claim refers to a hosted model endpoint.

The Write-Once Stamp Card

Print or paste this card so every pair fills it before a model proposes create_lab_order. The card is the grading surface; fluent model prose is not evidence that a mutation ran once. Students should keep the filled card beside the curl transcripts through the debrief block.

Stamp card fields

tool_name — exact function the host will allow, with no aliases or case folding.
arg_digest — SHA-256 of canonical JSON arguments, keys sorted, no extra whitespace.
idempotency_key — opaque string supplied by the caller, stable across one user intent.
side_effect_counter — integer the stub increments only on the first successful apply.
decision — one of APPLY, REPLAY_OK, or REJECT, copied from the stub body.

Decision table (lab oracle)

Incoming stamp	Store lookup	Required decision	Side-effect delta
Missing key	n/a	`REJECT`	0
New key, valid args	miss	`APPLY`	+1
Same key, same digest	hit	`REPLAY_OK`	0
Same key, different digest	hit	`REJECT`	0
Unknown tool name	n/a	`REJECT`	0

The table is the oracle for the whole session and should stay frozen while students edit code. If a student implementation returns APPLY twice for one key, the lab has failed regardless of how confident the model sounded. Grade the counter and the decision string, then ignore any narrative the model attached to the tool call.

Workshop clock

Keep a visible timer and refuse to open a model client until the curl baseline is green. The times below assume a small group that already reads HTTP status lines and JSON objects. Absolute beginners in JSON should get a longer schema block rather than a compressed demo.

Minutes 0–10 — Frame the failure

Show a timeout-and-retry story without naming a vendor, a model, or a hosted agent brand. A client posted create_lab_order, waited thirty seconds, then posted again with new JSON and no key. Ask students to write the observed inventory delta on paper before they see any code. Collect two or three guessed deltas, then state the oracle: without a stamp, the correct educational answer is undefined, therefore unfit for class credit.

Minutes 10–25 — Freeze the schema

Do not let the model invent argument shapes during the first coding block of the session. Hand out a frozen JSON Schema for one write tool only, and treat extra properties as a failed drill. A workable teaching slice is create_lab_order with sku, qty, and idempotency_key. Students must copy the schema into a file they will not edit until the debrief, because schema drift collapses the oracle table.

Proposed schema fixture, labeled unexecuted until students save it:

{
  "$id": "lab://create_lab_order/v1",
  "type": "object",
  "additionalProperties": false,
  "required": ["sku", "qty", "idempotency_key"],
  "properties": {
    "sku": { "type": "string", "minLength": 1, "maxLength": 32 },
    "qty": { "type": "integer", "minimum": 1, "maximum": 20 },
    "idempotency_key": { "type": "string", "minLength": 8, "maxLength": 64 }
  }
}

Minutes 25–50 — Build the stub and the stamp store

Students implement a local HTTP stub that never talks to a paid API during this block. The store is a process-local dictionary keyed by idempotency_key, guarded by a lock because retries can overlap. On APPLY, persist the digest and increment side_effect_counter before returning the order identifier. On REPLAY_OK, return the original body and leave the counter unchanged so a timeout retry is visible and safe. On REJECT, return HTTP 409 or 400 and leave the counter unchanged so a mutated payload cannot hijack a key.

Minutes 50–70 — Run the four replay drills

Each drill is a command the whole room can rerun from the same working directory. Grade the side_effect_counter, not the model's explanation of what it meant to do. If a pair cannot finish all four drills, keep Drill A and Drill B as the minimum passing packet and park Drill C as homework.

Minutes 70–80 — Debrief with the card

Students mark which drills their stub failed, using the oracle table instead of memory. Typical misses are canonical JSON that still depends on key order, keys generated inside the stub instead of the caller, and silent coercion of qty from string to integer. Record those misses on the stamp card so the next lab starts from evidence rather than from a fresh anecdote.

Worked example students can rerun

The following Python 3 fixture is a lab stub, not a framework and not a durable ledger. Save it as write_once_stub.py and run it on loopback only. Restarting the process resets the store, which is an intended limitation for an 80-minute room.

#!/usr/bin/env python3
"""Lab fixture: fail-closed write-once stub. Not a production store."""
from __future__ import annotations

import hashlib
import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Lock

ALLOWED_TOOL = "create_lab_order"
STORE: dict[str, dict] = {}
LOCK = Lock()
SIDE_EFFECTS = 0


def canonical_digest(args: dict) -> str:
    body = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(body.encode("utf-8")).hexdigest()


def decide(payload: dict) -> tuple[int, dict]:
    global SIDE_EFFECTS
    tool = payload.get("tool_name")
    args = payload.get("arguments")
    if tool != ALLOWED_TOOL or not isinstance(args, dict):
        return 400, {"decision": "REJECT", "reason": "unknown_tool_or_args"}
    key = args.get("idempotency_key")
    sku = args.get("sku")
    qty = args.get("qty")
    extra = set(args) - {"sku", "qty", "idempotency_key"}
    if extra or not isinstance(key, str) or not (8 <= len(key) <= 64):
        return 400, {"decision": "REJECT", "reason": "schema"}
    if not isinstance(sku, str) or not (1 <= len(sku) <= 32):
        return 400, {"decision": "REJECT", "reason": "schema"}
    if not isinstance(qty, int) or isinstance(qty, bool) or not (1 <= qty <= 20):
        return 400, {"decision": "REJECT", "reason": "schema"}
    digest_args = {"idempotency_key": key, "qty": qty, "sku": sku}
    digest = canonical_digest(digest_args)
    with LOCK:
        prior = STORE.get(key)
        if prior is None:
            SIDE_EFFECTS += 1
            record = {
                "decision": "APPLY",
                "arg_digest": digest,
                "side_effect_counter": SIDE_EFFECTS,
                "order_id": f"lab-{len(STORE) + 1}",
            }
            STORE[key] = record
            return 200, record
        if prior["arg_digest"] != digest:
            return 409, {
                "decision": "REJECT",
                "reason": "key_reuse_with_different_args",
                "side_effect_counter": SIDE_EFFECTS,
            }
        replay = dict(prior)
        replay["decision"] = "REPLAY_OK"
        replay["side_effect_counter"] = SIDE_EFFECTS
        return 200, replay


class Handler(BaseHTTPRequestHandler):
    def log_message(self, *_args) -> None:
        return

    def do_POST(self) -> None:
        if self.path != "/tool":
            self.send_error(404)
            return
        length = int(self.headers.get("Content-Length", "0"))
        raw = self.rfile.read(length)
        try:
            payload = json.loads(raw.decode("utf-8"))
        except (UnicodeDecodeError, json.JSONDecodeError):
            payload = {}
        status, body = decide(payload)
        encoded = json.dumps(body).encode("utf-8")
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(encoded)))
        self.end_headers()
        self.wfile.write(encoded)


if __name__ == "__main__":
    server = ThreadingHTTPServer(("127.0.0.1", 8088), Handler)
    print("write-once stub on http://127.0.0.1:8088/tool")
    server.serve_forever()

Start the stub in one terminal and leave it running through every drill:

python3 write_once_stub.py

Run the four drills from a second terminal without restarting the stub between calls. These curl commands are the reproducible artifact students should keep in version control next to the filled stamp card.

# Drill A — first apply (expect APPLY, counter 1)
curl -sS -D - http://127.0.0.1:8088/tool \
  -H 'Content-Type: application/json' \
  -d '{"tool_name":"create_lab_order","arguments":{"sku":"SKU-9","qty":2,"idempotency_key":"intent-alpha-01"}}'

# Drill B — retry with the same stamp (expect REPLAY_OK, counter still 1)
curl -sS http://127.0.0.1:8088/tool \
  -H 'Content-Type: application/json' \
  -d '{"tool_name":"create_lab_order","arguments":{"sku":"SKU-9","qty":2,"idempotency_key":"intent-alpha-01"}}'

# Drill C — same key, mutated qty (expect REJECT, counter still 1)
curl -sS -D - http://127.0.0.1:8088/tool \
  -H 'Content-Type: application/json' \
  -d '{"tool_name":"create_lab_order","arguments":{"sku":"SKU-9","qty":3,"idempotency_key":"intent-alpha-01"}}'

# Drill D — missing key (expect REJECT)
curl -sS -D - http://127.0.0.1:8088/tool \
  -H 'Content-Type: application/json' \
  -d '{"tool_name":"create_lab_order","arguments":{"sku":"SKU-9","qty":2}}'

A passing room has one APPLY, one REPLAY_OK, two rejects, and side_effect_counter equal to 1 after all four drills. Anything else is a teaching defect in the stub or in the client that minted the stamp. Do not “fix” a failing counter by restarting the process, because that erases the evidence the card is meant to capture.

Optional model lane after the stub is green

Do not invite a model into the loop until Drill A through Drill D pass on curl alone. After that, the model is only a JSON producer sitting behind the same schema and the same stamp card. The host still validates required fields, still computes arg_digest, and still refuses missing keys. That split is the lesson; fluent tool-call prose is not a substitute for the oracle table.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Classrooms that already have a local model or a vendor key should keep using that path for argument generation. Classrooms that do not can point the same host at MonkeyCode's free model access and free server option, then keep this stub as the only component allowed to mutate state. This article does not claim model names, token quotas, hardware, uptime, or benchmark numbers for that option, because those details must be checked on the project page at publish time. If you need a free-model lane for argument generation, read that current project page before class and keep this stub as the only mutation gate.

A minimal host wrapper, labeled as a proposal, looks like the following sketch. Replace generate_tool_json with whatever client you actually use after the curl baseline is already green.

# Proposal / unexecuted host sketch — wire your own generate_tool_json.
import json, urllib.request

def canonical_digest(args: dict) -> str:
    import hashlib
    body = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(body.encode("utf-8")).hexdigest()

def host_write(tool_json: dict) -> dict:
    args = tool_json["arguments"]
    if "idempotency_key" not in args:
        raise ValueError("fail closed: model omitted idempotency_key")
    stamp = {
        "tool_name": tool_json["tool_name"],
        "arg_digest": canonical_digest(args),
        "idempotency_key": args["idempotency_key"],
    }
    req = urllib.request.Request(
        "http://127.0.0.1:8088/tool",
        data=json.dumps(tool_json).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req) as resp:
        body = json.loads(resp.read().decode("utf-8"))
    body["stamp"] = stamp
    return body

If you use a free server to host the stub instead of loopback, bind it to a classroom-only network and keep the in-memory map. The stamp card does not become more true because the process moved off a laptop. Network placement is logistics; the oracle table is still the only passing condition.

Limitations

This lab ignores exactly-once delivery across process crashes, because the store lives in RAM and vanishes on restart. It also ignores partial downstream work, such as a payment capture that succeeded while an inventory write failed. Canonical JSON here does not normalize Unicode homoglyphs in sku, so two visually similar SKUs can mint different digests. Integer qty rejects bool, but a model that emits qty as a string still fails the schema on purpose.

The 80-minute clock assumes students can read HTTP status lines without a long detour into curl flags. Absolute beginners in JSON will overrun the schema block and should get a longer session rather than skipped drills. The drills do not measure latency, token cost, or model quality, and they should not be cited as a benchmark of any hosted product. They also do not prove safety for concurrent distinct keys under a multi-process deployment, because the lock is in-process only.

Who should not use this approach

Skip this outline if you need a real idempotency store for payments, inventory, or medical writes; use a durable key-value store with documented retention instead. Skip it if your tools are read-only, because a stamp card adds ceremony without a mutation to protect. Skip it if you want the model to retry until something happens to succeed, because that goal conflicts with fail-closed teaching. Skip it if you cannot pin a schema for even one tool, because the oracle table then has no meaning.

What to collect before you dismiss the room

Ask each pair for four artifacts only: the filled stamp card, the four curl transcripts, the final side_effect_counter, and one sentence naming the drill they failed first. That packet is enough to see whether the class learned write-once behavior or only learned to paste tool-calling snippets. If you later try the same packet against a model-produced JSON body, compare it to the curl baseline, not the other way around, so the stub remains the source of truth.

Workshop: Split Generation From Execution With a Two-Clock Tool Lab in 80 Minutes

Finley Zhu — Tue, 22 Sep 2026 15:50:14 +0000

A single elapsed-time figure for an AI-backed tool call still hides two different systems. Classroom labs should pin a two-clock card before anyone debates latency, cost, or correctness claims. Generation time belongs to the model path, while execution time belongs to the tool server. Mixing those clocks produces ungradeable homework, because a lucky live session is not a contract.

This eighty-minute workshop freezes a tool-call shape, then records both clocks in one ledger. Students leave a small harness they can rerun later without depending on live model variance. The grade attaches to those files, not to a demo that happened to work once. Each file is small enough to diff in review, which keeps weekend reruns comparable across machines.

What this lab is for

Tool-calling tutorials often print one duration beside a JSON payload and then call the demo finished. That number cannot tell a student whether the model stalled, the server stalled, or the path between them stalled. A teaching lab needs a split that survives weekend reruns, so grades attach to artifacts instead of luck. Without that split, two honest students can submit opposite conclusions from the same flaky live minute.

You will leave with three files: a frozen shape card, a two-clock recorder, and a local tool server. Students can start the server from a single command and grade traces from a JSONL ledger. The lab does not require a paid model endpoint on the first graded pass through the recorder. The notes stay useful if every commercial product name is later stripped from the text.

Timing box

Use this timing box as the session contract, and cut discussion first if a block starts to overrun.

00–10 min: name the mixed-clock failure and read the two-clock card aloud
10–25 min: freeze the tool-call shape against a tiny schema and three fixtures
25–45 min: wire generation and execution clocks in a JSONL recorder
45–65 min: rerun the worked inventory example and grade the traces
65–80 min: walk the failure matrix and decide who should skip this lab

Keep a visible timer on the projector. If a section overruns, cut discussion rather than the recorder, because the artifact is the grade surface.

The two-clock card

Pin this JSON document before any student is allowed to hit a live generation endpoint. The card is the assignment contract for the lab, not a dashboard widget for later screenshots. Required keys, forbidden keys, and clock ownership stay frozen even when the model path changes. Reviewers should reject traces that add fields to this card without a written amendment in the ledger.

{
  "lab_id": "two-clock-tool-lab-2026-09-22",
  "tool_name": "inventory.lookup",
  "shape_freeze": {
    "required_keys": ["sku", "warehouse"],
    "forbidden_keys": ["prompt", "raw_user_text"],
    "sku_pattern": "^[A-Z]{3}-[0-9]{4}$",
    "warehouse_enum": ["east", "west"]
  },
  "clocks": {
    "t_generate_ms": {
      "owner": "model_path",
      "includes_network_to_model": true
    },
    "t_execute_ms": {
      "owner": "tool_server",
      "includes_network_to_server": true
    }
  },
  "pass_rule": "both clocks recorded unless shape fails; shape_freeze holds; execute status is 200 or 404"
}

The pass rule is deliberately narrow so stopwatch luck cannot rescue a broken tool-call shape. A fast generation that emits a forbidden key still fails, because the lab grades the contract. Status 404 is allowed because unknown SKUs are valid execution outcomes, not recorder crashes. Status 500 is not allowed, because the frozen map should not throw when the shape already passed.

Exercise 1 — freeze the shape (15 minutes)

Students often let the model invent extra fields such as reason, comment, or a copied prompt. Those fields leak homework text into the tool server and make two traces incomparable after class. Freeze the shape first, then allow generation, or the later clocks will measure the wrong object. The next block is lab code for teaching, not production middleware and not a security boundary.

# shape_freeze.py — lab-only contract check; not a security boundary
import json
import re
import sys

REQUIRED = {"sku", "warehouse"}
FORBIDDEN = {"prompt", "raw_user_text"}
SKU = re.compile(r"^[A-Z]{3}-[0-9]{4}$")
WAREHOUSES = {"east", "west"}


def freeze_ok(payload: dict) -> list[str]:
    errors = []
    keys = set(payload)
    missing = REQUIRED - keys
    extra_forbidden = keys & FORBIDDEN
    if missing:
        errors.append(f"missing:{sorted(missing)}")
    if extra_forbidden:
        errors.append(f"forbidden:{sorted(extra_forbidden)}")
    sku = payload.get("sku")
    if not isinstance(sku, str) or not SKU.match(sku):
        errors.append("sku_pattern")
    if payload.get("warehouse") not in WAREHOUSES:
        errors.append("warehouse_enum")
    return errors


if __name__ == "__main__":
    data = json.loads(sys.stdin.read())
    errs = freeze_ok(data)
    print(json.dumps({"ok": not errs, "errors": errs}))
    raise SystemExit(0 if not errs else 2)

Issue three fixtures before opening any model path, and refuse to continue until all three behave.

Fixture A is a valid WID-2044 lookup against the east warehouse and must print ok: true.
Fixture B leaks prompt text into the payload and must exit nonzero with a forbidden-key error.
Fixture C uses a lowercase sku and must fail the pattern check even when other keys look fine.

echo '{"sku":"WID-2044","warehouse":"east"}' | python shape_freeze.py
echo '{"sku":"WID-2044","warehouse":"east","prompt":"need stock"}' | python shape_freeze.py
echo '{"sku":"wid-2044","warehouse":"east"}' | python shape_freeze.py

Expected exits are 0, then 2, then 2. Students who continue after a green Fixture A alone have not frozen the contract.

Exercise 2 — record two clocks (20 minutes)

The recorder never stores a single elapsed_ms field as the grade for a student attempt. It stores two clocks and a shape verdict, then writes one JSON line per attempt into a ledger. A missing execute clock is valid when shape freeze fails, because the server must not see bad payloads. Students who print only the sum of both clocks fail the exercise even if both systems look healthy.

# recorder.py — generation clock and execution clock stay separate
from __future__ import annotations

import json
import time
import urllib.request
from pathlib import Path

from shape_freeze import freeze_ok

LEDGER = Path("two_clock_ledger.jsonl")


def timed(fn):
    start = time.perf_counter()
    result = fn()
    ms = int((time.perf_counter() - start) * 1000)
    return result, ms


def generate_tool_call(user_text: str) -> dict:
    # Lab stub: swap later for a live free-model call.
    # First graded rerun stays deterministic.
    if "inject-prompt" in user_text:
        return {
            "sku": "WID-2044",
            "warehouse": "east",
            "prompt": user_text,
        }
    if "widget" in user_text.lower():
        return {"sku": "WID-2044", "warehouse": "east"}
    return {"sku": "UNK-0000", "warehouse": "east"}


def execute_tool(payload: dict, base: str) -> dict:
    body = json.dumps(payload).encode()
    req = urllib.request.Request(
        base + "/inventory/lookup",
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=3) as resp:
        return {"status": resp.status, "body": json.loads(resp.read().decode())}


def run_attempt(user_text: str, base: str) -> dict:
    payload, t_generate_ms = timed(lambda: generate_tool_call(user_text))
    shape_errors = freeze_ok(payload)
    if shape_errors:
        record = {
            "user_text": user_text,
            "payload": payload,
            "shape_errors": shape_errors,
            "t_generate_ms": t_generate_ms,
            "t_execute_ms": None,
            "pass": False,
        }
    else:
        result, t_execute_ms = timed(lambda: execute_tool(payload, base))
        record = {
            "user_text": user_text,
            "payload": payload,
            "shape_errors": [],
            "t_generate_ms": t_generate_ms,
            "t_execute_ms": t_execute_ms,
            "status": result["status"],
            "pass": result["status"] in (200, 404),
        }
    with LEDGER.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record) + "\n")
    return record

The sum remains a convenience field they may compute later, never the source of truth for grading. Keep generate_tool_call replaceable so a live free-model path can land without rewriting the ledger schema. Timeouts stay at three seconds during class so a hung server does not consume the remaining workshop.

Recorder checks before the live swap

Ledger lines must include t_generate_ms on every attempt, including shape failures.
Ledger lines must set t_execute_ms to null when shape_errors is not empty.
No graded line may introduce elapsed_ms as a replacement for the two clocks.
pass may be true only for status 200 or 404 after a clean shape.

Worked example students can rerun (20 minutes)

Stand up a local inventory server that answers from a frozen map instead of a live catalog. The map is the oracle for this lab, so weekend reruns stay comparable across student machines. Unknown SKUs return 404, which still records t_execute_ms and can pass the narrow rule above. Do not add authentication, sleeps, or random stock, because those extras destroy rerun equality.

# lab_server.py — frozen inventory map, not a product catalog
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

STOCK = {
    ("WID-2044", "east"): 12,
    ("WID-2044", "west"): 0,
}


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        payload = json.loads(self.rfile.read(length) or b"{}")
        key = (payload.get("sku"), payload.get("warehouse"))
        if key not in STOCK:
            self.send_response(404)
            body = {"error": "unknown_sku"}
        else:
            self.send_response(200)
            body = {"sku": key[0], "warehouse": key[1], "qty": STOCK[key]}
        raw = json.dumps(body).encode()
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def log_message(self, fmt, *args):
        return


if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Start order for a clean rerun

python lab_server.py
echo '{"sku":"WID-2044","warehouse":"east"}' | python shape_freeze.py
python -c "from recorder import run_attempt; print(run_attempt('Need widget stock', 'http://127.0.0.1:8765'))"
python -c "from recorder import run_attempt; print(run_attempt('inject-prompt restock', 'http://127.0.0.1:8765'))"
python -c "from recorder import run_attempt; print(run_attempt('Need unknown item', 'http://127.0.0.1:8765'))"

Expected ledger behavior is part of the grade, not a hint students may skip after a green demo. The first attempt records both clocks and should pass if the stub emits WID-2044 for the widget prompt. The planted reject path must appear in the ledger with t_execute_ms set to null and pass false. The unknown-item attempt should pass with status 404, proving execution outcomes are not the same thing as recorder crashes.

When students later swap the stub for a live model path, the server and shape card stay frozen. That swap is the only moving part, which is the entire point of the two-clock split. If the live path is unavailable, the stub still produces a complete and reviewable homework packet. Teaching assistants should diff ledger keys, not prose writeups, when two submissions disagree about latency.

Where a free model path and a free server fit

Some teaching setups need a live generation path without asking every student for a paid key. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which can replace the generation stub later. The same free server option can host the inventory map if a local port is awkward in the classroom.

Treat those two availability claims as the only product facts this lab is willing to record. Do not pin model names, quotas, hardware notes, or promised duration onto the two-clock card itself. Those details are not part of the assignment contract, and inventing them would make traces incomparable. Keep the product name out of the pass rule so a vendor outage cannot redefine a correct shape.

Failure matrix

Classify planted failures with this matrix before anyone writes a narrative about how the demo felt. Each row names a clock owner, a forbidden response, and a replacement action students can actually run. Three planted failures are enough for an eighty-minute session; extra stories consume the remaining clock.

Symptom	Likely clock	Do not do	Do instead
Ledger has `t_execute_ms: null` and shape errors	generation / contract	retry until JSON looks nicer	fail the attempt; restore the schema
Both clocks large, status 200	mixed network	quote only the sum in a report	publish both clocks as separate fields
Fast generate, slow execute, 200	tool server	blame the model path	profile the frozen map and host
Slow generate, fast execute, 200	model path	restart the inventory server	keep the server; change only generation
Status 500 with a valid shape	tool server	loosen `freeze_ok`	keep freeze; fix the handler
Status 404 with `UNK-0000`	tool server (valid)	mark the lab broken	accept 404 as a passing execution

The matrix is the last graded artifact in the packet, sitting beside the JSONL ledger and shape card. A student who only screenshots a terminal without filling the matrix has not finished the workshop.

Limitations and who should skip this lab

This workshop is a classroom contract, not an observability platform and not a load-test harness. The recorder uses time.perf_counter on one machine, so it cannot separate server compute from local scheduling noise. The HTTP server is a frozen map, so it will not teach pooling, heavy tails, or authentication design. Clock values are teaching signals for attribution, not service objectives you should paste into a status page.

Skip this lab if you need production SLOs, multi-region probes, or a security review of tool calling. Skip it if your course cannot accept a local stub as the first graded path through generation. Skip it if the assignment requires named models, published quotas, or a vendor comparison table. Those claims sit outside this contract, and forcing them back in would reopen the mixed-clock problem.

The two-clock card also refuses to measure token spend, because spend is a different ledger entirely. Mixing token counts into latency traces recreates the original confusion this workshop exists to prevent. If a later course needs spend control, pin a separate card instead of extending this one.

Close

Pin the shape, split the clocks, and grade the ledger rather than the live demo window. Students can rerun the inventory example from the five commands above on any quiet evening. If you want a live generation path after the stub passes, try MonkeyCode's free model access next. Leave that access on the generation clock only, and keep the frozen inventory server in place.

Workshop: Pin an Independent Acceptance Card for AI-Written API Clients in 90 Minutes

Finley Zhu — Mon, 21 Sep 2026 14:04:53 +0000

AI-written API clients often ship with green tests the same model drafted, which makes those tests a weak merge gate. The stronger control is an independent acceptance card: frozen fixtures, status contracts, and a latency floor the model does not author. This ninety-minute workshop gives instructors a timed outline, three exercises, and a worked example students can rerun locally or on a shared runner. The room should treat the card as the only merge signal that counts, even when the generated client looks tidy.

Why generated tests are a circular gate

Most coding assistants will happily emit a client module and a pytest file that asserts the client's own happy path. Those assertions usually replay the same JSON the prompt already described, so they rarely catch missing headers, wrong status mapping, or silent retries. Instructors see this pattern in lab submissions: the demo works on one laptop and fails when another student hits a slower shared host. An independent card flips the order of work, because the contract exists before any model output is allowed to land.

Community threads this week keep returning to a related gap: the tests used to score model output no longer bound the behavior teams actually ship. This workshop stays smaller than those essays and isolates one failure mode, a generated HTTP wrapper that looks complete while violating a tiny public contract. You will not rank models here, and you will not debate productivity culture. You will freeze a card, forbid the model from editing it, and score only that card.

What belongs on the card

Keep the card small enough that a student can read it in four minutes without a lecture. Each field must be checkable by a script that never imports the generated client as a source of truth. If a check needs to call a private helper inside client.py, it is not independent and should be rewritten.

The card should include:

base_url pinned to a fixture process, never to a live vendor host during class
paths with method, required headers, and a JSON Schema-style body predicate
status_map that names which client exceptions must fire on 4xx and 5xx
latency_ms_p95 as a classroom floor against the fixture, not as a production SLO
forbidden_behaviors such as swallowing errors, retrying POST, or following redirects on POST

Write the card in git before anyone opens a chat window. Students may generate clients, helpers, and extra tests, but the card file stays read-only on the exercise branch.

Workshop clock (90 minutes)

Use this schedule as a hard ceiling so the debrief is not squeezed off the timetable.

0–10 min — Frame the failure. Show a green pytest file that never sent a request. Ask what evidence is missing.
10–25 min — Exercise 1: freeze the card. Pairs write acceptance_card.json from a one-page API note. No client code yet.
25–50 min — Exercise 2: generate against a locked card. Students produce a client. The card is instructor-owned and not an editable prompt field.
50–75 min — Exercise 3: run the card twice. Once on the laptop, once on a shared runner. Diff the two receipts.
75–90 min — Debrief. Collect the three most common card failures and mark which ones are merge blockers.

If a pair finishes early, they add one forbidden behavior rather than polishing the client. Polishing the client is how rooms accidentally reintroduce circular tests.

Exercise 1 — Freeze the card before the client exists

Give each pair a short API note, not an OpenAPI dump. The note should fit on one slide so students cannot hide behind generated spec noise. Instructors should walk the room once, then stay quiet while pairs argue about status mapping.

Example note for the room:

POST /v1/widgets creates a widget and must send Idempotency-Key
Success body is { "id": string, "ok": true }
400 must raise ClientError; 500 must raise ServerError
Fixture p95 must stay under 250 ms on the classroom runner
POST must not retry and must not follow redirects

Pairs then fill a JSON card. Reject cards that encode library names, retry counts inside the client, or prose such as "looks reasonable." A card is a predicate, not a design document, and predicates have to fail closed.

Exercise 2 — Generate the client with the card locked

Students may use any coding assistant they already have, including a key-free model if the lab does not issue paid credentials. The assistant can write client.py, but acceptance_card.json and test_acceptance.py remain instructor-owned files. If a student pastes the card into the prompt, they still cannot commit a card change on the exercise branch.

A useful prompt constraint for the room is boring and explicit. Tell the model the public function names and nothing else about scoring.

Write client.py only.
Public API: WidgetClient.create_widget(payload: dict, idempotency_key: str) -> dict
Raise ClientError on HTTP 4xx. Raise ServerError on HTTP 5xx.
Do not add retries. Do not follow redirects on POST.
Do not write tests. Do not modify acceptance_card.json.

Collect the generated files without reading them first. The point of this block is to separate authoring from scoring, not to workshop prompt style.

Exercise 3 — Two receipts, one card

Run the same instructor tests against the fixture on the laptop and on one shared process. The shared run exists to catch environment stories that a single laptop hides, such as DNS delay, proxy injection, or a client that binds to localhost only. Students compare receipts, not vibes, and they record pass or fail per card field.

Suggested receipt fields:

card_sha256 so later debate cannot swap the contract
fixture_pid or container id for the shared process
per-path status, schema_ok, header_ok, p95_ms
forbidden_hit as a boolean list, not a paragraph

If local pass and remote fail disagree, the pair debugs the receipt before they debug the client. That order keeps the room from rewriting tests until the green bar returns.

Worked example students can rerun

The following files are a workshop fixture, not a production service. They are labeled examples so a room can rerun them without claiming a live vendor result. Create a directory widget_lab/ and keep the card outside the generated module.

`acceptance_card.json`

{
  "base_url": "http://127.0.0.1:8765",
  "latency_ms_p95": 250,
  "paths": [
    {
      "id": "create_widget",
      "method": "POST",
      "path": "/v1/widgets",
      "required_headers": ["Idempotency-Key", "Content-Type"],
      "ok_status": 201,
      "body_required_keys": ["id", "ok"],
      "body_ok_type": "boolean"
    }
  ],
  "status_map": {
    "400": "ClientError",
    "500": "ServerError"
  },
  "forbidden_behaviors": ["retry_post", "follow_redirect_on_post"]
}

`fixture_server.py`

This fixture is intentionally slow by 40 ms so latency is visible, and it refuses POST retries by counting Idempotency-Key values.

#!/usr/bin/env python3
"""Workshop fixture. Example only; not a production API."""
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import time

SEEN_KEYS = {}

class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def _json(self, code, payload):
        raw = json.dumps(payload).encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_POST(self):
        time.sleep(0.04)
        if self.path != "/v1/widgets":
            return self._json(404, {"ok": False})
        key = self.headers.get("Idempotency-Key")
        ctype = self.headers.get("Content-Type", "")
        if not key or "application/json" not in ctype:
            return self._json(400, {"ok": False, "error": "missing_headers"})
        SEEN_KEYS[key] = SEEN_KEYS.get(key, 0) + 1
        if SEEN_KEYS[key] > 1:
            return self._json(400, {"ok": False, "error": "retry_detected"})
        length = int(self.headers.get("Content-Length", "0"))
        body = json.loads(self.rfile.read(length) or b"{}")
        if "name" not in body:
            return self._json(400, {"ok": False, "error": "missing_name"})
        return self._json(201, {"id": "wgt_123", "ok": True})

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

`test_acceptance.py`

These tests talk to the fixture through the student client, then assert card fields. They do not import expected JSON from the client module.

"""Instructor-owned acceptance tests. Example lab code; rerun locally."""
import json
import statistics
import time
from pathlib import Path

import pytest

from client import ClientError, ServerError, WidgetClient

CARD = json.loads(Path("acceptance_card.json").read_text())

@pytest.fixture(scope="module")
def client():
    return WidgetClient(CARD["base_url"])

def test_create_widget_matches_card(client):
    path = CARD["paths"][0]
    samples = []
    last = None
    for i in range(8):
        t0 = time.perf_counter()
        last = client.create_widget({"name": "demo"}, idempotency_key=f"k-{i}")
        samples.append((time.perf_counter() - t0) * 1000)
    p95 = sorted(samples)[int(0.95 * (len(samples) - 1))]
    assert last["ok"] is True
    assert isinstance(last["id"], str) and last["id"]
    assert p95 <= CARD["latency_ms_p95"], p95

def test_400_maps_to_client_error(client):
    with pytest.raises(ClientError):
        client.create_widget({}, idempotency_key="bad-body")

def test_post_does_not_retry(client, monkeypatch):
    calls = {"n": 0}
    real = client._transport

    def wrapped(*args, **kwargs):
        calls["n"] += 1
        return real(*args, **kwargs)

    monkeypatch.setattr(client, "_transport", wrapped)
    client.create_widget({"name": "once"}, idempotency_key="no-retry")
    assert calls["n"] == 1

Reference client shape (example, not a scored solution)

Students replace this file. The _transport hook exists so the no-retry test can observe calls without scraping logs.

"""Example student-shaped client. Not a scored solution."""
import json
from urllib.error import HTTPError
from urllib.request import Request, urlopen

class ClientError(Exception):
    pass

class ServerError(Exception):
    pass

class WidgetClient:
    def __init__(self, base_url):
        self.base_url = base_url.rstrip("/")

    def _transport(self, req):
        return urlopen(req, timeout=3)

    def create_widget(self, payload, idempotency_key):
        data = json.dumps(payload).encode("utf-8")
        req = Request(
            self.base_url + "/v1/widgets",
            data=data,
            method="POST",
            headers={
                "Content-Type": "application/json",
                "Idempotency-Key": idempotency_key,
            },
        )
        try:
            with self._transport(req) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except HTTPError as exc:
            if 400 <= exc.code < 500:
                raise ClientError(str(exc.code)) from exc
            if exc.code >= 500:
                raise ServerError(str(exc.code)) from exc
            raise

Rerun commands for the room:

python fixture_server.py &
python -m pytest test_acceptance.py -q

A pair has finished Exercise 3 when they can paste two receipts with the same card_sha256 and explain every mismatch in one sentence each. If they cannot explain a mismatch, they do not get to change the client yet.

Decision table for merge blockers

Card field	Local fail	Shared-runner fail	Merge rule
Missing `Idempotency-Key`	Client omitted header	Proxy stripped header	Block either way; fix client first, then re-run remote
Body missing `ok`	Model invented a different JSON	Fixture and laptop disagree	Block; schema is the card, not the model
p95 above 250 ms	Laptop is overloaded	Shared host is noisy	Warn once, block on second consecutive remote fail
POST retried	Client-level retry loop	Library default retry	Block; forbidden behavior is not a performance knob
500 mapped to `ClientError`	Status map ignored	Partial deploy of student file	Block; exception types are part of the card

The table is the debrief artifact. Instructors should fill the last column with the room, then freeze it for the rest of the term so later labs do not renegotiate merge rules mid-demo.

Where a free model and a free server actually help

The method does not require a particular vendor. It does require two lab constraints that many classrooms lack: a model students can call without purchasing keys, and one shared process that every laptop can hit. MonkeyCode is an open-source coding assistant with free model access and a free server option, which is enough to host the fixture and to generate client.py during Exercise 2. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the free server as the remote receipt target, not as a claim about production capacity. Keep the fixture tiny, pin the card in git, and treat any extra product UI as out of scope for the ninety minutes. If the room already has a campus runner and campus model access, skip the product entirely and keep the card.

Limitations

This card does not prove the client is correct against a live vendor, because the fixture is a classroom double with a fixed 40 ms pause. It also does not replace contract testing tools that already generate consumer pacts, schema registries, or recorded traffic from staging. Shared free hosts add noise, so a single p95 miss is a warning rather than a diagnosis of the generated code.

Students can still cheat by editing the card if the branch is not protected, and a model can still write a client that passes the card while leaking secrets in headers. Eight latency samples are a teaching device, not a statistically serious percentile. Label those limits on the slide so nobody files the receipt as a benchmark.

Who should skip this workshop

Skip this outline if the team already merges on Pact, OpenAPI-generated servers, or a staging replay harness that authors did not generate in the same session. Skip it if the goal is a model leaderboard, a token-cost study, or an incident review for a production outage. Skip it for junior students who have not yet issued an HTTP request without a tutorial copy-paste, because the card will hide the networking basics they still need.

The useful close for instructors is operational, not promotional: lock the card, generate the client, and keep the only scored artifact outside the model's edit window. If a lab still needs a key-free model and one shared fixture host, the free model access and free server option above are sufficient to rerun this example with the room.

Workshop: Catch Shared-Host Drift With a Five-Probe Floor Card in 85 Minutes

Finley Zhu — Sun, 20 Sep 2026 12:00:24 +0000

Shared inference hosts fail quietly when a probe set is missing, not when a dashboard still looks green. A five-probe floor card records the cheapest behaviors you still require after a host, prompt, or routing change. This workshop times that card at eighty-five minutes, with rerunnable files and a pass-or-fail ledger. Students leave with a JSON pack, a local scorer, and a printed floor they can recheck after every lab swap.

Why a floor card beats a chat anecdote

Anecdotes from one lucky prompt hide regressions that only appear on the second or third frozen case. Shared hosts also move under you, because capacity, routing, and hidden system prompts are not a public version pin. You need a frozen input set, a deterministic scorer, and a stored floor, or next week's run cannot be compared at all. The method below treats the model as an untrusted function with a tiny, documented surface.

Measurement talk in developer circles often outruns the tests that still discriminate. When a host starts returning fluent prose, empty objects, or a constant label, yesterday's demo stops being evidence. A floor card is deliberately small so a classroom can finish it, not so a vendor can be ranked. If the card cannot fail, it cannot teach anything useful about drift.

What this workshop is not

This is not a leaderboard, a latency study, or a claim about production model quality on a named system. It is a teaching lab that detects silent capability loss on one narrow, field-shaped task. If your team already runs a versioned eval platform with owners and an SLA, skip this outline and keep that platform. If you cannot freeze inputs because the task is open-ended prose, shrink the contract before you schedule the lab.

Workshop clock

Keep a visible timer for eighty-five minutes and refuse to expand the probe set during the first pass through the files.

0–10 min — Frame the failure. Name one task, one JSON contract, and one host URL environment variable.
10–25 min — Write the probe pack. Freeze five inputs, expected fields, and scoring modes on disk.
25–45 min — Run the worked example. Score against a file fixture before any remote host is allowed.
45–65 min — Exercise: point at a shared host. Keep the scorer identical and record a second floor card.
65–80 min — Exercise: break one probe. Mutate a prompt or response shape and show the ledger turning red.
80–85 min — Recap limits. List who should not ship this card as a quality gate.

Artifact layout

Students should create a directory they can zip, copy to another machine, and rerun without editing expects.

probe-floor/
  probes.json
  score_probes.py
  floor_card.json
  Makefile

The Makefile is the only class entrypoint, which stops ad-hoc flags from becoming the real curriculum.

.PHONY: fixture remote diff

fixture:
    python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json

remote:
    python3 score_probes.py --probes probes.json --base-url "$$PROBE_BASE_URL" --out floor_card.remote.json

diff:
    python3 score_probes.py --diff floor_card.json floor_card.remote.json

Probe pack schema

Keep scoring modes boring on purpose. Exact field match plus JSON parse success catch more host drift than a long prose rubric.

{
  "task": "semver_bump_from_diff",
  "contract": {
    "type": "object",
    "required": ["bump", "reason"],
    "properties": {
      "bump": {"enum": ["major", "minor", "patch", "none"]},
      "reason": {"type": "string", "minLength": 8, "maxLength": 160}
    }
  },
  "floor": {"min_pass": 5, "max_parse_fail": 0},
  "probes": [
    {
      "id": "P1_docs_patch",
      "input": {"diff_summary": "docs: fix typo in README install block"},
      "expect": {"bump": "patch"}
    },
    {
      "id": "P2_optional_field_minor",
      "input": {"diff_summary": "feat: add optional timeout_ms to ClientConfig"},
      "expect": {"bump": "minor"}
    },
    {
      "id": "P3_removed_field_major",
      "input": {"diff_summary": "breaking: remove ClientConfig.retry_count"},
      "expect": {"bump": "major"}
    },
    {
      "id": "P4_empty_diff_none",
      "input": {"diff_summary": ""},
      "expect": {"bump": "none"}
    },
    {
      "id": "P5_chore_none",
      "input": {"diff_summary": "chore: reformat imports with no behavior change"},
      "expect": {"bump": "none"}
    }
  ]
}

Five probes are a floor, not coverage, and they exist to fail closed when a host collapses. Watch for fluent prose, empty JSON, or a constant minor returned for every distinct case. If a pair wants a sixth probe during the first hour, park it in a notes file instead of changing the pack.

Worked example students can rerun

Label: this example is a local teaching fixture, not a measured vendor benchmark and not a claim about any hosted model. The file backend returns canned JSON so the scorer can be graded without a network round trip. Remote calls below use a lab default path; change that path to match whatever route your host actually documents.

# score_probes.py — teaching example, not a production eval platform
from __future__ import annotations

import argparse, json, sys, urllib.request
from pathlib import Path

SYSTEM = (
    "Return only JSON with keys bump and reason. "
    "bump must be major, minor, patch, or none."
)

def load_probes(path: Path) -> dict:
    return json.loads(path.read_text())

def complete_file(probe: dict) -> str:
    bump = probe["expect"]["bump"]
    return json.dumps({"bump": bump, "reason": f"fixture:{probe['id']}"})

def complete_http(base: str, probe: dict, timeout: float = 30.0) -> str:
    payload = json.dumps({
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(probe["input"])},
        ]
    }).encode()
    req = urllib.request.Request(
        base.rstrip("/") + "/v1/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        body = json.loads(resp.read().decode())
    return body["choices"][0]["message"]["content"]

def score_one(contract: dict, probe: dict, raw: str) -> dict:
    row = {"id": probe["id"], "pass": False, "parse_ok": False, "detail": ""}
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        row["detail"] = "not_json"
        return row
    row["parse_ok"] = True
    if set(contract["required"]) - set(data):
        row["detail"] = "missing_keys"
        return row
    if data.get("bump") != probe["expect"]["bump"]:
        row["detail"] = f"bump:{data.get('bump')}"
        return row
    reason = data.get("reason", "")
    if not isinstance(reason, str) or not (8 <= len(reason) <= 160):
        row["detail"] = "reason_len"
        return row
    row["pass"] = True
    row["detail"] = "ok"
    return row

def run(probes: dict, base_url: str) -> dict:
    rows = []
    for probe in probes["probes"]:
        raw = (
            complete_file(probe)
            if base_url.startswith("file:")
            else complete_http(base_url, probe)
        )
        rows.append(score_one(probes["contract"], probe, raw))
    passed = sum(1 for r in rows if r["pass"])
    parse_fail = sum(1 for r in rows if not r["parse_ok"])
    floor = probes["floor"]
    return {
        "task": probes["task"],
        "passed": passed,
        "parse_fail": parse_fail,
        "floor_ok": passed >= floor["min_pass"] and parse_fail <= floor["max_parse_fail"],
        "rows": rows,
    }

def diff_cards(a: dict, b: dict) -> int:
    print(f"local_floor_ok={a['floor_ok']} remote_floor_ok={b['floor_ok']}")
    ids = {r["id"]: r for r in a["rows"]}
    rc = 0
    for row in b["rows"]:
        prior = ids.get(row["id"], {})
        if prior.get("pass") and not row["pass"]:
            print(f"REGRESS {row['id']} {prior.get('detail')} -> {row['detail']}")
            rc = 1
        elif prior.get("pass") != row["pass"]:
            print(f"CHANGE  {row['id']} pass {prior.get('pass')} -> {row['pass']}")
            rc = 1
    return rc

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--probes")
    p.add_argument("--base-url")
    p.add_argument("--out")
    p.add_argument("--diff", nargs=2)
    args = p.parse_args()
    if args.diff:
        a = json.loads(Path(args.diff[0]).read_text())
        b = json.loads(Path(args.diff[1]).read_text())
        return diff_cards(a, b)
    pack = load_probes(Path(args.probes))
    card = run(pack, args.base_url)
    Path(args.out).write_text(json.dumps(card, indent=2) + "\n")
    print(json.dumps({"floor_ok": card["floor_ok"], "passed": card["passed"]}, indent=2))
    return 0 if card["floor_ok"] else 2

if __name__ == "__main__":
    sys.exit(main())

Expected fixture command for every pair, before anyone exports a remote URL:

python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json

The teaching fixture always returns the expected bump, so floor_ok must be true before PROBE_BASE_URL is set. That order is the lab, not a ceremony around the lab.

Exercise 1 — freeze the contract, not the essay (15 minutes)

Students often want a long rubric because it feels more serious than five enum checks. Stop that impulse and ask each pair to delete any probe whose expect field is a free-text essay. A probe that cannot fail in one sentence is not frozen yet, and it will be edited to match a nicer model next week.

Checklist for the teaching assistant:

Every probe has a stable id that will survive later wording edits.
bump is an enum, never a list of allowed synonyms in natural language.
reason is length-bounded so empty strings and novels both fail the same way.
min_pass equals the probe count on day one; lowering it requires a written note.

Exercise 2 — change the host, not the scorer (20 minutes)

Export one URL and keep probes.json byte-identical. A local scorer should not care which process sits behind a compatible HTTP path, only whether the floor still holds. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can sit behind PROBE_BASE_URL once the file backend already prints floor_ok: true.

export PROBE_BASE_URL="http://127.0.0.1:8080"
make remote
make diff

Interpret the diff with a table, not with a vibe from a single chat window in another tab.

Diff signal	Meaning in this lab	Next action
`floor_ok` stays true	Floor still holds on this host	Record the card; do not add probes yet
parse failures greater than zero	Contract broke into prose or invalid JSON	Fix the prompt or reject the host
one bump mismatch	Task policy drifted on a frozen case	Keep the probe; do not edit `expect`
all bumps become `minor`	Classifier collapsed to a constant	Fail the floor; do not average scores

Do not invent a latency number, a token budget, or a quality rank from this table. The only honest outputs are floor_ok, per-probe detail codes, and whether a previously passing id regressed.

Exercise 3 — inject one honest failure (15 minutes)

Change the system string so it asks for Markdown fences, then rerun make remote without touching expects. The ledger should show not_json and floor_ok false on that host. If the card stays green, the scorer is too loose, and the rest of the workshop is invalid until the detail codes are strict again.

Optional mutation list, one change at a time:

Truncate reason to three characters and expect reason_len.
Force bump to the string Minor with surrounding spaces.
Return two JSON objects concatenated in one message body.
Echo the user input with no bump key at all.

Each mutation should map to a single detail code already printed by the scorer. Students who add a new code must document it beside the probe pack before they change Python.

Limitations

A five-probe floor card will not tell you that a host is good enough for production traffic. It only tells you that a host is still able to clear a tiny, frozen bar you wrote down in advance. Exact enum match is brittle if the real task is stylistic writing, multi-file refactors, or tool loops with side effects. Shared hosts can pass the card in the morning and fail it in the afternoon because routing is not a pin you control. This outline also assumes JSON contracts; if you cannot shrink the task to fields, do not fake a floor with a subjective one-to-five score.

The HTTP helper is a lab default, not a specification of any product. Timeouts, auth headers, response envelopes, and route names must follow the host you actually run. Do not paste secrets, customer diffs, or licensed source into probes.json just to make the cases feel realistic.

Who should not use this approach

Teams that need contractual uptime, data-handling terms, or a named model revision they can cite.
Classrooms that skip the file fixture and start on a public URL during the first half hour.
Workloads with secrets, personal data, or licensed code inside the probe inputs.
Anyone hoping five cases will replace unit tests, code review, or a staging environment.

Recap for the last five minutes

Write three lines on the board and stop talking over them.

Freeze inputs and expects on disk before you change hosts or prompts.
Score fields, not vibes, and fail closed on parse errors every time.
Treat a green floor card as permission to continue testing, not as proof of quality.

The same pack should rerun next week without editing expects to match a nicer answer. If you must change a probe, bump its id and record why the old floor died, because silent edits are how drift becomes folklore.

Workshop: Pin a Dual-Backend Tool Receipt in 80 Minutes

Finley Zhu — Sat, 19 Sep 2026 10:04:27 +0000

A live agent demo is not a lab until both backends emit the same canonical tool receipt. Students often treat a successful chat reply as evidence, even when the tool JSON cannot be replayed. This workshop isolates that failure with a dual-run harness, a fixture server, and a sorted-key canonicalizer. The core result is a failing or passing receipt file, not a screenshot of a fluent answer.

Recent discussion around vibe coding versus engineering is useful here only as a measurement problem. A fluent model turn can hide argument drift, extra keys, and type coercion that later labs cannot rerun. The fix is not a longer prompt. The fix is a receipt that two backends must both satisfy before the exercise is marked complete.

What this lab measures

The unit of evidence is a tool call, not assistant prose. A student prompt may produce a correct English summary while the tool payload silently changes shape. That split is why classroom reruns collapse after a backend swap. This outline treats that collapse as the bug under study.

You will leave with four artifacts:

A local fixture server that returns a frozen tool payload.
A canonicalizer that sorts keys and strips non-contract fields.
A dual-run script that hits fixture and live backends with one prompt hash.
A pass or fail receipt that classmates can rerun without the original chat UI.

Timing box

Total scheduled time is eighty minutes, including a short debrief. Keep a visible timer so discussion does not eat the dual-run.

Minutes 0–10 — Frame the failure. Show two JSON payloads that look equal in a chat log and fail after canonicalization.
Minutes 10–25 — Exercise A: capture. Record one tool name plus raw arguments from a single agent turn.
Minutes 25–45 — Exercise B: canonicalize. Implement sorted-key JSON and a deny list of extra fields.
Minutes 45–70 — Exercise C: dual-run. Compare fixture output against one live backend using the same prompt hash.
Minutes 70–80 — Decision table and debrief. Classify misses as shape drift, value drift, or transport noise.

Lab layout students can rerun

Keep the repository to three files so the receipt stays reviewable in a pull request. Extra notebooks hide the contract. Label the snippets below as a teaching harness, not as production agent infrastructure.

lab_tool_receipt/
  fixture_server.py
  canonicalize.py
  dual_run.py
  receipts/
    prompt_a.fixture.json
    prompt_a.live.json

Exercise A — Capture a raw tool turn

Start from a single intended tool, not a multi-step agent loop. Multi-step traces mix planning noise with contract noise. The worked example uses create_ticket with three fields that classrooms actually grade: title, priority, and project_id.

# Label: teaching example, not a live production client.
RAW_TURN = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "high",
        "project_id": "ops-214",
        "confidence": 0.81,          # not in the contract
        "Priority": "high",          # duplicate key after lowercasing
    },
}

Write the raw turn to disk before you clean it. Students who canonicalize in their head cannot later prove which field the model invented. A receipt that starts after cleanup is not replayable.

Exercise B — Canonicalize before you compare

Canonical form is the only equality that survives pretty-printers and key reordering. The function below lowercases object keys, drops unknown fields, and encodes with sorted keys. That trio catches the three classroom failures that look like “the model changed its mind.”

# canonicalize.py — teaching harness
import json
from typing import Any

CONTRACT_FIELDS = {
    "create_ticket": ("title", "priority", "project_id"),
}

ALLOWED_PRIORITY = {"low", "medium", "high"}


def canonicalize_args(tool: str, args: dict[str, Any]) -> dict[str, Any]:
    lowered = {str(k).lower(): v for k, v in args.items()}
    allowed = CONTRACT_FIELDS[tool]
    clipped = {k: lowered[k] for k in allowed if k in lowered}
    missing = [k for k in allowed if k not in clipped]
    if missing:
        raise ValueError(f"missing contract fields: {missing}")
    if clipped["priority"] not in ALLOWED_PRIORITY:
        raise ValueError("priority outside contract enum")
    if not isinstance(clipped["title"], str) or not clipped["title"].strip():
        raise ValueError("title must be a non-empty string")
    if not isinstance(clipped["project_id"], str):
        raise ValueError("project_id must be a string")
    return {k: clipped[k] for k in allowed}


def receipt_bytes(tool: str, args: dict[str, Any], backend: str, prompt_sha: str) -> bytes:
    body = {
        "prompt_sha256": prompt_sha,
        "tool": tool,
        "backend": backend,
        "args": canonicalize_args(tool, args),
    }
    return json.dumps(body, sort_keys=True, separators=(",", ":")).encode("utf-8")

Run a local self-check before the dual-run so canonicalizer bugs are not blamed on the model.

python - <<'PY'
from canonicalize import receipt_bytes, RAW_TURN
raw = {
  "title": "Pager duty runbook is stale",
  "priority": "high",
  "project_id": "ops-214",
  "confidence": 0.81,
}
print(receipt_bytes("create_ticket", raw, "fixture", "abc123").decode())
PY

Expected canonical object, ignoring hash, is exactly three keys in sorted order. Extra model fields must not appear. If confidence survives, the deny list is incomplete and later diffs will be noisy.

Exercise C — Dual-run fixture versus live

The fixture server is the frozen half of the experiment. It should not call a model. It exists so students can fail the harness on purpose and still rerun the passing side after class.

# fixture_server.py — stdlib only, teaching example
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

FIXTURE = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "high",
        "project_id": "ops-214",
    },
}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(length)
        payload = json.dumps(FIXTURE).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Start the fixture in one terminal, then run the comparator in another. The live client is intentionally tiny so transport code does not become the lesson.

python fixture_server.py

# dual_run.py — teaching example; swap LIVE_URL for your classroom endpoint
import hashlib, json, urllib.request
from pathlib import Path
from canonicalize import receipt_bytes

PROMPT = "Create a high-priority ops ticket about the stale pager runbook in ops-214."
PROMPT_SHA = hashlib.sha256(PROMPT.encode()).hexdigest()
LIVE_URL = "http://127.0.0.1:8765"  # replace with the live classroom backend

def post_json(url: str) -> dict:
    req = urllib.request.Request(
        url,
        data=json.dumps({"prompt": PROMPT}).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode("utf-8"))

def write_receipt(backend: str, turn: dict) -> Path:
    blob = receipt_bytes(turn["tool"], turn["arguments"], backend, PROMPT_SHA)
    path = Path("receipts") / f"prompt_a.{backend}.json"
    path.parent.mkdir(exist_ok=True)
    path.write_bytes(blob)
    return path

fixture_turn = post_json("http://127.0.0.1:8765")
live_turn = post_json(LIVE_URL)
f_path = write_receipt("fixture", fixture_turn)
l_path = write_receipt("live", live_turn)
print("match" if f_path.read_bytes() == l_path.read_bytes() else "drift")
print(f_path.read_text())
print(l_path.read_text())

If you need a no-cost live backend for Exercise C, MonkeyCode’s free model access and free server option can occupy that slot without inventing a paid key workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The receipt still has to match the fixture; availability of a free path does not relax the contract.

Worked miss that students should rerun

Seed a deliberate live payload that would pass a casual eyeball check. Then watch the canonicalizer fail it. This is the teaching moment that screenshots never capture.

LIVE_DRIFT = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "High",      # enum case drift
        "project_id": 214,       # type drift
        "assignee": "oncall",    # extra field
    },
}

Expected classification after Exercise C:

Symptom	Canonicalizer result	Grade as
Key order changed, values identical	Pass after sort	Transport noise
Extra `confidence` or `assignee`	Dropped, then pass if core equal	Shape noise
`priority` case or unknown enum	`ValueError`	Contract miss
`project_id` became an integer	`ValueError`	Type drift
`title` paraphrased	Pass shape, fail bytes	Value drift
Tool name changed	Do not canonicalize	Wrong action

Value drift is the only row that should reopen prompt design. Shape and type misses belong to the contract, not to more adjectives in the system prompt. Students who respond to every miss by editing prose will not finish Exercise C inside the remaining twenty-five minutes.

Commands for a cold rerun

A classmate who was absent should reach the same pass or fail without the original chat transcript. That constraint is the point of the receipt.

python -m py_compile fixture_server.py canonicalize.py dual_run.py
mkdir -p receipts
python fixture_server.py &
python dual_run.py
sha256sum receipts/prompt_a.fixture.json receipts/prompt_a.live.json

If the hashes differ, print a field-level diff of the decoded objects. Do not paste both files into a chat model and ask which is better. The lab already defined equality.

import json
from pathlib import Path

a = json.loads(Path("receipts/prompt_a.fixture.json").read_text())
b = json.loads(Path("receipts/prompt_a.live.json").read_text())
for key in sorted(set(a) | set(b)):
    if a.get(key) != b.get(key):
        print(key, a.get(key), b.get(key))

Limitations

Canonical byte equality is stricter than human agreement and weaker than a full behavioral specification. It will fail useful paraphrases of title and it will pass a wrong ticket that still uses the three required keys. The fixture server also cannot represent rate limits, auth failures, or streaming token events. Treat those as out of scope for an eighty-minute session.

Free live backends can still change outputs between class meetings. The receipt documents that drift; it does not freeze a third-party endpoint. If the live path moves, keep the fixture receipt as the gradeable object and record the live miss as a dated lab note. Do not silently retune the fixture to match a new live payload, because that erases the teaching signal.

Who should not use this outline

Skip this workshop if your tools have no stable argument schema, or if you are grading writing quality rather than tool effects. Skip it for production incident work that needs authentication, PII controls, and real service-level objectives. Skip it when the assignment is open-ended generation with no tool layer, because there is then no receipt to pin. Teams that already freeze OpenAPI contracts and run consumer-driven pact tests will find this lab redundant except as a student on-ramp.

The conclusion stays narrow. A classroom can use a free live path, but only the dual-run receipt turns that path into engineering evidence. Fluent text remains optional commentary beside the JSON, not the thing you grade.

Workshop: Map Agent Claims to a Failing Command Ledger in 90 Minutes

Finley Zhu — Fri, 18 Sep 2026 08:32:22 +0000

An agent-assisted lab only counts as engineering when each claim maps to a command that fails first. Natural-language success reports from a coding model are not evidence, because they can pass by deleting tests. This workshop gives students a ledger format, a Makefile contract, and a mutation check they can rerun. The same method still works if the lab later moves onto another provider or endpoint.

Who this session is for

The outline targets instructors and self-taught developers who already run short coding labs with language models. Students should be comfortable with Python 3.11 or newer, pytest, and a local POSIX shell. Teams that need production SLAs, private data handling, or multi-service orchestration should pick another control plane. This session is not a model bake-off, and it does not rank vendors or quote unpublished benchmarks.

Use the ledger when the problem has a small oracle that students can encode as JSON fixtures. Skip it when the task has no deterministic expected output, or when claim commands would execute untrusted strings. The decision table below is a teaching filter, not a capacity plan.

Situation	Use this ledger	Do not use it
Classroom lab with a known oracle	Yes	—
Agent output is prose without a fixture	—	Yes
Commands in the ledger are student-authored and allowlisted	Yes	—
Commands could include untrusted shell fragments	—	Yes until argv is allowlisted
Production incident work with private payloads	—	Yes

Timing

Treat the schedule as a hard budget rather than a loose suggestion for discussion.

0–10 min — State the rule: no claim enters the ledger without a currently failing command.
10–25 min — Write three claims and fixtures for a small interval-merge problem.
25–50 min — Request an implementation from a free coding endpoint, then run make prove.
50–70 min — Inject a mutation that drops one assertion and confirm the prove target goes red.
70–85 min — Review which ledger fields students actually filled, and which they left blank.
85–90 min — Recap limitations and name the claims this ledger cannot prove.

The core artifact

Students keep one JSON ledger next to the code under test, and they treat every row as an untrusted claim. Each row binds a claim id, a shell command, an expected initial exit code, and a fixture path. The Makefile refuses prove until fail-first has been recorded in the same working tree. That stamp is session memory, not a gradebook.

{
  "problem": "merge_intervals",
  "endpoint": "free-lab",
  "claims": [
    {
      "id": "C1",
      "statement": "Overlapping closed intervals collapse into a single span.",
      "command": "pytest -q tests/test_c1_overlap.py",
      "fixture": "fixtures/c1_overlap.json",
      "fail_first_exit": 1
    },
    {
      "id": "C2",
      "statement": "Touching intervals that share an endpoint also merge.",
      "command": "pytest -q tests/test_c2_touch.py",
      "fixture": "fixtures/c2_touch.json",
      "fail_first_exit": 1
    },
    {
      "id": "C3",
      "statement": "An empty input list returns an empty list without raising.",
      "command": "pytest -q tests/test_c3_empty.py",
      "fixture": "fixtures/c3_empty.json",
      "fail_first_exit": 1
    }
  ]
}

Label the JSON above as a teaching fixture, not as production telemetry from a live classroom. Students copy it into claims.json and must not edit fail_first_exit after the first red run. If a stub accidentally returns a plausible list, the claim is too weak, and the fixture must be tightened before the generate step starts.

Makefile contract

PYTHON ?= python3
LEDGER := claims.json
STAMP  := .fail-first.ok

.PHONY: fail-first prove mutate clean

fail-first:
    $(PYTHON) tools/check_ledger.py --mode fail-first --ledger $(LEDGER)
    touch $(STAMP)

prove: $(STAMP)
    $(PYTHON) tools/check_ledger.py --mode prove --ledger $(LEDGER)

mutate:
    $(PYTHON) tools/mutate_drop_assert.py tests/test_c1_overlap.py
    -$(PYTHON) tools/check_ledger.py --mode prove --ledger $(LEDGER); \
        status=$$?; \
        $(PYTHON) tools/mutate_drop_assert.py --restore tests/test_c1_overlap.py; \
        exit $$status

clean:
    rm -f $(STAMP)

The stamp file is the only session memory the lab needs to keep honest. If a student skips fail-first, prove does not run, which blocks the shortcut of writing tests after a green implementation. Instructors should collect the stamp and the ledger, not a chat export, when they grade the lab.

Exercise 1 — Write claims that already fail (15 minutes)

Create src/intervals.py with a stub that raises NotImplementedError so every first run is red for a known reason. Then add one pytest module per claim, each loading JSON rather than inlined literals that drift during copy. Keep the assertion count at one behavioral check per file so later mutations stay obvious to a reader. The empty-input claim exists to catch models that return None instead of [].

# tests/test_c1_overlap.py
import json
from pathlib import Path
from src.intervals import merge_intervals

def test_overlapping_closed_intervals_collapse():
    payload = json.loads(Path("fixtures/c1_overlap.json").read_text())
    assert merge_intervals(payload["input"]) == payload["expected"]

{
  "input": [[1, 3], [2, 6], [8, 10], [15, 18]],
  "expected": [[1, 6], [8, 10], [15, 18]]
}

{
  "input": [[1, 4], [4, 5]],
  "expected": [[1, 5]]
}

{
  "input": [],
  "expected": []
}

Run make fail-first and keep the checker exit at zero while pytest exits non-zero for every claim. If C2 is omitted, many generated functions will treat touching ranges as disjoint and still look correct on C1. That gap is a ledger defect, not a model defect, and it should be fixed before anyone pastes a prompt.

Exercise 2 — Generate an implementation on a shared free endpoint (25 minutes)

This is the point where a shared free coding setup matters, because a classroom cannot grade work that only ran inside one paid chat window. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which is enough to host the same prompt and the same runner for every student laptop in the room.

Treat that product as a lab backend, not as the subject of the lesson or as a scored vendor. Students paste a prompt that forbids extra files and forbids editing tests, fixtures, or the ledger. The prompt below is a proposal for this outline, not a measured template with claimed pass rates.

Implement merge_intervals(intervals) in src/intervals.py only.
Do not edit tests/, fixtures/, claims.json, or the Makefile.
Intervals are inclusive integer pairs [start, end] with start <= end.
Return merged spans sorted by start. Empty input returns [].

After the model writes the function, students run make prove against the unchanged tests. The ledger checker should load each claim, execute the command, and require exit code 0. Any extra file the model created is a lab failure even if tests pass, because the contract was file-scoped rather than chat-scoped.

Ledger checker (teaching code)

# tools/check_ledger.py
from __future__ import annotations

import argparse, json, subprocess, sys
from pathlib import Path

def load_claims(path: Path) -> list[dict]:
    data = json.loads(path.read_text())
    claims = data.get("claims") or []
    if not claims:
        raise SystemExit("ledger has no claims")
    return claims

def run_cmd(command: str) -> int:
    completed = subprocess.run(command, shell=True)
    return int(completed.returncode)

def fail_first(claims: list[dict]) -> int:
    bad = 0
    for claim in claims:
        code = run_cmd(claim["command"])
        expected = int(claim["fail_first_exit"])
        if code == 0:
            print(f"{claim['id']} already passes; claim is not a test")
            bad += 1
        elif code != expected:
            print(f"{claim['id']} exited {code}, expected {expected}")
            bad += 1
        else:
            print(f"{claim['id']} fails as required")
    return 1 if bad else 0

def prove(claims: list[dict]) -> int:
    bad = 0
    for claim in claims:
        code = run_cmd(claim["command"])
        if code != 0:
            print(f"{claim['id']} still failing with exit {code}")
            bad += 1
        else:
            print(f"{claim['id']} proved")
    return 1 if bad else 0

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--mode", choices=("fail-first", "prove"), required=True)
    parser.add_argument("--ledger", required=True)
    args = parser.parse_args()
    claims = load_claims(Path(args.ledger))
    status = fail_first(claims) if args.mode == "fail-first" else prove(claims)
    sys.exit(status)

if __name__ == "__main__":
    main()

This checker is intentionally small so students can read it in one sitting without a framework tour. It does not sandbox the shell, so untrusted commands do not belong in claims.json during a first offering. Instructors grading a second cohort should replace shell=True with an allowlisted argv map before they collect stamps.

Exercise 3 — Mutation that drops an assertion (20 minutes)

Green tests after an agent edit are still weak if the agent can delete the assertion and keep the file importable. The mutate target comments out the first assert in tests/test_c1_overlap.py and reruns prove without touching src/intervals.py. The expected teaching result is a non-zero exit from the prove path. If prove stays green after the drop, the claim never tested behavior and must be rewritten.

# tools/mutate_drop_assert.py
from pathlib import Path
import sys

MARKER = "# mutated-assert: "

def drop(path: Path) -> None:
    lines = path.read_text().splitlines(True)
    for i, line in enumerate(lines):
        stripped = line.lstrip()
        if stripped.startswith("assert "):
            indent = line[: len(line) - len(stripped)]
            lines[i] = f"{indent}{MARKER}{stripped}"
            path.write_text("".join(lines))
            return
    raise SystemExit(f"no assert found in {path}")

def restore(path: Path) -> None:
    text = path.read_text().replace(MARKER, "")
    path.write_text(text)

if __name__ == "__main__":
    restore_mode = "--restore" in sys.argv
    target = Path(sys.argv[-1])
    restore(target) if restore_mode else drop(target)

Students should log three numbers only: fail-first checker status, prove exit, and mutate exit. Do not collect latency, token counts, or model quality scores in this lab, because those numbers change by endpoint and distract from the ledger rule. A useful board note is a three-column table with claim ids, not a screenshot of a chat transcript.

Worked example students can rerun

Use this sequence on a clean tree after the files above exist. The commands assume a POSIX shell and a virtualenv that already has pytest.

python3 -m venv .venv
. .venv/bin/activate
pip install pytest
mkdir -p src tests fixtures tools
# copy the ledger, tests, fixtures, Makefile, and tools from this article
rm -f .fail-first.ok
make fail-first    # pytest red per claim; checker status 0
# paste the prompt into the shared free lab endpoint, write src/intervals.py only
make prove         # expect 0
make mutate        # expect non-zero from prove, then restored tests

A reference implementation that satisfies the three fixtures is shown next. Label it as a teaching solution, not as the only correct merge, and do not paste it into the prompt.

# src/intervals.py
def merge_intervals(intervals):
    if not intervals:
        return []
    ordered = sorted(intervals, key=lambda pair: (pair[0], pair[1]))
    merged = [list(ordered[0])]
    for start, end in ordered[1:]:
        last = merged[-1]
        if start <= last[1]:
            last[1] = max(last[1], end)
        else:
            merged.append([start, end])
    return merged

If a student's model returns a version that drops touching intervals, claim C2 stays red and the ledger remains honest. That failure is the lesson, not a reason to loosen the fixture or to delete C2 from claims.json. Students who “fix” the lab by editing expected JSON have left the engineering rule, even when the chat window still looks successful.

What the ledger does not prove

The method does not prove algorithmic optimality, concurrency safety, or prompt robustness across vendors. It also does not prove that a free server will remain available, because availability is an operational choice rather than a test assertion. Students who need property-based coverage should add Hypothesis after this lab, not during the ninety-minute window, because shrinking counterexamples will blow the timing budget.

Do not use this workshop when the task has no oracle, when fixtures would leak private data, or when shelling out from check_ledger.py would execute untrusted strings. Do not use it as a substitute for code review on security-sensitive diffs, because a passing ledger only shows that named fixtures stayed green. Instructors running large classes should allowlist commands before the second session, then keep the fail-first stamp as the gradeable artifact.

Closing

The transferable output is the ledger plus the fail-first stamp, not a chat transcript and not a vendor comparison. If a class needs one shared free endpoint for the generate step, MonkeyCode's free model access and free server option can host that lab runner without changing the Makefile contract.

Workshop: Score AI Refactors With a Frozen Oracle and Diff Budget in 75 Minutes

Finley Zhu — Thu, 17 Sep 2026 06:43:47 +0000

An AI refactor is not an engineering change until a frozen oracle and a bounded diff both accept it. Chat tone, shorter functions, and confident commit messages do not constitute acceptance criteria for production code. This seventy-five minute workshop builds a rerunnable scoring harness that grades patches independently of which host produced them. Students leave with a golden test file, a diff-budget checker, and a command sequence they can replay after class.

Why this workshop exists

Public threads keep asking whether models already outcode typical developers, which measures the wrong unit of work. A patch can delete logging, widen types, or skip the failing test that motivated the original change. Engineering accepts a change when behavior stays inside a declared envelope, not when a diff looks tidy. Cleaner chat output is not evidence that reserved-stock math, error paths, or invariants still hold.

This lab treats the model as an untrusted patch generator, not as a reviewer of its own work. The oracle is frozen before any prompt is sent, and the same scorer runs against local stubs and remote completions. If the remote host is unavailable, the local cassette still grades known-good and known-bad patches on schedule. Classroom time then depends on git and pytest, rather than on vendor latency or prompt fashion.

Audience, timing, and materials

The intended audience is backend or tooling developers who already write pytest and need a classroom gate for model-authored diffs. Facilitators should have a terminal, Python 3.11 or newer, git, and a working pytest install before the session starts. Groups without tests, repos that hold production secrets, or workshops that need a guaranteed model SLA should stop after Exercise 1. Those groups can still treat the oracle as a local linter, without sending source to any remote host.

Timing (75 minutes)

Minutes 0–10 — Freeze the golden tests and record a baseline hash.
Minutes 10–25 — Encode a diff budget and a forbidden-path list.
Minutes 25–40 — Score a known-good patch and a vibe-coded patch.
Minutes 40–60 — Optional remote run against a free hosted model.
Minutes 60–75 — Debrief false accepts, false rejects, and homework.

The kit is a tiny inventory module, one pytest file, and a scorer script that exits non-zero on oracle failure. Students should not edit the oracle after minute ten unless the facilitator records a new hash in the lab log. Late edits to tests are protocol violations, because they make later scores incomparable across machines.

Learning objectives

By the end of the session, a student should be able to:

Explain why a passing chat demo is not an acceptance test for a production refactor.
Freeze pytest output and a tree hash so later runs remain comparable across student machines.
Reject patches that exceed a line budget or that touch a forbidden path in the tree.
Replay the same scorer against a local fixture and an optional remote host without changing rules.

The artifact students will rerun

Create a directory named oracle_lab and keep every file below under version control for the full session. The module under test is intentionally small so the class can read the entire diff during debrief. A small surface also keeps the twelve-line budget meaningful, because a rewrite cannot hide inside noise.

`inventory.py`

"""In-memory SKU ledger used as the system under test."""

from dataclasses import dataclass


@dataclass
class Line:
    sku: str
    qty: int
    reserved: int = 0

    def available(self) -> int:
        return self.qty  # bug: reserved stock is still counted as available


class Ledger:
    def __init__(self) -> None:
        self._lines: dict[str, Line] = {}

    def add(self, sku: str, qty: int) -> None:
        if qty <= 0:
            raise ValueError("qty must be positive")
        current = self._lines.get(sku)
        if current is None:
            self._lines[sku] = Line(sku=sku, qty=qty)
            return
        current.qty += qty

    def reserve(self, sku: str, qty: int) -> None:
        line = self._lines[sku]
        if qty > line.available():
            raise ValueError("insufficient stock")
        line.reserved += qty

    def available(self, sku: str) -> int:
        return self._lines[sku].available()

The planted defect lives on Line.available, where reserved units remain inside the sellable count. Students must not fix the file by hand until the scorer is in place and the baseline commit exists. A manual fix would skip the comparison the lab is measuring and collapse Exercise 3 into a git checkout.

`test_inventory.py`

import pytest

from inventory import Ledger


def test_add_and_available_without_reserve():
    ledger = Ledger()
    ledger.add("sku-a", 10)
    assert ledger.available("sku-a") == 10


def test_reserve_reduces_available():
    ledger = Ledger()
    ledger.add("sku-a", 10)
    ledger.reserve("sku-a", 4)
    assert ledger.available("sku-a") == 6


def test_over_reserve_raises():
    ledger = Ledger()
    ledger.add("sku-a", 2)
    with pytest.raises(ValueError):
        ledger.reserve("sku-a", 3)


def test_second_add_accumulates():
    ledger = Ledger()
    ledger.add("sku-a", 4)
    ledger.add("sku-a", 6)
    assert ledger.available("sku-a") == 10

These four tests form the frozen oracle and must stay byte-stable after the first hash is recorded. After minute ten, facilitators should treat any edit to this file as a protocol violation, not as cleanup. A model that rewrites tests can manufacture a green run without restoring reserved-stock behavior at all.

`score_patch.py`

#!/usr/bin/env python3
"""Score a git diff against a frozen test oracle and a line budget."""

from __future__ import annotations

import argparse
import hashlib
import subprocess
import sys
from pathlib import Path

ORACLE_SHA256 = "REPLACE_AFTER_EXERCISE_1"
MAX_CHANGED_LINES = 12
FORBIDDEN_PREFIXES = ("test_inventory.py", "score_patch.py", ".git/")


def run(cmd: list[str]) -> subprocess.CompletedProcess[str]:
    return subprocess.run(cmd, text=True, capture_output=True, check=False)


def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def changed_lines(diff_text: str) -> int:
    total = 0
    for line in diff_text.splitlines():
        if line.startswith("+++") or line.startswith("---"):
            continue
        if line.startswith("+") or line.startswith("-"):
            total += 1
    return total


def forbidden_paths(diff_text: str) -> list[str]:
    hits: list[str] = []
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            if any(path.startswith(prefix) for prefix in FORBIDDEN_PREFIXES):
                hits.append(path)
    return hits


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", default="HEAD")
    args = parser.parse_args()

    oracle = Path("test_inventory.py")
    digest = sha256_file(oracle)
    if ORACLE_SHA256 == "REPLACE_AFTER_EXERCISE_1":
        print(f"oracle sha256 (record this): {digest}")
    elif digest != ORACLE_SHA256:
        print("oracle drift: test_inventory.py no longer matches the freeze")
        return 2

    diff = run(["git", "diff", args.base, "--", "."])
    if diff.returncode != 0:
        print(diff.stderr)
        return 3

    budget = changed_lines(diff.stdout)
    blocked = forbidden_paths(diff.stdout)
    pytest_result = run([sys.executable, "-m", "pytest", "-q", "test_inventory.py"])

    print(f"changed_lines={budget} max={MAX_CHANGED_LINES}")
    print(f"forbidden={blocked or 'none'}")
    print(pytest_result.stdout)
    print(pytest_result.stderr)

    failures = []
    if budget > MAX_CHANGED_LINES:
        failures.append("diff budget exceeded")
    if blocked:
        failures.append("forbidden path edited")
    if pytest_result.returncode != 0:
        failures.append("oracle tests failed")

    if failures:
        print("SCORE: REJECT")
        print("reasons:", "; ".join(failures))
        return 1

    print("SCORE: ACCEPT")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The scorer never asks the model whether the patch is good, which keeps generation and judgment on separate sides. Exit code one means reject, exit code two means oracle drift, and exit code zero is the only accept path. Students should read the failure list before they retry a prompt, because retries without logs hide systematic errors.

Exercise 1 — Freeze the oracle (0–10 minutes)

Initialize git so later diffs have a stable base, then record the oracle digest that the scorer prints. The first pytest run should fail on test_reserve_reduces_available, because reserved stock is still counted as free. Paste the printed sha256 into ORACLE_SHA256 and commit that constant as a second freeze commit before continuing.

cd oracle_lab
python3 -m venv .venv
source .venv/bin/activate
pip install pytest
git init
git add inventory.py test_inventory.py score_patch.py
git commit -m "freeze baseline before any model run"
python score_patch.py
pytest -q test_inventory.py; echo pytest_exit:$?

From this minute forward, a student who simplifies the tests has broken the protocol, not improved the module. Facilitators should write the digest on a shared board so late arrivals can catch up without regenerating hashes. If two machines print different digests, stop and compare line endings before continuing, because that drift poisons later scores.

Exercise 2 — Encode the budget (10–25 minutes)

Keep MAX_CHANGED_LINES at twelve for this module, since the correct fix is a one-line correction plus a comment. Ask students to list files the model must not touch, then confirm those prefixes match FORBIDDEN_PREFIXES in the scorer. A useful discussion prompt is why test_inventory.py is forbidden, since edited tests can fake a green run. Optional extra rules should stay in a table so each rule remains a measurable predicate rather than taste.

Rule	Predicate	Fail code
Oracle hash	`sha256(test_inventory.py)` matches freeze	2
Diff budget	changed lines ≤ 12	1
Path lock	patch does not edit tests or scorer	1
Behavior	pytest on the frozen file exits 0	1

Students should explain each row without referring to any model vendor, product name, or chat screenshot. If a row cannot be checked by a command, it does not belong in the acceptance table for this lab. Taste-based rules such as formatter churn can wait until the behavioral oracle is stable on every machine.

Exercise 3 — Score two fixtures (25–40 minutes)

Do not call a hosted model yet, because the scorer must be proven against fixtures before the network is involved. Hand the class two patches as files, then apply each one on a clean tree and record the scorer exit code. The known-good patch only subtracts reserved quantity, while the vibe patch adds helpers and rewrites tests.

Known-good patch (good.patch):

--- a/inventory.py
+++ b/inventory.py
@@ -10,7 +10,7 @@ class Line:
     reserved: int = 0

     def available(self) -> int:
-        return self.qty  # bug: reserved stock is still counted as available
+        return self.qty - self.reserved

Vibe patch (vibe.patch) — this one looks more complete in a chat window and must still reject:

--- a/inventory.py
+++ b/inventory.py
@@ -1,5 +1,6 @@
 """In-memory SKU ledger used as the system under test."""
+
+import json
 from dataclasses import dataclass
@@ -14,6 +15,18 @@ class Line:
         return self.qty  # bug: reserved stock is still counted as available
+
+    def as_json(self) -> str:
+        return json.dumps({"sku": self.sku, "qty": self.qty})
+
+
def dump_ledger(ledger: "Ledger") -> str:
+    return json.dumps(sorted(ledger._lines))
--- a/test_inventory.py
+++ b/test_inventory.py
@@ -10,11 +10,6 @@ def test_add_and_available_without_reserve():
     assert ledger.available("sku-a") == 10
-
-
def test_reserve_reduces_available():
-    ledger = Ledger()
-    ledger.add("sku-a", 10)
-    ledger.reserve("sku-a", 4)
-    assert ledger.available("sku-a") == 6

git apply good.patch
python score_patch.py; echo score_exit:$?
git checkout -- inventory.py
git apply vibe.patch
python score_patch.py; echo score_exit:$?
git checkout -- .

Record both exit codes in a shared sheet so the room can see accept and reject as data rather than opinion. Extra features are not credit when they blow the budget or rewrite the oracle that was frozen in Exercise 1. If any group accepts the vibe patch, inspect whether they ran pytest without score_patch.py, which is a process failure.

Expected transcripts look like the following pair, with hashes replaced by the classroom freeze value.

# good.patch
changed_lines=2 max=12
forbidden=none
SCORE: ACCEPT

# vibe.patch
changed_lines=18 max=12
forbidden=['test_inventory.py']
SCORE: REJECT
reasons: diff budget exceeded; forbidden path edited; oracle tests failed

Exercise 4 — Optional remote generation (40–60 minutes)

Only start this exercise after both fixtures have been scored, so remote noise cannot mask a broken harness. The model remains a patch source, and the scorer remains the only judge that can print SCORE: ACCEPT. One convenient option is MonkeyCode, which currently offers free model access and a free server option. Students can point that host at the same prompt after the local oracle is frozen.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the prompt narrow so the remote run stays comparable to the known-good patch from Exercise 3. Students may tune wording but must not paste test_inventory.py into an editable region of the prompt. A starter prompt follows, and the class should treat extra commentary from the model as a defect in the patch.

Return a unified diff against inventory.py only.
Do not edit tests. Do not add files. Do not exceed twelve changed lines.
Line.available must subtract reserved from qty.
Do not reformat unrelated code.

The request helper below is labeled as an unexecuted proposal. Wire TLS, auth, and timeouts in the lab, using whatever HTTP client the room already maintains.

# labeled example: unexecuted request helper, fill in the classroom endpoint
import os
import urllib.request

def request_patch(prompt: str, endpoint: str) -> str:
    # Proposal only: do not hard-code secrets; read them from the environment.
    raise NotImplementedError("connect the classroom endpoint here")

Apply and score exactly as in Exercise 3:

git apply model.patch
python score_patch.py; echo score_exit:$?
git checkout -- inventory.py

Apply model.patch with the same git apply and score_patch commands used on the local fixtures. If the free server is busy or the model returns prose instead of a diff, log a reject and continue. The oracle does not pause for vendor availability, which is the behavior a merge gate needs in practice. This workshop does not publish latency numbers, token quotas, or model names, because those figures drift quickly.

Exercise 5 — Debrief (60–75 minutes)

Walk the board through four failure classes and map each class to a scorer branch or a written protocol rule. Keep the discussion forensic, with transcripts and exit codes, rather than with impressions of how helpful the chat felt. Homework is a second module of the student's choice, scored with a budget taken from a hand-written fix.

Oracle drift. Someone edited tests to match a messy patch, and the sha256 guard should have blocked the run.
Budget overflow. A rewrite that cleans the file changes thirty lines and hides the one-line semantic fix.
Unparseable output. The model returned markdown fences, a lecture, or a partial hunk, which is a reject.
False confidence. The patch passes tests but touches score_patch.py, which is part of the trusted computing base.

Students submit the scorer transcript, not a screenshot of a chat window, because screenshots are not replayable evidence. Replay after class by resetting to the freeze commit and applying good.patch again on a clean tree. The accept path should be stable on every machine if line endings and the oracle hash were recorded correctly.

Limitations

This harness does not prove concurrency correctness, API compatibility, or runtime performance under production traffic. Four pytest cases cannot stand in for a production suite, and a twelve-line budget is a classroom constant. Binary files, generated code, and license headers will inflate changed_lines unless the class extends the counter. Semantic edits that stay under twelve lines can still be wrong in ways the oracle never observed.

The remote exercise depends on a host that may rate-limit, change, or refuse the prompt shape without notice. Students should keep local fixtures so the class still completes when the network path fails or stalls. Nothing in the scorer inspects prompt injection, secret leakage, or the license of generated code. Those reviews stay human, and they should happen before any patch is merged outside this classroom repository.

Who should not use this approach

Repositories that cannot run tests offline, because the oracle would then depend on the same network as the model.
Compliance-bound codebases that forbid sending source to a third-party host, even when that host is free.
Facilitators who need a fixed model identity, quota, or uptime number in the syllabus, which this lab does not provide.
Teams that already gate merges with mature CI covering tests, ownership, and size, for whom the budget table is redundant.

After class

Replay the lab by resetting to the freeze commit and applying good.patch; the accept path should stay stable. If a machine rejects that fixture, compare the oracle hash and line endings before blaming the scorer logic. For an optional Exercise 4 backend, try MonkeyCode's free model access and free server after the oracle hash is recorded. Leave the local fixtures in the repo so later cohorts can grade patches even when that host is offline.