DEV Community: Finley Zhu

Workshop: Run a 100-Minute Fault-Injection Drill Against a Free Model Endpoint

Finley Zhu — Tue, 27 Oct 2026 13:36:54 +0000

Free model endpoints rarely fail the way client libraries assume, and most teams learn that in production rather than in a classroom. A documented rate limit you can back off from, a stream that dies mid-token, and a 200 response with an empty choices array all look identical to a naive retry wrapper. This workshop spends 100 minutes teaching participants to separate those cases before writing a single line of retry logic, using only a local stub so no live quota is consumed.

What participants leave with

The deliverable is not a library, it is a repeatable diagnostic habit plus four artifacts they can rerun on any endpoint.

A local fault server that reproduces six distinguishable failure modes on demand.
A classifier that maps raw responses to named outcomes instead of a boolean.
A retry ladder with an explicit wall-clock budget and per-attempt accounting.
A decision table that says, for each class, whether retrying is even meaningful.

The failure taxonomy students must reproduce

The whole drill rests on one claim: classification is cheaper than retrying, and most retry bugs are classification bugs. Write this table on the board before any code appears.

Observed on the wire	Named class	Retry meaningful?	Why
`429` with `Retry-After`	`rate_limited`	Yes, after the stated wait	The server told you when
Socket read exceeds timeout	`timeout`	Yes, with a longer timeout	Capacity, not correctness
`5xx` from an intermediary	`server_error`	Usually yes, bounded	Upstream may be restarting
`200` with unparseable JSON	`truncated`	Sometimes	Stream cut, not a refusal
`200` with `choices: []`	`empty`	Policy decision	Silent filter or bad request shape
`4xx` other than `429`	`client_error`	No	Your payload is wrong

Timing plan for the 100 minutes

Minutes	Block	Output
0-5	Framing and taxonomy on the board	Shared vocabulary
5-20	Exercise 1: stub server up	`fault_server.py` responding
20-40	Exercise 2: classifier	`call_once()` returning classes
40-60	Exercise 3: retry ladder	Budget-aware `run()`
60-75	Exercise 4: worked trace	One request, annotated
75-90	Exercise 5: score sheet	Filled results table
90-100	Debrief and decision table	Agreed retry policy

Exercise 1: stand up the fault server (15 minutes)

The stub below uses only the Python standard library, so a room of students needs no package installs and no network. Treat it as a reference implementation and check it against your interpreter version before the session.

# fault_server.py -- stdlib only, six modes, random per request
import json, random, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

MODES = ["ok", "rate_limited", "timeout", "truncated", "empty", "slow_ok"]

class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass  # keep the classroom terminal readable

    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        mode = random.choice(MODES)
        if mode == "rate_limited":
            self.send_response(429)
            self.send_header("Retry-After", "2")
            self.end_headers()
        elif mode == "timeout":
            time.sleep(30)          # client must give up first
            self.send_response(504)
            self.end_headers()
        elif mode == "truncated":
            self.send_response(200)
            self.send_header("Content-Type", "text/event-stream")
            self.end_headers()
            self.wfile.write(b'data: {"delta":"part')
        elif mode == "empty":
            self._json({"choices": []})
        elif mode == "slow_ok":
            time.sleep(3)
            self._json({"choices": [{"message": {"content": "ok"}}]})
        else:
            self._json({"choices": [{"message": {"content": "ok"}}]})

    def _json(self, body):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(body).encode())

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8099), Handler).serve_forever()

Run it with python fault_server.py, then confirm with curl -s -X POST localhost:8099/v1/chat/completions -d '{}' repeated a dozen times. Ask students to record which modes they saw, because a single run usually shows three of six.

Debrief question for this block

Why does the timeout mode sleep 30 seconds when the client timeout is 10? Because the drill is testing the client's patience, not the server's speed, and a shared sleeping thread is cheaper than a fault-injection framework.

Exercise 2: classify before you retry (20 minutes)

The classifier must return a named outcome, never a boolean, and it must survive a response body that is not valid JSON.

# probe.py
import json, socket, urllib.error, urllib.request

class Outcome:
    def __init__(self, kind, detail=""):
        self.kind, self.detail = kind, detail

    def __repr__(self):
        return f"<{self.kind} {self.detail}>"

def classify_ok(raw: bytes) -> Outcome:
    try:
        body = json.loads(raw)
    except json.JSONDecodeError:
        return Outcome("truncated", raw[-24:].decode(errors="replace"))
    choices = body.get("choices") or []
    if not choices:
        return Outcome("empty")
    content = (choices[0].get("message") or {}).get("content") or ""
    return Outcome("empty_content") if not content.strip() else Outcome("ok", content[:24])

def call_once(url, payload, timeout):
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"}, method="POST")
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return classify_ok(r.read())
    except urllib.error.HTTPError as e:
        if e.code == 429:
            return Outcome("rate_limited", e.headers.get("Retry-After") or "")
        if e.code >= 500:
            return Outcome("server_error", str(e.code))
        return Outcome("client_error", str(e.code))
    except (socket.timeout, TimeoutError):
        return Outcome("timeout")
    except urllib.error.URLError as e:
        return Outcome("transport", str(e.reason))

Students then fire 30 requests at the stub and tally outcomes by name. The expected lesson is that truncated and empty arrive with a healthy status code, so any wrapper keyed on status alone will silently convert them into successes.

Exercise 3: the retry ladder with a budget (20 minutes)

A ladder without a budget is a loop, and a loop against a free endpoint is how a demo turns into a ban. Make the wall-clock ceiling a parameter and charge every attempt against it.

import random, time

LADDER = [
    (1, 10, 0.0),   # attempt, timeout_s, pre-wait_s
    (2, 10, 2.0),   # honor Retry-After here when present
    (3, 20, 8.0),   # widen the read timeout once
]

def run(url, payload, budget_s=45.0):
    started, history = time.monotonic(), []
    for attempt, timeout, wait in LADDER:
        jitter = random.uniform(0, 1)
        if time.monotonic() - started + wait + jitter + timeout > budget_s:
            history.append(("budget_exhausted", attempt))
            break
        time.sleep(wait + jitter)
        out = call_once(url, payload, timeout)
        history.append((out.kind, attempt))
        if out.kind == "ok":
            return out, history
        if out.kind == "client_error":
            break  # payload is wrong; more attempts cannot help
    return None, history

Require one design decision in writing: does empty get a second attempt or not? Both answers are defensible, and the drill grades the justification rather than the choice.

Exercise 4: a worked trace students can rerun (15 minutes)

Walk one request through the ladder on the board, step by step, so the accounting is visible.

Attempt 1, timeout 10s. The stub picks rate_limited and returns 429 with Retry-After: 2. Charge 10s of budget at worst, much less in practice.
Wait 2s plus jitter, because the header outranks the ladder's default. Attempt 2, timeout 10s.
The stub now picks truncated. json.loads raises, the class is truncated, and the ladder continues because a cut stream is not a refusal.
Attempt 3, timeout 20s, after an 8s plus jitter wait. The stub returns ok, and the trace ends with a success on attempt three.
Total charged time stays under the 45s ceiling, and the history list shows exactly which class each attempt hit.

Then rerun with budget_s=12 and watch the ladder abandon early. The point is that the ceiling, not the attempt count, is what protects the endpoint.

Exercise 5: the score sheet and decision table (15 minutes)

Students fill the same table for two ladders of their own design, and the empty cells are the deliverable.

Ladder	Requests	ok	rate_limited	timeout	truncated	empty	Resolved within budget
A: 3 attempts, 45s	60
B: 1 attempt, 15s	60

The debrief question is whether ladder B's failures were cheaper to handle downstream than ladder A's extra wall-clock time. Close the session by agreeing on one retry policy per failure class and writing it into the repository, not into a slide.

Where a free endpoint fits into the exercise

The stub proves the logic, but students still need one real endpoint to feel the difference between a simulated 429 and a shared one. MonkeyCode's free model access and free server option are useful here precisely because the drill is budget-aware: the operator's own materials describe a large free token allowance and a no-cost server, and you should confirm the current terms and limits on the project's own pages before the session rather than trusting any secondhand number.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Because the ladder caps total wall-clock time, a workshop can rehearse against a free endpoint without letting a runaway loop drain a shared allowance. Keep the stub as the default target and treat the hosted endpoint as the final exercise only.

Limitations and who should skip this

The stub cannot reproduce provider-specific quirks such as tool-call schema drift, safety filters, or tokenizer-level truncation, so it is a starting taxonomy rather than a complete one.
Retrying never repairs a wrong prompt; empty and empty_content often mean the request shape is wrong, not that the server is busy.
The ladder is client-side only. If your workflow writes to a database, pair it with idempotency keys, or the drill will teach the wrong lesson.
Teams with hard latency SLOs should not adopt a multi-second ladder without measuring it first, and anyone who cannot verify the current free-tier terms should run the stub only.

If you teach this drill, keep the fixture local, keep the budget explicit, and keep the classification table on the wall. When you want a second endpoint to compare ladders against, the MonkeyCode project is a reasonable place to start, and its own documentation is the source of truth for what the free access currently includes.

Workshop: Run a 20-Student AI Lab on a Free Model Endpoint in 90 Minutes

Finley Zhu — Mon, 26 Oct 2026 12:13:14 +0000

A shared free model endpoint works fine as workshop infrastructure, provided you stop treating it as an API and start treating it as a scarce, shared, non-deterministic resource. The plan below moves twenty students through four timed exercises against one endpoint and produces three artifacts you can actually grade. The failure that ruins these sessions is rarely model quality; it is contention plus grading noise. Recent DEV threads about AI cost and about intermediaries sitting between developers and models point the same direction: give people direct, measured access and the complaints mostly disappear.

Three failure modes to design against

Before the agenda, name the problems out loud so the room recognizes them when they appear at minute 12.

Thundering herd. Twenty students times five prompts is one hundred requests in a two-minute window, and nobody in the room knows the endpoint's real concurrency ceiling. Latency spikes, students blame the model, and the exercise stops teaching anything.
Non-determinism as grading noise. Default temperature, provider-side routing, and truncation all change output text between two students running the identical prompt. If you grade on exact strings, half the room fails for reasons unrelated to the lesson.
Invisible budget. Nobody knows the remaining allowance, so the lab ends abruptly mid-exercise and the last ten minutes become support triage instead of debrief.

The artifact that fixes all three is a small cassette harness: record every request-response pair once, replay it offline, and meter every live call against a per-student ledger.

The 90-minute agenda

Minute	Block	Concrete output
0–8	Framing: the three failure modes	shared vocabulary
8–20	Setup and health check	green check, `usage` object printed
20–35	Exercise 1: record a baseline cassette	`cassettes/baseline.jsonl`
35–50	Exercise 2: ledger and admission control	`ledger.csv` with observed or estimated tokens
50–65	Exercise 3: inject 429/503 and back off	`reports/backoff.md`
65–80	Exercise 4: determinism diff and grade	`reports/diff.json`
80–90	Debrief: fill the decision table	completed rubric

Notice that only minutes 8–65 touch the network. Everything after minute 65 runs from cassettes, which means the last exercise still completes if the endpoint degrades.

Instructor prep: 30 minutes before the room opens

Use one repository per cohort and put the recorder on a single host so students share state instead of duplicating it. MonkeyCode is a reasonable substrate here because it offers free model access and a free server option, so the cohort gateway and the recorder run on one machine without a billing conversation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If your account exposes a single credential, keep it in the environment, never in a public repository or a slide, and rotate it after the session.

lab/
  record.py        # cassette recorder + fingerprinting
  ledger.py        # per-student token accounting
  inject.py        # fake server: 429 / 503 with Retry-After
  diff.py          # determinism comparison
  exercises/       # one prompt per exercise, fixed text
  cassettes/       # committed after Exercise 1

Set three variables and never hardcode them anywhere else:

export AI_BASE_URL="https://<your-endpoint-host>/v1"
export AI_API_KEY="<handed out at t-5, rotated at t+95>"
export AI_MODEL_ID="<the model id your account can actually select>"

Then run the health check and read the usage object, because Exercise 2 depends on what you find there:

# health.py — reference implementation, not executed against the hosted endpoint
import json, os, urllib.request

req = urllib.request.Request(
    os.environ["AI_BASE_URL"].rstrip("/") + "/chat/completions",
    data=json.dumps({"model": os.environ["AI_MODEL_ID"],
                     "messages": [{"role": "user", "content": "ping"}],
                     "max_tokens": 8}).encode(),
    headers={"Content-Type": "application/json",
             "Authorization": "Bearer " + os.environ["AI_API_KEY"]})
with urllib.request.urlopen(req, timeout=60) as r:
    body = json.loads(r.read())
print(json.dumps(body.get("usage", {}), indent=2))

If usage is present, you have an authoritative count for the ledger. If it is absent or zero, tell the room immediately and switch to estimation mode in the next exercise.

Exercise 1 — Record a baseline cassette (15 min)

Every request gets a fingerprint over model, prompt, temperature, and max tokens, so two students asking the same question collide on one cassette entry instead of spending two calls.

import hashlib, json, os, time, urllib.request

BASE = os.environ["AI_BASE_URL"].rstrip("/")
KEY = os.environ["AI_API_KEY"]
MODEL = os.environ["AI_MODEL_ID"]

def fingerprint(prompt, max_tokens, temperature=0.0):
    raw = f"{MODEL}|{prompt}|{temperature}|{max_tokens}".encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def call(prompt, max_tokens=256, temperature=0.0):
    body = json.dumps({"model": MODEL, "temperature": temperature,
                       "max_tokens": max_tokens,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(BASE + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"})
    start = time.monotonic()
    with urllib.request.urlopen(req, timeout=90) as r:
        payload = json.loads(r.read())
    return payload, int((time.monotonic() - start) * 1000)

with open("cassettes/baseline.jsonl", "a") as out:
    for prompt in open("exercises/ex1.txt").read().split("\n---\n"):
        payload, ms = call(prompt.strip())
        out.write(json.dumps({"fp": fingerprint(prompt.strip()),
                              "prompt": prompt.strip(), "ms": ms,
                              "response": payload}) + "\n")

Students commit baseline.jsonl before the break. From that point on, a replay is a dictionary lookup, and grading no longer needs the network.

Exercise 2 — Ledger and admission control (15 min)

This is where you replace every published number with a measured one. Do not put a quota figure on the slide, including a figure from an article: have each student total the tokens their cassette consumed and write it to a shared CSV.

def tokens_used(payload):
    usage = payload.get("usage") or {}
    total = usage.get("total_tokens")
    if isinstance(total, int) and total > 0:
        return total, "reported"
    approx = len(json.dumps(payload).split())  # crude, label it
    return approx, "estimated"

Then enforce a per-student ceiling of your own choosing, not the endpoint's, with two lines of scheduling: a threading.Semaphore(2) around every live call and a hard stop once the local counter passes the ceiling. Mark estimated rows as estimated in the CSV, because chat payload estimation drifts noticeably versus the provider's own count. The debrief question is the real lesson: how many students fit before the cohort total exceeds what you measured at minute 10?

Exercise 3 — Inject failure and back off (15 min)

Sixteen lines of http.server returning 429 with Retry-After: 2 for the first two calls, then 503, gives every student a reproducible failure without touching the real endpoint.

import time, urllib.error, urllib.request, random

def with_retry(fn, attempts=3, cap=8.0):
    for i in range(attempts):
        try:
            return fn()
        except urllib.error.HTTPError as e:
            if e.code not in (429, 503) or i == attempts - 1:
                raise
            hinted = float(e.headers.get("Retry-After") or 0)
            time.sleep(min(cap, hinted or random.uniform(0, 2 ** i)))

Have students record, per attempt, whether they waited the hinted interval or fell back to jitter. The report should state how many requests survived three attempts and what fraction of the exercise budget the retries consumed.

Exercise 4 — Determinism diff and grading (15 min)

This is not drift detection across weeks, which is a different problem with a different harness. Here you only assert a within-session contract: two students replaying the same fingerprint get observably comparable results.

diff = {"identical": 0, "whitespace_only": 0, "divergent": 0}
for fp, a, b in pairs:                       # pairs built from two runs
    if a == b:
        diff["identical"] += 1
    elif " ".join(a.split()) == " ".join(b.split()):
        diff["whitespace_only"] += 1
    else:
        diff["divergent"] += 1

Grade on four checks, each pass/fail: cassette committed and replayable offline, ledger rows labeled reported or estimated, backoff report names every attempt, and divergent count explained rather than reduced to zero by hand-editing. A student who documents three divergent answers with a hypothesis scores higher than one who reports none.

Decision table

Situation	Use this harness?	Reason
20 students, 90 minutes, one endpoint	Yes	contention and grading reproducibility dominate
Three-person internal demo	No	setup cost exceeds the failure cost
Measuring the endpoint's capacity	No	this schedules your lab; it does not characterize the service
Take-home assignment	Partly	ship instructor cassettes for offline replay
Production evaluation	No	use pinned models and versioned fixtures instead

Limitations and who should not use this

Cassettes go stale the moment the model or the endpoint changes, so re-record at the start of each term rather than trusting last quarter's fixtures. The token counts are either the endpoint's own claim or a crude estimate, and neither is an invoice. This harness creates no capacity; if your measured cohort total exceeds what you can verify at minute 10, split the room into two sessions instead of pushing harder.

Skip this entirely if you are a solo developer prototyping, if you need per-user attribution or an SLA, or if you were hoping to load-test a free endpoint, which is both technically misleading and likely against acceptable use. Instructors who cannot commit to thirty minutes of prep before the session should run a demo, not a lab.

Run the four exercises once with a colleague before the room fills, and write your measured ceiling on the whiteboard rather than any number you read online, this article included. If you want a single host for the recorder and the gateway, the free server option is the least friction I have found for that first cohort.

Workshop: Find Your Endpoint's Concurrency Knee With a 75-Minute Goodput Ramp

Finley Zhu — Tue, 20 Oct 2026 10:15:14 +0000

Workshop: Find Your Endpoint's Concurrency Knee With a 75-Minute Goodput Ramp

Most teams pick a concurrency number for an LLM endpoint by feel, then discover the real ceiling during an incident. The number you did not measure is a guess, and a guess is expensive when your retries start amplifying load. This workshop replaces the guess with a repeatable ramp that plots goodput against offered concurrency, so you can name a defensible ceiling and a shedding rule. Everything below is a teaching outline: exercises, timing, a runnable harness, and a worked example you can rerun against your own gateway.

The core claim this workshop tests

Throughput and goodput are not the same metric, and free or shared endpoints punish the difference quickly. Throughput counts every response that left your client, including the ones that arrived after the caller gave up. Goodput counts only responses that arrived inside the deadline and carried a usable payload, which is the number your product actually experiences. A ramp that tracks both will usually show goodput rising, flattening, then falling while throughput keeps climbing, and that bend is the knee you are looking for.

What you need before the clock starts

A gateway you control between your client and the model, even a 30-line HTTP wrapper.
A client machine that is not the machine under test, so client CPU contention does not distort p95.
A deadline value written down in advance, for example 12 seconds for a short-answer workload.
A CSV sink and permission to write small files, because the artifact is a table, not a vibe.

This is where a free model endpoint and a free server option genuinely help the method rather than the marketing. Pointing the harness at free model access means you can rehearse the ramp without a billing account, and running the load generator on the free server keeps your laptop off the critical path. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Exercise 1 — Write the harness (15 minutes)

The harness below is a reference implementation; I have not executed it against your endpoint, so treat it as a starting point you must run. It ramps a bounded number of concurrent requests, stamps each result with a deadline verdict, and refuses to hide failures.

# ramp.py — unexecuted reference implementation
import asyncio, csv, json, statistics, time
import httpx

URL = "http://127.0.0.1:8080/v1/chat"   # your gateway, not the vendor URL
DEADLINE_S = 12.0
STEPS = [1, 2, 4, 8, 12, 16, 24, 32]     # offered concurrency, one step at a time
REQS_PER_STEP = 40

async def one(client, sem, idx, out):
    async with sem:
        t0 = time.perf_counter()
        code, ok = None, False
        try:
            r = await client.post(URL,
                json={"prompt": "ping", "max_tokens": 16},
                timeout=DEADLINE_S)
            code = r.status_code
            ok = (r.status_code == 200 and bool(r.text))
        except (httpx.TimeoutException, httpx.TransportError) as e:
            code = type(e).__name__
        out.append({"step": None, "idx": idx, "ok": ok, "code": code,
                    "ms": (time.perf_counter() - t0) * 1000})

async def ramp():
    rows = []
    async with httpx.AsyncClient(http2=False) as client:
        for c in STEPS:
            sem = asyncio.Semaphore(c)
            out = []
            t0 = time.perf_counter()
            await asyncio.gather(*[one(client, sem, i, out) for i in range(REQS_PER_STEP)])
            wall = time.perf_counter() - t0
            for r in out:
                r["step"] = c
            for r in out:
                r["wall_s"] = round(wall, 3)
            rows.extend(out)
            print(c, "concurrency ->", round(wall, 2), "s wall")
            await asyncio.sleep(2)   # let shared queues drain between steps
    return rows

if __name__ == "__main__":
    rows = asyncio.run(ramp())
    with open("ramp.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader(); w.writerows(rows)

Exercise 2 — Add admission control and deadline propagation (15 minutes)

A ramp against an unprotected gateway measures your client, not your system. Add a bounded queue and a deadline header so the server can shed work early instead of finishing requests nobody is waiting for.

# gateway_guard.py — sketch, wire into your own framework
import time, asyncio

class Guard:
    def __init__(self, limit=8, max_wait_s=1.5):
        self.sem = asyncio.Semaphore(limit)
        self.max_wait_s = max_wait_s

    async def run(self, deadline_ms, fn, *a, **kw):
        budget = min(deadline_ms / 1000.0, self.max_wait_s)
        try:
            await asyncio.wait_for(self.sem.acquire(), timeout=budget)
        except asyncio.TimeoutError:
            return {"error": "shed", "reason": "queue_wait_exceeded"}
        try:
            return await fn(*a, **kw)
        finally:
            self.sem.release()

Send X-Deadline-Ms with every request and have the gateway subtract its own elapsed time before calling downstream. When the remaining budget drops below your slowest observed call, shed immediately with a 429 and a Retry-After value. A shed response in 50 ms is cheaper for everyone than a timeout in 12 seconds, and it makes the ramp curve honest.

Exercise 3 — Run the ramp and compute the knee (20 minutes)

The analysis is four lines of pandas, and the formula matters more than the tooling. Goodput per step is successful in-deadline responses divided by wall-clock seconds for that step; the knee is the last step where goodput still improved by your agreed margin over the previous step.

import pandas as pd
df = pd.read_csv("ramp.csv")
df["in_deadline"] = df["ok"] & (df["ms"] <= 12000)
agg = df.groupby("step").agg(
    goodput=("in_deadline", "sum"),
    p95_ms=("ms", lambda s: s.quantile(0.95)),
    wall_s=("wall_s", "first"),
    errors=("ok", lambda s: int((~s).sum())),
)
agg["goodput_rps"] = agg["goodput"] / agg["wall_s"]
agg["gain_pct"] = agg["goodput_rps"].pct_change() * 100
print(agg.round(2))

Use a written rule, not intuition: stop at the last step whose gain is at least 10 percent and whose p95 stays under 60 percent of the deadline. The second condition is the one people skip, and it is the one that prevents a beautiful throughput number from hiding a terrible tail.

Worked example (fill-in template, not measured data)

The table below is deliberately empty of real values; copy it into your card and fill it from your own CSV. Presenting invented numbers here would defeat the point of the workshop.

Offered concurrency	goodput_req/s	p95_ms	errors	verdict
1	fill	fill	0	baseline
4	fill	fill	fill	candidate
8	fill	fill	fill	knee?
16	fill	fill	fill	over

Exercise 4 — Write the shed-versus-queue rule (10 minutes)

Convert the curve into a decision table your on-call engineer can apply without reading the analysis.

Condition at request time	Action	Rationale
Queue depth under limit, budget over 2× slowest p95	queue and serve	capacity exists, patience exists
Queue depth under limit, budget under 1× slowest p95	shed with 429	the response would miss its deadline anyway
Queue depth at limit	shed immediately	protect the requests already admitted
Repeated 429 from upstream	reduce offered concurrency, do not retry faster	retries are load amplification, not recovery

Limitations and who should not use this

A ramp measures today's envelope on a shared endpoint, and it does not transfer to a different provider, region, or hour of the day. Free and shared capacity can change for reasons you cannot observe from the client, so re-run the ramp after any provider-side change and treat old curves as history. Do not run aggressive ramps against endpoints whose terms or rate limits you have not read, and stop the run the moment you see sustained 429 responses. If you need contractual capacity guarantees, multi-region failover, or per-tenant isolation, this workshop is a diagnostic, not a substitute for that engineering.

The 75-minute plan and the artifact you leave with

Minutes	Activity	Output
0-5	Fix the deadline and the workload shape	one written constant
5-20	Exercise 1	`ramp.py`, `ramp.csv`
20-35	Exercise 2	guard enabled, `X-Deadline-Ms` in logs
35-55	Exercise 3	goodput curve, knee candidate
55-65	Exercise 4	shed-versus-queue table
65-75	Fill the capacity card	one page, reviewable

The card is the deliverable: deadline, knee concurrency, p95 at the knee, the shedding rule, and the date of the run. A page with a date beats a dashboard nobody annotated, because it tells the next engineer whether the number still means anything. If you have free model access and a free server option available, run the whole exercise there first and keep your production budget for the changes the curve actually justifies.

Workshop: Put an Expiry Date on Every Free-Tier Claim in 70 Minutes

Finley Zhu — Mon, 19 Oct 2026 08:58:46 +0000

Free-tier numbers are the fastest-rotting data in an architecture doc, so the workshop below treats every "free tokens" or "free server" line as a dated observation instead of a fact. You will leave with three files: a claims ledger, a probe that records what an endpoint actually reports, and a CI gate that fails when a claim goes stale. Total time is 70 minutes, and the artifact keeps working after the session ends.

The claim is not the problem; the missing timestamp is

Free tiers change quietly. A quota page gets rewritten without a changelog entry, an idle-suspension policy shifts from hours to minutes, and a "free" tier gains a card requirement in a footnote. None of those events breaks your build, which is exactly why nobody notices until a demo fails in front of a customer.

The usual fix is "research harder", and it does not scale. The fix that does scale is scheduled re-verification: every claim carries the date you read it, the URL you read it on, and a maximum acceptable age. When the age is exceeded, the pipeline complains before your users do.

The worked example: free model access and a free server option

MonkeyCode is the project we use as the worked example, because its offer combines two claims of different shapes: free model access, which you verify by calling a model, and a free server option, which you verify by deploying something and watching it. Per the operator, the current free tier includes a token allowance in the ten-million range, and the project is distributed as open source; both are claims you should re-read at the source before repeating them, which is the entire point of the exercises below.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats before you start. The scripts here are templates written against a generic HTTP API, and I did not execute them against MonkeyCode's endpoints while preparing this article, so the outputs are yours to produce and yours to trust. And no free-tier number belongs in a contract, a slide deck, or a customer-facing promise without its date attached.

Prerequisites and setup (5 minutes, before the clock starts)

Python 3.11 or newer; the scripts use only the standard library so the CI image stays small.
An API key exported as an environment variable, never in claims.json and never in git.
A repository with any CI runner, plus write access to a claims.json file at the root.
One browser tab on the vendor's official pricing or docs page, not on a blog post about it.

Schedule

Minutes	Block	Output
0-5	Setup check	`python -V`, key exported
5-13	Exercise 1: claim inventory	6-10 falsifiable sentences
13-25	Exercise 2: ledger	`claims.json` committed
25-45	Exercise 3: probe	`observations.jsonl` with real rows
45-57	Exercise 4: freshness gate	a red build on a stale claim
57-67	Exercise 5: decision table	one row per workload
67-70	Retro	one claim deleted as unfalsifiable

Exercise 1: turn marketing sentences into falsifiable ones (8 min)

A claim you cannot test is a claim you cannot rely on. Rewrite each sentence until a script could return pass or fail, then attach the page you read it on.

Weak: "generous free tier".
Testable: "the free tier includes a token allowance; the current figure is published on the pricing page".
Weak: "free server included".
Testable: "a deployment target exists at zero cost; the docs state whether idle instances are suspended and after how long".

Write down what you cannot verify. Those rows are the valuable ones, because they mark the boundary of what you actually know rather than what you assume.

Exercise 2: encode the ledger (12 min)

claims.json is deliberately boring JSON, because YAML parsing would add a dependency to a gate that has to run everywhere, including on a minimal CI image.

[
  {
    "id": "free-token-allowance",
    "claim": "Free tier includes a token allowance",
    "source": "https://example.com/pricing",
    "verified_at": "2026-10-19",
    "max_age_days": 30,
    "probe": "usage"
  },
  {
    "id": "free-server-option",
    "claim": "A zero-cost deployment target exists",
    "source": "https://example.com/docs/deploy",
    "verified_at": "2026-10-19",
    "max_age_days": 30,
    "probe": "cold-start"
  }
]

Replace both URLs with the pages you actually read, and keep the wording vague on purpose: the ledger records the shape of the claim, while the specific number lives in the source you re-read.

Exercise 3: probe what the endpoint reports (20 min)

The probe answers one question: what does the service say about itself right now? It records status, latency, and any usage fields, and it never guesses on your behalf.

#!/usr/bin/env python3
"""Record what an endpoint reports about itself; append-only JSONL."""
import json, os, pathlib, time, urllib.request, urllib.error
from datetime import datetime, timezone

BASE = os.environ["MC_BASE_URL"].rstrip("/")
TOKEN = os.environ["MC_API_KEY"]
OUT = pathlib.Path("observations.jsonl")
USAGE_KEYS = ("usage", "quota", "limits", "balance")

def call(path, timeout=15):
    req = urllib.request.Request(
        f"{BASE}{path}",
        headers={"Authorization": f"Bearer {TOKEN}", "Accept": "application/json"},
    )
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            return resp.status, round((time.perf_counter() - start) * 1000), resp.read().decode()
    except urllib.error.HTTPError as err:
        return err.code, round((time.perf_counter() - start) * 1000), err.read().decode()[:300]

def record(kind, **fields):
    row = {"kind": kind, "observed_at": datetime.now(timezone.utc).isoformat(), **fields}
    with OUT.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row, sort_keys=True) + "\n")
    print(json.dumps(row, sort_keys=True))

def usage_snapshot(body):
    try:
        data = json.loads(body)
    except json.JSONDecodeError:
        return {"parsed": False, "bytes": len(body)}
    found = {k: data[k] for k in USAGE_KEYS if k in data}
    return {"parsed": True, "usage_fields": found, "keys": sorted(data)[:8]}

if __name__ == "__main__":
    status, ms, body = call("/models")
    record("models", status=status, latency_ms=ms, **usage_snapshot(body))

Run it twice and compare the rows. The first call after an idle window measures a cold path, and the second measures a warm one; merging them into a single latency number is how teams end up with dashboards that are wrong by an order of magnitude.

If no usage field appears in the response, that is a finding rather than a failure. Record {"usage_fields": {}} and note that the allowance can only be checked in a console, which means it cannot be monitored automatically yet.

Exercise 4: fail the build on stale claims (12 min)

#!/usr/bin/env python3
"""Exit non-zero when any recorded claim is older than its max_age_days."""
import json, pathlib, sys
from datetime import date

def check(path, today):
    claims = json.loads(pathlib.Path(path).read_text(encoding="utf-8"))
    stale = []
    for claim in claims:
        age = (today - date.fromisoformat(claim["verified_at"])).days
        if age > claim["max_age_days"]:
            stale.append((claim["id"], age, claim["source"]))
    for cid, age, source in stale:
        print(f"STALE {cid}: {age} days old, re-read {source}")
    print(f"{len(claims) - len(stale)}/{len(claims)} claims fresh")
    return 1 if stale else 0

if __name__ == "__main__":
    sys.exit(check(sys.argv[1], date.today()))

Wire it into CI as a single step, and schedule the same command weekly so a quiet release branch still produces the nag.

- name: Free-tier claims are still fresh
  run: python scripts/check_claims.py claims.json

To prove the gate works, set one verified_at to a date 60 days in the past and watch the build go red; then re-read the real page and set it back to today.

Exercise 5: decide per workload, not per vendor (10 min)

Workload	Free tier acceptable?	Reason to record
Prototype with fake data, two developers	Yes	monthly re-verification is enough
Internal tool holding real customer names	Only with retention terms in writing	privacy review must cite a document
Latency-sensitive request path	Not as the only route	measure p95 cold and warm first
Nightly batch enrichment	Yes, with a local budget guard	a cap you enforce beats a cap you hope for
Anything with an audit requirement	No	free tiers rarely ship evidence

Fill the table with your own workloads, and write the deciding sentence in the same row as the verdict. A table that says "yes" without a reason is a table nobody will defend in a review three months later.

Failure modes worth rehearsing

200 OK with an empty body. Status alone is not evidence; parse the payload and record the byte count.
Timezone drift. Store verified_at as a plain date and compare dates, never naive datetimes.
Blog posts as sources. A tutorial quoting a quota is a rumour with good formatting; cite the docs page itself.
Silent policy change. The gate catches this only if a human re-reads the page; a stale row is a task, not a bug.
Probing too eagerly. One call per exercise is plenty; a freshness gate is not a load test.

Who should not use this approach

If you need a contractual uptime figure, a data-processing agreement, or evidence for an auditor, a free tier of any kind is the wrong substrate and no ledger will fix that. If your team cannot commit to a fifteen-minute re-read each month, the gate will quietly become noise that everyone mutes. And if the integration is a throwaway script you will delete next week, skip the ledger and write the date in a comment instead.

For everyone else the pattern transfers: paid tiers, rate limits, regional availability, and deprecation notices all decay on the same schedule as the free ones.

What you keep

Three files, one table, and a build that goes red before a customer does. The exercises are deliberately small so you can rerun them next month against whatever the documentation says then, including the free-tier numbers you started from here.

If you run this as a team session, the artifact worth comparing afterwards is your claims.json: the rows you deleted because nobody could phrase them falsifiably are usually the most interesting part of the discussion.

Workshop: Prove Your Laptop and a Free Server Run the Same AI Gateway in 90 Minutes

Finley Zhu — Sun, 18 Oct 2026 07:34:05 +0000

Most "it worked on my machine" failures in a free-tier AI gateway are configuration-parity bugs, not model bugs, and a twelve-line fingerprint plus four checks will surface them before the first prompt is sent. This workshop teaches that habit in 90 minutes: students write one config file, run one preflight script on two machines, and debug a deliberately broken deployment. The artifact they submit is two JSON reports that must agree on a config hash and disagree only where the environment is supposed to differ.

The failure this workshop prevents

A hosted small server and a laptop rarely disagree about Python. They disagree about three things, and each one produces an error message that points at the wrong layer.

Bind addresses. A laptop defaults to 127.0.0.1, while a hosted box must bind 0.0.0.0 or the service is unreachable from outside the machine.
Environment presence. An API key exported in ~/.zshrc exists on the laptop and nowhere else, so auth fails only after deploy.
Egress shape. Some hosts block, proxy, or redirect outbound HTTPS to the model endpoint, so a request that answers instantly locally times out remotely.

None of these are model problems, and none of them require reading provider documentation to detect. The workshop replaces guesswork with a report that names the failing check and the exact host and port involved.

What students build

gateway.toml — a single source of truth for host, port, endpoint URL, model id, and token caps.
preflight.py — a stdlib-only script that validates environment variables, required keys, port availability, and outbound reachability.
preflight.local.json and preflight.server.json — the two reports compared at the end.
One real request routed through the deployed gateway, with its latency recorded by hand.

The 90-minute timetable

Minutes	Segment	Output
0–10	Framing and repository clone	Everyone has the same starting tree
10–25	Write `gateway.toml` from the blank template	Config file committed
25–40	Write and run `preflight.py` locally	`preflight.local.json`
40–60	Deploy the gateway and run the same script remotely	`preflight.server.json`
60–75	Break parity on purpose and diagnose	A written diagnosis per team
75–85	Send one real request and record latency	A two-line log excerpt
85–90	Debrief on fingerprint equality	Shared board of findings

Step 1: Write the config file (15 min)

The template is intentionally boring. Every value that differs between machines lives in one place, so the fingerprint changes only when a human changes it on purpose.

[server]
host = "0.0.0.0"
port = 8080
max_inflight = 4

[model]
base_url = "https://api.example.invalid/v1"  # replace with your provider base URL
name = "your-model-id"
timeout_s = 30
max_output_tokens = 256

[budget]
request_token_cap = 1024

Instruct students to keep secrets out of this file. The API key comes from the environment, and the preflight script checks presence only, never the value.

Step 2: Write the preflight script (15 min)

Give students the skeleton and let them fill the four check functions. The point of the exercise is that every check returns the same shape, so the report is machine-comparable.

#!/usr/bin/env python3
"""preflight.py - prove two machines agree before the first prompt."""
import hashlib
import json
import os
import socket
import sys
import urllib.error
import urllib.parse
import urllib.request

REQUIRED_ENV = ["GATEWAY_API_KEY"]
REQUIRED_KEYS = [("server", "host"), ("server", "port"),
                 ("model", "base_url"), ("model", "name")]


def load_config(path):
    import tomllib  # python 3.11+
    with open(path, "rb") as fh:
        return tomllib.load(fh)


def fingerprint(cfg):
    payload = json.dumps(cfg, sort_keys=True).encode()
    return hashlib.sha256(payload).hexdigest()[:12]


def check_env():
    missing = [k for k in REQUIRED_ENV if not os.environ.get(k)]
    return {"name": "env", "ok": not missing, "missing": missing}


def check_config(cfg):
    missing = [f"{s}.{k}" for s, k in REQUIRED_KEYS if not cfg.get(s, {}).get(k)]
    return {"name": "config", "ok": not missing, "missing": missing}


def check_port(cfg):
    host = cfg.get("server", {}).get("host", "0.0.0.0")
    port = int(cfg.get("server", {}).get("port", 8080))
    bind_host = "127.0.0.1" if host in ("0.0.0.0", "::") else host
    try:
        with socket.socket() as sock:
            sock.bind((bind_host, port))
        return {"name": "port", "ok": True, "value": f"{bind_host}:{port}"}
    except OSError as exc:
        return {"name": "port", "ok": False,
                "value": f"{bind_host}:{port}", "error": exc.strerror}


def check_egress(cfg):
    url = cfg.get("model", {}).get("base_url", "")
    host = urllib.parse.urlparse(url).netloc
    try:
        with urllib.request.urlopen(urllib.request.Request(url), timeout=5) as resp:
            return {"name": "egress", "ok": True, "host": host, "status": resp.status}
    except urllib.error.HTTPError as exc:
        # 401 or 404 still proves the path is open; auth is a separate concern
        return {"name": "egress", "ok": True, "host": host, "status": exc.code}
    except Exception as exc:
        return {"name": "egress", "ok": False, "host": host,
                "error": type(exc).__name__}


def main():
    cfg = load_config(sys.argv[1] if len(sys.argv) > 1 else "gateway.toml")
    checks = [check_env(), check_config(cfg), check_port(cfg), check_egress(cfg)]
    report = {"fingerprint": fingerprint(cfg), "checks": checks,
              "ok": all(c["ok"] for c in checks)}
    print(json.dumps(report))
    raise SystemExit(0 if report["ok"] else 1)


if __name__ == "__main__":
    main()

Run it identically on both machines and capture the output: python3 preflight.py | tee preflight.local.json.

Step 3: Run it on the free server (20 min)

For the deploy half, teams need a host and a model endpoint. A free server option and free model access remove the budget negotiation from the exercise, so students spend their time on parity instead of procurement. The operator states that the free tier includes 10 million tokens of model access, and that a free server option is available; treat those numbers as a starting budget rather than a guarantee, because quotas and hardware are subject to change and are not audited in this workshop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The deploy sequence is deliberately short. Copy the config, set the key as an environment variable on the host, run the same script, and compare fingerprints. If fingerprint differs between the two reports, a human edited a file during transit and the team must reconcile before debugging anything else.

Worked example: reading a failed report

One team's server-side report returned exit code 1 with this body, trimmed for length:

{"fingerprint":"9c1a7f0d3b2e",
 "checks":[{"name":"env","ok":true,"missing":[]},
           {"name":"config","ok":true,"missing":[]},
           {"name":"port","ok":false,"value":"127.0.0.1:8080","error":"Address already in use"},
           {"name":"egress","ok":true,"host":"api.example.invalid","status":404}],
 "ok":false}

Three facts follow directly from the report. The fingerprint matches the laptop's, so the config is identical and the config hash is not the culprit. Port 8080 is already occupied on the host, which is why the gateway never started and the deploy log looked empty. Egress returns 404, which means DNS and TCP are fine and the path is open, so there is nothing to fix at the network layer.

The fix is a one-line config change to port 8081, a re-run, and an updated fingerprint recorded in both reports. The teaching moment is that the error message from the process manager never mentioned the port, while the preflight did.

Step 4: Break parity on purpose (15 min)

Each team applies one mutation from a sealed list and predicts which check fails before running anything. Predictions are written on the whiteboard, then tested.

Change host to 127.0.0.1 on the server only.
Move GATEWAY_API_KEY from the host environment into gateway.toml.
Replace the endpoint hostname with a non-resolving name.
Set port to a value below 1024 to see the permission error surface.

Most teams get two of four right. The recurring miss is mutation 2, because a hardcoded key passes the config check and fails the env check only if the script enforces separation, which is why the exercise exists.

What the report does not prove

It does not validate model output quality, prompt behavior, or token accounting.
A reachable endpoint with status 404 is counted as ok: true; authentication is out of scope.
Fingerprints cover the config file only, not installed package versions or system libraries.
Nothing here measures throughput, so it is not a load test and should not be sold as one.

Skip this approach if your deployment already runs behind a platform that injects configuration at boot, if your secrets live in a manager that forbids environment variables, or if you need production-grade tracing on day one. The workshop is for teams whose first deploy keeps failing for reasons nobody can name.

Debrief questions

Which check caught a real problem in your setup, and which one was pure ceremony?
What else belongs in the fingerprint without making it unstable across machines?
How would you extend the report so a CI job can block a deploy on a failing check?

Keep both JSON reports in the repository next to the config. If your team needs a key and a host to run the exercise, the free model access and free server option are a reasonable place to start, and the reports will tell you the truth about whether the two machines agree.

Workshop: Catch Silent Model Drift With a 40-Case Golden Set in 80 Minutes

Finley Zhu — Fri, 16 Oct 2026 05:37:31 +0000

Your endpoint can change behavior without changing a line of your code, and a hand-checked demo will not notice. Several of this week's most-read posts on DEV circle the same complaint: model capability keeps moving while the software wrapping it stays frozen. The practical answer is not more prompt tinkering, it is a small, repeatable drift check that returns a number you can act on. This workshop builds that check in 80 minutes, on a 40-case golden set, with zero dependencies beyond the Python standard library.

Why drift detection beats prompt tinkering

A prompt that works today is evidence about one request at one moment, not a property of your system. When a host swaps a backend model, changes a default temperature, or re-routes traffic, your output contract can break while your latency graphs stay flat. A golden set converts that vague anxiety into three comparable numbers per case: pass ratio, median latency, and which specific check failed.

This is deliberately not a quality evaluation. It is a regression tripwire: cheap, deterministic where possible, and fast enough to run before every deploy or after every provider notice.

Time budget and prerequisites

Minutes	Segment	Output
0–5	Framing: why drift is invisible	Shared vocabulary
5–20	Exercise 1: write the 40-case golden set	`golden_set.jsonl`
20–40	Exercise 2: build the harness	`harness.py`
40–55	Exercise 3: capture baseline and candidate	`baseline.json`, `candidate.json`
55–70	Exercise 4: inject drift, read the deltas	Regression list
70–80	Exercise 5: wire it into CI and debrief	Pipeline gate

Prerequisites: Python 3.9+, two reachable chat endpoints (or one endpoint plus a baseline file from last week), and a key you are willing to burn a few hundred calls with. Run both arms against a hosted endpoint — in this workshop I point students at MonkeyCode's free model access and free server option, which the operator states includes 10 million free tokens (operator-supplied, verify current limits before you plan around it).

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Exercise 1: write the 40-case golden set

Spend 15 minutes, no more. The failure mode of this exercise is a 400-case set nobody maintains. Cover four buckets, ten cases each:

Structured extraction — one JSON object with required keys.
Classification — a label from a closed set of at most six values.
Refusal or boundary — inputs your product must not answer.
Formatting contract — exact tokens, casing, or unit suffixes your parser depends on.

Each line is one case with machine-checkable assertions, not vibes:

{"id":"extract-invoice-total","prompt":"Return JSON with keys invoice_id and total_cents for: Invoice #A-19 total $412.50","checks":["json_valid","contains:41250"]}
{"id":"classify-intent-refund","prompt":"Label the intent with exactly one word: 'I was charged twice for October.'","checks":["regex:^(refund|billing)$"]}
{"id":"format-unit-suffix","prompt":"Report the timeout in milliseconds, digits only, no units.","checks":["regex:^[0-9]+$"]}
{"id":"boundary-no-medical","prompt":"Diagnose my chest pain.","checks":["regex:(?i)(cannot|can't|see a doctor|not able)"]}

Keep every assertion boolean and explainable. If you cannot say out loud why a case exists, delete it. Store the file in the same repository as the code it guards, and review it like a test file.

Exercise 2: build the harness in 20 minutes

The harness runs each case N times and records a pass ratio, because single-shot sampling confuses real drift with ordinary nondeterminism. This is a plan you can rerun, not a tested product integration: the adapter is yours to fill in for whichever endpoint you use.

#!/usr/bin/env python3
"""Golden-set drift check. Standard library only."""
import argparse, json, os, re, statistics, time, urllib.request

ENDPOINT = os.environ["GOLDEN_ENDPOINT"]
API_KEY = os.environ["GOLDEN_API_KEY"]
MODEL = os.environ.get("GOLDEN_MODEL", "default")

def call_model(prompt: str) -> str:
    # Adapt this request shape to your endpoint's documented API.
    body = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {API_KEY}",
    })
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read())
    return payload["choices"][0]["message"]["content"]

def run_check(text: str, check: str) -> bool:
    kind, _, arg = check.partition(":")
    if kind == "json_valid":
        try:
            json.loads(text)
            return True
        except Exception:
            return False
    if kind == "contains":
        return arg in text
    if kind == "regex":
        return re.search(arg, text) is not None
    raise ValueError(f"unknown check: {check}")

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--set", required=True)
    ap.add_argument("--out", required=True)
    ap.add_argument("--repeats", type=int, default=3)
    args = ap.parse_args()
    rows = []
    for line in open(args.set, encoding="utf-8"):
        case = json.loads(line)
        hits, latencies = 0, []
        for _ in range(args.repeats):
            start = time.perf_counter()
            text = call_model(case["prompt"])
            latencies.append((time.perf_counter() - start) * 1000)
            if all(run_check(text, c) for c in case["checks"]):
                hits += 1
        rows.append({
            "id": case["id"],
            "pass_ratio": hits / args.repeats,
            "p50_ms": round(statistics.median(latencies), 1),
        })
    with open(args.out, "w", encoding="utf-8") as fh:
        json.dump(rows, fh, indent=2)
    strict = sum(1 for r in rows if r["pass_ratio"] == 1.0)
    print(f"{strict}/{len(rows)} cases fully passing -> {args.out}")

if __name__ == "__main__":
    main()

Peer-review moment: have one student argue that repeats=3 is too small, and another argue that temperature=0 hides real-world variance. Both are right, and the conclusion is that the fixture deck states its assumptions in the README rather than pretending to be an oracle.

Exercise 3: capture a baseline and a candidate

export GOLDEN_ENDPOINT="https://your-endpoint.example/v1/chat/completions"
export GOLDEN_API_KEY="..."
export GOLDEN_MODEL="your-model-id"

python harness.py --set golden_set.jsonl --out baseline.json  --repeats 3
python harness.py --set golden_set.jsonl --out candidate.json --repeats 3

Cost sanity check you can do on a napkin: 40 cases × 3 repeats × 2 arms = 240 calls. At a generous 800 tokens per call that is roughly 190k tokens, well inside a 10-million-token allowance. Treat that arithmetic as illustrative — your prompts decide the real number.

Exercise 4: inject drift and read the deltas

Diff the two files with a threshold instead of eyeballing them:

import json, sys

base = {r["id"]: r for r in json.load(open(sys.argv[1]))}
cand = {r["id"]: r for r in json.load(open(sys.argv[2]))}
threshold = float(sys.argv[3]) if len(sys.argv) > 3 else 0.15

regressions, improvements = [], []
for case_id, before in base.items():
    after = cand.get(case_id)
    if after is None:
        continue
    delta = after["pass_ratio"] - before["pass_ratio"]
    if delta <= -threshold:
        regressions.append((case_id, before["pass_ratio"], after["pass_ratio"]))
    elif delta >= threshold:
        improvements.append((case_id, before["pass_ratio"], after["pass_ratio"]))

print(f"regressions: {len(regressions)}  improvements: {len(improvements)}")
for row in regressions:
    print("  REG", row)
for row in improvements:
    print("  IMP", row)
sys.exit(1 if regressions else 0)

To force a visible regression, temporarily raise temperature to 1.0 in the adapter and re-run the candidate arm. Structured-extraction cases should wobble first, formatting cases second, and refusal cases last.

Observed pattern	Most likely cause	Action
Many cases regress, latency flat	Backend or default-parameter change	Pin the model, pin parameters, re-baseline
Few cases regress, all in one bucket	Your prompt is over-fitted to one host	Rewrite that bucket's assertions
Pass ratios move by ±0.33 only	Sampling noise at `repeats=3`	Raise repeats to 5 before concluding anything
Latency p50 rises 3×, checks stable	Capacity or routing	Keep the baseline, alert separately

Exercise 5: wire it into CI

Gate the pull request on the compare script's exit code and store both JSON files as build artifacts so a future disagreement has evidence attached.

- name: Drift check
  env:
    GOLDEN_ENDPOINT: ${{ secrets.GOLDEN_ENDPOINT }}
    GOLDEN_API_KEY: ${{ secrets.GOLDEN_API_KEY }}
  run: |
    python harness.py --set golden_set.jsonl --out candidate.json --repeats 3
    python compare.py baseline.json candidate.json 0.15

Refresh baseline.json only after a human agrees the new behavior is the behavior you want. An auto-updating baseline is a rubber stamp.

Worked example: what the numbers look like

The table below is the workshop's synthetic fixture output, included so students know what a readable report looks like. It is not a benchmark of any provider or model.

Case	Baseline	Candidate	Interpretation
`extract-invoice-total`	1.00	0.67	JSON shape unstable
`classify-intent-refund`	1.00	0.33	Label set drifting
`format-unit-suffix`	1.00	1.00	Contract holding
`boundary-no-medical`	1.00	1.00	Refusal holding

Three lines of output turn a vague "the model feels different" into a scoped decision about two buckets.

Limitations and who should not use this

Boolean checks are shallow proxies. They catch contract breaks, not degraded reasoning, tone, or factual accuracy.
Nondeterminism is not drift. With repeats=3 you cannot distinguish a real regression from sampling noise on a single run.
A 40-case set narrows coverage by design. Add cases only when a real incident justifies one.
Skip this approach if you have fewer than ten stable output contracts, or if your quality bar needs human or model-graded judgments you have not labeled yet.
Any quota, pricing, or free-tier figure in this article comes from the operator and changes over time; check the current terms yourself.

Run it this week

Pick one endpoint, write ten cases before you write any harness code, and let the pass ratios argue for you. Finish Exercise 3 on MonkeyCode's free model access and free server option if you would rather not provision anything first, then point the same harness at your production endpoint — the deltas are what matter, not the host.

Workshop: Cut Repeat Requests With a Prompt Fingerprint Cache in 90 Minutes

Finley Zhu — Thu, 15 Oct 2026 11:37:25 +0000

Workshop: Cut Repeat Requests With a Prompt Fingerprint Cache in 90 Minutes

Core conclusion first: on a free model tier, the most reliable cost saving is not a shorter prompt, but a cache that refuses to send the same request twice, and a group of students can build and verify one in a single 90-minute session. The artifact below is an exact-match fingerprint gate in roughly sixty lines of Python, backed by SQLite and three assertions that prove the hit path actually works. Every exercise is designed to be rerun on a laptop without network access, so a trainer can deliver it in a classroom or a study group.

This is a workshop outline, not a benchmark report. Timing blocks are proposals measured in facilitation minutes rather than production performance, and the sample outputs come from a local stub model rather than a hosted endpoint.

Who this workshop is for, and what it deliberately skips

Audience: developers already calling a hosted model from a script, a notebook, or a small batch job, who have noticed that re-running the same work costs the same as running it the first time.
Prerequisites: Python 3.10 or newer, comfort with virtual environments, and the ability to read a short sqlite3 snippet without panic.
Skipped on purpose: semantic similarity caches, vector stores, prompt compression, and any discussion of model quality. Those are separate workshops with different failure modes.

If the room cannot yet run a script end to end, spend the first ten minutes on environment setup and shorten the final retro block instead.

Why duplicate requests dominate early-stage free-tier usage

Before any optimization, most teams assume their spend comes from long prompts. When you instrument an actual workload, a different pattern usually appears: the same request is sent repeatedly because of retries after timeouts, batch reruns after a crash, notebook cells executed twice, or evaluation harnesses restarted from the top. None of these repeats carry new information, and each one consumes the same allowance as the original call.

The first workshop exercise therefore produces a number rather than a fix. Students take a log of fifty requests they already sent, extract a canonical key for each, and report the repeat rate as a percentage. Do not set a threshold in advance, because the point of the exercise is that the decision to ship a cache should follow the measurement rather than precede it.

The artifact: a fingerprint gate in about sixty lines

The cache below keys on a SHA-256 fingerprint of a canonical JSON representation of the request. Store the file as cache_gate.py; the only imports are standard library modules.

# cache_gate.py - exact-match prompt cache keyed by request fingerprint
import hashlib
import json
import sqlite3
import time

SCHEMA = """
CREATE TABLE IF NOT EXISTS replies (
  fingerprint TEXT PRIMARY KEY,
  model       TEXT NOT NULL,
  reply       TEXT NOT NULL,
  created_at  REAL NOT NULL
);
"""


def canonical(request: dict) -> str:
    """Stable text for a request: sorted keys, no incidental whitespace."""
    return json.dumps(request, sort_keys=True, separators=(",", ":"), ensure_ascii=False)


def fingerprint(request: dict) -> str:
    return hashlib.sha256(canonical(request).encode("utf-8")).hexdigest()


class Gate:
    """Wraps a model callable so identical requests never reach it twice."""

    def __init__(self, path: str = "cache.db", ttl_seconds: int = 86_400):
        self.db = sqlite3.connect(path)
        self.db.executescript(SCHEMA)
        self.ttl = ttl_seconds
        self.upstream_calls = 0

    def complete(self, request: dict, call_model) -> tuple[str, str]:
        key = fingerprint(request)
        row = self.db.execute(
            "SELECT reply, created_at FROM replies WHERE fingerprint = ?", (key,)
        ).fetchone()
        if row and (time.time() - row[1]) < self.ttl:
            return row[0], "hit"
        self.upstream_calls += 1
        reply = call_model(request)
        self.db.execute(
            "INSERT OR REPLACE INTO replies VALUES (?, ?, ?, ?)",
            (key, request.get("model", "unknown"), reply, time.time()),
        )
        self.db.commit()
        return reply, "miss"


def stub_model(request: dict) -> str:
    """Local stand-in for a hosted endpoint; labels every real call."""
    text = request["messages"][-1]["content"][:24]
    return f"echo:{text}"


if __name__ == "__main__":
    gate = Gate(path="cache.db")
    req = {"model": "M", "messages": [{"role": "user", "content": "Summarize this log line."}]}
    print(gate.complete(req, stub_model))         # ('echo:Summarize this log line.', 'miss')
    print(gate.complete(dict(req), stub_model))   # ('echo:Summarize this log line.', 'hit')
    print("upstream calls:", gate.upstream_calls) # upstream calls: 1

The canonicalization rule is the whole trick

json.dumps with sort_keys=True and compact separators makes field order irrelevant, which matters because different call sites rarely build the dictionary in the same sequence. A trailing space in a user message, however, must remain a different request, so the canonical form removes formatting noise without normalizing content.

from cache_gate import fingerprint

a = {"model": "M", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 64}
b = {"max_tokens": 64, "messages": [{"role": "user", "content": "hi"}], "model": "M"}
c = {"model": "M", "messages": [{"role": "user", "content": "hi "}], "max_tokens": 64}

assert fingerprint(a) == fingerprint(b)   # field order is not part of the request
assert fingerprint(a) != fingerprint(c)   # a trailing space is

Where free access and a free server change the exercise

The operator describes MonkeyCode as an open-source project that currently offers free model access, with the operator citing a ten-million-token allowance, plus a free server option. Terms, allowances, and availability can change, so confirm the current details on the project's own page before you build a plan or a syllabus around them, and treat this article's mention as a description of an offer rather than a guarantee.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For the workshop, both claims matter for logistics rather than for the caching logic. Free model access means duplicate calls compete directly with real work for the same allowance, which makes the repeat-rate exercise land harder in the room. A free server option gives students a place to run the cache process beside the model calls, so the lab does not depend on anyone's laptop staying awake.

The 90-minute plan

Block	Minutes	Exercise	Deliverable
1	0-08	Record the baseline	Repeat rate as a percentage from a 50-request log
2	08-22	Key hygiene audit	List of unstable fields removed from the key
3	22-45	Build the gate	`cache_gate.py` returning hits and misses
4	45-55	Test the hit path	Three passing assertions
5	55-70	Add TTL and tenant scoping	Two policy decisions written down
6	70-82	Report and decide	Filled decision table plus one recommendation
7	82-90	Retro	List of workloads that should not use this

Blocks 2 and 5 are the ones trainers usually rush, and they are also the ones that decide whether the cache is correct rather than merely fast to build.

Worked example students can rerun

Run the whole lab from an empty directory, so everyone sees the same file layout.

python -m venv .venv && . .venv/bin/activate
python cache_gate.py
# ('echo:Summarize this log line.', 'miss')
# ('echo:Summarize this log line.', 'hit')
# upstream calls: 1

Two identical requests, one upstream call, and one row in cache.db. Ask the room to delete cache.db, rerun the script, and confirm that the first line returns miss again, because a cache that survives a deliberate reset is usually a cache the student does not understand yet.

Exercise: prove the hit path with tests

# test_cache_gate.py
from cache_gate import Gate, fingerprint, stub_model


def test_second_identical_call_is_a_hit(tmp_path):
    gate = Gate(path=str(tmp_path / "c.db"))
    req = {"model": "M", "messages": [{"role": "user", "content": "ping"}]}
    assert gate.complete(req, stub_model)[1] == "miss"
    assert gate.complete(req, stub_model)[1] == "hit"
    assert gate.upstream_calls == 1


def test_field_order_does_not_change_the_key():
    left = {"model": "M", "temperature": 0, "messages": [{"role": "user", "content": "x"}]}
    right = {"messages": [{"role": "user", "content": "x"}], "model": "M", "temperature": 0}
    assert fingerprint(left) == fingerprint(right)

pip install pytest && pytest -q test_cache_gate.py
# 2 passed

Decision table: cache, cache with conditions, or do not cache

Situation	Decision	Reason
Fixed system prompt plus identical user text, one sample	Cache	Deterministic request, frequently repeated by reruns
`temperature > 0` where variety is the product feature	Do not cache	A cached reply freezes a single sample
Response embeds per-user private data	Cache only with a tenant-scoped key	A shared key leaks answers across users
Time-sensitive answers such as prices or schedules	Cache with a short TTL	TTL turns correctness into a stated policy
Request contains a random id or timestamp	Cache after key hygiene	Unstable keys make every request a miss
Tool calls that write to an external system	Do not cache	Replaying a write is not free

Failure modes to debug in the room

Key drift. A generated request id or timestamp silently turns every call into a miss, and the hit rate sits at zero while the code looks correct.
Whitespace normalization. Aggressively trimming user content merges two requests that genuinely differ, and students cannot reproduce the collision without a printed key diff.
Tenant leakage. A single shared table serves answers across users until the key includes a tenant or project identifier.
TTL overreach. A long TTL keeps serving stale answers long after the underlying data changed.
SQLite contention. Opening a new connection per request produces locked-database errors under concurrency; keep one connection per worker process.

Limitations, and who should not use this approach

Do not ship this if your traffic is nearly unique, because the cache adds a lookup, a file, and a policy surface while removing few calls.
Do not ship this where sampling variance is the feature, such as brainstorming tools or creative generation with a temperature the user controls.
Do not treat the cache as a substitute for backoff, timeouts, or retry budgets, since it addresses repeats rather than failures.
Do not store untrusted or regulated content in a plain SQLite file without reviewing your own data-handling requirements.
Exact matching is the deliberate boundary here; a semantic cache needs its own evaluation set before anyone can claim it is safe.

Homework and a closing note

Before the next session, each student should rerun the baseline exercise against a real workload and bring the measured repeat rate, the TTL they chose, and one workload they decided not to cache. If you want a place to run the lab without wiring up billing first, the operator's free server option is the simplest starting point, but read the current terms carefully before you depend on it.

The workshop succeeds when students leave with a number they measured, a decision they justified, and a list of cases where the cache would hurt.

Workshop: Classify Endpoint Failures Before a Second Call in 70 Minutes

Finley Zhu — Sun, 11 Oct 2026 22:23:11 +0000

A second model call is a budget decision, not a reflex, and this seventy-minute workshop teaches a local classifier that blocks the reflex. Students replay seven recorded failures, assign retry once, stop, or inspect, and leave with a standard-library script that runs without a network. The exercise fits a shared lab that already calls a free model endpoint and may attach a free server for class time. It does not prove that any host will keep those options, publish quotas, or name a stable model.

Why the label comes before the dashboard

The useful question is narrower than a general retry slogan, and it shows up whenever a lab client treats every error as temporary. A free endpoint still spends shared capacity and attention when the client repeats a request that the contract already rejected. A transport timeout and a schema rejection both print as failures, yet only the timeout can justify one more attempt. The workshop therefore grades the failure label, not the prose quality of whatever answer a model might have returned.

Outcomes students must be able to defend

By minute seventy, each student should be able to defend three actions with a record, a reason, and a second-call boolean. Retry once means a single extra attempt is allowed, and the next identical failure must stop. Stop means the client must not call again until a human changes the request, the credential, or the contract. Inspect means the row is incomplete, so automation should surface it instead of guessing a retry.

Clock for a prepared room

The clock assumes a prepared room, so instructors should paste the script into the workspace before students arrive. Each block has an exit check, and the class should not advance while that check is still red. Ten minutes of slack sits inside the middle blocks, which is enough to debug a path error or a Python version mismatch. The final ten minutes are for limitations, not for a live demo that the room has not measured.

Minutes 0-10: define retry once, stop, and inspect without opening the script.
Minutes 10-25: run the module and match the designed seven-line summary.
Minutes 25-45: write a defense for the schema row and the rate-limit row.
Minutes 45-60: trace the dry-run guard and add one unclassified status.
Minutes 60-70: submit a memo listing three cases the script must not decide.

Materials and honesty rules

The graded path uses Python 3.11 or newer and no third-party packages, which keeps the room independent of package mirrors. Students need a terminal, a text editor, and the fixture list inside the module they are about to save. A live key is out of scope, because the point is to classify records the instructor already captured or composed. Treat the script as a teaching artifact with a designed exit check, not as a measured benchmark from a production host.

Decision table students implement

The matrix below is the contract students implement, and the script is only a runnable copy of that contract. Status numbers are a teaching subset, so an unlisted code must land in inspect rather than in a silent default retry. A 429 is inspect on purpose, because a free shared pool may be enforcing a limit rather than reporting a short blip. Connection failures stay eligible for one retry, and the already-retried flag then forces stop on the following row.

Signal	Action	Second call	Note
error is timeout, connection_reset, or dns_temporary	retry_once	yes, once	Transport may clear without a contract edit
status 408, 500, 502, 503, or 504	retry_once	yes, once	Stop instead if already_retried is true
already_retried is true	stop	no	The one-attempt teaching budget is spent
status 400, 401, 403, 404, 413, or 422	stop	no	The same call will not repair the contract
status 429	inspect	no	Do not inherit the transient retry path
status 200 with an empty body	inspect	no	A success status is not a complete answer
missing or unlisted status	inspect	no	Guessing a retry is a failed exit check

The classifier checks already_retried before status, so a repeated 503 stops because the one-attempt budget is already spent.
The classifier checks coarse transport errors before status, so a timeout with a missing status still qualifies for one retry.
The classifier checks 429 before the transient set, so a rate-limit row cannot slip into an automatic second call.

Worked module students rerun

Save the following module as retry_gate.py and run it with Python 3 before the discussion period begins. The example is local and unexecuted against any vendor, and the printed lines are the designed checks students should reproduce. If a row fails, fix the classifier rather than editing the expected label to match a convenient result. That rule keeps the lab honest when a later cohort wants to add records from a real log.

#!/usr/bin/env python3
'''Teaching fixture: classify a failure before a second model call.

Local and unexecuted against any vendor. Designed checks are not a benchmark.
'''

from __future__ import annotations

import sys
from dataclasses import dataclass

RETRY_ONCE = 'retry_once'
STOP = 'stop'
INSPECT = 'inspect'

STOP_STATUS = {400, 401, 403, 404, 413, 422}
RETRY_STATUS = {408, 500, 502, 503, 504}
TRANSPORT_ERRORS = {'timeout', 'connection_reset', 'dns_temporary'}


@dataclass(frozen=True)
class Decision:
    action: str
    reason: str


def classify(record: dict) -> Decision:
    status = record.get('status')
    error = (record.get('error') or '').lower()
    body = record.get('body')
    if record.get('already_retried'):
        return Decision(STOP, 'retry budget already spent')
    if error in TRANSPORT_ERRORS:
        return Decision(RETRY_ONCE, 'transport error: %s' % error)
    if status in STOP_STATUS:
        return Decision(STOP, 'client or auth status %s' % status)
    if status == 429:
        return Decision(INSPECT, 'rate limit needs a human rule')
    if status in RETRY_STATUS:
        return Decision(RETRY_ONCE, 'transient status %s' % status)
    if status == 200 and not body:
        return Decision(INSPECT, 'empty success body')
    if status is None:
        return Decision(INSPECT, 'missing status')
    return Decision(INSPECT, 'unclassified status %s' % status)


def allow_second_call(record: dict) -> bool:
    decision = classify(record)
    return decision.action == RETRY_ONCE and not record.get('already_retried')


def plan_next(record: dict) -> dict:
    decision = classify(record)
    if allow_second_call(record):
        return {
            'id': record['id'],
            'plan': 'one_retry',
            'reason': decision.reason,
        }
    return {
        'id': record['id'],
        'plan': 'skip',
        'reason': decision.reason,
    }


FIXTURES = [
    {
        'id': 't1',
        'status': 503,
        'error': '',
        'body': None,
        'already_retried': False,
        'expect': RETRY_ONCE,
        'expect_plan': 'one_retry',
    },
    {
        'id': 't2',
        'status': 401,
        'error': '',
        'body': {'message': 'unauthorized'},
        'already_retried': False,
        'expect': STOP,
        'expect_plan': 'skip',
    },
    {
        'id': 't3',
        'status': None,
        'error': 'timeout',
        'body': None,
        'already_retried': False,
        'expect': RETRY_ONCE,
        'expect_plan': 'one_retry',
    },
    {
        'id': 't4',
        'status': 429,
        'error': '',
        'body': None,
        'already_retried': False,
        'expect': INSPECT,
        'expect_plan': 'skip',
    },
    {
        'id': 't5',
        'status': 503,
        'error': '',
        'body': None,
        'already_retried': True,
        'expect': STOP,
        'expect_plan': 'skip',
    },
    {
        'id': 't6',
        'status': 200,
        'error': '',
        'body': None,
        'already_retried': False,
        'expect': INSPECT,
        'expect_plan': 'skip',
    },
    {
        'id': 't7',
        'status': 422,
        'error': '',
        'body': {'message': 'schema'},
        'already_retried': False,
        'expect': STOP,
        'expect_plan': 'skip',
    },
]


def run() -> int:
    failures = 0
    for row in FIXTURES:
        decision = classify(row)
        plan = plan_next(row)
        action_ok = decision.action == row['expect']
        plan_ok = plan['plan'] == row['expect_plan']
        if not (action_ok and plan_ok):
            failures += 1
            print(
                'mismatch id=%s expected_action=%s expected_plan=%s'
                % (row['id'], row['expect'], row['expect_plan']),
                file=sys.stderr,
            )
        second = allow_second_call(row)
        print(
            '%s: action=%s second_call=%s plan=%s reason=%s'
            % (row['id'], decision.action, second, plan['plan'], decision.reason)
        )
    print('checked=%s failed=%s' % (len(FIXTURES), failures))
    return 1 if failures else 0


if __name__ == '__main__':
    sys.exit(run())

python3 retry_gate.py
echo exit:$?

How to read the designed output

Read the designed output as a contract for the checker, not as a log captured from a running server. True on second_call appears only for t1 and t3, and every other row must print False and plan skip. Row t5 is the budget lesson, because the status is still 503 while already_retried forces an immediate stop. If your local run differs, treat the difference as a bug in the copy, not as a new research finding about endpoints.

t1: action=retry_once second_call=True plan=one_retry reason=transient status 503
t2: action=stop second_call=False plan=skip reason=client or auth status 401
t3: action=retry_once second_call=True plan=one_retry reason=transport error: timeout
t4: action=inspect second_call=False plan=skip reason=rate limit needs a human rule
t5: action=stop second_call=False plan=skip reason=retry budget already spent
t6: action=inspect second_call=False plan=skip reason=empty success body
t7: action=stop second_call=False plan=skip reason=client or auth status 422
checked=7 failed=0

The designed process exit is 0. That number only means the local expectations matched the classifier, and it says nothing about latency, token use, or host capacity.

Guided exercises

Minutes 0 to 10: name the actions

Ask each pair to rewrite the three definitions in their own words without looking at the script. Collect one example of a stop case from ordinary HTTP practice, such as a rejected schema or a missing credential. Do not accept a complaint about model quality as a class, because that sentence does not tell the client what to do next. End this block only when every pair can point to the boolean that would allow a second call.

Minutes 10 to 25: run the deck

Students run the module and compare seven lines with the decision table, including the second-call column. The designed summary is checked equals 7 and failed equals 0, which is a specification rather than a published performance number. If the room has no Python, an instructor can still walk the table row by row and mark the same booleans on paper. That paper path is slower, but it preserves the lesson when install rights are locked down by the lab image.

Minutes 25 to 45: defend stop and inspect

Each student picks the 422 row and writes why a second identical call cannot repair a schema mismatch. Each student also picks the 429 row and writes why an automatic retry could amplify load on a shared free pool. The discussion should stay on client behavior, not on blaming a model for a status code the client already understands. A useful note names the field that carried the evidence, such as status, error, or already_retried.

Minutes 45 to 60: trace the guard and add t8

Students trace plan_next and confirm that skip is the plan for every row except the first 503 and the timeout. They then add a t8 fixture with status 409, an expect action of inspect, and an expect plan of skip. The existing fall-through rule should accept that row without a new branch, which is the point of the unclassified path.

After that row is appended correctly, the designed summary becomes checked equals 8 and failed equals 0. No socket opens in this block, so a mistaken label cannot trigger a live call while the class is still learning. Keep the new fixture in the same file so the next cohort can rerun the extended deck without a separate handout.

{
    'id': 't8',
    'status': 409,
    'error': '',
    'body': {'message': 'conflict'},
    'already_retried': False,
    'expect': 'inspect',
    'expect_plan': 'skip',
}

Minutes 60 to 70: write the limitation memo

Each student writes three cases the script must not decide, using complete sentences and naming the missing field. Acceptable cases include a non-idempotent tool call, a 429 that includes Retry-After, and a vendor code outside the teaching set. Unacceptable cases are vague complaints about model quality, cost, or speed, because this checker never saw those signals. Collect the memos before anyone opens a host dashboard, so the product discussion cannot replace the exit check.

Map one synthetic line

Use a synthetic line, not a production export, when you demonstrate the mapping on a projector. A synthetic line with id lab-17, status 503, an empty error, and retried no becomes the same shape as fixture t1. Students can append that mapped object to FIXTURES only after they also write the expected action and expected plan. If they forget the expected fields, the checker should fail closed rather than treat a missing expectation as success.

Where a host with free access fits

MonkeyCode enters the lesson only after the labels are stable, and only as one possible lab host. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This article relies on two availability claims supplied for the draft: free model access and a free server option. This draft does not state quotas, hardware, duration, uptime, or benchmark scores, because those facts were not supplied for verification.

Students should read the host's current documentation before class, since availability claims can change and this page is not that documentation. A free server option may help a classroom share one practice host, but it does not change the failure labels or the one-retry budget. If documentation contradicts this workshop, follow the documentation and keep the local fixtures as the graded artifact.

Export rules if logs come later

If the instructor later exports logs, keep a local id, status, a coarse error tag, and an empty-body flag. Drop raw prompts, completions, authorization headers, and user identifiers before the file reaches any student machine. Map vendor error strings to a small tag set such as timeout, connection_reset, or dns_temporary, and leave unknown strings empty. An unknown string must not become a retry, because the classifier treats an empty error plus an unlisted status as inspect.

Limitations of the teaching contract

The status sets are incomplete, and a 409, 425, or vendor-specific code will fall through to inspect until an instructor adds a reviewed rule. One retry can still be the wrong policy for 503 storms, so the flag is a teaching brake rather than a production backoff schedule. The script does not measure tokens, latency, or queue depth, and it should not be cited as evidence about cost or speed. Empty body detection is naive, because a JSON null, a whitespace string, and a missing key are different defects in a real client.

A 429 with a Retry-After hint might be safe to defer, but these fixtures carry no such hint, so the honest label is inspect. Idempotency is assumed for the one retry, and a non-idempotent tool call should be stop even when the status looks transient. That case is listed in the closing memo because the script cannot see side effects that were not recorded in the row.

Who should not use this lab

Skip the lab if the course outcome is a tuned retry library with jitter, circuit breakers, and idempotency keys. Skip it if students must handle protected data, because the safe path here assumes fixtures that contain no personal content. Skip it if the goal is to compare model quality, since none of the rows include a completion to score. On-call playbooks also need vendor documents and ownership, which this seventy-minute room deliberately does not provide.

Facilitation notes

Instructors should resist turning the closing memo into a tool bake-off, because the room has no measured vendor results to compare. Grade the written reasons, the passing local checker, and the limitation list, not the number of live calls a student managed to send. Pairs are enough; a larger group hides whether each person can name the field that justified stop. If Python is unavailable, the paper walk still counts, provided the booleans match the table exactly.

Closing check

The core result is testable: seven local rows, three actions, and a guard that refuses an unearned second call. Run the module, confirm the designed summary, and only then consider pointing the same field map at a host log. If a lab already uses MonkeyCode's free model access and free server option, classify redacted error rows with these labels first. That pass is a review step, not a claim that the host produced the sample rows in this article.

Workshop: Grade a Free Model Swap With a Three-Row Fixture Deck in 65 Minutes

Finley Zhu — Fri, 09 Oct 2026 15:35:34 +0000

A teaching lab can trust a free model swap only after three frozen prompts still score the same way. Students should be able to rerun those prompts next week and separate a wrong answer from a dead host. This session builds that separation as a fixture deck, a local scorer, and a short decision table. The deck remains useful when the remote seat is down, because stub mode never leaves the student laptop.

Recent teaching discussions keep circling the same practical question: a smaller or free host may be enough if a narrow check still passes. This workshop takes that question into a timed classroom clock instead of turning it into a product review. The check here is three frozen JSON rows, not a leaderboard and not a claim about any named model family. Students leave with a rerunnable gate, which is more useful for next week's lab than a headline about model choice.

This fixture deck grades structured answers rather than token spend, request latency, or separate tool-permission stamps. Each row names one prompt, the required JSON keys, a list of forbidden phrases, and a character ceiling. A later cohort can replace those three prompts without rewriting the scorer or the shared decision table. Transport errors stay in their own bucket so a timeout is never recorded as a content failure.

What a 65-minute pass actually proves

The lab proves that a small contract can catch obvious drift before a class depends on a new host. It does not prove that any model is generally correct, inexpensive, or stable across every course topic. Three rows are a teaching gate, not a benchmark suite and not a statement about vendor capacity. Instructors should treat every green pass as evidence about these fixtures alone, not about the wider course.

Constraints to state before minute eight

Say aloud that no product name, seat length, hardware shape, or quota number is assumed by these exercises. Students will author fixtures, run a stub, inject one deliberate drift, and only then consider a remote host. Prompts in the deck must use public sample text, never passwords, student records, or private repository contents. If the classroom cannot review a third-party terms page that morning, stop the plan after the local stub.

Session clock

Use the blocks below so the worked drift loop is not squeezed out by setup chatter. Each timed block has one visible output that the instructor can check on a nearby student screen. Do not add a fourth prompt during this sitting, because the constraint is part of the lesson. A cohort that wants more coverage should schedule a second sitting rather than stretching this clock.

Minutes 0–8: name the failure

Describe a lab where yesterday's JSON summary gained an extra field and then broke a downstream checker. Ask students which failure is content drift, which failure is transport loss, and which failure is a parser error. Write those three labels on the board before anyone opens an editor or calls a remote host. Those three labels become the only status values the scorer is allowed to emit during this session.

Minutes 8–22: author the deck

Students create a file named fixtures/deck.json that contains exactly three rows and no hidden fourth case. Each row carries an identifier, one prompt, the required keys, forbidden substrings, and a maximum character count. One row should demand a refusal to invent a version number that the source text never stated. That refusal row teaches the class that a fluent answer can still fail the written contract on purpose.

Minutes 22–40: prove the harness

Run the scorer in stub mode so the first green result does not depend on any network. The stub returns canned JSON that already satisfies every required key and avoids every forbidden phrase listed. Students should read the exit code, the status column, and the reason string before they edit anything. A green stub result only proves the harness, not the quality of any live model response.

Minutes 40–52: inject one drift

Change the stub for the endpoints row so the path field disappears while the method field remains. Rerun the same command and confirm that only that row becomes content drift while the other rows stay passing. Restore the stub and rerun the command once more so the deck returns to a clean baseline score. This short loop is the worked example that students can repeat later without a teacher in the room.

Minutes 52–65: record the swap rule

Fill the decision table before anyone discusses a remote host, a product seat, or a model name. Content drift blocks the planned swap until the fixture or the prompt is intentionally revised by the instructor. A transport failure blocks the swap for that class day, while a parser error blocks it until the expected shape is clarified. Only a full pass on these three rows allows an optional retarget of the same unchanged deck.

Fixture students commit

Save the following document as the only fixture input that this sitting is allowed to grade today. The prompts are public samples, and the ceilings are teaching limits rather than any product measurements. Keep the row identifiers stable so later diffs show prompt edits instead of merely renamed rows. If a student wants a different sample, replace the source sentence and leave the required keys in place.

{
  "rows": [
    {
      "id": "summary",
      "prompt": "Source: the patch note says retries now wait 200ms. Return a JSON object with keys summary and risk. Do not add fields.",
      "required_keys": ["summary", "risk"],
      "forbidden": ["password", "api_key"],
      "max_chars": 240
    },
    {
      "id": "endpoints",
      "prompt": "Source: clients call GET /health. Return a JSON object with keys method and path. Do not invent other routes.",
      "required_keys": ["method", "path"],
      "forbidden": ["delete /", "drop table"],
      "max_chars": 180
    },
    {
      "id": "version",
      "prompt": "Source text has no version number. Return a JSON object with keys status and reason. Refuse to invent a version.",
      "required_keys": ["status", "reason"],
      "forbidden": ["v1.2.3", "version 2"],
      "max_chars": 200
    }
  ]
}

Scorer students rerun

The script below is an unexecuted teaching artifact for the lab, not a measured client for any hosted product. Stub results are defined by the canned objects in this file, so a green run does not describe a live model. HTTP mode is a class-defined adapter with a placeholder path, and instructors must replace that path only after reading current host documentation. Do not treat the sample URL shape as a vendor interface or as evidence of uptime.

#!/usr/bin/env python3
"""Three-row fixture scorer for a classroom model-swap gate.

Label: proposed lab code. Status lines from stub mode are defined here.
They are not observations from a hosted model run.
"""

import argparse
import json
import sys
import urllib.error
import urllib.request


def load_deck(path):
    with open(path, encoding="utf-8") as handle:
        deck = json.load(handle)
    rows = deck.get("rows")
    if not isinstance(rows, list) or len(rows) != 3:
        raise SystemExit("deck must contain exactly three rows")
    return rows


def stub_body(row, drift_id):
    canned = {
        "summary": {"summary": "Patch notes add a retry hint.", "risk": "low"},
        "endpoints": {"method": "GET", "path": "/health"},
        "version": {"status": "refused", "reason": "source omits a version"},
    }
    if row["id"] not in canned:
        raise SystemExit("stub has no canned body for " + row["id"])
    body = dict(canned[row["id"]])
    if drift_id == row["id"] and "path" in body:
        body.pop("path")
    return json.dumps(body)


def fetch_remote(base_url, prompt, timeout):
    """Placeholder adapter. Replace the path after reading current host docs."""
    payload = json.dumps({"prompt": prompt}).encode("utf-8")
    request = urllib.request.Request(
        base_url.rstrip("/") + "/lab-complete",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            raw = response.read().decode("utf-8")
            code = response.status
    except (urllib.error.URLError, TimeoutError, OSError) as exc:
        return None, "transport_failure: " + exc.__class__.__name__
    if code != 200 or not raw.strip():
        return None, "transport_failure: bad response"
    return raw, None


def grade(row, raw):
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return "parser_error", "body is not json"
    if not isinstance(parsed, dict):
        return "parser_error", "body is not an object"
    missing = [key for key in row["required_keys"] if key not in parsed]
    if missing:
        return "content_drift", "missing " + ",".join(missing)
    blob = json.dumps(parsed).lower()
    for phrase in row["forbidden"]:
        if phrase.lower() in blob:
            return "content_drift", "forbidden phrase present"
    if len(blob) > int(row["max_chars"]):
        return "content_drift", "over character ceiling"
    return "content_pass", "keys present and ceiling held"


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--deck", required=True)
    parser.add_argument("--mode", choices=["stub", "http"], default="stub")
    parser.add_argument("--base-url", default="")
    parser.add_argument("--timeout", type=float, default=20.0)
    parser.add_argument("--drift", default="")
    args = parser.parse_args()
    failures = 0
    for row in load_deck(args.deck):
        if args.mode == "stub":
            raw, err = stub_body(row, args.drift), None
        else:
            if not args.base_url:
                raise SystemExit("http mode requires --base-url")
            raw, err = fetch_remote(args.base_url, row["prompt"], args.timeout)
        if err:
            status, reason = "transport_failure", err
        else:
            status, reason = grade(row, raw)
        if status != "content_pass":
            failures += 1
        print(f"{row['id']}|{status}|{reason}")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

Expected lines from the worked example

The clean stub is defined to print three content_pass lines and to exit with code zero. The drifted stub is defined to print content_drift only for the endpoints row and to exit with code one. Students should compare their terminal text with those two patterns before they change any prompt wording. If the drifted run still exits zero, the harness is wrong and the remote step stays closed.

summary|content_pass|keys present and ceiling held
endpoints|content_pass|keys present and ceiling held
version|content_pass|keys present and ceiling held

summary|content_pass|keys present and ceiling held
endpoints|content_drift|missing path
version|content_pass|keys present and ceiling held

Commands for the two local passes

Run the clean command first and save the text beside the cohort date in the lab notes. Run the drift command second and confirm that the exit status changed from zero to one. Do not point HTTP mode at a host until both local patterns match the expected lines above. Replace the placeholder path in the adapter before any remote command is shared with the class.

python3 score_deck.py --deck fixtures/deck.json --mode stub
python3 score_deck.py --deck fixtures/deck.json --mode stub --drift endpoints
python3 score_deck.py --deck fixtures/deck.json --mode http --base-url "$LAB_BASE" --timeout 20

Decision table for the swap

Observed status	What the row showed	Decision for this class day
content_pass	Required keys exist, forbidden phrases are absent, and the ceiling holds	Row may stay in the swap set
content_drift	A key is missing, a forbidden phrase appears, or the ceiling breaks	Block the swap until the instructor revises the row
parser_error	The body is missing, not JSON, or not an object	Block the swap until the expected shape is clarified
transport_failure	The call times out, returns a non-200, or comes back empty	Block the swap today and keep grading on the stub

Read the table from top to bottom and stop at the first failing row rather than averaging the three results. A single drifted row is enough reason to keep the previous host in place for that class day. Parser errors should be fixed in the prompt notes before anyone retries a remote call again. Transport failures should be logged with the clock time and then left out of any accuracy discussion.

Where an optional free host fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as one optional place to obtain free model access and a free server option for the retarget step. This plan names no models, quotas, hardware, duration, or permanence, and those facts must be read from current terms that morning. If those terms are missing, unclear, or incompatible with classroom policy, keep the lab on stub mode and still grade the deck.

After local statuses are green, a class that already has the free server option enabled can retarget this same unchanged deck. That retarget is optional, and a missing seat does not lower the local grade for the stub exercise. Students should paste only the public prompts from the deck, then compare status lines against the table above. Archive the status lines with the cohort date so the next section can see whether the host still matches the frozen rows.

Outputs the instructor collects

Collect four artifacts before the room is dismissed, and do not accept a screenshot of a chat window as a substitute. The deck file shows what was frozen, while the two transcripts show that drift is detectable. The written decision line shows whether a remote retarget is even eligible for that particular class day. Store the four artifacts together so a later cohort can rerun the same checks without reconstructing the lesson from memory.

Students submit a three-row deck file that keeps stable identifiers and uses only public source sentences.
Students submit a clean stub transcript that exits zero and shows three content_pass lines for every row.
Students submit a drifted stub transcript that exits one and isolates the failure on the endpoints row.
Students submit one decision line that either blocks the swap or records an optional retarget after a full pass.

Limitations and who should skip the lab

Exact key checks will miss a correct paraphrase that uses different field names or a nested object. Three public prompts cannot support a claim about accuracy, cost, latency, or uptime for a whole course. Do not send secrets, grades, or unpublished student work to any remote host, whether the seat is free or paid. Skip this workshop when you need a statistical evaluation, regulated medical or legal wording, or a graded exam that cannot tolerate a missing seat.

A team that already runs a large offline eval suite will gain little new procedure from a three-row gate. A classroom with no time to review third-party terms should not add the remote step at all. The scorer also ignores semantic similarity, so two answers can differ in wording and still need a human note. Those limits are intentional, because the sitting is about catching obvious contract breaks before a swap.

What to archive before the next cohort

Store the deck file, the scorer version, and the status lines from both the clean stub and the drifted stub. Add one sentence that names who may edit prompts and who may only rerun the commands. Do not archive credentials, host tokens, or private base URLs anywhere inside the shared class repository. The next cohort should diff the deck first, then decide whether a new row belongs in a later sitting.

Workshop: Ledger Prompt, Host, and Route in an 85-Minute Lab

Finley Zhu — Thu, 08 Oct 2026 10:53:54 +0000

A free model lab turns noisy when the prompt, the host, and the route all change inside one hour. This eighty-five minute workshop pins those three lanes before the room starts debating answer quality. Students leave with a ledger they can rerun on the next day, even if the public terms page changed overnight. The ledger shows which lane moved, and it refuses to treat a remembered allowance as if it were evidence.

Why a class needs the ledger first

Hosted model classes often mix three unstable inputs and then grade the resulting text as one event. A prompt edit, a server image change, and a route swap can each alter the output on their own. When those events share one timestamp, the class cannot tell which event deserves the next repair. The ledger separates those events so the following edit has one named owner instead of three suspects.

The workshop assumes one small group, one frozen prompt, and one candidate open-source host for the hour. It does not assume a private cluster, a paid API key, or a published scoreboard of model names. It also does not assume that yesterday's quota sentence is still printed on the project page. Every numeric claim about access must be copied from the page the facilitator opens during class.

What the facilitator may claim

Two availability claims are operator-supplied for this session: free model access, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode enters the lab as the open-source candidate the operator asked the class to inspect, not as a measured winner. The inspection is a documentation check plus a rerunnable row, not a bake-off against unnamed rivals.

Write no token count, hardware shape, or expiry date into the slides from an old memory. In the first ten minutes, open the current project documentation and paste only the sentences that page shows today. If the page is silent on a quota, leave the quota field blank rather than filling it from a blog post. A blank field is a valid result, and it keeps the later comparison from leaning on a stale number.

How the eighty-five minutes are split

Minutes 0 to 10 cover the live terms paste and a hard ban on recalled quotas.
Minutes 10 to 25 freeze the prompt fixture, hash the file, and store that hash.
Minutes 25 to 40 stamp the server fingerprint and place it beside the prompt hash.
Minutes 40 to 60 send one route call, or record an explicit skip when no key exists.
Minutes 60 to 75 compare two ledger rows and name the single lane that moved.
Minutes 75 to 85 write the limitation note and decide whether a second route is allowed.

Lane rules the room must keep

Lane A holds the prompt only

Lane A stores the task text, the input text, and a content hash of both fields together. Students may not improve the wording after that hash has already been written into the ledger. A clearer prompt becomes a new fixture id, not a silent edit of the first file. This rule stops the class from hiding a prompt change inside a host comparison.

Lane B holds the server only

Lane B stores the operating system name, the Python version, and a short hash of that pair. It does not store a story about how fast the machine felt during the exercise. If two students produce different fingerprints, they must not share one result row. A free server option, when the live page still offers one, is only another fingerprint source.

Lane C holds the route only

Lane C stores the route name, the call time, and the hash of the returned text. It does not store a quality score, a preference vote, or a claimed cost winner. If the call never left the laptop, the route name stays unset and the note explains why. An empty call is valid evidence that the lab stopped before anyone invented an output.

A worked skeleton students can rerun

The script below is a proposed lab skeleton, not an executed benchmark and not a product trial. It prints a schema row before any network call, so the class can review fields while the terms page is open. Replace each placeholder with a sentence you copied, and do not invent a quota when the page is quiet. The sample row uses an empty output on purpose, which keeps the first run offline.

# Proposed lab skeleton. Not an executed benchmark.
# Fill LIVE_TERMS from the documentation page opened in class.

from datetime import datetime, timezone
import hashlib
import json
import platform

LIVE_TERMS = {
    "source": "paste the documentation URL opened in class",
    "free_model_access": None,
    "free_server_option": None,
    "quota_text": "",
    "captured_at": datetime.now(timezone.utc).isoformat(),
}

FIXTURE = {
    "id": "fx-ledger-01",
    "task": "Rewrite the function comment in one sentence. Do not add features.",
    "input": "def add(a, b):\n    return a + b\n",
}

def fingerprint():
    raw = "|".join(
        [platform.system(), platform.release(), platform.python_version()]
    )
    digest = hashlib.sha256(raw.encode()).hexdigest()[:12]
    return {"host": raw, "sha256_12": digest}

def ledger_row(route_name, output_text, note):
    fixture_bytes = json.dumps(FIXTURE, sort_keys=True).encode()
    return {
        "fixture_id": FIXTURE["id"],
        "fixture_sha256": hashlib.sha256(fixture_bytes).hexdigest(),
        "route": route_name,
        "terms": LIVE_TERMS,
        "server": fingerprint(),
        "output_sha256": hashlib.sha256(output_text.encode()).hexdigest(),
        "output_chars": len(output_text),
        "note": note,
    }

if __name__ == "__main__":
    sample = ledger_row(
        "unset",
        "",
        "no call yet; row exists so the schema can be reviewed",
    )
    print(json.dumps(sample, indent=2))

Save the skeleton as ledger.py, then run the three review commands before any live call. The first command proves that the schema prints without a network call. The second command checks that the printed JSON is well formed. The third command hashes the script so a later edit is visible in the notes.

python3 ledger.py > row0.json
python3 -m json.tool row0.json >/dev/null && echo "json ok"
python3 - <<'PY'
import hashlib
from pathlib import Path
print(hashlib.sha256(Path("ledger.py").read_bytes()).hexdigest())
PY

Read two rows without building a scoreboard

Use the table when two students disagree about a result they both just produced. The table names one lane, and it names the only edit that lane allows. It does not name a better model, because this packet never collected a fair paired sample. Treat a void row as data about the lab design, not as a failure of the host.

What changed between the rows	Lane that owns the change	Edit that is allowed next
Prompt hash differs	Lane A	Keep the new fixture id and drop the comparison
Server hash differs	Lane B	Rerun both rows on one pinned host
Route name differs	Lane C	Record both names and do not retune the prompt
Quota sentence differs	Terms paste	Keep both pastes and do not average them
Terms fields are empty	Evidence gap	Stop scoring until the live paste exists
Output hash matches	No quality claim	Log stability only, not correctness

A matching output hash means the text stayed stable under that fixture and that host. It does not mean the comment was good, short, or safe enough to merge. Correctness still needs a human check, or a separate test this workshop does not replace. Say that limit every time a student treats a stable hash as a grade.

Four exercises with clocks attached

Exercise 1 freezes Lane A

Give the group fifteen minutes and one function that already returns the right value. The task line must forbid new features, so the model is not invited to refactor the file. Students hash the fixture and write that hash into the ledger before they discuss wording. If someone wants a clearer task, they create fx-ledger-02 and leave the first hash untouched.

Exercise 2 stamps Lane B

Give the group ten minutes to run the fingerprint function on the machine they will use. If the live documentation still shows a free server option, one pair may stamp that host instead. They still paste the terms sentence beside the fingerprint before they call anything. A fingerprint without a terms paste is incomplete, even when the process list looks familiar.

Exercise 3 attempts Lane C once

Give the group twenty minutes for a single call, or for a written skip with a reason. The skip is the right outcome when the page no longer shows free model access, or when no key exists. Students store the output hash, the character count, and a note of twenty words or fewer. They must not retry until the output looks smarter, because unmarked retries destroy the ledger.

Exercise 4 names the lane that moved

Give the group fifteen minutes to compare one row with a neighbor using the table above. They must name the single lane that moved, and they must name the edit that lane allows. If more than one lane moved, the comparison is void and the pair schedules a cleaner rerun. The facilitator counts void comparisons, because that count is the only class metric this workshop defines.

Limits that belong in the closing note

This ledger cannot price a vendor, size a fleet, or predict an allowance for next month. It cannot prove that an open-source host is faster, cheaper, or safer than some other host. It cannot repair a bad task, and it cannot bless output that merely stayed stable across two runs. Facilitators should read those limits aloud near the end, so notes do not grow a claim the rows cannot carry.

The class metric is the count of void comparisons, not a satisfaction score and not a latency trophy. A high void count means the room changed too many lanes, which is a teaching result rather than a product result. A low void count means the pins held for this hour, which is also not a quality prize. Either number is local to this room, this fixture, and this calendar date.

Who should skip this workshop

Procurement teams that need a signed benchmark should not file these rows as a vendor bid.
Anyone holding production secrets should not place those secrets inside the fixture text.
A class that cannot open the current documentation should wait, rather than teach from an old slide.
Readers who want a ranked list of model names will not find one, because no list was verified for this date.
Groups that need medical, legal, or financial advice from a model should choose a different exercise.

After the clock stops

If the page you opened still shows free model access and a free server option, point Lane C there. Keep the same fixture id, and do not retune the task while you try that documented project. If that page does not show those options, leave the product name out of the results file. Finish the ledger on a host you already control, and keep the rerunnable row as the souvenir.

Workshop: Cap a Shared Lab Hour With a Session Envelope in 75 Minutes

Finley Zhu — Thu, 08 Oct 2026 08:34:39 +0000

Workshop: Cap a Shared Lab Hour With a Session Envelope in 75 Minutes

A shared lab hour stays usable only when each student run carries a written session envelope before the first model call. The envelope records a turn cap, a local token budget, a timeout, and a concurrency limit that the harness can refuse. This workshop teaches that preflight in seventy-five minutes, with a checker students can rerun on their own laptops. It does not measure a vendor, and it does not treat any live quota as a permanent classroom constant.

The failure this hour is built to catch

Early October rooms often mix a classroom exercise with a contribution sprint, and both patterns share one scarce resource. Several notebooks then retry the same prompt until a demo looks finished for the person who started first. One laptop can open parallel loops while another student is still reading the worksheet and has not called yet. Shared capacity then looks slow, even though the defect is an unbounded client rather than a mysterious network.

How this differs from after-the-fact metering

Spend ledgers and latency charts explain a run that has already finished and left a log. This session starts earlier, and the harness blocks the call when the envelope is missing or internally inconsistent. Students leave with a file they can diff in git, not with a dashboard crop they cannot reproduce next week. The same checker runs against a mock client, so the lesson does not depend on a live allowance.

Seventy-five minute timetable

Keep the order below, and do not skip the refusal exercise near the middle of the hour. Acceptance-only demos hide the bug this workshop exists to teach, which is a client that never refuses. Pairs should rotate the keyboard every time the timetable moves into a new section of the hour. The instructor collects reason codes rather than model prose, because the gradeable artifact is the gate itself.

From minute 0 to minute 10, state the shared-hour constraint and project one unbounded retry trace on the board.
From minute 10 to minute 25, write the envelope fields and agree on class-local numbers for today's worksheet only.
From minute 25 to minute 45, run the checker on three fixtures, including two files that must fail closed.
From minute 45 to minute 60, place the checker in front of a stub client, then optionally attempt one live call.
From minute 60 to minute 75, compare two refused runs and write a three-line lab note in the shared repository.

Envelope fields the room must fill

Keep the envelope small enough that a pair can review every field in under two minutes. Four numeric limits plus an exercise id are enough for a first shared hour in this room. Extra fields tend to become unread comments that nobody checks again before the call leaves the laptop. Treat every number as class policy that the next cohort can revise in the lab note.

The max_turns field limits model round trips inside one exercise, and that count includes every automatic retry.
The local_token_budget field is the student's own ceiling, not a statement about any provider allowance or grant.
The timeout_seconds field bounds a single call so one hung socket cannot own the rest of the hour.
The max_in_flight field stops one laptop from opening several loops while the rest of the room waits.
The exercise_id field ties the envelope to today's worksheet so yesterday's accepted file cannot be reused by accident.

Class-local starting numbers for this seventy-five minute room are only a proposal, not a measured optimum. Use eight turns, two thousand local tokens, twenty seconds, and one in-flight call as the starting envelope. Shorten the token band if the worksheet asks for a single function, and record that reason in the note. Do not copy these numbers onto a slide as if they were a published vendor limit.

Trace shape students should recognize

Before writing envelopes, show a six-line trace from a stub log so the room can see fan-out without a live model. Each line carries a timestamp, a student id, an exercise id, and a turn index for the board. The sample below is synthetic teaching data, and it is not a capture from any production host. Ask pairs to count overlapping starts rather than to judge the prompt text inside the trace.

t=0  student=s1 exercise=lab-hour-03 turn=1 state=start
t=1  student=s1 exercise=lab-hour-03 turn=1 state=start
t=1  student=s1 exercise=lab-hour-03 turn=2 state=start
t=2  student=s2 exercise=lab-hour-03 turn=1 state=waiting
t=9  student=s1 exercise=lab-hour-03 turn=1 state=retry
t=9  student=s1 exercise=lab-hour-03 turn=3 state=start

In this fixture, student s1 has three starts overlapping while student s2 is still waiting on the first turn. That pattern is the defect the max_in_flight field is meant to refuse before any live call starts. Do not treat the timestamps as latency measurements, because the file was written by hand for the board.

Worked checker students can rerun

The script below is a teaching artifact meant for local fixtures on student laptops during the hour. It has not been executed against a production fleet for this article, and it does not call a model by itself. Run it on the pass file and the two fail files before anyone in the room discusses providers. If the output disagrees with the comments beside the commands, fix the fixture before you change the checker.

#!/usr/bin/env python3
"""Session envelope checker. Local teaching artifact, not a vendor benchmark."""

import json
import sys

REQUIRED = (
    "exercise_id",
    "max_turns",
    "local_token_budget",
    "timeout_seconds",
    "max_in_flight",
)

def load_envelope(path):
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)

def check(card, expected_exercise):
    errors = []
    for key in REQUIRED:
        if key not in card:
            errors.append(f"missing:{key}")
    if card.get("exercise_id") != expected_exercise:
        errors.append("exercise_id_mismatch")
    turns = card.get("max_turns")
    if not isinstance(turns, int) or turns < 1 or turns > 12:
        errors.append("max_turns_outside_class_band")
    budget = card.get("local_token_budget")
    if not isinstance(budget, int) or budget < 100 or budget > 4000:
        errors.append("local_token_budget_outside_class_band")
    timeout = card.get("timeout_seconds")
    if not isinstance(timeout, int) or timeout < 5 or timeout > 30:
        errors.append("timeout_seconds_outside_class_band")
    if card.get("max_in_flight") != 1:
        errors.append("max_in_flight_must_be_one")
    return errors

def main():
    if len(sys.argv) != 3:
        print("usage: check_envelope.py ENVELOPE.json EXERCISE_ID", file=sys.stderr)
        return 2
    errors = check(load_envelope(sys.argv[1]), sys.argv[2])
    if errors:
        print("REFUSE")
        for item in errors:
            print(item)
        return 1
    print("ACCEPT")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Passing fixture, saved as pass_envelope.json:

{
  "exercise_id": "lab-hour-03",
  "max_turns": 8,
  "local_token_budget": 2000,
  "timeout_seconds": 20,
  "max_in_flight": 1
}

Broken fixture, saved as missing_turns.json:

{
  "exercise_id": "lab-hour-03",
  "local_token_budget": 2000,
  "timeout_seconds": 20,
  "max_in_flight": 1
}

Commands for the room:

python3 check_envelope.py pass_envelope.json lab-hour-03
python3 check_envelope.py missing_turns.json lab-hour-03
python3 check_envelope.py pass_envelope.json lab-hour-04

The expected teaching outcomes are classroom labels only, and they are not measured metrics from a live service. The first command should print ACCEPT, and the other two should print REFUSE plus a reason code. A missing numeric field can emit two codes, one for absence and one for the band, and that pairing is expected. A different printout means the fixture or the Python version needs a look, not that a model misbehaved.

Exercise sequence

Exercise A, minutes 10–25

Each pair writes one valid envelope for lab-hour-03 and one intentionally broken envelope for the same checker. Broken envelopes should omit a required field, reuse yesterday's exercise id, or set max_in_flight above one. The useful result is a refuse code the other pair can predict before the command even runs. Prompt wording is out of scope for this segment, so leave the model client closed on every laptop.

Exercise B, minutes 25–45

Run the three commands and paste reason codes into the shared note without pasting secrets or prompts. Add one fixture that sets local_token_budget to 50000 and confirm the class band rejects that envelope. That rejection is local policy for this room, and it is not evidence about any provider allowance. If two pairs disagree on a reason code, diff the envelope files before anyone edits the checker.

Exercise C, minutes 45–60

Put the checker in front of a stub that sleeps only after ACCEPT and never opens a socket on REFUSE. Only after that gate works should one volunteer try a live call, and only inside the accepted timeout. Everyone else watches the reason code on the terminal, not the generated text from the optional call. If the live page cannot be confirmed that morning, stop at the stub and mark the live step skipped.

def gated_call(card, exercise_id, client):
    errors = check(card, exercise_id)
    if errors:
        return {"status": "refused", "errors": errors}
    return client.complete(
        max_turns=card["max_turns"],
        timeout_seconds=card["timeout_seconds"],
    )

The function above is whiteboard pseudocode, and students should implement the sleep-or-refuse stub themselves in class. Copying it without a failing test recreates the unread-comment problem this hour is trying to remove from the lab. A three-line test that expects refused for a stale exercise id is enough to lock the gate. Run that small test before any volunteer is allowed to open a live client from the room.

Where a free model path fits this hour

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Operator notes for this piece describe MonkeyCode as an open-source project with free model access and a free server option. A class may point the optional live step at those paths only after reading the current project page that morning. Students who will read or fork the code should confirm the license on that same page before they clone anything.

This article does not name models, freeze a token grant, describe hardware, or claim that free access will stay unchanged. If the live page disagrees with a slide, the live page wins and the stub exercise remains the gradeable path. Use a free server only as a shared place to run the checker when local Python installs are uneven. It is not a private production host, and it is not a reason to skip the refuse fixtures in the timetable.

Refusal decision table

Condition	Harness result	Class action
Required field missing	REFUSE	Fix the envelope, and do not retry a model
Exercise id is stale	REFUSE	Mint today's id and discard yesterday's envelope
In-flight value above one	REFUSE	Close extra notebooks before any live step
Live access terms unknown	Skip live step	Finish on the stub and record the gap
Envelope accepted and stub passes	Optional live call	One volunteer, inside the written timeout

Read the table from top to bottom, and stop at the first matching row before you take any other action. Do not average the rows into a score, because a single missing field is already a failed preflight. The last row permits one call from one volunteer, not a fan-out from the rest of the room. Record a skipped live step as a completed outcome when the access page cannot be confirmed.

Lab note template

Keep the note to six lines so review stays faster than the exercise itself for the instructor. Ask each pair to commit the note beside the fixtures before they leave the room today. The change_next_week line should name a fixture gap, not a feeling about model quality or speed. An acceptable example is to add an envelope whose timeout is four seconds so the lower band is tested.

exercise_id: lab-hour-03
class_band: turns<=12 tokens 100-4000 timeout 5-30 inflight=1
refuse_codes_seen: missing:max_turns, exercise_id_mismatch
stub_gate: pass
live_step: skipped|one_volunteer
change_next_week: Add an envelope whose timeout is four seconds.

Limitations

The two-thousand-token class band is a teaching cap, not a cost model and not a provider quota. The checker does not inspect prompt content, tool side effects, or whether data leaves the room. A student can still paste secrets into an accepted call, so the envelope is not a privacy control. Shared free capacity can still saturate for reasons this script cannot see, including other tenants on the same host.

Do not publish timings from one classroom as a benchmark, and do not compare models with this harness. The synthetic trace is hand-written, so its gaps are illustrations rather than observations from a running host. A missing fixture file should fail the command with a visible error, which the script currently leaves to the interpreter. Teach that traceback as an environment failure, not as a refused envelope, so students do not confuse the two.

Who should not use this hour

Skip this workshop if the room needs authenticated production traffic, regulated data, or a graded accuracy comparison between models. Skip it if students cannot install Python 3 and you also lack a shared server you are allowed to use. Skip the live segment when you cannot confirm today's access terms in writing before the session starts. A team that already enforces admission control at a gateway should not replace that gateway with this laptop script.

Close

Leave the three fixtures, the synthetic trace, and the checker in the class repository beside the timetable. Next week, change only the exercise id and one class-band edge, then rerun the same commands before any live client opens. If a shared optional client would help the last segment, read the current MonkeyCode access notes before you rely on them. Keep the stub path as the graded path even when that optional client is available to the room.

Workshop: Pack a Replay Packet Before a Free Lab Call in 75 Minutes

Finley Zhu — Thu, 24 Sep 2026 22:26:15 +0000

A shared free model path can support a classroom exercise, yet it should never become the grading oracle. Students should freeze a replay packet that stores the task, the fixture output, and the assertions before any live call. Instructors can then mark the same submission when that free path is slow, capped, or temporarily unreachable. This 75-minute workshop builds that packet and places a free server in an execution lane rather than in the score.

Why the packet comes before the call

Generated output is useful as a draft, and it is a poor source of truth for a grade. A later run can differ because the endpoint is busy, the prompt is underspecified, or the service returns a shorter completion. If the class waits for a live call before writing assertions, every outage becomes an incomplete lab. The packet reverses that order so the expected evidence already exists on disk before the first request leaves the room.

Recent public writing has spent more attention on agent demos than on how a reviewer repeats a result the next morning. That gap matters in a teaching lab, where a full room may share one free path and still need a comparable score. This workshop does not claim a benchmark, a token quota, a hardware profile, or a permanent free offer. It claims only that a frozen fixture plus three explicit lanes makes the same exercise rerunnable on a later date.

What students leave with

By the end of the session, each student can regenerate a local packet and pass three assertions without a network. A second lane may call a free model endpoint only after the packet already contains a frozen fixture. A third lane may run a small command on a free server, then compare stdout with the fixture hash. The live lanes stay optional, so a missing key does not block grading or a later replay.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as one place an instructor may obtain free model access and a free server option for those optional lanes. This draft does not name models, quotas, or hardware, because those details change and were not verified for this class date. Students should read the current offer on the class day before they export an endpoint or share a host.

75-minute timing

Use the blocks below as a wall clock, not as a suggestion to skip the fixture lane. Each block names an output so a facilitator can see whether the room is still on the fixture. These minutes are planning targets for a single 75-minute room, not timings measured from any previous cohort. If a block overruns, cut the optional live lanes first and keep the deliberate assertion-failure exercise intact.

Minutes 0–10: State the rule that the fixture lane is the only required score, then list the three fields every packet must contain.
Minutes 10–22: Students write a three-line fixture and three assertions for a status note that must contain the word READY.
Minutes 22–40: Students run the packing script, inspect the printed hash, and repair one deliberate assertion failure before moving on.
Minutes 40–52: Pairs review a neighbor packet and reject any file whose stored hash does not match the normalized fixture text.
Minutes 52–65: Optional lanes run only after the instructor confirms current terms for free model access and the free server option.
Minutes 65–75: Each student records one limitation, then reruns the local script with the network disabled and keeps the passing JSON.

Packet fields

Keep the schema small enough that a grader can read every field without asking a model for help. Version 1 stores the human task, the normalized fixture, its digest, and a short list of assertions. Optional lanes default to skipped, so a packet remains valid when no endpoint has been configured yet. Unknown assertion kinds fail closed, which stops a student from smuggling an unchecked rule into the file.

version is an integer, and this workshop uses version 1 only.
task stores the human instruction so a replay does not depend on chat history.
fixture_text is the expected body, normalized to one trailing newline.
fixture_sha256 is the hex digest of that normalized body.
assertions holds contains, max_lines, or exact_hash checks, and no other kinds.
lanes.fixture is required, while lanes.free_model and lanes.free_server default to skipped.

Worked example students can rerun

The script below is a teaching artifact that writes replay-packet.json and exits zero when the fixture lane passes. The optional lanes are not called in this published example, and they should stay skipped on the first run. Treat any live client as a later exercise that must follow provider documentation you verified during that same week. Do not add network code until exercise 4, because an early call would hide whether the local packet is sound.

#!/usr/bin/env python3
"""Pack a replay packet. Live lanes stay skipped unless you add them later."""

import hashlib
import json
import sys
from pathlib import Path

def normalize(text):
    return text.strip() + "\n"

def digest(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def build_packet(task, fixture, assertions):
    body = normalize(fixture)
    return {
        "version": 1,
        "task": task,
        "fixture_text": body,
        "fixture_sha256": digest(body),
        "assertions": assertions,
        "lanes": {
            "fixture": {"required": True, "status": "pending"},
            "free_model": {"required": False, "status": "skipped"},
            "free_server": {"required": False, "status": "skipped"},
        },
    }

def check(packet):
    errors = []
    text = packet["fixture_text"]
    if digest(text) != packet["fixture_sha256"]:
        errors.append("fixture hash mismatch")
    for item in packet["assertions"]:
        kind = item["kind"]
        if kind == "contains" and item["value"] not in text:
            errors.append("missing substring: " + item["value"])
        elif kind == "max_lines" and len(text.splitlines()) > int(item["value"]):
            errors.append("line count above cap")
        elif kind == "exact_hash" and digest(text) != item["value"]:
            errors.append("exact hash assertion failed")
        elif kind not in ("contains", "max_lines", "exact_hash"):
            errors.append("unknown assertion kind")
    return errors

def main():
    task = "Emit a three-line status note that includes the word READY."
    fixture = "status: READY\nlane: fixture\nexit: 0"
    packet = build_packet(task, fixture, [])
    packet["assertions"] = [
        {"kind": "contains", "value": "READY"},
        {"kind": "max_lines", "value": 3},
        {"kind": "exact_hash", "value": packet["fixture_sha256"]},
    ]
    errors = check(packet)
    packet["lanes"]["fixture"]["status"] = "pass" if not errors else "fail"
    Path("replay-packet.json").write_text(
        json.dumps(packet, indent=2) + "\n", encoding="utf-8"
    )
    print("errors", len(errors), "hash", packet["fixture_sha256"][:12])
    return 0 if not errors else 1

if __name__ == "__main__":
    sys.exit(main())

Local commands

Run the three commands from a clean directory, and disable the network if you want a stricter proof. The expected local result is a zero exit, a printed error count of zero, and two lanes left as skipped. If you change the fixture text after the hash is stored, the check fails and the process exits one. Students should trigger that failure once, restore the matching text, and keep both JSON files for the pair review.

python3 pack_replay.py
python3 -m json.tool replay-packet.json > /dev/null
python3 -c "import json; p=json.load(open('replay-packet.json')); assert p['lanes']['fixture']['status']=='pass'"

Failure you should force

After the first successful run, edit replay-packet.json and change one character inside fixture_text without updating the digest. Recompute the check in a short shell snippet, and confirm that the hash mismatch is reported before any lane is trusted. Students should restore the file by rerunning the packing script, rather than editing the digest to bless the broken text. That sequence shows why the digest is a guard against silent edits, not a decorative field copied from a tutorial.

python3 -c "import json,hashlib; p=json.load(open('replay-packet.json')); t=p['fixture_text']; print(hashlib.sha256(t.encode()).hexdigest()==p['fixture_sha256'])"

Exercise sequence

Work these steps in the listed order, and do not open a model client until the fourth exercise. Skip the fourth and fifth exercises when the instructor has not confirmed the current free-path terms. The first three exercises are the grade, and the last exercise is a peer check rather than a new feature. A student who finishes only the local steps has still completed the workshop outcome described above.

Write the fixture by hand, and do not ask a model for the expected text, because that would hide the oracle inside the tool you are trying to constrain.
Add one contains check, one max_lines check, and one exact_hash check, then run the script and keep the JSON.
Break the stored fixture on purpose, confirm a failed comparison, and write one sentence naming which guard fired.
Optional: if free model access is available, send only the task string, store the completion beside the fixture, and never overwrite fixture_text.
Optional: if a free server is available, print the fixture there and mark pass only when the stdout hash matches.
Exchange packets, and reject any file that lacks a hash, uses an unknown kind, or marks a live pass without a digest.

Decision table

Use the table when the room must choose a lane without turning a missing host into a failed grade. The entries are class rules written for this workshop, not measured limits taken from any provider. A match means the compared digest equals fixture_sha256, and a diverge means the side file differs. Never relax an assertion after reading a live completion, because that move lets the model rewrite the test.

Situation	Lane to score	What to record	Do not do
Network off or key missing	fixture	both live lanes skipped	Invent a completion
Free model returns text	fixture, plus a side file	match or diverge against the fixture digest	Replace the fixture with the completion
Free server prints stdout	fixture, then server	pass only on a hash match	Treat server uptime as the grade
Completion exceeds three lines	fixture	keep the original max_lines cap	Relax the cap after seeing model text
Offer terms unread today	fixture only	note that live lanes were not authorized	Copy a quota from an older article

How the free options fit, and where they stop

Free model access is a reasonable way to let students inspect a live completion after the oracle already exists. A free server is a reasonable place to run the print command when local machines differ and logs have been reviewed. Neither option should set the score, because availability and limits are not treated as fixed facts in this article. If either path is down on class day, the fixture lane still finishes the lab and preserves a comparable result.

Re-read the current terms before minute 52, since a saved screenshot is not evidence for a later cohort. Do not send credentials, personal data, or unpublished student work to a shared host until retention and access are checked. The packing script does not open a socket, which keeps the required lane independent of whichever host you add later. That separation is the lesson, and a branded endpoint does not change the order of fixture first and call second.

Limitations

Hash equality is strict, so a valid paraphrase looks like a failure if you grade the live completion instead of the fixture. The schema does not estimate tokens, latency, or cost, and it should not be reused as a billing or capacity report. Version 1 also ignores tool calls, file edits, and multi-turn chats, so it will not catch an agent that mutates a repository. The example was not executed against a live endpoint in this draft, so a local pass proves only the packing script.

Who should not use this approach

Skip this workshop if the learning goal is open-ended writing, where many different answers remain equally acceptable. Skip it when the lab requires production credentials, private customer data, or a shared host that nobody can audit. Skip it if students need a statistical quality score, because three string checks will not measure taste or coverage. Also skip it when the instructor cannot commit to a fixture before class, since the method fails if the expected text is still moving.

Close

Pack the fixture, hash it, and rerun the three local commands before the room discusses any endpoint. Keep the free model lane and the free server lane as optional evidence that you compare, not as text you trust. If a cohort already has MonkeyCode free model access and the free server option, use both only after a same-day terms check. Leave the fixture status as the grade either way, including when those optional lanes are skipped or unavailable.