DEV Community: Charlie Xu

Budget the Free Tokens Like Lab Chemicals: A Bootcamp Lab on Metered Agent Runs

Charlie Xu — Tue, 27 Oct 2026 13:08:02 +0000

Your free quota is not a budget until something refuses to spend past it. That single sentence is the whole lab.

Most agent tutorials teach students to call a model and print the answer. Then the cohort discovers, the hard way, that a retry loop can eat an entire free allowance before anyone notices. Sound familiar?

This is a 90-minute bootcamp lab I run as a discipline exercise: build the meter before you build the agent. It uses only the Python standard library, so a clean clone runs with zero installs. The one thing it cannot give you is exact billing numbers — and that limitation is part of the grade.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The 2 a.m. failure this prevents

Free model access and a free server option are genuinely useful for a classroom: nobody has to collect credit cards from twenty students before the first session. MonkeyCode's open-source project advertises both, and the operator describes an allowance on the order of ten million tokens. Verify the current terms yourself before you build a syllabus on them; free-tier numbers move, and I am not going to pretend otherwise.

But free access fails quietly. It does not raise an error at minute two. It raises one at minute forty, in the middle of a demo, with no record of which step spent what.

There is a running DEV thread this week about tools that sit between a developer and the model. Fair complaint. A spend ledger is the smallest honest answer to it: you can see what actually crossed the wire, and what it cost.

What students build (three parts, in this order)

A ledger. Append-only rows: run id, step, model, prompt size, output size, estimated tokens, wall time.
A guard. A pre-flight check that refuses to start a step once spending reaches the budget minus a reserve.
An adapter. One function that actually talks to an endpoint. Everything else must work without it.

That third point matters for grading. If the adapter is a stub, the lab is testable offline and reproducible on a plane.

Setup (about 10 minutes)

python3 -m venv .venv && . .venv/bin/activate
mkdir -p lab && cd lab
python3 -c "import sqlite3, json, time; print('stdlib ok')"

No pip install. Students write ledger.py and runner.py, then commit both.

Checkpoint 1 — the ledger writes before the guard exists

# ledger.py — stdlib only. Not run against any live endpoint; the adapter is yours.
import sqlite3, time

DB = "runs.sqlite"

def init(db=DB):
    con = sqlite3.connect(db)
    con.execute("""CREATE TABLE IF NOT EXISTS spend(
        run_id TEXT, step INTEGER, model TEXT,
        prompt_chars INTEGER, completion_chars INTEGER,
        est_tokens INTEGER, wall_ms INTEGER, ts REAL)""")
    con.commit()
    return con

def estimate_tokens(text: str) -> int:
    # Crude on purpose: ~4 characters per token for English prose.
    # Swap in the provider's own tokenizer if it exposes one.
    return max(1, len(text) // 4)

def record(con, run_id, step, model, prompt, completion, wall_ms):
    est = estimate_tokens(prompt) + estimate_tokens(completion)
    con.execute("INSERT INTO spend VALUES (?,?,?,?,?,?,?,?)",
        (run_id, step, model, len(prompt), len(completion), est, wall_ms, time.time()))
    con.commit()
    return est

def spent(con, run_id):
    (n,) = con.execute(
        "SELECT COALESCE(SUM(est_tokens),0) FROM spend WHERE run_id=?", (run_id,)).fetchone()
    return n

Why a database and not a counter in memory? Because the question students ask at hour two is "what did step seven do?" A counter cannot answer it.

Checkpoint 2 — the guard must fail on request

# runner.py
import time
import ledger

BUDGET  = 10_000_000   # headline figure as described by the operator; confirm current terms
RESERVE = 50_000       # never spend the last slice, so a demo can always finish

class BudgetExceeded(RuntimeError):
    pass

def call_model(prompt, model):
    raise NotImplementedError("Wire your endpoint here. Keep it behind this one function.")

def step(con, run_id, n, prompt, model="default"):
    if ledger.spent(con, run_id) >= BUDGET - RESERVE:
        raise BudgetExceeded(f"run={run_id} step={n} stopped at the reserve line")
    t0 = time.monotonic()
    out = call_model(prompt, model)
    ms = int((time.monotonic() - t0) * 1000)
    used = ledger.record(con, run_id, n, model, prompt, out, ms)
    print(f"step {n}: +{used} est, total {ledger.spent(con, run_id)}/{BUDGET}")
    return out

Now the red test. Monkeypatch call_model to return the same fixed string every time, set BUDGET = 400 inside the test, and assert three things:

the run raises BudgetExceeded rather than returning a short answer;
no row exists for the step that was refused;
spent() equals exactly the sum of the rows printed.

If a student's guard "passes" by lowering the answer quality instead of stopping, that is a fail. A guard that degrades quietly is worse than no guard.

Checkpoint 3 — resume without re-spending

Add one function: read MAX(step) for the run and continue from there. Then kill the process mid-run and restart it. The ledger must show no duplicate step numbers and no re-charged work.

This is where free servers earn their keep in the lab. A free host is a fine place to discover that your runner is slow, that cold starts blow up your wall_ms column, and that your resume path was never actually tested. It is not a place to discover cleverness. Measure, don't assume — and if the host sleeps or throttles, your ledger tells you when, in milliseconds, not in vibes.

Stretch goals (pick two)

Tokens per accepted step. Divide estimated tokens by steps that produced a usable artifact. The number is usually ugly.
Two-model A/B on identical prompts. Same prompts, same order, compare spend at equal acceptance.
Hard step cap. A token budget does not stop a loop that fails cheaply. Add MAX_STEPS and a test for it.
Estimator error report. Twenty samples, compare estimate_tokens against provider-reported usage if available, publish the mean error.

A fair grading rubric

Weight	Criterion	Meets the bar
25%	Guard actually fires	Refusal raises; no row written; test committed
20%	Ledger reconciles	Stored sum matches printed totals; arithmetic shown
20%	Resume correctness	Restart adds no duplicate steps, no double spend
15%	Failure write-up	Names the real cause, not "the API was slow"
10%	Clean-clone setup	Runs offline from the repo with no installs
10%	Stated limitations	Owns the estimator's error

Note what is not on the rubric: whether the agent was impressive. Impressive is ungradable. Repeatable is not.

Who should not use this approach

The estimator is deliberately crude. At four characters per token it under-counts code, JSON, and non-Latin scripts, so anyone who needs invoicing-grade numbers should use provider-reported usage, not this. If your assessment depends on a free tier staying available for the full exam window, don't — keep a paid fallback or run the adapter in a recorded-fixture mode. And if a student only wants to call an API once to see it work, this lab is overkill and will feel like paperwork.

For a cohort that needs a shared default without a procurement conversation, the free model access and free server option are a reasonable on-ramp — with the ledger as the source of truth and the terms you measured written at the top of the repo. Try it with one group of five before you commit a syllabus to it.

The Demo Dies at Minute Nine: A Bootcamp Lab on Rehearsing Agent Failures

Charlie Xu — Mon, 26 Oct 2026 11:39:42 +0000

Your agent demo will not fail because the model is dumb. It will fail at minute nine, when the room goes quiet and the second request hangs.

So this lab grades the slow path. Students build one small tool-using agent, park it on a host, and then spend the entire session rehearsing the two things that actually break live: latency and a rejected request.

No leaderboard for tokens. No prize for the cleverest prompt. If the demo cannot survive a bad minute, the prompt never mattered.

For the lab I point students at two concrete pieces of the stack: MonkeyCode's free model access, and its free server option for hosting the rehearsal runner. The outreach brief also lists a free 10M-token allowance — students read the current numbers in their own dashboard, not on my slide, because quotas move. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The rule of the lab

Rehearse the failure, not the happy path. Students write three scripted prompts, then prove what happens when the network misbehaves.

Half the agent stacks shipping this month are proxies with a personality. Every proxy is another place a demo can stall — and another place a student can blame the wrong component.

So the artifact is not the agent. The artifact is the rehearsal log.

Setup: 30 minutes, no dependencies

Students need three files and one terminal. Everything below is standard library only, on purpose — a lab that starts with a 400MB install teaches the wrong lesson.

mkdir -p lab04 && cd lab04
python3 -m venv .venv && source .venv/bin/activate
python3 --version   # 3.9+ is enough

The agent sits behind a tiny local service so the harness never needs to know which provider is answering. That indirection is the point: swap the model, keep the contract.

# agent_server.py  (student-written wrapper; ~40 lines)
# POST /turn  {"prompt": "..."}  ->  {"text": "..."}
# Inside, call whichever model endpoint your key belongs to.

Scripted prompts live in script.json:

[
  {"turn": 1, "prompt": "List the three files changed in this diff."},
  {"turn": 2, "prompt": "Now summarize the risk of the second one."},
  {"turn": 3, "prompt": "Give me the one-line rollback command."}
]

Checkpoint 1 — Record the baseline

The harness times every turn and writes a machine-readable log. No vibes, no stopwatch.

# rehearse.py — stdlib only
import json, os, time, urllib.request, urllib.error, statistics

AGENT = os.environ.get("AGENT_URL", "http://127.0.0.1:8080/turn")
TURNS = json.load(open("script.json"))

def one_call(prompt, timeout=20.0):
    req = urllib.request.Request(
        AGENT,
        data=json.dumps({"prompt": prompt}).encode(),
        headers={"Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            body = json.loads(r.read())
        return {"ok": True, "ms": (time.perf_counter() - t0) * 1000, "text": body.get("text", "")}
    except urllib.error.HTTPError as e:
        return {"ok": False, "ms": (time.perf_counter() - t0) * 1000, "status": e.code}
    except Exception:
        return {"ok": False, "ms": (time.perf_counter() - t0) * 1000, "status": "timeout"}

results = [dict(turn=t["turn"], **one_call(t["prompt"])) for t in TURNS]
ok_ms = sorted(r["ms"] for r in results if r["ok"])

def pct(xs, p):
    return xs[min(len(xs) - 1, int(len(xs) * p))] if xs else None

summary = {
    "turns": len(results),
    "failures": [r for r in results if not r["ok"]],
    "p50_ms": pct(ok_ms, 0.50),
    "p95_ms": pct(ok_ms, 0.95),
}
json.dump({"summary": summary, "results": results}, open("rehearsal.json", "w"), indent=2)
print(json.dumps(summary, indent=2))

Three turns is not statistics. It is a smoke test — say that out loud in the write-up, or the log becomes a lie with good formatting.

Checkpoint pass condition: rehearsal.json exists, contains three turns, and the student can state p50 and p95 without opening the file.

Checkpoint 2 — Inject two failures

Now break it deliberately. Two toggles, both cheap:

Latency. Add a fixed sleep inside the wrapper before the model call: time.sleep(6). The demo must still finish under the clock.
Rejection. Make the wrapper return HTTP 429 for the second turn only. Students must handle it without editing rehearse.py.

Why 429 and not a generic error? Because rate limits are the failure students will meet most often on any free tier, and the correct response — back off, or fall back — is a design decision, not a try/except.

Checkpoint pass condition: the harness reports two distinct failure modes, and the student can name which component owns each one.

Checkpoint 3 — Build the playback path

A demo needs an escape hatch that is honest. Recorded replies are fine, as long as the audience is told.

# playback.py — serve recorded turns when live calls fail
def turn(n, live_fn, recorded):
    try:
        out = live_fn(recorded[n]["prompt"])
        if out["ok"]:
            return out["text"], "live"
    except Exception:
        pass
    return recorded[n]["text"], "recorded"

Label it on screen. source: recorded in the corner of the slide beats a silent lie every time.

Checkpoint pass condition: the student shows a run where turn 2 is recorded and turn 3 is still live.

Checkpoint 4 — The nine-minute clock

Run the whole script against a real timer. If it overruns, cut a turn rather than talking faster.

This is where most teams discover their demo was really a monologue with an API call attached.

Stretch goals

Add a fourth turn that requires memory of turn 1. Prove state survives the playback switch.
Hold the six-second latency injection and still land under the clock.
Publish the runner on a remote host so a grader can hit it — this is the least-friction use of the free server option in the lab, and it removes "works on my laptop" from the grade entirely.
Write a two-line postmortem for a run that failed. Best artifact in the cohort, usually.

A grading rubric that is actually fair

Weight	Criterion	What earns full marks
20%	Baseline log	`rehearsal.json` with 3 turns, p50/p95 quoted correctly
25%	Failure injection	Latency and 429 both reproduced, each attributed to a component
20%	Playback path	Mixed live/recorded run, visibly labelled
20%	Nine-minute clock	Finishes with margin, or a turn was cut deliberately
15%	Write-up honesty	States what the sample size does not prove

Two fairness rules, stated before anyone starts:

Latency outside the student's control is never penalised — only the response to it is graded.
Partial credit is real: reproducing a failure and explaining it earns most of the 25%, even if the fix is wrong.

The rubric explicitly refuses to reward raw speed, model-name dropping, or longer prompts. Those are not demos.

Limitations, and who should skip this

This lab measures one client, one host, three turns. That is a rehearsal, not a benchmark.

Do not run it as a load test. Parallel calls will get you rate-limited, and the log will tell you nothing useful.
Free-tier latency varies by time of day. Compare runs from the same hour or don't compare them.
The playback path can hide real regressions if you let it. Log every fallback.
Not for anyone who needs an SLA, production traffic numbers, or a capacity plan. This is a teaching harness, and it should stay small.

If you run a cohort, replace script.json with your own three prompts, grade the logs, and watch where minute nine lands. Then tell me which turn broke first — I keep a running list.

Free Breaks in Two Places: A Bootcamp Lab on Attributing Agent Failures

Charlie Xu — Tue, 20 Oct 2026 09:46:02 +0000

Free Breaks in Two Places: A Bootcamp Lab on Attributing Agent Failures

A student's agent run dies on a free stack. They spend forty minutes rewriting prompts. The actual cause? The box went to sleep. Or the reverse — they restart the server three times to fix an expired key.

That's the whole lab in one sentence: on a free stack, "it broke" is two different bugs wearing the same shirt.

This is the lab I hand out for a 90-minute pair session. It teaches failure attribution before it teaches anything about prompts, and it works because attribution is cheap to practice and expensive to get wrong.

What the lab is, and what it isn't

It is a drill: given a broken run, name the layer before you touch a line of code.
It is two probes, one table, four checkpoints, and a rubric you can grade in ten minutes.
It isn't a benchmark. We measure nothing about model quality here.
It isn't a vendor review. Free tiers change; the skill of telling layers apart does not.

Two budgets, two clocks

Here's the framing that makes the lab click. A free agent stack usually gives you two independent things, each with its own way of running out:

A metered allowance — you consume work, and the work has a ceiling.
A hosted runtime — a machine exists for you, and machines have a lifetime.

They fail differently. One fails as a refusal, the other as an absence. And the fix for a refusal (slow down, check credentials) is not the fix for an absence (wake it, reconnect).

So — have you ever restarted a server to fix a bad API key? Congratulations, you've debugged the wrong clock.

For the lab substrate I use MonkeyCode, an open-source project whose operator-supplied brief to me describes free model access plus a free server option. That combination is exactly what the drill needs: two failure surfaces, no invoice. The brief also cites a free token allowance — treat any such number as perishable. Date-stamp it on your handout and re-check the current terms before class, because nothing dates a syllabus faster than a quota printed in September.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Artifact 1 — the two-probe script

One script, two probes, one verdict. Standard library only, so nobody spends the lab fighting pip. This version is provided as a lab artifact and was not executed for this post — you run it, you paste the JSON into your log.

#!/usr/bin/env python3
# lab_probe.py — one run, two probes, one verdict.
import json, os, socket, time, urllib.request

BASE = os.environ["PROBE_BASE_URL"].rstrip("/")     # model endpoint
KEY  = os.environ.get("PROBE_API_KEY", "")
HOST = os.environ["PROBE_HOST"]                      # the free server

def run(label, fn):
    t0 = time.monotonic()
    try:
        detail = fn()
        return {"probe": label, "ok": True, "detail": detail,
                "seconds": round(time.monotonic() - t0, 2)}
    except Exception as exc:  # lab code: we want every failure
        return {"probe": label, "ok": False,
                "detail": f"{type(exc).__name__}: {exc}"[:240],
                "seconds": round(time.monotonic() - t0, 2)}

def host_probe():
    port = int(os.environ.get("PROBE_PORT", "443"))
    with socket.create_connection((HOST, port), timeout=5):
        return "tcp-handshake-ok"

def model_probe():
    req = urllib.request.Request(BASE + "/models")
    if KEY:
        req.add_header("Authorization", f"Bearer {KEY}")
    with urllib.request.urlopen(req, timeout=20) as resp:
        return f"http-{resp.status}"

def verdict(host, model):
    if host["ok"] and model["ok"]:
        return "both-up: the bug is yours, not the stack's"
    if not host["ok"] and not model["ok"]:
        return "both-down: network or credentials before anything else"
    if not host["ok"]:
        return "host-side: asleep, restarted, or gone"
    return "model-side: allowance, key, or endpoint — not the box"

if __name__ == "__main__":
    h, m = run("host", host_probe), run("model", model_probe)
    print(json.dumps({"host": h, "model": m, "verdict": verdict(h, m)}, indent=2))

A warning to students: seconds is a hint, not a diagnosis. A slow model probe and a dead host probe can both take ten seconds. Classify on the ok flags first; use timing only to separate "slow" from "refused".

Artifact 2 — the attribution table

After the probes, the pairs fill this in from real failures. Empty cells are the point — a blank cell is a bug in their observation, not in the stack.

Symptom	Host-side tell	Model-side tell	Cheapest next test
Run stalls with no output	probe times out, TCP refused	probe returns 4xx quickly	re-run `lab_probe.py`
Sudden auth error	stop... nothing host-side explains it	401/403 right after key change	re-check the key only
Worked yesterday, dead today	lifetime ended softly	allowance consumed	both probes, in order
Long silence, then success	cold start	throttling	log `seconds` twice
Fine locally, broken in the runtime	env vars missing there	endpoint not reachable	print env, never the key

The four checkpoints

Green baseline (10 min). Both probes ok: true, pasted into the log with a timestamp. No baseline, no lab.
Break it on purpose (20 min). Point PROBE_BASE_URL at a closed port → verdict must say host. Use a deliberately wrong key → verdict must say model. If the script can't tell sabotage apart, the script is wrong, not the stack.
Collect ten real failures (40 min). Every row gets probes, timestamp, and the layer you claim. Guessing is allowed; unrecorded guessing is not.
Cold handoff (20 min). Swap logs with another pair. They predict the layer from the log alone. Your notes are the product now.

Stretch goals

Add one retry-with-backoff, but only for the layer you can name. If it helps both, you've added jitter, not understanding.
Count how many past failures in your own project were misattributed. That number is usually embarrassing and always useful.
Swap in a different free host or model endpoint and confirm the table survives the swap.

The grading rubric

Criterion	Points	What full marks looks like
Baseline exists and is timestamped	4	two `ok: true` probes, reproducible
Sabotage tests pass	4	verdict flips correctly both ways
Ten logged failures	5	probes + claimed layer + outcome per row
Handoff predicts correctly	4	partner's prediction matches your log ≥ 7/10
Honesty about unknowns	3	cells left blank rather than invented
Total	20

Notice what earns no points: a beautiful prompt. The rubric grades observation, and observation is what transfers to the next stack.

Limitations, and who should skip this

The probe tests reachability and auth. It says nothing about output quality, latency under load, or cost.
Classification is heuristic. Two-layer failures exist, and the script will pick one.
Free-tier terms move. Any number in your handout needs a date and a link.

Skip this lab if your team already runs structured tracing with per-layer error budgets — you have this instrument, just bigger. Skip it too if your project is a single script you run once; attribution only pays off when you debug repeatedly.

Run it before you teach it

Set the three environment variables, run the script once against a live setup, then break it on purpose. If your own verdict surprises you, that surprise is the first slide of your class.

If you want to run checkpoints 1–3 without a purchase order, the free models and free server option in MonkeyCode are enough scaffolding for a first session — just verify the current terms the morning you print the handout.

Probe the Free Tier Before You Teach With It: A Bootcamp Lab in Three Checkpoints

Charlie Xu — Mon, 19 Oct 2026 08:28:44 +0000

If your bootcamp lab leans on a free tier nobody has measured, you are not grading skill. You are grading luck. That is the whole thesis of this lab: before students write a single prompt, they write a probe, and the probe sheet is what gets graded.

I have watched cohorts where two students submitted near-identical agent runs and got wildly different scores, because one ran at 09:00 and the other at 21:00. Neither of them could explain the gap. Could your rubric survive that? Mine could not, until I moved the measurement to the front of the exercise.

So here is the lab, end to end: setup, three checkpoints, stretch goals, and a rubric that a second grader can apply without arguing about vibes.

What the lab assumes

A small cohort (4–12 people) with individual accounts on whatever free tier you are using.
Python 3.9+ and nothing else. No SDK, no framework, no lockfile archaeology.
One shared repo per cohort, one probes.jsonl per student, committed as-is.
Rule zero: observed numbers belong to the observer. Nobody generalizes their measurements to the whole tier.

That last rule matters more than it looks. A probe tells you what your account saw, from your network, in your region, at that hour. It is evidence, not a specification.

Checkpoint 1 — Characterize the endpoint

Students write a probe script that does one thing: sends a request, records what happened, appends a JSON line. No retries on the first pass. No clever abstractions either — you want the raw behavior visible.

The snippet below is labelled as a starting point, not a tested artifact on your provider: it speaks an OpenAI-compatible chat-completions shape, so if your provider's API is not compatible, keep the record format and rewrite post().

# probe.py - one probe, one JSON line, stdlib only.
import json, os, time, urllib.request, datetime

BASE  = os.environ["PROBE_BASE_URL"].rstrip("/")
KEY   = os.environ["PROBE_API_KEY"]
MODEL = os.environ["PROBE_MODEL"]
PATH  = os.environ.get("PROBE_PATH", "/chat/completions")  # adjust to your provider


def post(payload, timeout=90):
    req = urllib.request.Request(
        BASE + PATH,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
        method="POST",
    )
    return urllib.request.urlopen(req, timeout=timeout)


def record(**kw):
    kw.setdefault("at", datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds"))
    kw.setdefault("model", MODEL)
    line = json.dumps(kw, sort_keys=True)
    with open("probes.jsonl", "a") as f:
        f.write(line + "\n")
    print(line)


def probe(name, text, stream=False):
    payload = {"model": MODEL,
               "messages": [{"role": "user", "content": text}],
               "stream": stream}
    t0 = time.perf_counter()
    chunks, body, err = None, None, None
    try:
        r = post(payload)
        if stream:
            chunks = sum(1 for _ in r)   # crude line count, not tokens
        else:
            body = json.loads(r.read())
        ok = True
    except Exception as e:
        ok, err = False, f"{type(e).__name__}: {e}"
    ms = round((time.perf_counter() - t0) * 1000)
    record(probe=name, ok=ok, latency_ms=ms, streamed=stream,
           chunks=chunks, error=err)


if __name__ == "__main__":
    probe("c1-short", "Reply with the single word: ready")
    probe("c1-stream", "Count to five, one number per line.", stream=True)
    probe("c1-toolish", "Call the `add` tool to compute 2+2.")

Run it once per student and check the shape of the output before anything else:

python probe.py
wc -l probes.jsonl      # expect 3, not 0 and not 12

Checkpoint 1 passes when every student has three lines and can point at which of them returned ok: false and why. A student who "fixes" a failing probe by deleting it has failed the checkpoint. That is the single most common failure I see.

Checkpoint 2 — Repeat until the spread is visible

One run is an anecdote. Five runs is a distribution, and a distribution is what a rubric can point at.

for i in $(seq 1 5); do python probe.py; done

python - <<'PY'
import json, statistics
rows = [json.loads(l) for l in open("probes.jsonl")]
ok = [r["latency_ms"] for r in rows if r["probe"] == "c1-short" and r["ok"]]
print(f"n={len(ok)} min={min(ok)} median={statistics.median(ok)} max={max(ok)}")
PY

Do not paste my output into your lab notes, because I do not have your output and neither does anyone else in the room. The deliverable is your n, your spread, and one sentence about what would make you distrust it (small n, one slow hour, a laptop on hotel wifi).

Checkpoint 2 passes when the student reports n honestly and names one confounder. Extra credit for noticing that c1-toolish may fail for a boring reason: the provider never promised tool calls.

Checkpoint 3 — Break it on purpose

Happy-path probes teach nothing. Students inject three faults and record recovery behavior:

Bad key. PROBE_API_KEY=not-a-key python probe.py — does the client surface a clear error, or silently write an empty success?
Oversized input. head -c 200000 /dev/urandom | base64 -w0 > big.txt then send it. Record truncation, refusal, or a timeout — not a guess about which.
Mid-flight interruption. Wrap the call in timeout 3 python probe.py. Then check: did a partial record get written, or nothing at all?

The point is a taxonomy, not a fix list. "Fails loudly" and "fails silently" are different grades of engineering, and only one of them will wake you up at 3 a.m.

The rubric, built to be fair under wobble

This is the part most lab write-ups skip, so here it is as a table a second grader can reuse.

Criterion	Weight	Evidence required	Scores zero if
Fixture sheet	20%	Enumerated probes with expected outcome, written before running	Probes described in prose only
Records	25%	`probes.jsonl`, one line per attempt, append-only	Any hand-edited line
Variance summary	20%	`n`, min/median/max, one named confounder	Only one run exists
Failure taxonomy	20%	All three injected faults, observed behavior quoted	No injected faults
Re-run note	15%	A classmate ran it and got a comparable record shape	"Works on my machine"

Two rules keep it honest when the free tier wobbles:

Never reward a lucky success. If a probe passed once and failed twice, the graded artifact is the record, not the win. Success rate is part of the evidence, not a bonus.
Grade artifacts that exist. Effort, intent, and a screenshot of a chat window are not artifacts. This is also what makes it fair: two graders reach the same score because they are reading the same JSON lines.

If your cohort is big enough for two graders, grade blind and compare. Any disagreement wider than one point on a 4-point scale means the criterion was written too vaguely — fix the rubric, not the student.

Stretch goals

Run the probe at three different hours and plot median latency per hour. Now you can talk about traffic dependence with evidence instead of folklore.
Add --repeat N and an automatic summary, so checkpoint 2 is one command.
Generate a Markdown fact sheet from probes.jsonl with a script. Hand-typed summaries drift from the data within a week.
Put the probe on a scheduled CI job. Fail the job when the record schema changes, not when latency moves — schemas are contracts, latency is weather.

Where MonkeyCode fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project, and the operator states that it offers free model access plus a free server option — at the time of writing, a 10M free-token allowance as part of that offer. For a bootcamp, that combination is genuinely useful for one narrow reason: checkpoints 1 and 2 need a place to run and a model to hit, and neither should require a budget approval email. Students can probe, record, and re-run on day one.

But please keep the recording discipline. Treat the advertised allowance as the campaign's claim, not as your measurement — put your n, your failures, and your spread in the repo, and re-run the probe before every cohort. Free tiers move; your fact sheet should move with them. And if you need guarantees on throughput, data handling, or uptime for a production workload, this is the wrong tool and the wrong lab — go get a contract with an SLA instead.

If you want to run the three checkpoints as written, the free model access and free server option are enough to get through them, and the repo README is the place to check current terms before you promise anything to students.

Pin the Contract, Not the Model: A Bootcamp Lab on Provider-Swappable Agent Runs

Charlie Xu — Sun, 18 Oct 2026 07:08:12 +0000

A lab that grades an agent run should never care which model produced that run. If your rubric silently changes when the free tier changes, you were grading the vendor, not the student.

That is the whole thesis of this lab, and it fits in one sentence: record once, replay forever, grade offline.

The failure I keep watching students repeat

Every bootcamp cohort hits the same wall around week two. A student builds an agent loop against whatever model was free that morning. It works. They submit. Two days later the grader re-runs the submission and gets a different answer, a different tool call, a timeout, or a rate-limit page.

So what did the rubric actually measure? Not the design. It measured how lucky the re-run was.

And honestly — is that the student's fault? Partly. But mostly it's a lab-design bug. If the grader has to call a live model to decide a grade, the grader is non-deterministic by construction, and no rubric can survive that.

The rule: pin the contract, not the model

Here's what I now require from every submission:

Prompts are hashed. Same prompt string, plus or minus whitespace, maps to the same key.
Responses are recorded to a plain JSON cassette, committed to the repo.
Grading runs on replay only. No network, no API key, no clock dependency.
One live run at the end, as a demo, never as the source of truth.

That's it. Four rules, and the whole class stops fighting flaky re-runs.

Setup: three files, fifteen minutes

python -m venv .venv && . .venv/bin/activate
pip install httpx pytest
mkdir -p cassettes lab && touch lab/__init__.py

The contract is a Protocol. Not an abstract base class, not a framework — a protocol, so any object with a complete method qualifies.

# lab/providers.py
import hashlib, json
from pathlib import Path
from typing import Protocol

def prompt_key(prompt: str) -> str:
    # normalize whitespace so "hi" and "hi " are the same recording
    return hashlib.sha256(prompt.strip().encode("utf-8")).hexdigest()[:16]

class Provider(Protocol):
    name: str
    def complete(self, prompt: str) -> str: ...

class CassetteMiss(KeyError):
    """Raised when the replay provider has no recording for a prompt."""

class Replay:
    name = "replay"

    def __init__(self, path: Path):
        self.path = path
        self.data = json.loads(path.read_text()) if path.exists() else {}

    def complete(self, prompt: str) -> str:
        k = prompt_key(prompt)
        if k not in self.data:
            raise CassetteMiss(f"no recording for {k}: re-record, do not blind-retry")
        return self.data[k]

The recorder wraps any live provider and appends what it sees. Note the ordering: it checks the cassette before calling out, so re-running a half-finished session doesn't burn calls.

class Recorder:
    name = "recorder"

    def __init__(self, inner: Provider, path: Path):
        self.inner, self.path = inner, path
        self.data = json.loads(path.read_text()) if path.exists() else {}

    def complete(self, prompt: str) -> str:
        k = prompt_key(prompt)
        if k in self.data:
            return self.data[k]
        out = self.inner.complete(prompt)
        self.data[k] = out
        self.path.write_text(json.dumps(self.data, indent=2, sort_keys=True))
        return out

The live adapter below is a sketch, not verified code — endpoints, auth headers, and response shapes differ per provider, so copy field names from that provider's own docs.

# lab/live.py — sketch, adapt before use
import httpx

class Live:
    name = "live"

    def __init__(self, base_url: str, api_key: str, model: str):
        self.base_url, self.api_key, self.model = base_url, api_key, model

    def complete(self, prompt: str) -> str:
        r = httpx.post(
            f"{self.base_url}/chat",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"model": self.model, "prompt": prompt},
            timeout=60,
        )
        r.raise_for_status()
        return r.json()["text"]

Checkpoints: three, each pass/fail in under a minute

Checkpoint 1 — whitespace is not a new prompt. Two calls, different padding, one recording.

def test_whitespace_normalized(tmp_path):
    c = tmp_path / "c.json"
    c.write_text(json.dumps({prompt_key("summarize this"): "ok"}))
    assert Replay(c).complete("summarize this") == "ok"
    assert Replay(c).complete("  summarize this\n") == "ok"

Checkpoint 2 — a miss is loud. No empty strings, no None, no swallowed exception. The CLI must exit non-zero.

def test_miss_raises(tmp_path):
    p = Replay(tmp_path / "empty.json")
    with pytest.raises(CassetteMiss):
        p.complete("never recorded")

Checkpoint 3 — two providers, one grade. Feed the same cassette through the grader twice; the output hash must match. This is the checkpoint that proves the rubric is model-independent, and it's the one most students fail on the first attempt because they hid a model_name inside their scoring logic.

Stretch goals, for the students who finish early

Drift detector: hash the prompt template file itself; if the template changes, warn that cassettes are stale.
Cassette hygiene: flag recordings older than the last template change, and refuse to grade on them.
Structured output guard: if the agent is supposed to return JSON, validate the replayed string against a schema and fail the checkpoint on malformed output — a replay should still be parsed, not trusted.

Pick one. Document the failure mode you found. Three half-finished stretch goals score worse than one finished one, and I say that in the syllabus.

The rubric I actually use

Criterion	Weight	Full marks looks like	Typical loss
Contract stability	30	identical grade across two providers	only passes on the model they happened to use
Loud failure	25	`CassetteMiss` propagates, exit code ≠ 0	returns `""` and looks successful
Trace	20	log line per call: key, provider, elapsed ms	only the final answer is logged
Write-up	15	one honest paragraph on what broke	“it worked”
Stretch	10	one goal, documented, with the failure it surfaced	three started, none finished

Notice what's absent: no points for model choice, no points for tokens spent, no points for speed. Those are environment variables, not skills.

Running the lab when the budget is zero

A room full of students on personal laptops needs two things a lab budget usually can't buy: model calls, and somewhere to run the grader that isn't a sleeping laptop on hostel Wi-Fi.

This is where I point students at MonkeyCode's open-source project, which offers free model access and a free server option — enough to record a cassette on day one and replay it for the rest of the term. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat I'd give any cohort: free allowances move. Check the current allowance in the console on the day you record, and don't copy a number from a blog post — including this one. The recorded cassette is the durable artifact; the quota is weather.

When this lab is the wrong tool

Replay hides variance, and sometimes variance is the subject.

Don't use it for tool-calling semantics. Providers disagree on how function arguments are serialized; a replayed call can hide a real incompatibility.
Don't use it for latency or quality benchmarks. You'd be benchmarking a JSON file.
Don't use it as a production eval harness. Production needs stochastic sampling; this is a grading device, not an evaluation platform.

Boundaries are part of the deliverable. If a student can't say what their lab can't measure, they haven't finished the write-up.

What the students actually learn

The lesson isn't about providers. It's that anything you can't re-run identically, you can't grade fairly — and that includes your own assumptions about the model. Record once, replay forever, keep the rubric boring.

If you run this lab with a cohort, start a FAILURES.md in the repo on day one. The second entry is always more interesting than the first.

The Second Host Is the Test: A Bootcamp Lab on Handoff-Ready Agent Runs

Charlie Xu — Sat, 17 Oct 2026 06:12:12 +0000

A run that only works on your laptop is not a result. It's a screenshot with extra steps.

That's the whole lab. Students build a small agent loop, then they have to prove it works on a machine they don't own — the free server option, a classmate's box, whatever the grader hands them. The grade comes from the second host, not the first.

Why so strict? Because "it ran on my machine" is the most expensive sentence in software, and bootcamp demos are where it breeds.

The brief, in one screen

Each team ships a repo that a stranger can run without asking a single question. The repo must contain three things:

env.manifest.json — what the run needs, in machine-readable form.
preflight.py — a script that reports whether the host matches the manifest.
checkpoint.json — a resumable state file plus the code path that resumes from it.

No chat logs. No "ask me for the API key". No 12-minute screen recording. If the grader has to read your history to run your code, that's a zero on reproduction, and there is no partial credit for enthusiasm.

Artifact 1: the manifest you can diff

Keep it boring and declarative. Boring survives a handoff.

{
  "entrypoint": "python run_agent.py --steps 5",
  "python": ">=3.10,<3.13",
  "env_vars": ["MODEL_BASE_URL", "MODEL_NAME"],
  "expected_outputs": ["out/run.jsonl", "out/trace.md"],
  "network": "outbound https required during run",
  "time_box_seconds": 300
}

The env_vars list holds names only, never values. That one rule kills most key-leak incidents before they happen.

Artifact 2: preflight that prints evidence

This is the core of the lab. Run it on both hosts, diff the JSON, and explain the diff in your README. Here's the version I hand out (labeled: not yet executed against your environment — run it before you trust it):

#!/usr/bin/env python3
"""preflight.py — run on BOTH hosts, then diff the two reports."""
import json, os, platform, shutil, subprocess, sys
from pathlib import Path

REQUIRED_FILES = ["run_agent.py", "checkpoint.json", "requirements.lock"]
REQUIRED_ENV = ["MODEL_BASE_URL", "MODEL_NAME"]  # names only, never values

def sh(cmd: str) -> str:
    p = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    return p.stdout.strip() if p.returncode == 0 else "error"

report = {
    "host": platform.node(),
    "os": f"{platform.system()} {platform.release()}",
    "python": platform.python_version(),
    "cwd": str(Path.cwd()),
    "files_present": {f: Path(f).exists() for f in REQUIRED_FILES},
    "env_present": {k: bool(os.environ.get(k)) for k in REQUIRED_ENV},
    "free_disk_gb": round(shutil.disk_usage(".").free / 1e9, 1),
    "packages": sorted(sh(f"{sys.executable} -m pip freeze").splitlines()),
}

Path("preflight.json").write_text(json.dumps(report, indent=2))
print(json.dumps({k: v for k, v in report.items() if k != "packages"}, indent=2))

Then the diff — this is the step teams skip and regret:

# laptop
python preflight.py && mv preflight.json preflight.laptop.json

# server
python preflight.py && mv preflight.json preflight.server.json

diff <(jq -S 'del(.host,.cwd)' preflight.laptop.json) \
     <(jq -S 'del(.host,.cwd)' preflight.server.json)

Two outcomes are acceptable: no diff, or a diff you predicted in writing. An unexplained diff is a finding, and findings cost points only when they're a surprise.

Artifact 3: a checkpoint the grader can resume

Small schema, strict rules:

{
  "run_id": "lab-042",
  "step": 3,
  "messages": [{"role": "user", "content": "..."}],
  "pending_tool": null,
  "budget": {"steps_left": 2},
  "written_at": "2026-10-17T09:00:00Z"
}

Write it with an atomic replace so a crash can't leave half a file — tmp file, then os.replace. And resume with an explicit cap:

python run_agent.py --resume checkpoint.json --steps 2

If resuming twice changes the output bytes, that's a different lab. Here, we only ask that resuming once produces out/run.jsonl that the grader can read and attribute to a step number.

The three checkpoints

Gate 1, end of day one: manifest committed, preflight.py runs locally, both files parse.

Gate 2, end of day two: the run has been executed on the free server option, preflight.server.json exists, and the README names every diff line.

Gate 3, submission: a classmate you've never paired with runs your entrypoint verbatim, on a host you didn't touch, and gets your documented output shape.

Gate 3 is the one that hurts. That's the point.

A rubric that can't be gamed

Weight	Criterion	What counts as evidence
40	Reproduces on grader host	documented entrypoint, clean exit, expected files
20	Preflight diff explained	two JSON reports + prose that matches them
15	Resumable checkpoint	one resume, attributable output
15	Honest limitations	says what did not work, in its own words
10	README clarity	stranger can start within 5 minutes

Fairness rules the graders follow, not the students:

Grader runs only the documented command. Improvising is not allowed, even if it would work.
Same fixture, same time box (300 s), same step cap for every team.
Timeouts count as failures. A hung run is not "almost working".
No live demos. The repo is the submission; your laptop stays closed.

Stretch goals that don't buy back a failure

Stretch work is capped at 5 bonus points and cannot rescue reproduction. Otherwise the fast teams just do more features and skip the boring part — which is exactly the behavior the rubric exists to price.

Add --dry-run that emits the planned tool calls without calling a model.
Emit a trace.md a non-programmer can follow, one line per step.
Add a preflight --strict mode that fails on any package drift.
Write a one-page incident note for the run that broke, in the format of a real post-mortem.

Where the free tier actually earns its place

This lab needs a second host and a lot of throwaway model calls, and that's the honest reason the free model access and free server option matter here: students get a foreign host and a budget to fail on, without a credit card conversation in week one. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I'm deliberately not quoting a token number or a hardware spec. Quotas and server terms move, and a stale number in a lab handout is worse than no number — check the project's own current pages before you plan a cohort around them.

Limits worth stating up front: a free host is not your production baseline, model behavior can shift between hosts so pin what you can, and this rubric measures reproducibility, not model quality. Nobody here is claiming your agent is good. Only that a stranger can run it.

Who should skip this lab: teams shipping to a fixed, already-provisioned environment, and anyone whose run legitimately needs more compute than a shared free box. Don't contort the exercise — write your own gate.

If you teach this, run preflight.py on your own machine first. Then ask yourself the only question that matters: could a stranger reproduce your green checkmark without you in the room?

If you try the lab on the free server option, send me the diff that surprised you — that's the part I want to read.

Rubric First, Box Second: A Bootcamp Lab Where the Setup Is the Deliverable

Charlie Xu — Fri, 16 Oct 2026 05:10:18 +0000

The short version: in a bootcamp lab, the environment is part of the answer. If a classmate can't rebuild your box from your repo and get the same bytes back, then your clever prompt never mattered. So this lab grades setup, checkpoints, and a rubric — before it grades the agent at all.

Why flip the order?

Most AI labs hand students a working sandbox and a problem. The sandbox is invisible, so the lesson students actually absorb is "the sandbox was just there." That's a bad lesson to carry into a job.

Free tiers make it worse. A free model sitting behind a shared queue is not guaranteed to be the same model tomorrow. A free server is not the same machine next week. Neither of those is a defect — it is the honest shape of free infrastructure. But it means the reproducible part of your work has to be the contract: what you pinned, what you recorded, and what you can rebuild from scratch.

Hence the flip. Deliverable one is a shell script. Deliverable four is a rubric. The agent run sits in the middle, where it belongs.

The lab contract

Four artifacts, one repo:

bootstrap.sh — rebuilds the box from nothing.
manifest.json — pins inputs, model identity, and budget.
checkpoints/ — three named checks a stranger can run.
rubric.json — the scoring rules, written by the student.

Nothing here is exotic. That's the point. If it feels boring, you're doing it right.

Artifact 1: a bootstrap that assumes amnesia

#!/usr/bin/env bash
set -euo pipefail
# lab/bootstrap.sh — rebuild from an empty box. Bash 5+, Debian-ish target.
LAB_ROOT="${LAB_ROOT:-$HOME/lab}"
python3 -m venv "$LAB_ROOT/.venv"
"$LAB_ROOT/.venv/bin/pip" install --quiet -r "$LAB_ROOT/requirements.txt"
mkdir -p "$LAB_ROOT/runs"
git -C "$LAB_ROOT" rev-parse --short HEAD > "$LAB_ROOT/runs/git-sha.txt"
date -u +%Y-%m-%dT%H:%M:%SZ        > "$LAB_ROOT/runs/built-at.txt"
echo "built $(cat "$LAB_ROOT/runs/git-sha.txt") at $(cat "$LAB_ROOT/runs/built-at.txt")"

Two rules for students:

It must run on a box where $LAB_ROOT does not exist yet.
It must print a build stamp. No stamp, no credit.

Why so strict? Because "it works on my machine" is a sentence, not evidence.

Artifact 2: pin the model, don't assume it

This is where free access actually changes the design. The project this lab is built around — the open-source coding agent from MonkeyCode — is described by its operator as offering free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am deliberately not quoting a token quota or a machine spec here. Those numbers move, and a bootcamp that hard-codes them teaches students to trust a stale README. Check the project's own page the morning you run the lab, and write down what you found. That check is itself worth a point.

What you do pin is which model actually answered:

{
  "schema": 1,
  "runtime": "python3.11",
  "model": { "alias": "student-default", "recorded_id": "fill-me-in", "fallback": "fill-me-in" },
  "inputs": { "fixture": "fixtures/issue-0007.md", "sha256": "fill-me-in" },
  "budget": { "max_calls": 40, "max_wall_clock_s": 900 },
  "rebuild": { "script": "bootstrap.sh", "verified_at": null }
}

rebuild.verified_at stays null until the student has run bootstrap.sh twice, on two fresh boxes. That one field kills more hand-waving than any rubric line I've ever written.

Artifact 3: three checkpoints

Not one checkpoint. Three, and each must be cheap enough to run on a whim.

Cold — bootstrap.sh on an empty box, then print the build stamp.
Frozen — re-run the agent with the fixture hash unchanged; assert the output hash equals run one's output hash.
Fenced — assert the diff touches only paths listed in allowed_paths.

A checkpoint is a command, not a paragraph:

$ ./checkpoints/frozen.sh
frozen: output hash 3f9c...  == run-001   OK

If a checkpoint needs three sentences of explanation to run, it isn't a checkpoint. It's a wish.

Artifact 4: stretch goals

Stretch goals are where the lab stops being a tutorial and starts being work:

Swap the model alias and keep the same score.
Cut wall clock by a third without losing a single checkpoint.
Add one rubric criterion that would catch a specific cheat, then explain who it punishes unfairly by accident.

That third one is the best hour of the week. Students grumble, then quietly rewrite the criterion twice.

The fair rubric

A rubric is only fair if two people applying it in the same room land on the same number. Otherwise you're not grading work, you're grading mood.

{
  "criteria": [
    {"id": "R1", "name": "rebuild", "weight": 3, "pass": "bootstrap.sh runs on an empty box and prints a build stamp"},
    {"id": "R2", "name": "repro",   "weight": 3, "pass": "two runs from the same manifest produce identical output bytes"},
    {"id": "R3", "name": "log",     "weight": 2, "pass": "runs/latest.json names model id, fixture hash, and call count"},
    {"id": "R4", "name": "scope",   "weight": 2, "pass": "no edits outside allowed_paths"}
  ],
  "disagreement_rule": "if two graders differ on any criterion, neither score counts until both re-read the manifest"
}

And a twenty-line grader, so the rubric is executable instead of decorative:

import json, sys

def grade(rubric_path, run_path):
    rubric = json.load(open(rubric_path))
    run    = json.load(open(run_path))
    earned = 0
    for c in rubric["criteria"]:
        ok = bool(run.get("checks", {}).get(c["id"]))
        print(f"{c['id']:>3}  {'PASS' if ok else 'FAIL'}  {c['name']}")
        earned += c["weight"] if ok else 0
    total = sum(c["weight"] for c in rubric["criteria"])
    return earned, total

if __name__ == "__main__":
    earned, total = grade(sys.argv[1], sys.argv[2])
    print(f"score {earned}/{total}")
    sys.exit(0 if earned == total else 1)

Then the drill that makes it real: two students grade the same run file, separately, no talking. Different score? The rubric is broken, not the run. Fix the wording of the criterion and try again. That loop — not the agent output — is the skill being taught.

Limitations, honestly

Free infrastructure recycles. A box can vanish between checkpoint two and three. That is the lesson, but it also eats lab time.
Shared queues make wall clock noisy. So don't grade speed. Grade recorded call counts.
The grader reads logs, not code. Pair it with a human read, or students optimize for the log format.
bootstrap.sh assumes network access and a Debian-ish base. Porting it elsewhere is part of the assignment, not a footnote.
These scripts are a skeleton. I haven't validated them across a distro matrix — adapt, then stamp your own manifest.

Who should not use this

Teams that need one fixed production model with compliance paperwork. Pin something stable instead.
Courses where the environment isn't the lesson. A pure algorithms unit gets overhead and nothing back.
Anyone who can't tolerate a free tier that changes shape mid-semester. Grade the contract, or pay for predictable hardware.

Try it, then break it

Run the lab once with a classmate grading you blind. Then ask the only question that survives contact with a deadline: could a stranger rebuild this from your repo on a Tuesday you didn't plan for?

If you'd rather not provision anything yourself just to find out, the free server option and the free model access are the cheapest way to start — check the project page for what's actually available today.

The Budget Is Part of the Answer: A Bootcamp Lab on Grading Agent Runs on a Free Tier

Charlie Xu — Thu, 15 Oct 2026 11:23:24 +0000

The Budget Is Part of the Answer: A Bootcamp Lab on Grading Agent Runs on a Free Tier

Your agent run isn't finished when the tests go green. It's finished when the tests go green inside the budget you were handed. That's the whole lab.

Most students I've watched treat a free tier as an infinite sandbox. It isn't. A free allowance is a constraint you get for free, which makes it a much better teaching tool than a credit card.

So here's the exercise: one prompt, one free model access tier, one free server box, and a ledger that has to reconcile. The grade is not "did it work." The grade is "can you prove what it cost, and where the cost went."

Why this lab exists (and what it is not)

Three things changed the shape of student agent projects recently. Model output got cheaper and more verbose; runs got longer; and the retry loop became the default debugging strategy. That last one is where budgets die.

This lab is not a benchmark. It's not a comparison of providers. It's a metering discipline you can run on any free tier, including the one this lab assumes.

That assumption: MonkeyCode offers free model access and a free server option, which is what makes a no-money lab possible at all. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm not going to restate a quota number here — quota terms shift, and a stale number makes a terrible grading key. The source of truth is the project's own current terms page, which every student screenshots at the start of the lab as their budget contract.

Setup: five steps, twenty minutes

Screenshot the terms page. Name it budget_contract.png. This is your ceiling for the whole lab.
Pick one task with a hard oracle. Parsing a log format, fixing an off-by-one in a paginator, adding a failing edge case. Something a test can settle yes or no.
Refuse to touch the task until logging exists. No logging, no run. That rule saves more time than any prompt trick.
Log every call as one JSON line. One line per model call, one line per tool call, never batched.
Export the log before anything else. Free servers get recycled. Logs that lived only on the box are logs you don't have.

The event schema is small on purpose:

{"step":"fix_offbyone","attempt":1,"kind":"model_call","prompt_tokens":812,"completion_tokens":204,"reason":"initial"}
{"step":"fix_offbyone","attempt":2,"kind":"model_call","prompt_tokens":1490,"completion_tokens":331,"reason":"test_1_failed"}
{"step":"fix_offbyone","attempt":2,"kind":"tool_call","prompt_tokens":0,"completion_tokens":0,"reason":"pytest"}

Note the reason field. A retry without a reason is, by lab rule, an unlabeled retry, and unlabeled retries cost double in the rubric. You'll see why below.

The artifact: a ledger you can run

This script reduces the log to a per-attempt ledger and prints a retry-overhead ratio. It's deliberately boring code, because boring code is what you can trust at 2 a.m.

# ledger.py -- reduce a JSONL run log into a per-attempt budget ledger.
# Usage: python ledger.py run.jsonl > ledger.csv
import csv, json, sys
from collections import defaultdict

REQUIRED = {"step", "attempt", "kind", "prompt_tokens", "completion_tokens"}
KINDS = {"model_call", "tool_call", "other"}

def load(path):
    rows, bad = [], 0
    with open(path, encoding="utf-8") as fh:
        for lineno, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                ev = json.loads(line)
            except json.JSONDecodeError:
                print(f"skip line {lineno}: not json", file=sys.stderr)
                bad += 1
                continue
            missing = REQUIRED - ev.keys()
            if missing:
                print(f"skip line {lineno}: missing {sorted(missing)}", file=sys.stderr)
                bad += 1
            elif ev["kind"] not in KINDS:
                print(f"skip line {lineno}: unknown kind {ev['kind']!r}", file=sys.stderr)
                bad += 1
            else:
                rows.append(ev)
    return rows, bad

def main():
    rows, bad = load(sys.argv[1])
    agg = defaultdict(lambda: {"calls": 0, "prompt": 0, "completion": 0})
    for ev in rows:
        a = agg[(ev["step"], int(ev["attempt"]))]
        a["calls"] += 1
        a["prompt"] += int(ev["prompt_tokens"])
        a["completion"] += int(ev["completion_tokens"])

    w = csv.writer(sys.stdout)
    w.writerow(["step", "attempt", "calls", "prompt_tokens", "completion_tokens", "total_tokens"])
    totals = {"all": 0, "retry": 0}
    for (step, attempt) in sorted(agg):
        a = agg[(step, attempt)]
        total = a["prompt"] + a["completion"]
        w.writerow([step, attempt, a["calls"], a["prompt"], a["completion"], total])
        totals["all"] += total
        if attempt > 1:
            totals["retry"] += total
    ratio = totals["retry"] / totals["all"] if totals["all"] else 0.0
    print(f"# unparsed_events={bad}", file=sys.stderr)
    print(f"# total_tokens={totals['all']} retry_overhead={ratio:.2f}", file=sys.stderr)

if __name__ == "__main__":
    main()

Run it and hand-check two rows against the raw log. If unparsed_events is above zero, you have a logging bug, not a model bug — that distinction alone is worth the whole lab.

The free server changes the checkpoint question

On a box you own, the working directory is durable. On a free server, assume it is not. So the lab forces one habit:

# before you close the tab, before you retry anything:
tar czf ledger_$(date +%s).tgz run.jsonl ledger.csv budget_contract.png
sha256sum ledger_*.tgz >> ledger_hashes.txt

Two files leave the box: the archive and its hash. The hash matters because a grade you can't verify is a rumor. If the box dies mid-run, you still hold the evidence of what was spent.

Rubric: 100 points

Weight	Criterion	What earns full marks
25	Task oracle	A test that fails before and passes after, reproducible by a classmate
25	Ledger integrity	Every call logged, `unparsed_events=0`, hand-checked rows match
15	Retry discipline	Each retry has an `attempt >= 2` line and a non-empty `reason`
15	Budget verdict	Run reported as within or over the screenshot contract, with a number
10	Replayability	A classmate reruns the archive and gets the same ledger totals
10	Failure write-up	One paragraph on what was not measured

No points exist for "it felt fast." No points exist for a pretty demo. A run that passes the test but blows the budget scores below a run that fails the test and reports the overspend honestly — that's the point of the lab.

Failure modes students hit every cohort

Retry amnesia. The fix lands on attempt 4, but only attempt 4 is logged. Total cost now reads as one call. This is the single most common way a ledger lies.
Reason inflation. reason: "retry" on every line is technically non-empty and completely useless. Grade it as unlabeled.
Hash drift. The ledger is edited after the run "to fix a column name." The hash proves it. Regenerate, don't patch.
Oracle smuggling. The test is weakened so the run finishes early. Compare the test file before and after; a shrinking assertion count is a fail.
Budget cliff. The task passes at attempt 9, after 8 identical failures. Log it and move on — you've just learned more than a passing run would teach.

Stretch goals

Add a --fail-over flag to ledger.py that exits nonzero when total_tokens exceeds a value you pass in. Wire it into your run script.
Split retry_overhead by failure class (test_failed, timeout, tool_error) and see which class actually eats your allowance.
Write the same task twice, once stopping at attempt 2 and once at a budget cap, and compare which one a classmate finds more useful.
Add a second, independent check of the ledger — a shell one-liner using jq that sums the same fields. Two counters that agree beat one counter you trust.

What this lab can't tell you, and who should skip it

This lab says nothing about latency, throughput ceilings, or whether a model is good at your domain. It measures one thing: whether you can account for what a run consumed. Also, don't bother if:

your workload can't leave your own network, so an external free tier is off the table;
you're on committed paid capacity where the budget genuinely doesn't bind;
or the grader never looks at your log, in which case metering is theater and you should spend the time on the oracle instead.

Numbers in your report are yours, not mine. Don't quote a total you didn't compute from your own JSONL — and re-check the free tier's current terms the day you run it, because allowances change and your screenshot is the only budget contract that counts.

If you run this, the useful write-up isn't the passing test. It's the ledger line where you realized the retries, not the model, were the expensive part.

The Hint Was Wrong on Purpose: A Bootcamp Lab on Disagree-Before-Retry

Charlie Xu — Tue, 13 Oct 2026 01:18:27 +0000

The second click is not a strategy. I grade two decisions before I look at a green test: did you reject the bad hint, and did you write why another model call was worth the lab budget?

A fluent retry can hide a lazy read. If the only change between runs is hope, what exactly would I be scoring?

This is a lab proposal for a bootcamp cohort, not a write-up of a class I already scored. The checker below is unexecuted example code. Use it as a worksheet. Do not cite it as a benchmark.

Grade the disagreement, not the vibe

The bug is small on purpose. The trap is the comment sitting on top of it.

I want the room to notice how fast a coding agent agrees with the nearest sentence. A wrong comment is the cheapest yes-man you can plant in a starter repo. Why would a model argue with a hint that looks like a spec?

So a passing patch is not the whole grade. A ledger row has to exist before any retry. No row, no second call. Would you trust a classmate's green test if the rerun left no trace?

Setup before anyone prompts

Give this one lab block. I would plan about 90 minutes for a group that already knows pytest. That is a planning guess, not a measured class time.

Fork the starter. Do not "clean up" the misleading comment. The comment is the fixture.
Branch from a clean tree: lab/disagree-retry.
Run the public tests once, before any model call, and save the output.
Use a shared runtime if you have one. If you do not, write the patch by hand. The grade has to survive a closed vendor window.

File layout I would ship:

discount.py — the function under test
tests/test_discount.py — three assertions, dates included
lab_out/retry_gate.py — the checker in the next section
retry_ledger.md — created by the student, empty at the start

git checkout -b lab/disagree-retry
python -m pytest tests/test_discount.py -q | tee attempt-0.txt
mkdir -p lab_out
printf '# retry ledger\n' > retry_ledger.md

The student handout should not include the diagnosis. I am including it here so an instructor can see the trap before assigning it.

from datetime import date

def apply_discount(price_cents: int, code: str, today: date) -> int:
    # HINT: the bug is integer division. Divide after the cast.
    if code == "SAVE10" or today.year < 2000:
        return price_cents - (price_cents // 10)
    return price_cents

The spec I would publish, without the hint:

SAVE10 takes 10% off only when today is on or before 2024-12-31.
Every other input returns price_cents unchanged.
Integer division of cents is acceptable. The hint is wrong about the bug class.

from datetime import date
from discount import apply_discount

def test_expired_code_is_rejected():
    assert apply_discount(1000, "SAVE10", date(2026, 10, 13)) == 1000

def test_active_code_still_discounts():
    assert apply_discount(1000, "SAVE10", date(2024, 1, 1)) == 900

def test_unrelated_code_never_discounts():
    assert apply_discount(1000, "OTHER", date(1999, 1, 1)) == 1000

Run those three before you believe me. On the starter, the expired-code test fails, and the unrelated-code test fails too, because year < 2000 still discounts. The active-code test already passes. That pass is the trap. The function is right for the wrong reason: any SAVE10 discounts, expiry or not.

What is the lie, then? The hint blames division. The failures are an or that skips the date check, plus a year branch that discounts the wrong code. Cents and // can be perfectly fine.

Checkpoint A: name the lie in writing

Before a model sees the file, the student adds one row. Blank line between rows. One - key: value per line. The parser is strict, and that strictness is part of the lab.

attempt: 0
observed: expired SAVE10 still returns 900; OTHER in 1999 also discounts
disagreed: yes
retry_reason: not a retry; the hint blames division and the failures do not

Can you explain both failures without treating the hint as the spec? If the sentence still quotes the hint as the task, you are not ready to call a model.

A shared runtime, and the fallback

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The lab stays useful if you delete every product name. You need some way to run a model, and a way to finish with no model at all.

MonkeyCode's free model access and free server option are the shared runtime I would offer so the cohort is not blocked on who owns a strong laptop. I am not naming models, token quotas, hardware, or a duration. This draft has no primary source to re-check those numbers, and a stale cap in a syllabus is worse than silence. If the current docs disagree with this article, trust the docs.

No model call is required to pass. A hand-written patch plus an honest ledger is a full submission. Do not stall the room because a free window is closed.

I am not describing a license, a repo layout, or a client API here. I have not verified those for this draft.

The artifact: a gate with no network

Proposed checker. Python 3.10+ syntax. It reads markdown. It does not call a model. I have not executed it in a class.

"""Proposed lab gate. Unexecuted example, not a class result."""
import re
import sys
from pathlib import Path

REQUIRED = ("attempt", "observed", "disagreed", "retry_reason")

def parse_rows(text: str) -> list[dict]:
    rows, current = [], {}
    for line in text.splitlines():
        match = re.match(r"-\s+(\w+):\s+(.*)$", line.strip())
        if not match:
            if current:
                rows.append(current)
                current = {}
            continue
        current[match.group(1)] = match.group(2).strip()
    if current:
        rows.append(current)
    return rows

def gate(rows: list[dict], next_attempt: int) -> tuple[bool, str]:
    if next_attempt <= 1:
        return True, "first model call is allowed"
    if not rows:
        return False, "retry blocked: ledger is empty"
    last = rows[-1]
    missing = [key for key in REQUIRED if not last.get(key)]
    if missing:
        return False, "retry blocked: missing " + ", ".join(missing)
    if last["disagreed"].lower() not in {"yes", "no"}:
        return False, "retry blocked: disagreed must be yes or no"
    if len(last["retry_reason"]) < 20:
        return False, "retry blocked: reason is too thin to grade"
    if last["disagreed"].lower() != "yes":
        return False, "retry blocked: planted hint was not rejected"
    return True, "retry allowed"

if __name__ == "__main__":
    attempt = int(sys.argv[1])
    path = Path("retry_ledger.md")
    rows = parse_rows(path.read_text()) if path.exists() else []
    ok, message = gate(rows, attempt)
    print(message)
    sys.exit(0 if ok else 2)

Why does attempt 1 skip the file check? Because I want a human to read the first note, not a regex. The script only locks the retry. The rubric still scores the pre-call row.

python lab_out/retry_gate.py 2
# exit 2 until the ledger rejects the hint
python lab_out/retry_gate.py 2 && echo "call the model now"

Pair the script with two tiny tests so a classmate can check the gate without a network:

def test_blocks_retry_without_disagreement():
    rows = [{
        "attempt": "1",
        "observed": "same fail text as attempt 0",
        "disagreed": "no",
        "retry_reason": "trying again because the first call was vague",
    }]
    ok, _ = gate(rows, 2)
    assert ok is False

def test_allows_retry_after_disagreement():
    rows = [{
        "attempt": "0",
        "observed": "hint blames division; failures are the date check",
        "disagreed": "yes",
        "retry_reason": "hint does not match the expired-code assertion",
    }]
    ok, message = gate(rows, 2)
    assert ok and message == "retry allowed"

A table I would leave on the board

What you saw	Disagree?	Retry?	Why
Tests fail and the hint matches your story	Not yet	No	You may be grading the comment, not the code
One test already passes on the starter	Yes	Maybe	Say which pass is accidental before you call a model
Same failure text as the previous attempt	Already yes	No	Another call will not invent a new observation
Tests pass, ledger never disagreed	n/a	n/a	Still a fail. Luck is not a rubric row

Checkpoint B: one boring patch

The instructor key is dull. Good. Excitement is not the learning goal.

def apply_discount(price_cents: int, code: str, today: date) -> int:
    if code == "SAVE10" and today <= date(2024, 12, 31):
        return price_cents - (price_cents // 10)
    return price_cents

Is 2024-12-31 magic? Only if the tests do not name it. They do. Hidden dates are an unfair rubric. Published dates are fine.

Append a closing row after the patch. A retry is allowed, not required. Why force a second call when the first read was enough?

attempt: 1
observed: expired and unrelated codes no longer discount; hint blamed division
disagreed: yes
retry_reason: no retry; the patch covers the date check the hint ignored

Checkpoint C: someone else can replay

Leave these on the branch:

attempt-0.txt
retry_ledger.md
the patch
attempt-final.txt

A classmate should get the same pass, or the same fail, without your chat log. If they need a screen recording, the lab failed, even when pytest is green.

Stretch, after the core path

Do these only after the rubric path is solid.

Plant a second wrong hint in a README and require the ledger to name the file.
Hard-cap model calls at two. A third needs a TA note, not a longer prompt.
Trade patches with a partner. The gate should still pass. It scores the ledger, not the author's tone.

Rubric I would publish first

Ten points. Hand this out before the lab, not after the first dispute.

2 points: failing output saved before any model call.
3 points: the ledger rejects the planted hint in the student's own words, and mentions both failing cases.
2 points: the gate's exit code matches the decision. No retry can still be full credit.
2 points: the patch satisfies the three published tests.
1 point: a classmate replays from the branch alone.

Partial credit if the patch is wrong and the disagreement is right. I would rather reward "I did not obey the hint" than reward a lucky green run.

Skipping the hosted runtime is not a deduction. Offline is a first-class path. Nondeterminism is the other reason. A free model can wander between Monday and Tuesday. Saved output outranks the chat.

Questions I ask in review

I read the ledger out loud and stop on thin lines.

You wrote "the model was confused." Confused compared to which assertion?
You retried with the same error text. What new fact did the second call have?
The active-code test passed on the starter. Did you notice, or did you only chase the red line?
If I delete the chat, can another student still see why you disagreed?

Short answers lose the point. Specific answers keep it.

Who should skip this

Skip it if the cohort cannot yet read a 15-line function. The or versus and trap will feel like a prank, and it is only fair once they can see the condition.

Skip it if you need a guaranteed free quota in the syllabus. I am not stating one.

Skip it for on-call training. A planted comment is a classroom fixture, not an incident model.

Do not treat the gate as a security control. The student can edit the markdown. The human still has to read the note. A regex cannot tell a real disagreement from a copied sentence, which is why the rubric still has a human point.

What this draft refuses to invent

I did not time a server. I did not count tokens. I did not compare models. If your tool prints a usage line, paste that raw line into the ledger and grade the habit of recording it. Do not grade against a number this article made up.

The approach is a bad fit for anyone who wants a scoreboard based on model cleverness. It is also a bad fit if you cannot spare a human pass over the notes. The script will not save you there.

Before lab day, confirm on the current MonkeyCode docs that the free server option is actually open, write that date into the assignment, and keep the hand-written path so a closed window cannot zero the grade.

Second Run, Same Bytes: A Bootcamp Lab on Idempotent Fixes

Charlie Xu — Sun, 11 Oct 2026 21:25:50 +0000

Second run, same bytes. Otherwise I mark a zero. That is the conclusion, and I am not interested in how smooth the demo felt.

If src/ still moves when the fixer runs again, the submission fails. Why would I pass a script that only behaves on a fresh tree?

The failure I want in the room

Bootcamp agents are fluent on the first edit. They are sloppy on the second.

You ask for a return-string change, run the script, and the file looks done. Run it again. Did it append another helper? Did it flip quotes? Did it "clean up" a comment it just wrote? Then you do not own a fix. You own a generator with no stop condition.

This lab asks one question. After the change is already present, does another run stay still? If you cannot show that with a hash, do not submit.

Setup

Build a fixture so small that excuses sound silly. I would rather debug a stop condition than a fake product.

Commands I put on the board

Create lab-twice/ and initialize git.
Add src/greet.py that returns "hello".
Commit it with the message freeze base.
Add student_fix.py. That file is what I grade.
Add tools/grade_twice.py from the scaffold below. Call it an unexecuted lab proposal, not a measured class result.

mkdir -p lab-twice/src lab-twice/tools
cd lab-twice
git init
printf 'def greet():\n    return "hello"\n' > src/greet.py
git add src/greet.py
git commit -m "freeze base"

The target behavior is tiny on purpose. Make greet() return "hello, lab", and do it so a second run does not change one byte under src/.

Why tiny? I want the miss to be idempotency, not domain trivia. Enlarge the fixture next week if you must. Starting large hides the bug inside a story.

One gate before any model call: git status --porcelain is empty. A dirty start is a zero. I will not sort your scratch files from your submission.

Checkpoint 1 — hand-write the stop condition

Ten minutes. No model. Can you stop yourself before you stop a chatbot?

# student_fix.py — illustrative submission, not a class result
from pathlib import Path

PATH = Path("src/greet.py")
OLD = 'return "hello"'
NEW = 'return "hello, lab"'

def main() -> None:
    text = PATH.read_text()
    if NEW in text:
        return
    if OLD not in text:
        raise SystemExit("fixture not recognized")
    PATH.write_text(text.replace(OLD, NEW, 1))

if __name__ == "__main__":
    main()

See the early return? That branch is the lab. A blind rewrite can still be stable, if and only if the template itself never drifts.

What happens if you append a helper whenever your check is slightly wrong? The second hash moves, and the grade falls over. Which design are you actually choosing: rewrite-from-template, or edit-if-absent? Pick one. Make run two prove it.

Checkpoint 2 — grade the rerun, not the session

Students may sketch a second draft with a hosted model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. For this lab I rely on MonkeyCode only for free model access and the free server option, so a first sketch does not require a private runner or a paid seat.

I am not stating a token quota, a machine size, a speed number, or a promise that the free option lasts. If a syllabus needs those figures, read that week's docs. Do not copy a number out of a chat and into a rubric.

The session is optional. If the tab dies, write the script locally. I would not move a deadline because a draft bench blinked.

What the checker proves

The script below is a scaffold. I have not run it as a study, and its print line is not a benchmark. It assumes Python 3.9+ and a clean git worktree.

#!/usr/bin/env python3
"""Run student_fix.py twice. Unexecuted lab scaffold."""
import hashlib
import subprocess
import sys
from pathlib import Path

def src_hash() -> str:
    root = Path("src")
    if not root.is_dir():
        sys.exit("ZERO: src/ is missing")
    digest = hashlib.sha256()
    files = sorted(p for p in root.rglob("*") if p.is_file())
    for path in files:
        digest.update(path.relative_to(root).as_posix().encode())
        digest.update(b"\0")
        digest.update(path.read_bytes())
    return digest.hexdigest()

def porcelain() -> str:
    proc = subprocess.run(
        ["git", "status", "--porcelain"],
        check=True,
        text=True,
        capture_output=True,
    )
    return proc.stdout.strip()

def changed_paths(status: str):
    paths = []
    for line in status.splitlines():
        # Two status characters, one space, then the path.
        # Renames and quoted paths are outside this scaffold.
        if len(line) < 4:
            continue
        paths.append(line[3:])
    return paths

def run_fix() -> None:
    proc = subprocess.run([sys.executable, "student_fix.py"], check=False)
    if proc.returncode != 0:
        sys.exit("ZERO: student_fix.py failed")

def main() -> None:
    if porcelain():
        sys.exit("ZERO: dirty tree at start")
    before = src_hash()
    run_fix()
    mid = src_hash()
    if mid == before:
        sys.exit("ZERO: first run did not change src/")
    run_fix()
    after = src_hash()
    if after != mid:
        sys.exit("ZERO: second run changed src/")
    for path in changed_paths(porcelain()):
        if not path.startswith("src/"):
            sys.exit("ZERO: unexpected path " + path)
    print("PASS", mid)

if __name__ == "__main__":
    main()

python3 tools/grade_twice.py

A pass means the start was clean, run one changed src/, and run two did not. Status also did not grow a file outside src/.

Does a pass mean the string is right? No. Hash stability is not an oracle. I still open src/greet.py and read it. Keep that read on its own rubric row, or a stable wrong answer will borrow points from a clean rerun.

Checkpoint 3 — force a zero

Sabotage the script before you trust the grader. If you cannot make it fail, you do not understand the points.

Delete the early return and append the new string on every run. The second hash should move.
Also write NOTES.txt. The path check should fire.
Leave an uncommitted edit, then start the grader. The dirty-start check should fire.

Did the sabotaged script still print PASS? Then the grader is the bug. Fix that before you argue with a model.

Stretch goals

Try these only after the three checkpoints are honest.

Run a third time. Same hash, or it is still a zero.
Refuse the submission if student_fix.py contains http:// or https://. A fixer that calls out to a host is not a lab artifact.
Cap the diff with git diff --numstat. I would use 40 changed lines as a teaching cap, not as a law of nature. Over the cap, stop and ask why the draft rewrote the file.
Preload the correct string, run the grader, and confirm a no-op first run is a zero. Doing nothing is not the same as being safe to rerun.

How I would mark it

Check	Points	Zero when
Clean tree before run 1	15	Porcelain was not empty
Run 1 changes only `src/`	25	No hash change, crash, or extra path
Run 2 keeps the `src/` hash	35	Any byte under `src/` changes
Human read sees `"hello, lab"`	15	Stable, but wrong
Sabotage notes	10	They never produced a real zero

No points for a screenshot. No points for "the model agreed." Partial credit stops at the first zero in that row. I am not averaging vibes.

Oral check

I would ask three questions, out loud, with the file open.

Where does the script stop?
Which paths may change?
If src/greet.py is missing, do you recreate it or exit?

Recreate can be idempotent. Only if run two writes the same bytes and run three writes nothing new. If you choose recreate, add that case to the checker. Do not leave it as a feeling.

Decision list when you are tired

Use this when the hour gets noisy. Do not invent a fourth outcome.

Start dirty? Zero. Do not tidy up and pretend the start was clean.
First hash equals the base hash? Zero. Empty work is not a pass.
Second hash differs? Zero. That is the whole lab.
Unexpected path? Zero, even if src/ was stable.
String wrong, hashes stable? Lose the 15 behavior points. Keep the idempotency points.
Hosted tab died? Irrelevant. Grade the file on disk.

Who should not use this

Do not hash-grade work that should change every run. Timestamps, random IDs, and training logs fail for honest reasons. You would be punishing the assignment, not the student.

Do not use this on a one-shot migration. Those should refuse a second apply with a lock, and side files may still move. Teach the lock. Do not smash it into this rubric.

Do not point the grader at secrets. Hashing bytes is fine. Printing a credential while you debug is not. Would you paste that terminal into a model session? Then do not debug that way.

Skip it when you meant to grade taste. This rubric punishes drift, not dullness. A careful one-line edit can outscore a clever rewrite. That is the deal.

Limits, said plainly

Free model access will not insert a stop condition you forgot. It also must not sit on the grading path. The free server is a draft bench, not a witness, and uptime is not part of the score.

This scaffold does not understand renames, quoted git status paths, or binary fixtures. Say that in the README. Otherwise a filename with a space becomes a fake model failure.

I do not have a measured pass rate for this write-up. Run the scaffold on your own fixture before lab day, and change the path rule if you assigned something wider than src/.

A green second run can still be the wrong function. Schedule a later lab with a human oracle. Do not fold the two scores into one mood.

Close the tab, keep the script

If you want a sandbox for the draft, use that free model on the free server, save student_fix.py, and close the session.

Then answer the only question that counts. You ran it twice. Did src/ stay still?

Leave a Trace a Classmate Can Run: A Bootcamp Lab on Checkpoint Replay

Charlie Xu — Sat, 10 Oct 2026 17:49:28 +0000

If your checkpoint log will not replay, the lab is not done. A polished portfolio screenshot will not talk me out of that.

People are busy arguing whether a vibe-coded site still counts as a portfolio. Fair question. Wrong assignment for this week. I care whether the next student can rerun your setup without borrowing your laptop.

The grade is a replay, not a vibe

I do not grade the homepage. I grade a trace.

You ship a bootstrap, one JSONL log, and a checker I can run with the network off. That is the whole submission. The page, if you built one, is an exhibit. Exhibits do not pass labs.

Why so strict? Bootcamp demos fail in a boring, expensive way. The terminal looked finished at 4pm, the box was reset by morning, and the draft that "worked" was never saved. I do not want that failure mode in the gradebook.

So the lab asks a rude question. Can I clone your repo and prove the setup happened, in order, without you in the room?

What you set up before any model call

Keep the tree dull. Dull is gradable.

lab/
  fixtures/tiny.md
  expected/checkpoints.json
  scripts/bootstrap.sh
  scripts/replay-check.mjs
  runs/.gitkeep
  README.md

fixtures/tiny.md is about twenty lines. A heading, a bug note, one expected phrase. Not a startup. If the fixture is a novel, you are hiding from the assignment.

The contract is a list, not a mood:

{
  "required": [
    "prereq_ok",
    "fixture_loaded",
    "draft_requested",
    "oracle_compared",
    "log_sealed"
  ]
}

Order is the point. Sealing the log before the fixture loads is fiction. A later reader should not have to trust your memory of which step came first.

Bootstrap stays boring on purpose.

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
test -f fixtures/tiny.md
mkdir -p runs
node --version >/dev/null
echo "prereq_ok"

Run it. Run it again. If the second run needs you to delete a folder by hand, you do not have a setup. You have a ritual.

I also want a missing-file drill before anyone celebrates. Rename the fixture, run bootstrap, put the file back. Did the script fail before any draft step? If it still "succeeds," the check is optimistic, and optimism is not a checkpoint.

Two roles, and they do not share a fate

Students may draft the script with a model. They may also run the lab on a remote box, so the grade does not depend on a personal laptop nobody else can reset.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The only product facts I am using are these: MonkeyCode offers free model access, and it offers a free server option. I am not stating a token quota, a hardware profile, a session length, a model name, or a promise that free access remains. Those details move. If a syllabus needs them, check the current docs and write down only what you verified.

Here is why those two options belong in this lab at all.

A free model may draft bootstrap.sh. It may not grade the lab.
A free server may be the student box you reset between attempts.
The grade runs on replay-check.mjs against the committed log. Offline. Local. No model in the loop.

What if free access is down during class? You write the script yourself, set "assisted": false, and keep going. The checker never phones home. Assistance is a convenience. The trace is the assignment.

That split is the lesson. Not a slogan. A boundary. Who typed the script can change. The order of evidence cannot.

The checker you will actually submit

Treat the following as a proposed scaffold, not as a timed benchmark. I am not publishing latency numbers. A green run is not a model evaluation, and it should not be quoted like one.

import { readFileSync } from "node:fs";

const expected = JSON.parse(
  readFileSync(new URL("../expected/checkpoints.json", import.meta.url))
);
const logPath = process.argv[2];
if (!logPath) {
  console.error("usage: node scripts/replay-check.mjs runs/latest.jsonl");
  process.exit(2);
}

const events = readFileSync(logPath, "utf8")
  .trim()
  .split("\n")
  .filter(Boolean)
  .map((line, index) => {
    const row = JSON.parse(line);
    if (!row.name || !row.at) {
      throw new Error(`line ${index + 1} missing name or at`);
    }
    return row;
  });

const names = events.map((event) => event.name);
let cursor = 0;
for (const required of expected.required) {
  const found = names.indexOf(required, cursor);
  if (found === -1) {
    console.error(`missing checkpoint: ${required}`);
    process.exit(1);
  }
  cursor = found + 1;
}

if (events.at(-1).name !== "log_sealed") {
  console.error("log is not sealed");
  process.exit(1);
}
console.log(`replay_ok events=${events.length}`);

A passing log is plain. Timestamps below are examples for the fixture, not a claim that I ran this on a particular day in production.

{"name":"prereq_ok","at":"2026-10-10T15:00:00Z"}
{"name":"fixture_loaded","at":"2026-10-10T15:00:01Z","bytes":412}
{"name":"draft_requested","at":"2026-10-10T15:00:04Z","assisted":true}
{"name":"oracle_compared","at":"2026-10-10T15:00:05Z","pass":true}
{"name":"log_sealed","at":"2026-10-10T15:00:06Z"}

Notice what I refused to store. No model essay. No API key. No "the UI felt modern." The local oracle is a student-written check against tiny.md, recorded as pass. If the model grades its own draft, you have wandered into a different course. Come back.

Appending a line can be ugly. Ugly is readable.

now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
printf '%s\n' "{\"name\":\"fixture_loaded\",\"at\":\"$now\",\"bytes\":412}" >> runs/latest.jsonl

On macOS, date -u is enough for this format. If the image is stripped down, write the timestamp from Node instead of fighting the shell. Same log either way. Do not invent a second schema just to feel modern.

Commands the README must show

I do not accept "run the project" as a step. I accept copy-paste. A classmate should not have to guess your directory layout.

git clone <your-repo> lab-replay && cd lab-replay
bash scripts/bootstrap.sh
node scripts/replay-check.mjs runs/latest.jsonl

Expected last line: replay_ok events=5. If your count is higher because you logged extra notes, say so in the README. Extra events are fine. Missing required ones are not.

What if node is missing on the free server image? Bootstrap should fail at the prerequisite check, not halfway through a draft. Write that failure in one README line. Classmates hit missing runtimes. Pretending they will not is how labs melt.

Stops I walk in class

Four stops. Each one has a question you should answer without slides.

Clean tree. Does bootstrap exit 0 on a fresh clone, then again with no cleanup? If not, why are you already drafting prompts?
Fixture. Does the log store a byte count? If the file is missing, does the script die before draft_requested?
Help flag. Did a model touch the script? Then assisted is true. Did you type it? Then false. Either can pass. A missing flag cannot.
Airplane mode. I copy the repo and run the checker offline. If it tries to call a model, grading stops. Not "partial." Stops.

At stop 3 I ask one spoken sentence. What did you keep from the draft, and what did you throw away? The script cannot hear you. I can. A green log plus a blank stare becomes a review note, not an automatic pass, and not a silent zero either.

A failure I want practiced, not theorized

Rename fixtures/tiny.md to fixtures/tiny.md.bak. Run bootstrap. You should stop. Write nothing that claims fixture_loaded. Restore the file. Append the real events only after the file is back.

Then aim the checker at a bad log on purpose:

node scripts/replay-check.mjs runs/failure.jsonl
echo "exit=$?"

I want missing checkpoint: fixture_loaded and a non-zero exit. If your checker prints a stack trace and still exits 0, the catch path is theater. Theater does not replay.

Questions I will ask while you do this: where did the script notice the missing file? Did the model draft hide that check inside a comment? Are you logging the absence, or only the happy path?

Happy-path-only logs are how screenshots happen. I am allergic to them this week.

Stretch, after the core is green

Do not start here. I mean it.

Commit runs/failure.jsonl where fixture_loaded never appears. The checker must exit 1 and name the gap.
Hand the repo to a classmate. README only. Ten minutes. No chat.
Produce two logs from the same contract: one assisted, one hand-written. The checker should not care who typed the script.
If you also shipped a portfolio page, link it under "exhibit." It adds zero points unless replay still passes.

That last stretch is my small answer to the portfolio argument. A generated site can be worth showing. It is not evidence that your setup exists.

Rubric I can defend to the class

Slow model calls are not a character flaw. Fake flags are.

Row	Points	What pass means
Idempotent bootstrap	25	Second run exits 0 with no hand cleanup
Ordered checkpoints	25	Checker prints `replay_ok`
Honest failure path	20	Missing fixture exits 1 and names the event
Honest `assisted` flag	15	Flag matches the work you actually did
README a classmate can follow	15	They finish without pinging you

The order row is all or nothing. You do not get partial credit for three of five events. Why? A skipped fixture with a sealed log is exactly the demo I am refusing to trust.

The README row allows partial credit. Seven minutes and one stuck command can be a 10, not a 0, if the classmate still finishes from your notes.

A late free-server reset does not deduct points if the committed log still replays on my machine. You lost a box. You did not lose the trace. That is the fair part. Availability wobble is not the same thing as a missing fixture.

Who should skip this lab

This will not make you hireable by itself. It will not audit CSS, and it will not tell you which model is best. I have not measured that, so I will not fake a ranking.

Skip it when:

The week is visual design, accessibility, or a public portfolio critique
You need a named model, a quota, a GPU shape, or a permanence clause in the syllabus
The log would contain secrets, tokens, or another student's private notes
You wanted a leaderboard — this checker will not rank models, and I will not add that

A green replay proves one claim: this repo can retell its setup in order. It does not prove you understand the code a model drafted. That is a real hole. The spoken sentence at stop 3 is how I patch it, and it is a human patch. Do not automate it into a fake sentiment score.

I also would not drop this rubric onto a hiring review. Hiring is a different promise. This promise is narrower: leave a trace the next person can run.

One way to try the boundary

If you run a lab this week, put drafts and remote boxes on the student side. Keep the grade on the log. MonkeyCode's free model access and free server option are a practical way to try that split, provided you treat them as availability, not as the rubric.

Same Fixture, Different Free Model: A Bootcamp Lab on Swap-Safe Grades

Charlie Xu — Fri, 09 Oct 2026 14:19:14 +0000

If I cannot swap the free model and still give you the same score, the score was never about your code. It was about a lucky sentence. This lab exists to kill that kind of grade.

You already know the failure mode. Two laptops. Same fixture. Same prompt. One free model emits a tidy JSON object, and the other buries the same fields inside a cheerful paragraph. One student "passes." The other eats a zero for vibes. Fair? No.

So here is the rule I would print on the whiteboard before anyone opens a chat box. Grade the pin and the structure, not the prose. If the run cannot name the model id and the server image, it does not count. If a second model on that same image flips your pass/fail bit, you do not get credit for the prettier demo.

I have not executed this harness against a live cohort in this draft. Treat the script as a proposed lab starter, not as a benchmark, a leaderboard, or a promise about any vendor plan.

The grade I actually want

You are not proving that a model is smart. You are proving three smaller, meaner things.

The fixture is frozen.
The checker ignores wording.
The run can be named, hashed, and repeated after a model swap.

That is a one-week bootcamp lab. It is not a production eval, and it is not a beauty contest between free models. Why start there? Because a cohort that cannot separate "the model spoke" from "the harness checked" will grade luck for the rest of the term.

What I will not argue about

I will not argue about tone. I will not argue about which reply "felt more senior." I will argue about missing keys, empty strings, a floating image tag, and a fixture whose hash moved after the run.

If that sounds harsh, good. Harsh and checkable beats kind and fuzzy.

Setup, before the model speaks

I want one shared box, not twelve snowflake laptops. A free server is useful here only when every student can land on the same image digest. A free model list is useful only when the manifest can name the id you actually called.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator behind this draft describes MonkeyCode as an open-source project and points at free model access plus a free server option. That is the whole product claim I am willing to put in a rubric. I am not pasting model names, token quotas, hardware sizes, or "how long free lasts." Those are live-plan facts. If the docs on lab morning disagree with a screenshot in Slack, the docs win, and the screenshot loses.

Make a fresh directory. Do not reuse last week's agent workspace.

mkdir -p swaplab/fixtures swaplab/runs swaplab/checker
cd swaplab
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
printf '%s\n' 'pytest==8.3.4' > requirements.txt
python -m pip install -r requirements.txt
python -c 'import sys; print(sys.version)'

Pin the interpreter in the manifest too. I do not care which patch you have. I care that the grade names it.

export LAB_MODEL_ID="${LAB_MODEL_ID:?set the model id from today's free list}"
export LAB_SERVER_IMAGE="${LAB_SERVER_IMAGE:?set an image digest, not a floating tag}"
export LAB_FIXTURE="fixtures/ticket.json"

See the :?? Forget the pin and the shell dies before the model spends a single token. That is checkpoint zero, and it is rude on purpose.

Secrets stay out of the repo. If your client needs a key, it comes from the server environment, not from a committed .env. I am not teaching a leak scanner this week. I am refusing to grade a diff that already contains one.

The fixture stays boring

One ticket. No customer data. No real company. The model must return a JSON object with owner, severity, and next_step. Everything else is trash, not signal.

{
  "id": "T-14",
  "title": "Checkout button ignores keyboard focus",
  "notes": "Reported on the staging box. No customer data in this fixture."
}

Save that as fixtures/ticket.json. Do not "improve" it mid-lab. A moving fixture is a moving grade, and I will not chase it.

Hash it before the first run.

sha256sum fixtures/ticket.json | tee runs/fixture.sha256

Why a digest, not a tag

latest is a mood, not a pin. If I rebuild the free server overnight and your grade still says latest, I cannot tell whether you passed on Tuesday's image or Thursday's. A digest is ugly. Ugly is the point. Can you replay the run without asking the student what they "think" they used? If not, the manifest is decoration.

The checker

This is the artifact. It does not call a vendor SDK, and it does not know MonkeyCode's API. You bring a tiny adapter that returns a string. The checker only accepts a JSON object with the three required keys. Extra keys are fine. Missing keys are not. Prose is not.

Save it as checker/grade.py.

#!/usr/bin/env python3
"""Proposed lab starter. Not executed against a live model in this draft."""
import hashlib, json, os, sys
from pathlib import Path

REQUIRED = ("owner", "severity", "next_step")
SEVERITIES = {"low", "medium", "high"}

def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def extract_object(raw: str) -> dict:
    raw = raw.strip()
    start, end = raw.find("{"), raw.rfind("}")
    if start < 0 or end < start:
        raise ValueError("no JSON object")
    obj = json.loads(raw[start:end + 1])
    if not isinstance(obj, dict):
        raise ValueError("JSON was not an object")
    missing = [k for k in REQUIRED if k not in obj]
    if missing:
        raise ValueError(f"missing {missing}")
    if str(obj["severity"]).lower() not in SEVERITIES:
        raise ValueError("severity not in allowed set")
    if not str(obj["owner"]).strip() or not str(obj["next_step"]).strip():
        raise ValueError("empty owner or next_step")
    return obj

def main() -> int:
    model = os.environ["LAB_MODEL_ID"]
    image = os.environ["LAB_SERVER_IMAGE"]
    fixture = Path(os.environ["LAB_FIXTURE"])
    obj = extract_object(sys.stdin.read())
    manifest = {
        "model_id": model,
        "server_image": image,
        "fixture_sha256": sha256(fixture),
        "python": sys.version.split()[0],
        "fields": sorted(obj),
    }
    out = Path("runs") / f"{model.replace('/', '_')}.json"
    out.parent.mkdir(exist_ok=True)
    out.write_text(json.dumps(manifest, indent=2) + "\n")
    print(json.dumps({"ok": True, "manifest": str(out)}))
    return 0

if __name__ == "__main__":
    try:
        raise SystemExit(main())
    except (KeyError, ValueError, json.JSONDecodeError) as exc:
        print(json.dumps({"ok": False, "error": str(exc)}))
        raise SystemExit(1)

Run the local cases first. No network. No model. If these fail, you are not ready to touch a free quota.

# pass: bare object
printf '%s\n' '{"owner":"sam","severity":"high","next_step":"restore focus ring"}' \
  | python checker/grade.py

# pass: fenced blob, wording ignored
printf '%s\n' 'Sure, here you go:' \
  '{"owner":"sam","severity":"high","next_step":"restore focus ring"}' \
  'Hope that helps!' | python checker/grade.py

# fail: prose only
printf '%s\n' 'The button is bad, please fix focus.' | python checker/grade.py

# fail: empty next_step
printf '%s\n' '{"owner":"sam","severity":"high","next_step":"  "}' \
  | python checker/grade.py

The prose case must exit non-zero. If you "helpfully" pass any non-empty string, I grade the checker, not the model. You built a compliment machine. I do not want one.

Checkpoints

Do these in order. A shiny demo does not unlock the next box.

Pin or stop. LAB_MODEL_ID and LAB_SERVER_IMAGE are set. The image is a digest, not latest. Show env | grep '^LAB_'.
Fixture hash. The SHA-256 in runs/fixture.sha256 matches fixture_sha256 after a stdin run.
Structure only. Fenced JSON passes. Prose fails. Empty next_step fails. severity: "urgent" fails, because it is not in the allowed set.
Swap drill. Same fixture, same image digest, two model ids from today's free list. Both manifests exist. Both pass, or both fail, for the same assertion. A pass/fail split is a lab fail.
Clean tree. git grep -nE 'api_key|BEGIN PRIVATE KEY|sk-' is empty. No committed token budget either. A hardcoded quota is a smell, not a feature.

Checkpoint 4 is the whole week. A warmer voice does not earn a higher score. A pass on model A and a fail on model B means your harness was coupled to one voice. Fix the contract, or drop the assertion that only one model can satisfy. Do not "pick the winner" and hide the other manifest. I will ask for both files.

A conversation I do not want

"But model B is worse, so my code is fine." Maybe. Maybe not. This lab cannot see that. It can see that you graded a single lucky reply and called it engineering.

Want to discuss model quality? Fine, after the swap drill, in a separate note, with both manifests attached. That note is not the grade.

Stretch goals

Pick one. I would rather see one finished drill than three half demos.

Add blocked_by as an optional field. Do not loosen severity. Update one fixture and the checker together.
Record elapsed milliseconds in the manifest, and do not grade on it. Free servers hiccup. A slow cold start is not a character flaw.
Add a 15-line pytest that feeds three stdin strings into extract_object. Those tests grade the checker. They do not call a model.

If a test needs the network, it is an integration drill. It does not belong in the default score. Label it, skip it in CI, and keep the unit path offline.

Rubric

Checkpoint	Full marks	Zero for that row
Pins	Model id plus image digest in the manifest	Floating tag, missing env, or a made-up quota
Fixture	SHA-256 matches the file you hashed	Fixture edited after the run
Checker	Fenced JSON passes; prose and empty fields fail	Any non-empty string passes
Swap	Two ids, same pass/fail, both manifests saved	Only the pretty model was shown
Hygiene	No secrets and no hardcoded budget in the diff	A key, or a token number copied from a blog

Partial credit exists, but not for charm. A broken swap with a clean checker is a 60. A perfect demo on one model, with the second manifest "lost," is a zero on that row. I am not negotiating with a screenshot of a chat window.

Limits, and who should skip this

This will not tell you which free model is best. I am not publishing latency numbers, token prices, or hardware, because I will not invent figures a live plan can contradict next week.

Do not use this lab for semantic grading. "Is next_step actually a good fix?" is a human question. The checker only asks whether the field exists, is non-empty, and sits in an allowed set. That is a floor. It is not a code review.

Do not use it as a security audit. The fixture has no customer data on purpose. If your notes include a real inbox, a real company, or a real secret, stop and strip them before any model call.

Do not park production traffic on a teaching server. A free image can sleep, reset, or change terms. That is a limitation, not a footnote you skip in the README.

Skip the lab entirely if the cohort cannot yet read a JSON error. Teach json.loads first. A swap drill on top of panic is just noise with extra steps.

Also skip it if you need a stable score across months. Free model lists move. When an id disappears, the lab pauses. You do not substitute a guess, and you do not edit yesterday's manifest to match today's menu.

What I would actually run

I would put the venv and the checker on the free server so the image digest is real, not "works on my laptop." I would copy two ids from the current free model list into the shell for checkpoint 4, then throw the list away at the end of the day. No id gets baked into the repo. No quota gets baked into the script.

Before you photocopy this rubric, open the current free-model and free-server notes, cross out anything this draft refused to invent, and run checkpoint 4 with whatever two ids are actually listed that morning. The checkpoint your cohort fails is the part worth keeping.