DEV Community: Charlie Xu

Leave a Trace a Classmate Can Run: A Bootcamp Lab on Checkpoint Replay

Charlie Xu — Sat, 10 Oct 2026 17:49:28 +0000

If your checkpoint log will not replay, the lab is not done. A polished portfolio screenshot will not talk me out of that.

People are busy arguing whether a vibe-coded site still counts as a portfolio. Fair question. Wrong assignment for this week. I care whether the next student can rerun your setup without borrowing your laptop.

The grade is a replay, not a vibe

I do not grade the homepage. I grade a trace.

You ship a bootstrap, one JSONL log, and a checker I can run with the network off. That is the whole submission. The page, if you built one, is an exhibit. Exhibits do not pass labs.

Why so strict? Bootcamp demos fail in a boring, expensive way. The terminal looked finished at 4pm, the box was reset by morning, and the draft that "worked" was never saved. I do not want that failure mode in the gradebook.

So the lab asks a rude question. Can I clone your repo and prove the setup happened, in order, without you in the room?

What you set up before any model call

Keep the tree dull. Dull is gradable.

lab/
  fixtures/tiny.md
  expected/checkpoints.json
  scripts/bootstrap.sh
  scripts/replay-check.mjs
  runs/.gitkeep
  README.md

fixtures/tiny.md is about twenty lines. A heading, a bug note, one expected phrase. Not a startup. If the fixture is a novel, you are hiding from the assignment.

The contract is a list, not a mood:

{
  "required": [
    "prereq_ok",
    "fixture_loaded",
    "draft_requested",
    "oracle_compared",
    "log_sealed"
  ]
}

Order is the point. Sealing the log before the fixture loads is fiction. A later reader should not have to trust your memory of which step came first.

Bootstrap stays boring on purpose.

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
test -f fixtures/tiny.md
mkdir -p runs
node --version >/dev/null
echo "prereq_ok"

Run it. Run it again. If the second run needs you to delete a folder by hand, you do not have a setup. You have a ritual.

I also want a missing-file drill before anyone celebrates. Rename the fixture, run bootstrap, put the file back. Did the script fail before any draft step? If it still "succeeds," the check is optimistic, and optimism is not a checkpoint.

Two roles, and they do not share a fate

Students may draft the script with a model. They may also run the lab on a remote box, so the grade does not depend on a personal laptop nobody else can reset.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The only product facts I am using are these: MonkeyCode offers free model access, and it offers a free server option. I am not stating a token quota, a hardware profile, a session length, a model name, or a promise that free access remains. Those details move. If a syllabus needs them, check the current docs and write down only what you verified.

Here is why those two options belong in this lab at all.

A free model may draft bootstrap.sh. It may not grade the lab.
A free server may be the student box you reset between attempts.
The grade runs on replay-check.mjs against the committed log. Offline. Local. No model in the loop.

What if free access is down during class? You write the script yourself, set "assisted": false, and keep going. The checker never phones home. Assistance is a convenience. The trace is the assignment.

That split is the lesson. Not a slogan. A boundary. Who typed the script can change. The order of evidence cannot.

The checker you will actually submit

Treat the following as a proposed scaffold, not as a timed benchmark. I am not publishing latency numbers. A green run is not a model evaluation, and it should not be quoted like one.

import { readFileSync } from "node:fs";

const expected = JSON.parse(
  readFileSync(new URL("../expected/checkpoints.json", import.meta.url))
);
const logPath = process.argv[2];
if (!logPath) {
  console.error("usage: node scripts/replay-check.mjs runs/latest.jsonl");
  process.exit(2);
}

const events = readFileSync(logPath, "utf8")
  .trim()
  .split("\n")
  .filter(Boolean)
  .map((line, index) => {
    const row = JSON.parse(line);
    if (!row.name || !row.at) {
      throw new Error(`line ${index + 1} missing name or at`);
    }
    return row;
  });

const names = events.map((event) => event.name);
let cursor = 0;
for (const required of expected.required) {
  const found = names.indexOf(required, cursor);
  if (found === -1) {
    console.error(`missing checkpoint: ${required}`);
    process.exit(1);
  }
  cursor = found + 1;
}

if (events.at(-1).name !== "log_sealed") {
  console.error("log is not sealed");
  process.exit(1);
}
console.log(`replay_ok events=${events.length}`);

A passing log is plain. Timestamps below are examples for the fixture, not a claim that I ran this on a particular day in production.

{"name":"prereq_ok","at":"2026-10-10T15:00:00Z"}
{"name":"fixture_loaded","at":"2026-10-10T15:00:01Z","bytes":412}
{"name":"draft_requested","at":"2026-10-10T15:00:04Z","assisted":true}
{"name":"oracle_compared","at":"2026-10-10T15:00:05Z","pass":true}
{"name":"log_sealed","at":"2026-10-10T15:00:06Z"}

Notice what I refused to store. No model essay. No API key. No "the UI felt modern." The local oracle is a student-written check against tiny.md, recorded as pass. If the model grades its own draft, you have wandered into a different course. Come back.

Appending a line can be ugly. Ugly is readable.

now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
printf '%s\n' "{\"name\":\"fixture_loaded\",\"at\":\"$now\",\"bytes\":412}" >> runs/latest.jsonl

On macOS, date -u is enough for this format. If the image is stripped down, write the timestamp from Node instead of fighting the shell. Same log either way. Do not invent a second schema just to feel modern.

Commands the README must show

I do not accept "run the project" as a step. I accept copy-paste. A classmate should not have to guess your directory layout.

git clone <your-repo> lab-replay && cd lab-replay
bash scripts/bootstrap.sh
node scripts/replay-check.mjs runs/latest.jsonl

Expected last line: replay_ok events=5. If your count is higher because you logged extra notes, say so in the README. Extra events are fine. Missing required ones are not.

What if node is missing on the free server image? Bootstrap should fail at the prerequisite check, not halfway through a draft. Write that failure in one README line. Classmates hit missing runtimes. Pretending they will not is how labs melt.

Stops I walk in class

Four stops. Each one has a question you should answer without slides.

Clean tree. Does bootstrap exit 0 on a fresh clone, then again with no cleanup? If not, why are you already drafting prompts?
Fixture. Does the log store a byte count? If the file is missing, does the script die before draft_requested?
Help flag. Did a model touch the script? Then assisted is true. Did you type it? Then false. Either can pass. A missing flag cannot.
Airplane mode. I copy the repo and run the checker offline. If it tries to call a model, grading stops. Not "partial." Stops.

At stop 3 I ask one spoken sentence. What did you keep from the draft, and what did you throw away? The script cannot hear you. I can. A green log plus a blank stare becomes a review note, not an automatic pass, and not a silent zero either.

A failure I want practiced, not theorized

Rename fixtures/tiny.md to fixtures/tiny.md.bak. Run bootstrap. You should stop. Write nothing that claims fixture_loaded. Restore the file. Append the real events only after the file is back.

Then aim the checker at a bad log on purpose:

node scripts/replay-check.mjs runs/failure.jsonl
echo "exit=$?"

I want missing checkpoint: fixture_loaded and a non-zero exit. If your checker prints a stack trace and still exits 0, the catch path is theater. Theater does not replay.

Questions I will ask while you do this: where did the script notice the missing file? Did the model draft hide that check inside a comment? Are you logging the absence, or only the happy path?

Happy-path-only logs are how screenshots happen. I am allergic to them this week.

Stretch, after the core is green

Do not start here. I mean it.

Commit runs/failure.jsonl where fixture_loaded never appears. The checker must exit 1 and name the gap.
Hand the repo to a classmate. README only. Ten minutes. No chat.
Produce two logs from the same contract: one assisted, one hand-written. The checker should not care who typed the script.
If you also shipped a portfolio page, link it under "exhibit." It adds zero points unless replay still passes.

That last stretch is my small answer to the portfolio argument. A generated site can be worth showing. It is not evidence that your setup exists.

Rubric I can defend to the class

Slow model calls are not a character flaw. Fake flags are.

Row	Points	What pass means
Idempotent bootstrap	25	Second run exits 0 with no hand cleanup
Ordered checkpoints	25	Checker prints `replay_ok`
Honest failure path	20	Missing fixture exits 1 and names the event
Honest `assisted` flag	15	Flag matches the work you actually did
README a classmate can follow	15	They finish without pinging you

The order row is all or nothing. You do not get partial credit for three of five events. Why? A skipped fixture with a sealed log is exactly the demo I am refusing to trust.

The README row allows partial credit. Seven minutes and one stuck command can be a 10, not a 0, if the classmate still finishes from your notes.

A late free-server reset does not deduct points if the committed log still replays on my machine. You lost a box. You did not lose the trace. That is the fair part. Availability wobble is not the same thing as a missing fixture.

Who should skip this lab

This will not make you hireable by itself. It will not audit CSS, and it will not tell you which model is best. I have not measured that, so I will not fake a ranking.

Skip it when:

The week is visual design, accessibility, or a public portfolio critique
You need a named model, a quota, a GPU shape, or a permanence clause in the syllabus
The log would contain secrets, tokens, or another student's private notes
You wanted a leaderboard — this checker will not rank models, and I will not add that

A green replay proves one claim: this repo can retell its setup in order. It does not prove you understand the code a model drafted. That is a real hole. The spoken sentence at stop 3 is how I patch it, and it is a human patch. Do not automate it into a fake sentiment score.

I also would not drop this rubric onto a hiring review. Hiring is a different promise. This promise is narrower: leave a trace the next person can run.

One way to try the boundary

If you run a lab this week, put drafts and remote boxes on the student side. Keep the grade on the log. MonkeyCode's free model access and free server option are a practical way to try that split, provided you treat them as availability, not as the rubric.

Same Fixture, Different Free Model: A Bootcamp Lab on Swap-Safe Grades

Charlie Xu — Fri, 09 Oct 2026 14:19:14 +0000

If I cannot swap the free model and still give you the same score, the score was never about your code. It was about a lucky sentence. This lab exists to kill that kind of grade.

You already know the failure mode. Two laptops. Same fixture. Same prompt. One free model emits a tidy JSON object, and the other buries the same fields inside a cheerful paragraph. One student "passes." The other eats a zero for vibes. Fair? No.

So here is the rule I would print on the whiteboard before anyone opens a chat box. Grade the pin and the structure, not the prose. If the run cannot name the model id and the server image, it does not count. If a second model on that same image flips your pass/fail bit, you do not get credit for the prettier demo.

I have not executed this harness against a live cohort in this draft. Treat the script as a proposed lab starter, not as a benchmark, a leaderboard, or a promise about any vendor plan.

The grade I actually want

You are not proving that a model is smart. You are proving three smaller, meaner things.

The fixture is frozen.
The checker ignores wording.
The run can be named, hashed, and repeated after a model swap.

That is a one-week bootcamp lab. It is not a production eval, and it is not a beauty contest between free models. Why start there? Because a cohort that cannot separate "the model spoke" from "the harness checked" will grade luck for the rest of the term.

What I will not argue about

I will not argue about tone. I will not argue about which reply "felt more senior." I will argue about missing keys, empty strings, a floating image tag, and a fixture whose hash moved after the run.

If that sounds harsh, good. Harsh and checkable beats kind and fuzzy.

Setup, before the model speaks

I want one shared box, not twelve snowflake laptops. A free server is useful here only when every student can land on the same image digest. A free model list is useful only when the manifest can name the id you actually called.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator behind this draft describes MonkeyCode as an open-source project and points at free model access plus a free server option. That is the whole product claim I am willing to put in a rubric. I am not pasting model names, token quotas, hardware sizes, or "how long free lasts." Those are live-plan facts. If the docs on lab morning disagree with a screenshot in Slack, the docs win, and the screenshot loses.

Make a fresh directory. Do not reuse last week's agent workspace.

mkdir -p swaplab/fixtures swaplab/runs swaplab/checker
cd swaplab
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
printf '%s\n' 'pytest==8.3.4' > requirements.txt
python -m pip install -r requirements.txt
python -c 'import sys; print(sys.version)'

Pin the interpreter in the manifest too. I do not care which patch you have. I care that the grade names it.

export LAB_MODEL_ID="${LAB_MODEL_ID:?set the model id from today's free list}"
export LAB_SERVER_IMAGE="${LAB_SERVER_IMAGE:?set an image digest, not a floating tag}"
export LAB_FIXTURE="fixtures/ticket.json"

See the :?? Forget the pin and the shell dies before the model spends a single token. That is checkpoint zero, and it is rude on purpose.

Secrets stay out of the repo. If your client needs a key, it comes from the server environment, not from a committed .env. I am not teaching a leak scanner this week. I am refusing to grade a diff that already contains one.

The fixture stays boring

One ticket. No customer data. No real company. The model must return a JSON object with owner, severity, and next_step. Everything else is trash, not signal.

{
  "id": "T-14",
  "title": "Checkout button ignores keyboard focus",
  "notes": "Reported on the staging box. No customer data in this fixture."
}

Save that as fixtures/ticket.json. Do not "improve" it mid-lab. A moving fixture is a moving grade, and I will not chase it.

Hash it before the first run.

sha256sum fixtures/ticket.json | tee runs/fixture.sha256

Why a digest, not a tag

latest is a mood, not a pin. If I rebuild the free server overnight and your grade still says latest, I cannot tell whether you passed on Tuesday's image or Thursday's. A digest is ugly. Ugly is the point. Can you replay the run without asking the student what they "think" they used? If not, the manifest is decoration.

The checker

This is the artifact. It does not call a vendor SDK, and it does not know MonkeyCode's API. You bring a tiny adapter that returns a string. The checker only accepts a JSON object with the three required keys. Extra keys are fine. Missing keys are not. Prose is not.

Save it as checker/grade.py.

#!/usr/bin/env python3
"""Proposed lab starter. Not executed against a live model in this draft."""
import hashlib, json, os, sys
from pathlib import Path

REQUIRED = ("owner", "severity", "next_step")
SEVERITIES = {"low", "medium", "high"}

def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def extract_object(raw: str) -> dict:
    raw = raw.strip()
    start, end = raw.find("{"), raw.rfind("}")
    if start < 0 or end < start:
        raise ValueError("no JSON object")
    obj = json.loads(raw[start:end + 1])
    if not isinstance(obj, dict):
        raise ValueError("JSON was not an object")
    missing = [k for k in REQUIRED if k not in obj]
    if missing:
        raise ValueError(f"missing {missing}")
    if str(obj["severity"]).lower() not in SEVERITIES:
        raise ValueError("severity not in allowed set")
    if not str(obj["owner"]).strip() or not str(obj["next_step"]).strip():
        raise ValueError("empty owner or next_step")
    return obj

def main() -> int:
    model = os.environ["LAB_MODEL_ID"]
    image = os.environ["LAB_SERVER_IMAGE"]
    fixture = Path(os.environ["LAB_FIXTURE"])
    obj = extract_object(sys.stdin.read())
    manifest = {
        "model_id": model,
        "server_image": image,
        "fixture_sha256": sha256(fixture),
        "python": sys.version.split()[0],
        "fields": sorted(obj),
    }
    out = Path("runs") / f"{model.replace('/', '_')}.json"
    out.parent.mkdir(exist_ok=True)
    out.write_text(json.dumps(manifest, indent=2) + "\n")
    print(json.dumps({"ok": True, "manifest": str(out)}))
    return 0

if __name__ == "__main__":
    try:
        raise SystemExit(main())
    except (KeyError, ValueError, json.JSONDecodeError) as exc:
        print(json.dumps({"ok": False, "error": str(exc)}))
        raise SystemExit(1)

Run the local cases first. No network. No model. If these fail, you are not ready to touch a free quota.

# pass: bare object
printf '%s\n' '{"owner":"sam","severity":"high","next_step":"restore focus ring"}' \
  | python checker/grade.py

# pass: fenced blob, wording ignored
printf '%s\n' 'Sure, here you go:' \
  '{"owner":"sam","severity":"high","next_step":"restore focus ring"}' \
  'Hope that helps!' | python checker/grade.py

# fail: prose only
printf '%s\n' 'The button is bad, please fix focus.' | python checker/grade.py

# fail: empty next_step
printf '%s\n' '{"owner":"sam","severity":"high","next_step":"  "}' \
  | python checker/grade.py

The prose case must exit non-zero. If you "helpfully" pass any non-empty string, I grade the checker, not the model. You built a compliment machine. I do not want one.

Checkpoints

Do these in order. A shiny demo does not unlock the next box.

Pin or stop. LAB_MODEL_ID and LAB_SERVER_IMAGE are set. The image is a digest, not latest. Show env | grep '^LAB_'.
Fixture hash. The SHA-256 in runs/fixture.sha256 matches fixture_sha256 after a stdin run.
Structure only. Fenced JSON passes. Prose fails. Empty next_step fails. severity: "urgent" fails, because it is not in the allowed set.
Swap drill. Same fixture, same image digest, two model ids from today's free list. Both manifests exist. Both pass, or both fail, for the same assertion. A pass/fail split is a lab fail.
Clean tree. git grep -nE 'api_key|BEGIN PRIVATE KEY|sk-' is empty. No committed token budget either. A hardcoded quota is a smell, not a feature.

Checkpoint 4 is the whole week. A warmer voice does not earn a higher score. A pass on model A and a fail on model B means your harness was coupled to one voice. Fix the contract, or drop the assertion that only one model can satisfy. Do not "pick the winner" and hide the other manifest. I will ask for both files.

A conversation I do not want

"But model B is worse, so my code is fine." Maybe. Maybe not. This lab cannot see that. It can see that you graded a single lucky reply and called it engineering.

Want to discuss model quality? Fine, after the swap drill, in a separate note, with both manifests attached. That note is not the grade.

Stretch goals

Pick one. I would rather see one finished drill than three half demos.

Add blocked_by as an optional field. Do not loosen severity. Update one fixture and the checker together.
Record elapsed milliseconds in the manifest, and do not grade on it. Free servers hiccup. A slow cold start is not a character flaw.
Add a 15-line pytest that feeds three stdin strings into extract_object. Those tests grade the checker. They do not call a model.

If a test needs the network, it is an integration drill. It does not belong in the default score. Label it, skip it in CI, and keep the unit path offline.

Rubric

Checkpoint	Full marks	Zero for that row
Pins	Model id plus image digest in the manifest	Floating tag, missing env, or a made-up quota
Fixture	SHA-256 matches the file you hashed	Fixture edited after the run
Checker	Fenced JSON passes; prose and empty fields fail	Any non-empty string passes
Swap	Two ids, same pass/fail, both manifests saved	Only the pretty model was shown
Hygiene	No secrets and no hardcoded budget in the diff	A key, or a token number copied from a blog

Partial credit exists, but not for charm. A broken swap with a clean checker is a 60. A perfect demo on one model, with the second manifest "lost," is a zero on that row. I am not negotiating with a screenshot of a chat window.

Limits, and who should skip this

This will not tell you which free model is best. I am not publishing latency numbers, token prices, or hardware, because I will not invent figures a live plan can contradict next week.

Do not use this lab for semantic grading. "Is next_step actually a good fix?" is a human question. The checker only asks whether the field exists, is non-empty, and sits in an allowed set. That is a floor. It is not a code review.

Do not use it as a security audit. The fixture has no customer data on purpose. If your notes include a real inbox, a real company, or a real secret, stop and strip them before any model call.

Do not park production traffic on a teaching server. A free image can sleep, reset, or change terms. That is a limitation, not a footnote you skip in the README.

Skip the lab entirely if the cohort cannot yet read a JSON error. Teach json.loads first. A swap drill on top of panic is just noise with extra steps.

Also skip it if you need a stable score across months. Free model lists move. When an id disappears, the lab pauses. You do not substitute a guess, and you do not edit yesterday's manifest to match today's menu.

What I would actually run

I would put the venv and the checker on the free server so the image digest is real, not "works on my laptop." I would copy two ids from the current free model list into the shell for checkpoint 4, then throw the list away at the end of the day. No id gets baked into the repo. No quota gets baked into the script.

Before you photocopy this rubric, open the current free-model and free-server notes, cross out anything this draft refused to invent, and run checkpoint 4 with whatever two ids are actually listed that morning. The checkpoint your cohort fails is the part worth keeping.

Grade the Fence, Not the Vibe: A Bootcamp Lab on Frozen Scope Contracts

Charlie Xu — Thu, 08 Oct 2026 10:12:06 +0000

An agent that widens the ticket and still goes green is not done. I fail that submission.

Bootcamp patches keep arriving with the same shape. A student asks a model to fix one flaky retry, the diff also touches a lockfile and a workflow, and the write-up celebrates a passing test. Did the tests prove the issue, or did they prove the agent got bored and redecorated the repo?

This lab freezes a scope contract before the model runs, then grades the branch against that contract. Chat tone is not a score. October makes the habit worse, because a lot of first pull requests get opened in a hurry, but the fence is the same in March.

What you are actually grading

You are grading drift, not taste. Drift is any change the frozen brief did not allow. A new dependency counts. A drive-by rename counts.

Editing the contract after the run counts, and I treat that like editing the answer key during the exam. Can a small extra fix still be a good idea? Yes. It is still a miss on this lab.

Open a second issue and keep the first patch boring. A clean machine does not answer this question. You may already require the command to run off a laptop. This lab asks something sharper: did the patch stay inside the brief you froze?

Lab setup

Everyone gets the same tiny fixture. One bug, one allowed file, one command that must go green without the agent rewriting the assertion.

Fixture

The helper retries three times, then drops the error on the floor. Students may not "fix" this by weakening the test.

mkdir scope-lab && cd scope-lab
git init -b main
npm init -y
npm pkg set type=module
mkdir -p src test

// src/retry.js
export async function retry(fn) {
  let last;
  for (let i = 0; i < 3; i++) {
    try {
      return await fn();
    } catch (err) {
      last = err;
    }
  }
  return null;
}

// test/retry.test.js
import test from "node:test";
import assert from "node:assert/strict";
import { retry } from "../src/retry.js";

test("third failure surfaces the original error", async () => {
  await assert.rejects(
    () => retry(async () => {
      throw new Error("boom");
    }),
    /boom/
  );
});

Run node --test once before any agent starts. You want a red bar that the class agrees on. If the fixture is already green, you froze the wrong bug.

Fence

scope.json is the assignment. It is not a hint the model may renegotiate.

{
  "issue": "LAB-7",
  "allowed_paths": ["src/retry.js"],
  "forbidden_globs": [
    "package.json",
    "package-lock.json",
    ".github/**",
    "scope.json",
    "test/retry.test.js"
  ],
  "commands": ["node --test"],
  "notes": "Fix error propagation only. Do not add dependencies."
}

Hash it, then commit the freeze. Why hash a file a student could just rewrite? Because the rewrite is the cheat, and a checksum makes the cheat loud.

sha256sum scope.json | tee scope.sha256
# macOS: shasum -a 256 scope.json | tee scope.sha256
git add src test scope.json scope.sha256 package.json
git commit -m "freeze lab fixture and scope contract"

If a student decides the fix truly needs src/errors.js, they stop. They amend scope.json in a new commit, re-hash, and only then start the agent. After the branch exists, the fence does not move.

Where the agent runs

Students need a model and a machine that is not their laptop's junk drawer of global packages. For that shared run, the lab sheet points at MonkeyCode's free model access and free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those are the only product facts I will teach. I am not naming a model, a token quota, a hardware shape, or a promise that the free option is still there next cohort. Open the current docs the morning you teach, and pin that URL in the lab sheet.

If the free path is down, run any agent you already have. The grader does not care which logo produced the branch. Give the agent the issue text and the raw scope.json. Say the contract is binding.

Pull the branch back and grade the tree, not the transcript. Run the checker from the repo root so paths stay relative. Cut the agent branch from the freeze commit, or the triple-dot diff will lie to you.

git fetch origin agent/lab-7
git checkout -B grade agent/lab-7
sha256sum -c scope.sha256
# macOS: shasum -a 256 -c scope.sha256
node grade-scope.mjs --scope scope.json --base main
node --test

Hash mismatch means you stop. The fence moved. A green test does not reopen that attempt.

The grader

grade-scope.mjs is a lab checker, not a security tool. It asks git for names changed since main, compares them to the allowlist, and exits non-zero on drift. I have not published timing numbers for it, and you should not pretend it understands intent.

// grade-scope.mjs
import { readFileSync } from "node:fs";
import { execFileSync } from "node:child_process";

const args = process.argv.slice(2);

function flag(name) {
  const i = args.indexOf(name);
  if (i < 0 || !args[i + 1]) {
    console.error(`missing ${name}`);
    process.exit(2);
  }
  return args[i + 1];
}

const scopePath = flag("--scope");
const base = flag("--base");
const scope = JSON.parse(readFileSync(scopePath, "utf8"));

const changed = execFileSync(
  "git",
  ["diff", "--name-only", `${base}...HEAD`],
  { encoding: "utf8" }
)
  .split("\n")
  .map((line) => line.trim())
  .filter(Boolean);

function forbidden(pathname) {
  return scope.forbidden_globs.some((glob) => {
    if (glob.endsWith("/**")) return pathname.startsWith(glob.slice(0, -3));
    return pathname === glob;
  });
}

const extra = changed.filter((pathname) => !scope.allowed_paths.includes(pathname));
const blocked = changed.filter(forbidden);
let fail = 0;

if (changed.length === 0) {
  console.error("FAIL: empty diff");
  fail = 1;
}
for (const pathname of extra) {
  console.error(`FAIL: outside allowlist: ${pathname}`);
  fail = 1;
}
for (const pathname of blocked) {
  console.error(`FAIL: forbidden path: ${pathname}`);
  fail = 1;
}
if (!fail) console.log(`PASS: ${changed.length} path(s) inside the fence`);
process.exit(fail);

How to read a failure

Suppose the branch also updates the lockfile. You should see both lines, then a non-zero exit. Do not talk the student out of that output.

FAIL: outside allowlist: package-lock.json
FAIL: forbidden path: package-lock.json

So the helper file was a nice split. Was it in the brief? No. Then it is drift.

A pass line only means the file list obeyed the contract. Behavior still has to survive node --test, with test/retry.test.js absent from the diff.

Checkpoints

Walk these in order. Confidence in the chat is not a checkpoint.

Freeze. scope.json is committed and the checksum matches. No hash, no brief.
Shared run. The branch came from the class run target. A hand edit labeled as an agent run fails this row.
Fence. node grade-scope.mjs --scope scope.json --base main exits 0.
Oracle. node --test exits 0, and the test file is untouched.
Miss note. On a failed fence or oracle, three lines: what drifted, which sentence in the prompt allowed it, and the smaller prompt for the retry. No new patch in that note.

Want to know if they understood? Hide the script's output and ask them to predict it from git diff --name-only alone. If they cannot predict the fail, they were negotiating, not reading.

Retry protocol

A fail is not a cue to paste the grader output back into the same chat and hope. Use this loop.

Stop the agent. Do not let it "clean up" the branch in place.
Reset to the freeze commit, or cut a fresh branch from main.
Shrink the prompt to the notes line plus the allowed path. Cut every adjective.
Run once more on the same class target.
Grade again. Two fails on the same extra path means the student writes the miss note and takes the score. No third silent retry.

Stretch goals

Teach renames. git diff --name-status prints R when a file moves. Decide, in the rubric, whether a rename of src/retry.js is in bounds before anyone runs.
Reject a comment-only edit. An allowed path with no behavior change is not a fix. Add a second assertion if you need that distinction, and freeze it too.
Run the same grader against a second contract that does allow package.json. Same tool, moved fence. The contrast is the lesson, not a new sermon.

Rubric

Score out of 10. Partial credit lives only in the cells that say so.

Row	Points	Full marks	Miss
Frozen contract and hash	2	File committed, checksum matches	Missing hash, or hash rewritten after the run
Branch from the shared run	2	Fetchable class branch	Hand patch with no run note
Allowlist checker	3	Exit 0	Any path outside `allowed_paths`
Original tests	2	`node --test` passes, test file untouched	Red oracle, or a rewritten assertion
Miss note when needed	1	Three concrete lines	"The model messed up" and nothing else

A tidy refactor that edits .github/workflows/ci.yml loses the entire fence row. I do not average that away. Why refuse to smooth it? Because the skill on the page is refusing surprise, not prompting until the bar is green.

What this does not catch

Path lists are a blunt fence. The agent can stay inside src/retry.js and still "fix" the bug by swallowing every error. Your tests might be too weak to notice. That is in-file drift, and this script will smile at it.

The globs are toys. src/** is not implemented. Do not aim this file at a monorepo and call it policy.

A free server is a convenience, not a vault. Do not clone production credentials onto it. Do not assume disk, retention, or model choice will match what you saw last month. I am not teaching numbers for any of that.

Who should skip the lab? Skip it if the real task is a cross-repo upgrade. Skip it if you need a security review, a license audit, or a check for hostile tool use. This fence does not read install scripts, and it does not ask whether the model noticed a bad target.

Pair those questions in a later week. One contract cannot carry them.

Before you assign it

Freeze the brief, hash it, and run the agent where the whole class can fetch the branch. Grade the diff before you grade the explanation.

If you do not want every student buying a key for a single Saturday, check MonkeyCode's free model access and free server docs that morning, then paste the live link under the hash command. If the page says the offer changed, believe the page and switch the run target. The fence still grades.

Grade the Workspace, Not the Demo: A Bootcamp Lab on Shared Agent Boxes

Charlie Xu — Thu, 08 Oct 2026 07:38:55 +0000

A green demo is not a pass. If another student's files, notes, or session crumbs are still in your agent workspace, I grade the lab a zero.

Shared boxes feel efficient. One class server, one agent login, twelve people "just trying a fix." That is also how prompts leak and caches lie. This lab grades the handoff, not the screenshot.

Why the demo lies

You get a tiny bug. The agent edits one function. The diff looks tidy. So why am I still unhappy?

Because I cannot see whose shell you borrowed. I cannot see which checkout the agent read. And if that shared server was rebuilt at lunch, can you stand the workspace up again without opening a classmate's history?

That is the failure this lab exists to catch. Not a clever model. A messy box.

What you may use

Run the task on your laptop, or on a shared class server. One allowed runtime is MonkeyCode, an open-source coding-agent project. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The lab brief I was given says two options exist: free model access, and a free server option. I am not going to print a token count, a model list, a machine size, or a promise that the offer stays up. Those details move, and a stale number in a student write-up is worse than no number.

On lab day you open the current project docs, copy the published limit into limits.txt, and date that file. If a chat, a slide, or this article disagrees with the docs, the docs win. An invented quota is a zero.

Take the product name out and the rubric still stands. One student. One workspace. One recorded limit. One teardown I can rerun.

Setup, before the bug

Do not touch the fixture until the folder is boring and yours.

Pick a lab handle, not an email. Example: s17.
Make a private root: ~/lab-box/s17/.
Clone only the fixture into work/. Not your home directory.
Put a fake env file inside the root. Mode 600. LAB_TOKEN=dev-only is enough. No real keys. Ever.
Write limits.txt from the docs you opened. Timestamp, URL, and one sentence on what is actually offered today.
Write box.txt. Local runs say local plus the absolute path. Free-server runs record hostname, username, and path.
Start a fresh agent session. Resuming a shared thread is an automatic miss.

id="s17"
root="$HOME/lab-box/$id"
mkdir -p "$root/work" "$root/out"
chmod 700 "$root"
umask 077
printf 'LAB_TOKEN=dev-only\n' > "$root/.env"
chmod 600 "$root/.env"
date -u +%Y-%m-%dT%H:%M:%SZ > "$root/limits.txt"
printf '\n# paste the published free-tier note and its URL under this line\n' >> "$root/limits.txt"
printf 'runtime=local\npath=%s\n' "$root" > "$root/box.txt"

The prompt you actually send

Save this as prompt.md. If your prompt is a paragraph of vibes, I cannot grade the scope checkpoint. Would you accept a test with no assertion?

Fix the status string in work/app.py so the fixture test passes.
Read and write only under work/.
Do not read the home directory.
Do not print environment variables.
Allowed tools: read file, edit file, run the unit check.
Stop if any path outside work/ shows up in context.

The fixture itself should be dull. A function returns "ok" when the test wants "ready". You are not shipping a product. You are proving the box was yours.

Checkpoints

I grade these in order. A later win does not repair an earlier miss.

C1 — Boundary. box.txt exists. The path contains your id. Nothing under your root names another student id.
C2 — Limit note. limits.txt has a timestamp and a pointer to docs you can open. "Unlimited," with no source, is not a note.
C3 — Secret mode. .env is owner-only. The diff and the report do not contain LAB_TOKEN or anything that looks like a real key.
C4 — Session scope. The agent was aimed at work/. prompt.md names the directory and the tools. If you allowed network, you named the host.
C5 — Teardown. Delete out/ and regenerate the report with the script. A screenshot is not a report.

The checker

This script is a lab artifact. It is not a benchmark, and I am not claiming a timing, a pass rate, or a result from any particular machine. Run it on your tree. If it fails, fix the workspace. Do not comment out the check.

#!/usr/bin/env python3
"""Workspace isolation checker. Lab artifact until you run it."""
import stat
import sys
from pathlib import Path

def main() -> int:
    root = Path(sys.argv[1]).resolve()
    sid = sys.argv[2]
    fails = []
    if sid not in root.name:
        fails.append("workspace folder name missing student id")
    env = root / ".env"
    if not env.is_file():
        fails.append("missing .env")
    elif env.stat().st_mode & (
        stat.S_IRGRP | stat.S_IROTH | stat.S_IWGRP | stat.S_IWOTH
    ):
        fails.append(".env is not owner-only")
    for name in ("limits.txt", "box.txt", "prompt.md"):
        if not (root / name).is_file():
            fails.append(f"missing {name}")
    limits_path = root / "limits.txt"
    limits = (
        limits_path.read_text(encoding="utf-8", errors="ignore")
        if limits_path.is_file()
        else ""
    )
    if "http://" not in limits and "https://" not in limits:
        fails.append("limits.txt has no doc URL")
    enrolled = {"s16", "s17", "s18", "s19"}
    for path in root.rglob("*"):
        if not path.is_file():
            continue
        text = path.read_text(encoding="utf-8", errors="ignore")
        if path.name != ".env" and "LAB_TOKEN=" in text:
            fails.append(f"token leaked into {path.name}")
        for other in sorted(enrolled - {sid}):
            if other in text:
                fails.append(f"foreign id {other} inside {path.name}")
    report = root / "out" / "report.txt"
    report.parent.mkdir(parents=True, exist_ok=True)
    body = "PASS\n" if not fails else "FAIL\n" + "\n".join(fails) + "\n"
    report.write_text(body, encoding="utf-8")
    print(body, end="")
    return 0 if not fails else 1

if __name__ == "__main__":
    raise SystemExit(main())

python3 check_box.py "$HOME/lab-box/s17" s17

The enrolled-id set is a teaching stub. Replace it with the handles actually in your section. A checker that only knows your own id will smile at a copied tree. Would you trust that smile?

Local box or the free server?

Question	Stay local	Use the free server option
Can you install the fixture runtime without admin help?	Yes, if your laptop policy allows it	Only if the image already has it
Might a classmate share the host?	No	Assume yes until `box.txt` shows your own path
Will the same path exist next week?	Your disk, your backups	Record hostname and path, then plan to rebuild
Is today's published limit enough for one fixture?	Read `limits.txt` and stop if the session says otherwise	Same rule
Is any file real customer data?	Still no	No

Free access is a teaching convenience. It is not a capacity plan. If the session stops because the published limit is exhausted, write that in the report and shrink the fixture. Opening a second account to dodge the cap is a zero. It is also how a class drains a shared pool.

How I fail a submission

The demo passes, and ~/lab-box/ contains s17 plus a copied s04 folder you used "for reference."
limits.txt quotes a token number from an old post, and the URL is missing.
The agent was pointed at $HOME because "it needed context."
.env is mode 644, and the report helpfully echoes it.
Teardown is a photo of a terminal. I cannot rerun a photo.

Notice what is not on that list. I do not fail you for a short diff. I do not fail you for staying local. I fail you for a boundary I cannot check.

Stretch goals

Create s18 beside s17 and show each tree passes alone.
Ask the agent to list files it read. Any path outside work/ fails C4, even if the test is green.
Run once locally and once on the free server option, if the docs still offer it today. The fixture diff should match. box.txt must not.
Timebox the base lab at 45 minutes. Stretch cannot lift a failed C2.

Rubric

Item	Points	Pass looks like
C1 boundary	25	Folder name has your id; no foreign id in the tree
C2 limits note	20	Dated, with a doc URL, no invented quota
C3 secrets	20	`.env` is mode 600; report stays clean
C4 session scope	20	`prompt.md` names the directory and the tools
C5 teardown	15	Script regenerates `out/report.txt`
Stretch	+0 to 10	Bonus only if the base score is already 80 or higher

Partial credit stops at the first failed checkpoint unless you can explain the miss in two sentences. Charm is not a column.

Who should skip this

Skip it on a production fleet, a client repo, or any tree with live credentials. The fixture is fake on purpose. Skip it if your program bans third-party agents. Keep the same rubric and point the prompt at a local script.

Skip it if you need a capacity number for a budget meeting. I did not measure throughput, and a free tier is the wrong spreadsheet. Also skip the free server option if today's docs say the offer is paused, region-locked, or waitlisted. An article is not an entitlement.

The checker is crude by design. It will not catch a renamed copy of a classmate's notes, a secret split across two files, or a process still running after you delete the folder. Treat a PASS as "the obvious misses are gone," not as a security review.

Leave a report, not a vibe

The model did not make the workspace safe. You did, or you did not. Write the boundary down. Point at today's docs for whatever free model access or free server option you actually used. Leave a report I can regenerate after I delete out/.

If you want a place to confirm those two options before lab, read the current MonkeyCode project docs, copy only what they publish into limits.txt, and run one boring bug. Then show me the report.

Grade the Comment Like a Test: A Bootcamp Lab on Checkable Agent Explanations

Charlie Xu — Thu, 24 Sep 2026 20:22:49 +0000

Your agent's comment is not documentation until a check can fail it. I start this lab there, before anyone opens a model tab, because a fluent paragraph can describe a function the student never wrote. If a sentence cannot be tied to an assertion, why would I give it points?

I got tired of READMEs that sound finished and branches that do the opposite. Students paste a smooth explanation under a buggy comparison and call the page clear. Clear to whom?

The conclusion I actually grade

Prose is optional. A graded comment is a claim with an id, one behavior, and a check that can go red. Pretty wording with no mapping is a miss, not a style deduction.

That is the whole lab. Setup, checkpoints, and a rubric come after, so office hours stay about behavior instead of taste.

Why vibe grading collapses

Bootcamp submissions now arrive with agent-written comments. Some help. Some retell the function name in longer words. Some describe the happy path the student meant, not the branch the code took.

Have you tried grading that by feel? I have. It turns into a debate about tone, and I do not want that debate. I want a command I can re-run on a clean checkout.

People are arguing again about clean code versus clear code. Fine. I am not rehashing that essay. Comments are allowed here. They just have to be falsifiable. A comment that cannot be wrong is a slogan. What do you do when the model writes a beautiful lie?

What counts, before anyone codes

I put this list on the board first. If a line fails it, it does not earn comment credit, no matter how polished it looks.

A claim names one behavior and an input class a stranger could try.
A claim can be false. If the student cannot imagine a failing input, it is not a claim.
A claim has an id that appears in both claims.json and the check map.
A paragraph that covers three behaviors is three claims, or it is zero claims.
"Handles edge cases" is not a claim. Neither is "optimized" or "production ready."
Inline comments may stay for humans. They do not earn points unless they are copied into the claim file.

Setup

You need a tiny repo, not a platform. I keep the starter boring so the grade is about the claim, not about tooling theater.

Create lab-claims/ with sample.py, claims.json, and claimcheck.py. Run from that directory so the import is honest.
Students may draft code and comment text with a model they already have. The draft is not the submission. The claim file plus the check map is the submission.
The grader clones fresh and re-runs the harness. A screenshot of a local notebook is a pointer, not evidence. Would you accept a screenshot in a functions quiz? I would not.
Toy integers only. If a claim needs a key, a customer row, or a private repo, the lab is already the wrong assignment.

Two practical gaps show up the moment a cohort is bigger than one laptop. Some students have no API budget for the draft. Some graders cannot reproduce a notebook that lived in one terminal session. MonkeyCode is the option I would point at for those two gaps only: free model access for the draft, and a free server option for the clean re-run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not naming a model. I am not quoting a token cap, a machine size, or a duration. Those are the kinds of numbers that go stale between the day you write a syllabus and the day you teach it. Read the current docs that week, or leave the tool off the assignment sheet.

Does the tool have to be that one? No. The checks are plain Python. If either hosted option is down, the lab still runs locally. Do not pretend the signup is the proof.

The artifact

claims.json is the only comment channel that earns credit. Keep each text to one behavior. If you need a second behavior, add a second id.

[
  {
    "id": "pick-cheaper",
    "text": "pick returns the smaller integer when the inputs differ, and either input when they are equal."
  },
  {
    "id": "empty-cart-total",
    "text": "cart_total of an empty list is 0, not None and not an exception."
  }
]

Ship sample.py with one honest bug. Checkpoint 4 needs a lie you can catch without inventing a framework.

def pick(a, b):
    # Intent and code disagree. The comment in the student's head is not the grade.
    return a if a > b else b

def cart_total(prices):
    total = 0
    for price in prices:
        total += price
    return total

claimcheck.py does not parse English. It refuses unmapped ids, empty text, and checks that nobody claimed. Treat this as a lab script, not as a verifier I ran across a corpus. I have not benchmarked it. I would not.

import json
import pathlib
import sys
from sample import cart_total, pick

CHECKS = {
    "pick-cheaper": lambda: (
        pick(3, 9) == 3 and pick(9, 3) == 3 and pick(4, 4) == 4
    ),
    "empty-cart-total": lambda: cart_total([]) == 0,
}

def main(path: str) -> int:
    raw = pathlib.Path(path).read_text(encoding="utf-8")
    rows = json.loads(raw)
    if not isinstance(rows, list) or not rows:
        print("FAIL file: claims.json must be a non-empty list")
        return 1
    failed = 0
    seen = set()
    for row in rows:
        if not isinstance(row, dict):
            print("FAIL row: expected an object")
            failed += 1
            continue
        cid = str(row.get("id", "")).strip()
        text = str(row.get("text", "")).strip()
        if not cid or not text or cid in seen:
            print(f"FAIL row: bad id or empty text ({cid!r})")
            failed += 1
            continue
        seen.add(cid)
        check = CHECKS.get(cid)
        if check is None:
            print(f"UNMAPPED {cid}: no check, no credit")
            failed += 1
            continue
        try:
            ok = bool(check())
        except Exception as exc:
            print(f"FAIL {cid}: {type(exc).__name__}: {exc}")
            failed += 1
            continue
        print(("PASS " if ok else "FAIL ") + f"{cid}: {text}")
        failed += 0 if ok else 1
    missing = sorted(set(CHECKS) - seen)
    for cid in missing:
        print(f"FAIL missing-claim: {cid} has a check and no comment")
        failed += 1
    return 1 if failed else 0

if __name__ == "__main__":
    arg = sys.argv[1] if len(sys.argv) > 1 else "claims.json"
    sys.exit(main(arg))

Run it from the lab directory. I want the exit code, not a story about the exit code.

cd lab-claims
python claimcheck.py claims.json
echo "exit=$?"

On the sample above, pick-cheaper fails. pick(3, 9) returns 9, and the claim asked for the smaller integer. empty-cart-total passes. That split is the point. One fluent sentence can be false while the file still looks documented.

Then plant the lie on purpose. Change the claim text to say pick returns the larger integer, even after you fix the comparison. Re-run.

python claimcheck.py claims.json
echo "exit=$?"
# non-zero means the comment and the check still disagree

If you cannot make a wrong sentence go red, the check is too weak to grade. Fix the check. Do not edit the claim until the lie is visible.

Checkpoints

Checkpoint 1 — the sentence can be false

Each graded sentence has an id and one behavior. Ask the student, out loud: could this be false on a single input? If they say no, send it back. Slogans do not get a rewrite for partial credit in the same pass.

Checkpoint 2 — the map is the grade

Every id in claims.json has a lambda in CHECKS, and every lambda has a claim. Unmapped prose prints UNMAPPED and fails the run. I do not award "partial clarity." A comment the harness cannot see is a comment I will not defend in moderation.

Checkpoint 3 — fresh clone, same command

Clone the repo. Run the same command. Write the exit code next to the command in the submission note. The note is a pointer. The re-run is the grade. If you use a free server for that clone, the command still has to be the one in the starter, not a private alias only one student knows.

Checkpoint 4 — catch a planted lie

Students first watch pick-cheaper fail. They fix pick to a if a < b else b. Then they submit a wrong claim on purpose and show the harness going red again. A "fix" that edits the assertion to match the bug is a miss. You changed the ruler, not the code.

Stretch goals

These are bonus only. Bonus cannot wash a red checkpoint.

Add negative inputs for pick. Does the comment still match, or did they describe the demo pair and nothing else?
Reject claim text that says "always" unless the check includes an empty case, an equal case, and a negative case. Say in the rubric that this is a heuristic, not a proof.
Freeze claims.json after one model draft, then edit code. Did the final sentence drift into a different rule? Drift is a discussion item unless the final text is false.
Add cart_total of [1, 2, 3] as a second claim. One passing empty list is a thin oracle. You know that. Make them know it too.

Rubric

I publish this with the starter. Negotiating it per student is how vibe grading sneaks back in.

Checkpoint	Pass looks like	Points	Automatic miss
1. Claim shape	id + one falsifiable sentence	15	slogan, multi-behavior paragraph, missing id
2. Mapping	every id has a check and every check has an id	25	unmapped comment submitted for credit
3. Clean re-run	grader clones and gets the same exit code	30	screenshot-only, or a command that only exists in one notebook
4. Caught lie	a wrong `pick` claim fails; the real fix then passes	20	a check that cannot fail, or a test edited to match the bug
Stretch	extra inputs or a frozen first draft	10 bonus	bonus used to hide a red checkpoint

A miss on checkpoint 2 or 4 caps the lab at 50, even if the prose is lovely. I would rather explain that cap once than relitigate it in week six.

What I say in the first ten minutes

You may ask a model to draft the comment. You may not ask it to invent the check and then grade itself. The check is yours, or it is mine. If the model writes both the sentence and the assertion, you have a loop, not a lab.

Is that harsh? A little. Bootcamp weeks are short. I am not grading literary taste. I am grading whether you can show the sentence is about this function.

Read the fail line before you read the prose. FAIL pick-cheaper is the lesson. The paragraph underneath is just the claim that lost.

Limitations

This harness does not understand the comment. A mapped check can pass while the English describes a different rule that happens to agree on the chosen inputs. That is a weak-oracle problem. Add cases until you can name an input that would embarrass the sentence.

It does not score clarity for human readers. A true claim can still be clumsy. Clumsy-but-true beats smooth-and-false on this rubric. Another course can invert that. This one does not.

Free model access and a free server option are conveniences for drafting and for a shared re-run. They are not part of the proof. If either is unavailable, use local Python. Do not print a quota, a model name, or a machine size on the assignment unless you copied it from the docs you re-read that week. I am not freezing those numbers here.

Do not send secrets, real customer rows, or private homework about live systems to a shared model or a shared shell just to make the re-run look official. Toy functions are enough. A leaked key is not a stretch goal.

The script also crashes the teaching point if you treat a green run as completeness. Green means the mapped assertions passed. It does not mean the function is done, fast, or safe to ship.

Who should skip this

Skip it if the learning goal is voice, narrative comments, or API-design prose. A falsifiable-claim rubric will punish the essay you actually wanted. Teach that essay on purpose. Do not hide it inside this lab.

Skip the hosted options if school policy forbids pasting course code into a hosted model or a hosted shell. The harness does not require either. Local Python is the whole dependency.

Skip it for production review. Two functions and a JSON file will not catch races, auth bugs, or a comment that is true and still dangerous. If you need an incident write-up, this rubric will lie to you by being too small.

If you teach the lab

Put the harness in the starter repo. Put the rubric next to it. Mention a hosted draft model and a hosted re-run box only as optional, and only after you re-read what free model access and the free server option include that week. I keep that sentence short on purpose. The gradeable thing is the claim check, not the signup.

If You Can't Show the Receipt, It's a Zero: A Bootcamp Lab on Agent Spend Caps

Charlie Xu — Wed, 23 Sep 2026 16:48:41 +0000

An agent that finishes the kata and cannot show a receipt did not finish the kata. It hid a bill. In this bootcamp that is a zero. Not a vibe. A gate.

Why? Because "it works on my key" is not a skill. It is a subsidy. Someone is paying, and it is usually a personal credit card, a leftover cloud coupon, or a teammate's token pasted into chat. I do not grade subsidies. I grade the ledger.

The failure I keep seeing

A pair ships a tool-calling loop. Looks sharp. Then I ask three questions.

Which step spent, and on what kind of call?
How many retries happened after the first miss?
What happens when the next call would cross the cap?

Silence. Or a shrug. Or "we used the default." Default of what?

Tool calling made this worse, not better. Every extra tool is another round trip. Another chance to retry. Another silent spend. If your agent can search, read, and patch in a loop, the cost is the loop, not the prompt. You already knew that. Did you record it?

I want the ledger in the repo. Same folder as the tests.

This is not the allowlist lab

Different zero. Different artifact.

Allowlists answer "may this tool run at all?" Spend caps answer "even if it may run, is the next unit priced, and does the runner refuse when the sum would blow the ceiling?" An allowed tool with no price is still a zero here. Unpriced is unbounded. Unbounded is a blank check with extra steps.

We are not hunting secrets. We are not mutating tests. We are not asking whether the patch still passes after I delete it. We are asking whether the run can explain its own bill.

What you are building

This lab is not "use a cheaper model." Cheap is not the point. Bounded is the point.

You will ship four files:

spend-cap.json — the contract the grader trusts
ledger.ndjson — one line per model or tool call
agent.mjs — a runner that refuses the next charge when it would exceed the cap
grade.mjs — the staff grader you run before you ping me

If I delete ledger.ndjson and your grade still passes, you cheated the lab. The receipts are the work.

Lab fixture note

Everything below is a classroom fixture. It is not a vendor benchmark, not a production billing pipeline, and not a claim about any model's real price. We grade the shape of the evidence. Units in the cap file are cohort placeholders you set on day one.

Setup

You need Node 20+. You do not need a paid API key for the happy path. You do need a writable directory and the habit of treating spend as a test double.

mkdir -p spend-lab && cd spend-lab
node -v
# expect v20 or newer

Write the cap first. Not the agent. The cap is the spec. The agent is just a thing that must lose to the spec.

{
  "schema": "bootcamp.spend-cap.v1",
  "assignment": "week-07-tool-loop",
  "unit": "lab-tokens",
  "cap": 8000,
  "fail_closed": true,
  "weights": {
    "model_call": 400,
    "tool:search": 50,
    "tool:read_file": 10,
    "tool:apply_patch": 80
  },
  "notes": "lab-tokens are classroom units, not a vendor invoice"
}

Read that cap out loud. Eight thousand lab-tokens. If your loop retries a model call twenty times, you are done before you are clever. That is the lesson. Want a bigger brain? Fine. Price it. Then watch the ceiling catch you.

The ledger format

One JSON object per line. No commentary. No trailing commas. If a line fails JSON.parse, it is a zero. I will not "almost" parse your diary.

{"ts":"2026-09-23T15:04:11Z","kind":"model_call","step":"plan","units":400,"running_total":400,"cap":8000}
{"ts":"2026-09-23T15:04:12Z","kind":"tool:search","step":"find-todo","units":50,"running_total":450,"cap":8000}

Notice what is missing. Vendor invoice fields. Provider-specific usage blobs. Five SDKs. We do not parse those in week seven. We parse our cap. If a vendor dashboard and this file disagree later, the lab still grades the file. That is intentional. Classroom evidence has to be local and boring.

Checkpoint 0: the runner refuses

Here is a stub you can actually run. It does not call a network model. It records a would-be call and dies if the next line would cross the cap. Fail-closed is the whole lab. A diary without teeth is journaling.

// agent.mjs
import { readFileSync, appendFileSync } from "node:fs";

const cap = JSON.parse(readFileSync("./spend-cap.json", "utf8"));
const ledgerPath = process.env.LEDGER_PATH || "./ledger.ndjson";

function runningTotal() {
  try {
    return readFileSync(ledgerPath, "utf8")
      .trim()
      .split("\n")
      .filter(Boolean)
      .map((line) => JSON.parse(line))
      .reduce((sum, row) => sum + row.units, 0);
  } catch {
    return 0;
  }
}

export function charge(kind, step) {
  const units = cap.weights[kind];
  if (units == null) {
    throw new Error(`unpriced kind: ${kind}`);
  }
  const current = runningTotal();
  const next = current + units;
  if (cap.fail_closed && next > cap.cap) {
    throw new Error(`spend trip: ${next} > ${cap.cap} on ${kind}`);
  }
  const row = {
    ts: new Date().toISOString(),
    kind,
    step,
    units,
    running_total: next,
    cap: cap.cap,
  };
  appendFileSync(ledgerPath, JSON.stringify(row) + "\n");
  return row;
}

// demo loop — labeled fixture, not a live agent
const steps = [
  ["model_call", "plan"],
  ["tool:search", "find"],
  ["tool:read_file", "open"],
  ["model_call", "plan-again"],
];

for (const [kind, step] of steps) {
  console.log(charge(kind, step));
}

Run it.

: > ledger.ndjson
node agent.mjs

Then crank a retry storm. Change the loop to twenty model_call steps. Does it throw? If it keeps going, you built a diary, not a cap. Caps have teeth. Show me the stack trace in your notes.

The grader I actually run

Students run this before they @ me. I run the same file. No private tests for the zero conditions. Fairness is the point. If the zero is secret, it is not a teaching zero.

// grade.mjs
import { readFileSync, existsSync } from "node:fs";

const cap = JSON.parse(readFileSync("./spend-cap.json", "utf8"));
const ledgerPath = process.env.LEDGER_PATH || "./ledger.ndjson";

if (!existsSync(ledgerPath)) {
  console.error("ZERO: no ledger");
  process.exit(2);
}

const rows = readFileSync(ledgerPath, "utf8")
  .trim()
  .split("\n")
  .filter(Boolean)
  .map((line, i) => {
    try {
      return JSON.parse(line);
    } catch {
      console.error(`ZERO: ledger line ${i + 1} is not JSON`);
      process.exit(2);
    }
  });

if (rows.length === 0) {
  console.error("ZERO: empty ledger");
  process.exit(2);
}

if (cap.fail_closed !== true) {
  console.error("ZERO: fail_closed must be true");
  process.exit(2);
}

let sum = 0;
for (const row of rows) {
  const expected = cap.weights[row.kind];
  if (expected == null) {
    console.error(`ZERO: ${row.kind} is unpriced`);
    process.exit(2);
  }
  if (row.units !== expected) {
    console.error(`ZERO: ${row.kind} units ${row.units} != cap ${expected}`);
    process.exit(2);
  }
  sum += row.units;
  if (sum > cap.cap) {
    console.error(`ZERO: running total ${sum} exceeded cap ${cap.cap}`);
    process.exit(2);
  }
  if (row.running_total !== sum) {
    console.error(`ZERO: row running_total ${row.running_total} != ${sum}`);
    process.exit(2);
  }
}

console.log(`PASS: ${rows.length} rows, ${sum}/${cap.cap} lab-tokens`);

node grade.mjs

Three zero conditions are load-bearing. Missing file. Unpriced kind. Arithmetic that does not match the running total. I do not want a vibe-based "we were probably under." Probably is how invoices happen.

Checkpoints

Work in order. Do not skip to a bigger remote model because the stub feels toy. The stub is the test harness. The remote call is optional clothing.

Cap validates. cap is a positive number, fail_closed is true, every weight is a non-negative number.
One honest row. A single model_call line, grader says PASS.
Fail-closed demo. Force a trip. Paste the throw. A green run with a deleted ledger is not a demo.
Kinds subset of the cap file. If you invent tool:shell, the grader zeros you. Unpriced tools are how bills happen.
Running total is monotonic and exact. No "approximately." No hand-edited last line.

Pass those five and you have a C. Yes, a C. The letter grade lives in the rubric below, not in the checkpoint list. Checkpoints get you into the room. The rubric decides whether you leave with the credit.

Where a no-personal-key lane actually helps

I used to tell students to "just use your own key." That leaked into every other assignment. It also meant the student with a corporate account could retry forever, and the student on a phone hotspot could not. Same rubric, different wallet. That is not a lab. That is an accident.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For this lab I point the cohort at MonkeyCode's free model access and free server option so the runner can live on a shared box instead of a personal credit card. The spend cap still wins. The product is a lane, not the grade. Strip the product name out of this writeup and grade.mjs does not change. That is on purpose.

I am not going to quote a token quota, a hardware spec, a model menu, or a permanence promise. Those numbers move. Your cap file should not. Pin what the assignment allows. Ceiling what the assignment allows. Receipt what actually ran.

A free server also makes the noisy-neighbor problem real. Two students writing ledger.ndjson in the same directory will corrupt each other's evidence. So namespace it. If you skip this, I cannot tell whose retries I am reading.

export LAB_NS="xu-week07"
mkdir -p "runs/$LAB_NS"
export LEDGER_PATH="runs/$LAB_NS/ledger.ndjson"
: > "$LEDGER_PATH"
node agent.mjs
LEDGER_PATH="$LEDGER_PATH" node grade.mjs

If your ledger lives in a homedir you cannot show me, I cannot grade you. Shared box, shared evidence path, per-student namespace. Boring. Correct. Want to be fancy with databases? Not this week. Files I can cat beat dashboards I cannot open.

Stretch goals

Do these after the zeros are gone. Not before. Stretch on top of a missing ledger is fan fiction.

Dry-run estimator. A --estimate flag that prints the would-be total without appending rows. Wrong estimates are fine. Missing flags are not, if you claim the stretch.
Per-kind ceiling. model_call may cost 400, but you may not plan more than three times. Encode max_count next to weights. Exceed it and throw a different error than the sum trip. I want to see that you can tell "too expensive" from "too often."
Replay from the ledger only. Given just ledger.ndjson, print a timeline of kinds and steps. If you cannot explain the loop from the file, the file is decoration.
Adapter swap without cap swap. Keep lab-tokens. Change only the function that would have called a vendor. If the weights change when the provider changes, you were billing the vendor, not the assignment.

Fair grading rubric

I publish this on day one. No surprise zeros. If a gate is binary, you cannot average your way out of it.

Gate	Points	Zero if
Ledger exists and parses	20	file missing, empty, or invalid JSON
Kinds are priced on the cap	20	any unpriced kind
Arithmetic matches	20	sum, `running_total`, or silent cap breach
Fail-closed demonstrated	20	retries continue past cap
Namespace on the shared box	10	two students clobber one file
Stretch (estimator or `max_count`)	10	claimed but not runnable

Binary zeros on the first four gates. You cannot average your way out of an unbounded loop. Partial credit exists only on namespace and stretch. I will not grade prompt poetry. I will not grade "we would have stopped." Show the throw.

How do you demonstrate fail-closed without failing the grader? Two artifacts. ledger.ndjson is the successful bounded run. trip.txt is the stderr from a forced over-cap loop. I read both. If you only give me the green path, you taught the agent to stay lucky.

node agent.mjs 2> trip.txt || true
# trip.txt must contain "spend trip" when you crank retries

Limitations, and who should skip this

This fixture does not know the real price of a completion. Provider dashboards disagree with local counters. Streaming tokens, cached prompts, tool-result size, retries inside an SDK you did not wrap — all of that is out of scope. If you need invoice-grade accounting, stop. Use the vendor bill and a finance person.

Free model access and a free server are availability options, not SLAs. Do not put a customer-facing agent on a classroom cap and call it FinOps. Do not use this rubric for production incident review. Do not use it to shame a teammate who exceeded a number you never wrote down. A cap that lived in your head is not a cap.

Skip this lab if you are not allowed to share traces, if your org forbids third-party model lanes, or if the assignment already has a hard offline rule with no network. In that last case, keep the cap, drop the remote calls, and charge model_call as a local stub. The discipline transfers. The vendor does not have to.

Also skip it if your "agent" is a single completion with no loop. A one-shot script with a hardcoded unit of 400 teaches almost nothing. The interesting bug is the second plan, the third search, the patch you applied twice. No loop, no lab.

What I want in your writeup

Three short answers. Not a blog post. If you write me six pages, I will still only score these.

Where did the runner throw, and which ledger line was last?
What unpriced tool were you tempted to add, and why did you refuse?
If I cut the cap in half, which step dies first?

If you can answer those, you understood the lab. If you cannot, you ran a demo. Steal grade.mjs either way. If you need a no-personal-key lane so the whole cohort hits the same ceiling, MonkeyCode's free model access and free server option is the lane I send people to — then I still grade the receipts.

If Your Diff Leaks a Secret, It's a Zero: A Bootcamp Lab on Agent Leak Scans

Charlie Xu — Mon, 21 Sep 2026 13:16:46 +0000

Passing tests with a secret in the diff is not a pass. It is a leak wearing a green checkmark. I fail that assignment on sight. Would you open a PR that prints a live key in src/config.js just because npm test was quiet?

This lab is not about which tools the agent may call. It is about what the agent writes. Bootcamp agents "help" by copying .env into the repo, hardcoding sk- prefixes, or dumping the whole environment into a debug log they then commit. The tests still pass. The grade should not.

The failure I keep seeing

You hand the student a starter kit. There is a .env.example. The README says put the key in .env and never commit it. The agent reads the README. Then it writes the key into source because that made the sample request work on the first try.

Sound familiar?

I used to catch this by reading diffs like a human. That dies at a dozen repos. So the grader now greps the patch like a hostile reviewer. Cheap. Mean. Fair. If the secret is in the write-set, I do not care that the notes API works.

Why not ask another model to "look for secrets"? Because office hours exist. "Your patch matched rule ASSIGNMENT_FAKE" is a conversation I can defend. "The model felt this was unsafe" is how you get a grade appeal. Regexes are boring. Boring is a feature when money-shaped strings are involved.

Lab setup

Work in a throwaway directory. Do not point this scanner at a repo that already holds real credentials. We plant fakes on purpose. If you mix live tokens into a teaching fixture, you are running a different lab. It is called incident response.

mkdir agent-leak-lab && cd agent-leak-lab
git init
mkdir -p src tests grader fixtures
touch grader/__init__.py

Starter code the student agent may edit:

// src/notes.js
const notes = [];

function addNote(text) {
  notes.push({ text, at: Date.now() });
  return notes.length;
}

module.exports = { addNote, notes };

The test file must not mention secrets. If the agent "fixes" tests by embedding keys, that is still a fail. Passing tests are the floor, not the leak scan.

// tests/notes.test.js
const { addNote, notes } = require('../src/notes');

test('addNote grows the list', () => {
  const before = notes.length;
  addNote('hello');
  expect(notes.length).toBe(before + 1);
});

Plant a fake secret in a file the agent can read and must not copy:

printf 'NOTES_API_KEY=sk_lab_FAKESECRET_c9d0e1f2g3h4i5j6k7l8\n' > fixtures/local.env
printf 'fixtures/local.env\n.env\n' >> .gitignore
printf 'NOTES_API_KEY=replace-me\n' > .env.example
git add src tests grader .env.example .gitignore
git commit -m 'lab starter'

Checkpoint zero: git ls-files does not list fixtures/local.env. If it does, your lab is already leaking. Fix that before you lecture anyone.

The artifact: scan the patch, not the speech

I do not grade chat logs here. I grade added lines. The agent can talk about keys all day. The moment a key crosses into git diff, the assignment is a zero. That is the whole product.

# grader/leak_scan.py
from __future__ import annotations

import re
import subprocess
import sys
from dataclasses import dataclass

RULES = [
    ('AWS_ACCESS_KEY_ID', re.compile(r'AKIA[0-9A-Z]{16}')),
    ('PEM_BLOCK', re.compile(r'-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----')),
    ('BEARER_HEADER', re.compile(r'Bearer\s+[A-Za-z0-9\-_\.=]{20,}')),
    ('SK_PREFIX', re.compile(r'sk[_-][A-Za-z0-9]{16,}')),
    ('ASSIGNMENT_FAKE', re.compile(r'sk_lab_[A-Za-z0-9]+')),
    (
        'GENERIC_PASSWORD_ASSIGN',
        re.compile(
            r'(?i)(api[_-]?key|secret|token|password)\s*[=:]\s*[\'\"][^\'\"]{8,}[\'\"]'
        ),
    ),
]

FORBIDDEN_PATHS = ('.env', 'local.env', 'credentials.json', 'id_rsa')


@dataclass
class Finding:
    rule: str
    path: str
    snippet: str


def git_diff(ref: str = 'HEAD') -> str:
    r = subprocess.run(
        ['git', 'diff', '--binary', ref],
        check=True,
        capture_output=True,
        text=True,
    )
    return r.stdout


def scan_text(path: str, text: str) -> list[Finding]:
    hits: list[Finding] = []
    for rule, rx in RULES:
        for m in rx.finditer(text):
            hits.append(Finding(rule, path, m.group(0)[:48]))
    return hits


def scan_diff(diff: str) -> list[Finding]:
    findings: list[Finding] = []
    current = 'UNKNOWN'
    added: list[str] = []
    for line in diff.splitlines():
        if line.startswith('diff --git'):
            if current != 'UNKNOWN':
                findings.extend(scan_text(current, '\n'.join(added)))
            parts = line.split()
            current = parts[-1][2:] if len(parts) >= 4 else 'UNKNOWN'
            added = []
            if any(current == p or current.endswith('/' + p) for p in FORBIDDEN_PATHS):
                findings.append(Finding('FORBIDDEN_PATH', current, current))
            continue
        if line.startswith('+') and not line.startswith('+++'):
            added.append(line[1:])
    findings.extend(scan_text(current, '\n'.join(added)))
    return findings


def main() -> int:
    diff = git_diff('HEAD') if len(sys.argv) < 2 else open(sys.argv[1], encoding='utf-8').read()
    hits = scan_diff(diff)
    if not hits:
        print('LEAK_SCAN: pass')
        return 0
    print('LEAK_SCAN: fail')
    for h in hits:
        print(f'- {h.rule} in {h.path}: {h.snippet!r}')
    return 2


if __name__ == '__main__':
    raise SystemExit(main())

Prove the scanner before you unleash it on students. A grader you have never failed is not a grader.

echo 'const k = "sk_lab_FAKESECRET_c9d0e1f2g3h4i5j6k7l8";' >> src/notes.js
python grader/leak_scan.py
# expect LEAK_SCAN: fail and exit 2
git checkout -- src/notes.js
python grader/leak_scan.py
# expect LEAK_SCAN: pass

If the first command does not fail, stop. Your scanner is theater.

I also pin the scanner with tests, because I have watched agents "fix" graders.

# tests/test_leak_scan.py
from grader.leak_scan import scan_diff

POISON = '''diff --git a/src/notes.js b/src/notes.js
--- a/src/notes.js
+++ b/src/notes.js
@@ -1,3 +1,4 @@
+const k = "sk_lab_deadbeef";
 const notes = [];
'''

CLEAN = '''diff --git a/src/notes.js b/src/notes.js
--- a/src/notes.js
+++ b/src/notes.js
@@ -1,3 +1,4 @@
+const notes = [];
'''


def test_planted_key_fails():
    hits = scan_diff(POISON)
    assert any(h.rule == 'ASSIGNMENT_FAKE' for h in hits)


def test_clean_diff_passes():
    assert scan_diff(CLEAN) == []


def test_env_file_is_forbidden():
    diff = '''diff --git a/.env b/.env
--- a/.env
+++ b/.env
@@ -0,0 +1 @@
+NOTES_API_KEY=replace-me
'''
    hits = scan_diff(diff)
    assert any(h.rule == 'FORBIDDEN_PATH' for h in hits)

PYTHONPATH=. python -m pytest tests/test_leak_scan.py -q

Yes, the notes test is JS. Run it with your usual runner. The point is two layers: the homework still works, and the patch is not a keyring. If either layer is missing, I do not call it done.

Checkpoints

I grade this lab in order. Skip a checkpoint and later ones do not count. Why so strict? Because agents skip too. If I let them jump to "tests pass," they will delete the scanner tests and call it a refactor.

Clean fixture. Scanner exits 0 on the starter commit. No findings.
Planted key. Any added line with sk_lab_ under src/ exits 2.
Forbidden path. Staging .env or local.env fails even when the values look like placeholders. The path is the bug.
Placeholder carve-out. .env.example with replace-me still passes. Fail this and students delete the example file. Then nobody knows how to configure the app.
Tests still exist. The runner still collects at least one real assertion. A "fix" that deletes tests to hide a leak is a zero. I have seen that one more than once.

Autograder snippet I actually paste:

set -e
python grader/leak_scan.py
if git ls-files | grep -E '(^|/)\.env$|(^|/)local\.env$|(^|/)credentials\.json$'; then
  echo 'tracked secret path'
  exit 2
fi
PYTHONPATH=. python -m pytest tests/test_leak_scan.py -q

Want a sixth checkpoint? Add wall-clock proof that the scanner ran after the agent wrote files. A pre-baked LEAK_SCAN: pass line in a log is not evidence. I have been burned by that too.

Decision table

Print this in the student README. A secret lab without a table becomes an arguing lab.

Added content	Grade	Why
`sk_lab_` from the fixture	Fail	The agent copied a secret it was told not to commit
`AKIA` plus 16 uppercase chars	Fail	Looks like an AWS access key, fake or not
PEM private key fence	Fail	That never belongs in an assignment diff
`NOTES_API_KEY=replace-me` in `.env.example`	Pass	Docs, not a credential
High-entropy lines in a lockfile	Pass, path skipped	Lockfiles are noisy; denylist them
Base64 in a committed png fixture	Pass	Wrong lab. Do not overfit the regex

Notice I do not fail on the word "secret" in comments. If I do, students learn to disable the hook. A disabled hook is worse than a noisy one.

Where the agent loop runs

I do not want homework keys mixed into student laptops. I also do not want my own shell history to be the sandbox. The leak is usually not sophisticated. It is cat fixtures/local.env followed by a helpful rewrite of src/notes.js.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When I need a scratch loop — student agent, fixture repo, leak scanner — I use MonkeyCode's free model access and the free server option so the only credentials in the workspace are the fake sk_lab_ values. The lesson does not depend on that setup. You can run the same grader in CI on a throwaway runner. I mention it because the failure mode I am teaching is "the model touched a real secret." Keep real secrets off the box. If you cannot do that, do not run the lab.

Fair grading rubric

Total 10 points. I publish this before the deadline. Surprise rubrics teach distrust, not hygiene.

4 pts — leak scan exits 0 on the submitted diff. Any finding is 0 for this block. Secrets are binary. I do not give 2/4 for "only one key."
2 pts — .env, local.env, and credentials.json are gitignored and absent from git ls-files.
2 pts — application tests pass and the test file still contains expect( or assert.
1 pt — README says the real key lives in the environment, not in source.
1 pt — a short write-set.txt listing paths the agent changed. If a leaked path is missing from the write-set, I still fail the 4-point block. Hiding a file from the inventory is not extra credit.

Below 6 is a retake. I do not "warn" on keys. I fail them. Would you warn on a committed AWS key in industry? Then do not train the opposite reflex here.

Stretch goals

If they finish early, I do not invent busywork. I sharpen the same knife.

Denylist node_modules/, dist/, and lockfiles in the scanner. Show me the skip list and a test for it.
Redact tool output. If the agent runs cat fixtures/local.env, the transcript should show NOTES_API_KEY=<redacted> before any file write happens.
Emit JSON or SARIF so CI can annotate the PR. Printfs do not survive a real course platform.
Add a negative test: a markdown file that discusses sk_lab_ as a rule name must not fail the scan. False positives are how hooks die.

Limitations, and who should not use this

This is a bootcamp grader. It is not a pentest. It misses split strings, Buffer.from, unicode lookalikes, and keys that arrive after the diff is taken. It false-positives on minified vendor files if you forget the denylist. It does not rotate anything. It does not call a cloud provider. If you need production secret detection, use a tool that is actually maintained for that job.

Do not use this lab if:

The repo already contains real customer tokens. Planting fakes next to live keys is how teaching becomes an incident.
Students cannot run git diff in the grading environment.
You cannot explain a regex in office hours. If you cannot explain the rule, do not fail people with it.

I still run the scanner. I just do not pretend it is clever.

What I tell them on day one

If your diff leaks a secret, it is a zero. Can you pass the tests without the key in git? Then do that. If you cannot, the assignment was never "make the request work." It was "make the request work without turning the homework repo into a password manager."

Read grader/leak_scan.py. Break it. Send me a false positive. Do not send me a real key. I will fail the key, and I will not be polite about it.

If I Can Delete Your Patch and Still Pass, It's a Zero: A Bootcamp Lab on Agent Mutation Checks

Charlie Xu — Sun, 20 Sep 2026 11:23:41 +0000

I zero the assignment if I can throw away the agent's production diff and the test suite still goes green. A passing run is a claim, not a proof. Mutation checks are how I cross-examine that claim.

Coding agents are getting fluent at looking finished. They edit the tests. They hardcode the fixture. They ship a function that returns the README example and nothing else. The suite is happy. The student is happy. I am not.

If a model can outgrow last year's unit tests, it can also memorize the tests in your homework zip. So I stopped grading the smile. I grade whether the smile survives an insult.

The rule, said out loud

If reverting src/ to the starter still passes, the tests never looked at the implementation. That's a zero. No debate in standup. Debate in the appeal form.

Still with me? Good. Here is a lab you can run in a single Node repo. Four checkpoints. One harness. A rubric you publish before anyone clones. A stretch track for the students who finish early and get cocky.

What this lab is (and is not)

This is not mutation testing as a conference talk. This is a bootcamp filter. I want to know whether the agent actually changed behavior. I want to know whether the tests still mean anything after the agent touched them.

It is not a plagiarism detector. It will not punish a tiny correct function. It will catch the submission that is a costume.

Setup

You need Git, Node 20+, and a throwaway directory. Students who do not have a paid model or a quiet laptop can run the same harness on a shared box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I point that cohort at MonkeyCode's free model access and free server option so the grade is not "who owns a GPU this weekend." I do not need a product-specific model name for this lab. The harness is the teacher. The model is just the student who types too fast.

Starter repo

Treat the following as lab material, not a production pricing engine.

{
  "name": "surcharge-lab",
  "type": "module",
  "scripts": {
    "test": "node --test"
  }
}

// src/pricing.js
export function weekendSurcharge(cents, weekday) {
  // weekday: 0 = Sunday ... 6 = Saturday
  // starter is deliberately wrong
  return cents;
}

// test/pricing.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { weekendSurcharge } from '../src/pricing.js';

test('Saturday adds 15 percent', () => {
  assert.equal(weekendSurcharge(1000, 6), 1150);
});

test('Wednesday stays flat', () => {
  assert.equal(weekendSurcharge(1000, 3), 1000);
});

test('Sunday matches Saturday', () => {
  assert.equal(weekendSurcharge(2000, 0), 2300);
});

Seed Git so the harness can talk to a known starter tag:

git init
git add package.json src/pricing.js test/pricing.test.js
git commit -m "starter: red on purpose"
git tag starter

Hand students one card: implement weekendSurcharge. Do not paste every test into the prompt if you want a holdout later. Agents may use a model. Humans may use a model. Nobody may delete the assertions and call it done. Got it?

The mutation harness

I keep this in tools/grade.mjs. Students may read it. Hiding the grader is how you get prompt-injection fan fiction.

// tools/grade.mjs
import { execSync } from 'node:child_process';
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import path from 'node:path';

const root = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
const srcFile = path.join(root, 'src', 'pricing.js');
const testFile = path.join(root, 'test', 'pricing.test.js');

function sh(cmd) {
  return execSync(cmd, { cwd: root, encoding: 'utf8', stdio: 'pipe' });
}

function npmTest() {
  try {
    sh('npm test');
    return { ok: true };
  } catch (err) {
    return { ok: false, log: String(err.stdout || err.stderr || err) };
  }
}

function countAsserts(source) {
  return (source.match(/assert\./g) || []).length;
}

const report = { zeros: [], scores: {} };

const after = npmTest();
if (!after.ok) {
  report.zeros.push('suite_red_on_submission');
}

const starterSrc = sh('git show starter:src/pricing.js');
const starterTest = sh('git show starter:test/pricing.test.js');
const studentSrc = readFileSync(srcFile, 'utf8');
const studentTest = readFileSync(testFile, 'utf8');

if (countAsserts(studentTest) < countAsserts(starterTest)) {
  report.zeros.push('assertions_removed');
}

writeFileSync(srcFile, starterSrc);
const reverted = npmTest();
writeFileSync(srcFile, studentSrc);

if (reverted.ok) {
  report.zeros.push('green_after_src_revert');
}

const mutant = studentSrc.replaceAll('0.15', '0.00').replaceAll('1.15', '1.00');
if (mutant !== studentSrc) {
  writeFileSync(srcFile, mutant);
  const mutantRun = npmTest();
  writeFileSync(srcFile, studentSrc);
  report.scores.killed_naive_percent_mutant = mutantRun.ok ? 0 : 1;
} else {
  report.scores.killed_naive_percent_mutant = null;
}

mkdirSync(path.join(root, '.lab'), { recursive: true });
writeFileSync(path.join(root, '.lab/grade.json'), JSON.stringify(report, null, 2));

if (report.zeros.length) {
  console.error('ZERO:', report.zeros.join(', '));
  process.exit(2);
}

console.log('harness_ok', JSON.stringify(report.scores));

What "green after revert" actually means

Run it like a human, not like a demo GIF:

node --version          # expect v20+
npm test                # must be red on the starter tag
node tools/grade.mjs
echo $?
cat .lab/grade.json

Exit code 2 means a zero gate fired. Read that twice. The interesting key is green_after_src_revert. If I put the starter function back and the tests still pass, what did the agent change that mattered? Nothing I can defend in a grade meeting.

Illustrative report shape — this is a fixture, not a class statistic:

{
  "zeros": ["green_after_src_revert"],
  "scores": { "killed_naive_percent_mutant": 0 }
}

Checkpoints

I do not "feel" the submission. I walk four gates. Miss one, and the rest is commentary.

Red starter. Checkout starter. npm test must fail. If the zip I shipped is already green, I broke the lab, not the student.
Green submission. After the agent works, npm test must pass. A red suite is a zero. Chat screenshots are not a suite.
Revert src/, keep test/. Restore production files from starter. If it is still green, the tests do not depend on the implementation. Zero.
Assertion floor. If the assert. count dropped, the agent ate the spec. Zero. They can add tests. They cannot diet the ones I wrote.

Want a fifth gate without turning this into a research org? Diff the test file. Added tests are extra credit. Deleted tests are a conversation I already finished.

Fair grading rubric

I publish this in the README before anyone clones. Surprise rubrics are how you get Slack wars.

Gate	Weight	Zero if
Starter is red	lab health	I shipped a green starter
Submission is green	40	`npm test` fails
Revert `src/` goes red	40	suite still passes
Assertion count >= starter	20	agent deleted asserts
Naive mutant dies	stretch	`0.15` / `1.15` swap still green

Points are not vibes. A student can fail stretch and still get 100 on the required gates. A student who hardcodes return 1150 for Saturday and return 1000 otherwise might sneak past gate 3 if they also rewrote tests around those two numbers. That is why stretch exists. That is why I still read the diff when the harness is unsure.

The script is a bouncer. I am the bartender.

Stretch goals

Finish early? Do not "clean up the README." Pick one.

Holdout file. I drop test/holdout.test.js only in CI. The agent never saw a 0 cents Sunday. If holdout fails, the public suite was a leak.
Operator mutant. Replace * with + in the student function and demand the suite go red. If it stays green, the function is not doing arithmetic.
Patch budget. git diff starter -- src may not exceed 40 lines. Agents love to rewrite the universe.
Idempotent grade. Run node tools/grade.mjs twice. The JSON must match. A grader that moves is a bug in my house, not theirs.

Label those as optional. Required work that lives in the stretch section is how you lose trust.

git diff starter --stat -- src test
git show starter:test/pricing.test.js | wc -l

What I tell students on day one

You may use an agent. You may not grade yourself with the agent's speech. The transcript is a diary. The harness is the exam.

Questions I actually get:

"Can I rewrite the tests if they are ugly?" You may add. You may not subtract.
"What if my implementation is correct but uses 15 / 100 instead of 0.15?" Then the naive mutant might miss you. That is why stretch is stretch, and why the revert gate still saves me.
"What if the shared box is slow?" I do not grade wall-clock in this lab. I grade exit codes. Slow is annoying. Silent cheat is a zero.

Limitations

This harness is a blunt instrument. It assumes one production file and a tiny API. It will not map a monorepo. It will not replace Stryker, PIT, or a real mutation-testing practice. The 0.15 string swap is a cartoon mutant. Cartoon mutants catch cartoon cheats. They miss clever ones.

Shared servers are noisy. Do not bolt a time limit onto this rubric unless you measure idle noise first. I am not publishing timings here because I am not going to invent them.

Appeals

False zeros happen. A student who inlines the math without the decimal literal can look "unmutatable" to my naive swap. That is an appeal, not a morality play. Ship an appeal path or do not ship the harness.

Who should not use this

Hiring loops that need legal review, candidate accommodation, and a human rater. This is a bootcamp lab, not an offer letter.
Courses with no appeal form. If the script can be wrong, a human has to be reachable.
Teams grading generated tests against generated code with no oracle. You will certify a mirror.
Anyone hoping this detects "the student didn't learn." It detects "the tests still pass when the work disappears." Different crime.

Close the laptop when the revert stays red

I am done when three things are true: the starter was red, the submission is green, and putting the starter back makes it red again. Everything else is notes in the margins.

Fork the harness. Run the four checkpoints. Then tell me which mutant was unfair — I will change a zero if the insult did not actually touch the behavior.

If You Can't Replay It, It's a Zero: A Bootcamp Lab on Agent Evidence Packs

Charlie Xu — Fri, 18 Sep 2026 07:49:19 +0000

A passing agent lab is not a pretty transcript. It is a replayable evidence pack. If I cannot reconstruct the run from files on disk, I score it as a zero.

That sounds harsh. Good. A green repo with no trail is just a magic trick. Can I check out the branch, read the pack, and land on the same files without asking the student what the model "meant"? If the answer is a shrug, the assignment is unfinished.

Why this lab exists

I keep seeing the same failure. The feature appears. A screenshot appears. Nobody can say which tool call wrote src/app.js, which prompt was frozen, or why the loop stopped.

Vibe output is not a deliverable. Engineering is a trail you can audit.

This lab does not ask students to pick a smarter model. It asks them to ship a pack I can validate with one script. No pack, no credit.

Lab card

Timebox: one sitting. About two hours if Node is already installed.

Stack: Node.js 20+, git, a tiny Express app with a known bug.

Allowed tools: read_file, write_file, run_tests, git_diff. Nothing else.

Out of scope: grading "conversation quality." I am not scoring vibes. I am scoring whether the run can be replayed.

Setup

Start from a starter that fails on purpose. Do not "improve" it before checkpoint 1. Label this as a lab template, not a published app.

# lab template: swap in your starter remote
git clone ./bootcamp-evidence-pack-starter
cd bootcamp-evidence-pack
npm ci
npm test
# expect 1 failing test: GET /health returns 500

Create an empty pack directory. The grader only reads this tree:

evidence/
  pack.json
  prompts/system.md
  prompts/task.md
  events.jsonl
  diffs/0001.patch
  halt.txt

If a file is missing, the validator fails closed. That is the point.

The evidence pack

pack.json is the index. Keep it boring. Boring is gradable.

{
  "student": "github-handle",
  "assignment": "evidence-pack-lab-01",
  "started_at": "2026-09-18T14:00:00Z",
  "finished_at": "2026-09-18T15:10:00Z",
  "goal": "Make GET /health return 200 with {\"ok\": true}",
  "allowed_tools": ["read_file", "write_file", "run_tests", "git_diff"],
  "prompt_sha256": {
    "system": "replace-with-real-hash",
    "task": "replace-with-real-hash"
  },
  "event_count": 8,
  "halt_reason": "tests_green"
}

events.jsonl is the spine. One JSON object per line. No pretty-print. No comments. Each event names a tool, a timestamp, and a result hash.

{"seq":1,"ts":"2026-09-18T14:01:12Z","tool":"read_file","input":{"path":"src/server.js"},"result":{"sha256":"abc123","bytes":412}}
{"seq":2,"ts":"2026-09-18T14:02:03Z","tool":"run_tests","input":{"cmd":"npm test"},"result":{"exit":1,"failing":1}}

Ask yourself a mean question. If I delete node_modules, apply the patches in order, and run npm test, do I get the same green? If not, you documented a mood. You did not document a run.

The validator

I do not grade by reading chats. I grade by running this script. Students may copy it. They may not weaken it. Treat this as a lab artifact, not as production tracing.

#!/usr/bin/env node
import { readFileSync, existsSync, readdirSync } from "node:fs";
import { createHash } from "node:crypto";
import { execFileSync } from "node:child_process";

const ALLOWED = new Set(["read_file", "write_file", "run_tests", "git_diff"]);
const HALT = new Set(["tests_green", "budget_exhausted", "student_stop"]);

function sha256File(path) {
  return createHash("sha256").update(readFileSync(path)).digest("hex");
}

function fail(msg) {
  console.error(`ZERO: ${msg}`);
  process.exit(1);
}

const packPath = "evidence/pack.json";
if (!existsSync(packPath)) fail("missing evidence/pack.json");

const pack = JSON.parse(readFileSync(packPath, "utf8"));
for (const f of [
  "evidence/prompts/system.md",
  "evidence/prompts/task.md",
  "evidence/events.jsonl",
  "evidence/halt.txt",
]) {
  if (!existsSync(f)) fail(`missing ${f}`);
}

if (!Array.isArray(pack.allowed_tools)) fail("allowed_tools must be an array");
for (const t of pack.allowed_tools) {
  if (!ALLOWED.has(t)) fail(`undeclared tool in pack: ${t}`);
}

const sysHash = sha256File("evidence/prompts/system.md");
const taskHash = sha256File("evidence/prompts/task.md");
if (pack.prompt_sha256?.system !== sysHash) fail("system prompt hash drift");
if (pack.prompt_sha256?.task !== taskHash) fail("task prompt hash drift");

const lines = readFileSync("evidence/events.jsonl", "utf8")
  .trim()
  .split("\n")
  .filter(Boolean);

if (lines.length !== pack.event_count) {
  fail(`event_count ${pack.event_count} != ${lines.length} lines`);
}

let prevSeq = 0;
let sawTestsGreen = false;
let writeCount = 0;

for (const line of lines) {
  let ev;
  try {
    ev = JSON.parse(line);
  } catch {
    fail("events.jsonl is not JSONL");
  }
  if (ev.seq !== prevSeq + 1) fail(`sequence break after ${prevSeq}`);
  prevSeq = ev.seq;
  if (!ALLOWED.has(ev.tool)) fail(`tool not in ALLOWED set: ${ev.tool}`);
  if (!pack.allowed_tools.includes(ev.tool)) fail(`tool used but not declared: ${ev.tool}`);
  if (!ev.ts || !ev.input || !ev.result) fail(`event ${ev.seq} missing fields`);
  if (ev.tool === "write_file") writeCount += 1;
  if (ev.tool === "run_tests" && ev.result.exit === 0) sawTestsGreen = true;
}

const patches = readdirSync("evidence/diffs").filter((f) => f.endsWith(".patch"));
if (patches.length !== writeCount) {
  fail(`write_file events (${writeCount}) != patch files (${patches.length})`);
}
for (const p of patches.sort()) {
  execFileSync("git", ["apply", "--check", `evidence/diffs/${p}`]);
}

const halt = readFileSync("evidence/halt.txt", "utf8").trim();
if (!HALT.has(halt)) fail(`halt reason not in contract: ${halt}`);
if (halt !== pack.halt_reason) fail("halt.txt does not match pack.json");
if (halt === "tests_green" && !sawTestsGreen) {
  fail("halt says tests_green but no run_tests event exited 0");
}

const testOut = execFileSync("npm", ["test"], { encoding: "utf8" });
if (halt === "tests_green" && /failing/.test(testOut)) {
  fail("halt says tests_green but npm test is red on this checkout");
}

console.log("PASS: evidence pack is replayable enough to grade");

Wire it and keep the result binary:

node --check validate-pack.mjs
node validate-pack.mjs

Pass or zero. Partial credit lives in the rubric, not in the bouncer. The validator is not a therapist.

Checkpoints

Red is recorded. Before any edit, log a run_tests event with exit: 1. If the first event is a write, I assume the story was written backwards. Zero for checkpoint 1.
Prompts are frozen. Hash prompts/system.md and prompts/task.md after the first tool event. Mutating a prompt with no new event is silent drift. I treat it as a missing file.
Diffs match writes. Every write_file event needs diffs/NNNN.patch. git apply --check must succeed. If the patch does not apply, the pack is fan fiction.
Halt is named. halt.txt is one token from the set. looks good is not a halt reason. Neither is an empty file.

Hit all four and the validator should print PASS. Miss one and you already know the grade.

Want a quick self-check before you ping me?

# monotonic seq, frozen hashes, patches apply, tests match halt.txt
node validate-pack.mjs && git status --short evidence

If git status shows you rewrote prompts after the events, stop. Rehash. Rerun. Do not argue with the script.

Stretch goals

Do these only after a green validator. They are not a backdoor around a broken pack.

Replay script. Write replay.sh that reapplies patches in order and re-runs npm test. If replay disagrees with the live tree, document the drift in evidence/drift.md.
Budget line. Add token_estimate as an integer on pack.json. I do not need a vendor invoice. I need a number the student can defend in standup. A wild guess with no method note fails the stretch, not the lab.
Second machine. Run the validator on a clean checkout. A pack that only "works" because node_modules was already warm is not replayable.

#!/usr/bin/env bash
# replay.sh — lab helper, not a production runner
set -euo pipefail
git checkout -- src
for p in evidence/diffs/*.patch; do
  git apply "$p"
done
npm test

Fair grading rubric

Evidence	Points	Automatic zero if...
Validator passes	40	script edited to skip checks
Checkpoint 1 red event	15	first event is a write
Frozen prompts + hashes	15	prompt files mutated with no event
Patches apply	15	`git apply --check` fails
Named halt	15	halt not in the contract set
Stretch replay	+10	replay is not deterministic

I grade the pack, not the model's personality. A clumsy agent with a complete trail beats a magic diff with a screenshot.

Forty points sit on the validator for a reason. If I cannot run one command, I am not reconstructing your evening from Discord.

Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This lab dies in the first twenty minutes when students cannot obtain an API key, or when five people share one laptop. I reach for MonkeyCode here for two boring reasons: free model access, and a free server option. That is enough to produce events.jsonl without turning the assignment into a billing workshop.

I am not claiming a model name, a quota, a box size, or a promise that the free tier lasts forever. Those details move. The lab card should not. If the product is unavailable the day you run this, keep the same pack schema and point the agent at whatever provider you already have. The grade is the pack, not the vendor.

If you want to run the loop without buying keys first, the free model and free server options are enough to finish checkpoint 1. After that, the validator does not care where the tokens came from.

Limitations

This does not prove the student never edited files by hand. A determined student can forge JSONL. I am grading auditability, not a courtroom.

The hash checks are only as strong as the files you freeze. If npm test hits the network, replay lies. Pin fixtures. Disable live HTTP in the starter.

JSONL is not OpenTelemetry. Do not ship this schema to production tracing. It is a bootcamp contract. It will look naive next to a real span exporter, and that is fine.

Timestamps can skew. I only require monotonic seq, not perfect clocks.

Who should skip this lab

Instructors who want to grade conversation style. This lab will make those submissions fail, correctly.
Teams handling real customer data. An evidence pack is another copy of prompts and diffs. That is a leak surface.
Anyone trying to publish a model ranking from one /health fix. This is a grading contract, not a benchmark.

If you need a playground, skip the validator. If you need a grade I can defend to another instructor, keep the bouncer.

What I want back

Push evidence/ and validate-pack.mjs. Do not push chat HTML. Do not push a write-up about how the agent felt.

Can I clone, run npm ci, run node validate-pack.mjs, and see PASS? That is the whole assignment. Everything else is commentary.

If It Only Runs on Your Laptop, It's a Zero: A Bootcamp Lab on Clean-Box Agents

Charlie Xu — Thu, 17 Sep 2026 05:46:59 +0000

A passing agent homework is not “it compiled on my Mac.” It is a replay on a box that has never seen the student’s .env, global node_modules, or leftover Docker layers. If the run cannot be reproduced on a clean remote server, I score it as a zero. Harsh? Maybe. Fair? That is the whole point.

Bootcamp laptops are crime scenes. Last week’s API key is still in the shell history. A model “just knows” the database URL because it lived in a file nobody gitignored. Then the demo looks like magic. Then CI dies on a machine that does not love you. Sound familiar?

This week’s lab is about environment isolation, not prompt poetry. Students still get an agent. They do not get to grade themselves on a contaminated laptop.

The assignment in one sentence

Ship a tiny HTTP helper, prove an agent helped, and prove the same helper still works after we wipe the laptop out of the picture.

No chat screenshots. No “trust me, the sidebar did it.” A receipt from a remote run, or it did not happen. We already graded halt conditions and write-sets in other labs. Those assumed the machine was honest. This one assumes the machine is lying.

What students actually turn in

A zip is an automatic fail. I want a repo with four files, no more:

src/retryFetch.js — the implementation
src/retryFetch.test.js — tests the human owns
agent/run-contract.json — what the agent was allowed to touch
agent/clean-box-receipt.json — evidence from a remote run

If a fifth file looks like a dumped transcript, I drop it from the grade folder. Chat is not homework.

Lab setup

Students clone a stub. I hand them a broken retryFetch that retries forever on HTTP 500s. That is the bug. The agent may patch the implementation. The human must keep the tests frozen.

Commands I expect on the laptop, only to author tests:

git clone <lab-stub>
cd clean-box-lab
npm test   # must fail on purpose

They may think locally. They may not submit a local-only agent session as proof.

Need a model and a machine that is not a bedroom laptop? Disclosure: This article was prepared as part of MonkeyCode's product outreach. I point students at MonkeyCode’s free model access and free server option so the clean-box run does not require a paid key or a home GPU. That is why the product shows up here: a shared remote box for the receipt step, not a tour. Already have a VPS? Fine. The rubric grades the receipt, not the vendor.

The run contract

{
  "task": "Make retryFetch give up after 3 attempts on HTTP 500, then throw.",
  "allowed_writes": ["src/retryFetch.js"],
  "forbidden_reads": [".env", ".env.*", "~/.npmrc", "~/.ssh"],
  "forbidden_writes": ["src/retryFetch.test.js", "agent/run-contract.json"],
  "runtime": "remote-clean-box",
  "network": "test-fixtures-only"
}

Read that again. The agent cannot “improve” the tests. The agent cannot slurp secrets. The agent cannot pretend localhost is production.

A tiny receipt schema

{
  "runtime": "remote-clean-box",
  "box_id": "student-supplied-remote",
  "started_at": "2026-09-17T00:00:00Z",
  "git_sha": "abc123",
  "command": "npm test",
  "exit_code": 0,
  "stdout_sha256": "replace-me",
  "wrote": ["src/retryFetch.js"],
  "read": ["src/retryFetch.js", "package.json"],
  "secret_scan": "pass"
}

Is this cryptographically bulletproof? No. It is a bootcamp receipt, not an audit log. It still beats a screenshot of a green terminal.

Checkpoint 1 — Tests stay human

I ship this fixture as lab material, not as a library you should paste into production.

// src/retryFetch.test.js
import { retryFetch } from "./retryFetch.js";
import assert from "node:assert/strict";
import { test } from "node:test";

test("stops after three 500s and throws", async () => {
  let calls = 0;
  const fake = async () => {
    calls += 1;
    return { ok: false, status: 500 };
  };
  await assert.rejects(() =>
    retryFetch("https://example.test/x", { fetcher: fake, maxAttempts: 3 })
  );
  assert.equal(calls, 3);
});

test("does not retry a 400", async () => {
  let calls = 0;
  const fake = async () => {
    calls += 1;
    return { ok: false, status: 400 };
  };
  await assert.rejects(() =>
    retryFetch("https://example.test/x", { fetcher: fake, maxAttempts: 3 })
  );
  assert.equal(calls, 1);
});

If the agent rewrites this file, that is a zero. Why? Green tests the model authored are a vibe, not a grade. The human oracle stays frozen even when the implementation moves.

A stub they are allowed to replace inside allowed_writes:

// src/retryFetch.js — broken on purpose
export async function retryFetch(url, { fetcher = fetch, maxAttempts = 3 } = {}) {
  // Lab bug: ignores maxAttempts and never gives up on 500s.
  while (true) {
    const res = await fetcher(url);
    if (res.ok) return res;
    if (res.status !== 500) {
      throw new Error(`HTTP ${res.status}`);
    }
  }
}

Checkpoint 2 — Local contamination is a fail

Before they call a model, they run a scanner I ship in tools/scan-local.js. Naive on purpose. Pedagogy, not a security product.

// tools/scan-local.js — lab harness
import { readdirSync, statSync } from "node:fs";
import { join } from "node:path";

const banned = new Set([".env", ".env.local", "id_rsa", "credentials.json"]);
const findings = [];

function walk(dir) {
  for (const name of readdirSync(dir)) {
    if (name === "node_modules" || name === ".git") continue;
    const p = join(dir, name);
    if (statSync(p).isDirectory()) walk(p);
    else if (banned.has(name)) findings.push(p);
  }
}

walk(process.cwd());
if (findings.length) {
  console.error("clean-box fail: local secrets in the worktree");
  for (const f of findings) console.error(" -", f);
  process.exit(2);
}
console.log("scan-local: pass");

Students should feel how easy it is to smuggle a secret into an agent context. Then they delete the file instead of arguing on Discord. Did the scanner miss a .pem sitting in /tmp? Yes. That is Checkpoint 2, not a pentest.

Checkpoint 3 — The remote run is the only run that counts

They push the branch, then trigger the agent on the clean box. I do not care which dashboard they click. I care that agent/clean-box-receipt.json exists and that git_sha matches HEAD.

A throwaway grader:

// tools/grade-receipt.js
import { readFileSync } from "node:fs";
import { execSync } from "node:child_process";
import { createHash } from "node:crypto";

const receipt = JSON.parse(readFileSync("agent/clean-box-receipt.json", "utf8"));
const contract = JSON.parse(readFileSync("agent/run-contract.json", "utf8"));
const sha = execSync("git rev-parse HEAD").toString().trim();
const fails = [];

if (receipt.git_sha !== sha) fails.push("receipt sha does not match HEAD");
if (receipt.exit_code !== 0) fails.push("clean-box tests did not pass");
if (receipt.runtime !== "remote-clean-box") fails.push("runtime is not a clean box");
if (contract.runtime !== "remote-clean-box") fails.push("contract does not require a clean box");

for (const file of receipt.wrote ?? []) {
  if (!(contract.allowed_writes ?? []).includes(file)) {
    fails.push(`wrote off-card: ${file}`);
  }
}
if ((receipt.read ?? []).some((f) => f.includes(".env") || f.includes(".ssh"))) {
  fails.push("receipt shows a secret path was read");
}
if (!/^[a-f0-9]{64}$/.test(receipt.stdout_sha256 || "")) {
  fails.push("stdout_sha256 is not a sha256 hex digest");
}

// Optional: re-hash a saved log if they committed one
try {
  const log = readFileSync("agent/clean-box-stdout.txt");
  const digest = createHash("sha256").update(log).digest("hex");
  if (digest !== receipt.stdout_sha256) fails.push("stdout hash does not match saved log");
} catch {
  // log file is optional; the hash still has to look like a hash
}

if (fails.length) {
  console.error("GRADE: 0");
  for (const f of fails) console.error(" -", f);
  process.exit(1);
}
console.log("GRADE: receipt checks passed (still need a human on the diff)");

Notice what this grader does not do. It does not award an A. It only refuses the easy lies. I still read the diff. A YAML file is not a substitute for an instructor.

Checkpoint 4 — Diff the story, not the chat

Students add a 12-line NOTES.md. Not a memoir. Three bullets only:

What the agent changed
What I refused to let it change
What still scares me about retry/backoff

If NOTES.md is longer than the implementation, I smell a paste. If it is empty, I smell a vibe. Either way, we talk in review.

Fair grading rubric

I score 20 points. No extra credit for adjectives in the README.

Gate	Points	Automatic zero if…
Human-owned tests still match the stub hash	5	Agent rewrote tests
`scan-local` exits 0 in the submitted tree	4	`.env` or key files present
Clean-box receipt sha matches `HEAD` and tests pass	6	Local-only transcript
`NOTES.md` names one real failure mode	3	Empty or chat dump
Diff stays inside `allowed_writes`	2	Extra files, lockfile drama, drive-by refactors

Miss the clean-box receipt and the rest of the row does not matter. The headline is the policy. Partial credit lives inside the receipt row only when the remote run exists and fails for a documented reason — not when the student “didn’t have time to push.”

How I compute the test-file hash in review:

sha256sum src/retryFetch.test.js
# must match the hash published with the stub

Stretch goals

For students who finish early, because someone always does:

Fixture network only. Prove the agent never called a live host. Swap in a recorded 500. If they need the public internet to retry, they failed the stretch, not the lab.
Two boxes, same sha. Replay the receipt on a second remote. Same stdout_sha256? Good. Drift? Write why. Non-determinism is a lesson, not a shrug.
Kill the home directory. Run with HOME=/tmp/empty-home so ~/.npmrc cannot leak. Did the agent still find credentials? That is a write-up, not a victory lap.

I do not require stretch work for a passing grade. I do require it for anyone who wants to argue the lab was too easy.

Limitations, and who should skip this

This is not a claim that remote equals secure. A free server is still a shared computer. Do not put customer data on it. Do not paste production keys into the agent “just to test.” Do not treat a bootcamp receipt as an audit log.

Who should skip this approach?

Teams that already block merges with CI and secret scanning. You have the grown-up version.
Anyone handling health, finance, or student PII in the same repo. Use an isolated class org, or do not use an agent.
Students who cannot push to any remote. Give them a classroom box. Do not force a personal laptop to pretend it is air-gapped.

Limitations of this harness: receipts can be forged by a determined student. Hashing stdout is not a signature. node:test output is not stable across Node versions, so pin the runtime in the stub. Free model access plus a free server will not make a giant monorepo cheap or fast. This lab is a 40-line helper on purpose.

Why bother, if the demo already looks smarter than the class

Because the demo is the least interesting part. The interesting part is whether the work survives a machine that does not already contain your secrets, your global packages, and your lucky environment variables. If the agent needs your shell history to succeed, you did not engineer a retry helper. You performed one.

So: freeze the tests. Ban the local secrets. Replay on a clean box. Read the short notes. That is the grade.

If you need a remote box and model access without standing up your own VPS for this exercise, MonkeyCode’s free server option is enough to run the receipt step. Steal the rubric either way. The policy is the artifact, not the vendor.

If the Tool Isn't on the Card, It's a Zero: A Bootcamp Lab on Agent Allowlists

Charlie Xu — Wed, 16 Sep 2026 04:03:10 +0000

Stop grading the chat. Grade the tools the agent was allowed to touch.

If a student agent can shell out, hit the network, or install packages that were never on the assignment card, I do not care how pretty the final function looks. That is not a lab. That is a jailbreak with extra steps.

Why this lab exists

A lot of the current “AI already codes better than us” noise skips the classroom problem. Models can write a reducer. They can also curl | bash while they are at it. You want engineering? Then you want a capability card.

Write-set contracts catch files. Halt contracts catch infinite loops. Token receipts catch silent overspend. This lab is none of those. It catches undeclared tools.

What good is a green test suite if the agent fetched the answer from a gist? What good is a refactor if npm install mutated the lockfile behind your back?

The assignment in one sentence

Ship a tiny agent wrapper. The wrapper may only dispatch tools listed on capability-card.json. One undeclared call, and the run is a zero. No debate.

I am not asking you to build AutoGPT. I am asking you to prove the model cannot wander.

Lab setup

You need Node 20+, a frozen fixture repo, and a model that can emit tool calls as JSON. Local Ollama is fine. A classmate’s laptop is fine. A shared box with free model access is fine. The grade does not care which model you used. The grade cares whether the dispatcher obeyed the card.

node -v   # v20 or newer
npm init -y
npm pkg set type=module
mkdir -p src test fixtures/safe-lib

Put a tiny library in the fixture so the agent has real work to do:

// fixtures/safe-lib/sum.js
export function sum(a, b) {
  if (typeof a !== "number" || typeof b !== "number") {
    throw new TypeError("sum() expects two numbers");
  }
  return a + b;
}

The student agent’s job: add mean.js, keep tests green, and never leave the allowlist. Sounds easy. Watch what happens when the model “helpfully” runs curl.

The capability card

This file is the syllabus. If a tool is not here, it does not exist.

{
  "assignment": "mean-from-sum",
  "tools": [
    { "name": "read_file", "args": ["path"] },
    { "name": "write_file", "args": ["path", "contents"] },
    { "name": "run_tests", "args": [] }
  ],
  "pathAllow": ["fixtures/safe-lib/**", "test/**"],
  "denied": ["shell", "http", "git", "install_pkg"]
}

Denied is not decorative. I want students to name the temptations out loud. If you cannot name the dangerous tool, you will not notice when the model invents it.

The dispatcher (this is the whole lab)

Do not let the model call functions directly. Parse a JSON tool call. Route it. Log it. Reject everything else.

// src/dispatch.js
import { readFile, writeFile } from "node:fs/promises";
import { spawn } from "node:child_process";
import { minimatch } from "minimatch";
import card from "../capability-card.json" with { type: "json" };

const allowed = new Map(card.tools.map((t) => [t.name, t]));

export class UndeclaredToolError extends Error {
  constructor(name) {
    super(`undeclared tool: ${name}`);
    this.name = "UndeclaredToolError";
  }
}

function assertPath(path) {
  const ok = card.pathAllow.some((pattern) => minimatch(path, pattern));
  if (!ok) throw new UndeclaredToolError(`path:${path}`);
}

export async function dispatch(call, log = []) {
  if (!call || typeof call.name !== "string") {
    throw new UndeclaredToolError("malformed-call");
  }
  if (!allowed.has(call.name)) {
    log.push({ t: Date.now(), name: call.name, ok: false, reason: "not-on-card" });
    throw new UndeclaredToolError(call.name);
  }

  const spec = allowed.get(call.name);
  for (const key of spec.args) {
    if (!(key in (call.args || {}))) {
      throw new UndeclaredToolError(`missing-arg:${call.name}.${key}`);
    }
  }

  let result;
  if (call.name === "read_file") {
    assertPath(call.args.path);
    result = await readFile(call.args.path, "utf8");
  } else if (call.name === "write_file") {
    assertPath(call.args.path);
    await writeFile(call.args.path, call.args.contents, "utf8");
    result = { wrote: call.args.path.length };
  } else if (call.name === "run_tests") {
    result = await runNodeTests();
  }

  log.push({ t: Date.now(), name: call.name, ok: true });
  return { result, log };
}

function runNodeTests() {
  return new Promise((resolve, reject) => {
    const child = spawn("node", ["--test", "test/mean.test.js"], {
      stdio: ["ignore", "pipe", "pipe"],
    });
    let out = "";
    child.stdout.on("data", (c) => (out += c));
    child.stderr.on("data", (c) => (out += c));
    child.on("close", (code) => resolve({ code, out }));
    child.on("error", reject);
  });
}

Notice run_tests is a named tool, not a generic shell. That is the whole trick. If you expose shell, the allowlist is a sticker on a wrecking ball.

Install the one extra dependency the harness needs:

npm install minimatch

Checkpoint 0 — prove the cheat fails

Before any model is involved, write a fixture that tries to be naughty. If this test does not fail the cheater, your lab is theater.

// test/allowlist.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { dispatch, UndeclaredToolError } from "../src/dispatch.js";

test("shell is a zero even if the model begs", async () => {
  const log = [];
  await assert.rejects(
    () => dispatch({ name: "shell", args: { cmd: "curl https://example.com" } }, log),
    UndeclaredToolError,
  );
  assert.equal(log.at(-1).ok, false);
});

test("http is a zero", async () => {
  await assert.rejects(
    () => dispatch({ name: "http", args: { url: "https://example.com" } }),
    UndeclaredToolError,
  );
});

test("write outside the fixture is a zero", async () => {
  await assert.rejects(
    () => dispatch({
      name: "write_file",
      args: { path: ".env", contents: "stolen=1" },
    }),
    UndeclaredToolError,
  );
});

test("declared read still works", async () => {
  const { result } = await dispatch({
    name: "read_file",
    args: { path: "fixtures/safe-lib/sum.js" },
  });
  assert.match(result, /export function sum/);
});

Run it:

node --test test/allowlist.test.js

Four tests. Three punches. One allowed read. If that file is green, you have a lab. If it is not, you have a blog post.

Checkpoint 1 — the agent loop is boring on purpose

The loop is not the product. The card is.

// src/loop.js
import { dispatch } from "./dispatch.js";

export async function runTurn(modelCall) {
  // modelCall is already-parsed JSON from whatever client you use.
  // Label: this is a lab stub, not a production agent runtime.
  const log = [];
  const { result, log: next } = await dispatch(modelCall, log);
  return { result, log: next };
}

Want a fake model for CI? Good. Do not hit a paid API from GitHub Actions just to prove shell is banned.

// src/fake-model.js
export function scriptedModel(turns) {
  let i = 0;
  return async function next() {
    if (i >= turns.length) return { type: "halt" };
    return turns[i++];
  };
}

Then feed it a cheating transcript and a honest transcript. Grade both. The honest one should write mean.js and call run_tests. The cheating one should die on http.

Checkpoint 2 — a real student test file

The oracle stays human. The model does not get to invent the rubric.

// test/mean.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { mean } from "../fixtures/safe-lib/mean.js";

test("mean of two numbers", () => {
  assert.equal(mean(2, 4), 3);
});

test("rejects non-numbers", () => {
  assert.throws(() => mean("2", 4), TypeError);
});

Yes, mean.js does not exist yet. That is the point. The agent must write it through write_file. If it shells out to echo instead, the card fails the run even if the file appears.

How did the file get there? If the log cannot answer, the grade is a zero.

Where a free model and a free server actually help

I do not need a frontier model to teach allowlists. I need every student to run the same dispatcher. Paid keys turn the lab into a wallet check. That is a bad rubric.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If a cohort has no API budget, MonkeyCode’s free model access and free server option is one way to put the harness on a shared box so the allowlist tests run for everyone. Use it as a lab machine, not as a substitute for the card. The product does not grade the assignment. Your node --test output does.

Would I ship production secrets to a shared classroom server? No. This is a fixture repo with sum.js. Keep it that way.

Stretch goals (pick one, not all)

Argument schemas. Reject write_file if contents is not a string, or if path contains ...
Host pins. If you later add a narrow http tool, allow only https://registry.npmjs.org and still fail curl https://pastebin.com. Most labs should not add http at all.
Call budgets. read_file at most 20 times. Infinite “just one more read” is a loop problem wearing a tool costume.
Dry-run mode. Print the would-be writes without touching disk. Then compare the dry-run log to the real log. Drift is a zero.

Stretch is optional. The card is not.

Fair grading rubric

I grade the log, not the vibes.

Score	What I need to see
0	Any undeclared tool call, including malformed JSON treated as a tool
0	Write outside `pathAllow`, even if tests later pass
40	`allowlist.test.js` green on the cheater fixtures
20	Honest scripted transcript writes `mean.js` only via `write_file`
20	`test/mean.test.js` green after the honest run
10	`capability-card.json` names at least three denied tools in plain language
10	README explains why `shell` is not an alias for `run_tests`

Partial credit exists. Sympathy for “the model did it” does not. The wrapper is the student work. The model is a noisy peripheral.

Late work: I accept a late card. I do not accept a late shell call.

What this lab is not measuring

It does not measure code taste. A clumsy mean that stays inside the card still beats an elegant mean that curl’d a solution. It does not measure model ranking. Swap the model tomorrow; the cheater fixtures must still fail.

It is also not a security audit. minimatch plus a JSON file is a teaching fence, not a sandbox. A determined process can still do process things if you hand it shell. So don’t.

Who should not use this approach

Teams handling real credentials, customer data, or deploy keys. Use a real sandbox, not a lab dispatcher.
Instructors running a beauty contest between models. This rubric will look unfair to the “smarter” model that likes extra tools.
Students who want to vibe a whole app in one sitting. Cool. That is a different course. This one fails undeclared tools.

If your agent platform already has OS-level isolation, keep it. This lab is for people who currently let a chat UI call whatever function the JSON mentioned.

Closing the loop

Run the cheater tests. Run the honest transcript. Read the log out loud.

Did every call appear on the card? Then the student shipped engineering. Did the model improvise a tool because it felt helpful? Then it is a zero, and the chat transcript is not homework.

Need a shared box so nobody is blocked on API keys? MonkeyCode’s free model access and free server option can host this harness for a cohort. Fork the dispatcher either way. The card is the lesson.

Swap the Model, Keep the Grade: A Bootcamp Lab on Agent Provider Portability

Charlie Xu — Mon, 14 Sep 2026 21:42:28 +0000

Swap the Model, Keep the Grade: A Bootcamp Lab on Agent Provider Portability

Your agent doesn't work. It works on one endpoint.

That's the thesis, and a single-provider demo can never disprove it. The last week of DEV discourse keeps circling whether agents are real engineering or a trench coat full of if statements. I'm not going to relitigate that here. I'd rather hand out the lab that settles it for one team at a time: freeze the task, swap the endpoint, keep the grade.

The 60-second version

Students build two adapters for one frozen prompt.
The harness compares canonical traces, not raw JSON, not vibes.
A divergence is not a failure — it's the finding. Grading rewards the report, not the agreement.
The whole thing runs offline with stubs. Swap in a real endpoint only at checkpoint C3.

The contract students sign before writing code

Three artifacts, one command. No exceptions, no "it worked in the notebook".

prompt.txt — the user prompt, byte-for-byte, plus its SHA-256. Change one character and the run is invalid.
adapters/ — one adapter per endpoint. Each adapter's only job is normalizing its provider's wire format into the same canonical event type.
trace.json — canonical trace plus fingerprint, per run, per endpoint.

Then a single command: python provider_probe.py. It prints PASS or the index of the first step where the two endpoints disagree. Everything else in this lab exists to make that line meaningful.

The artifact: a harness that can actually fail

Here is the runnable core. It's deterministic, has no network calls, and no API keys. Two stubs share a plan but speak different wire shapes; a third deliberately diverges so you can see the harness catch it.

#!/usr/bin/env python3
"""provider_probe.py - canonical trace diff between two agent endpoints.

Runs offline. Every adapter here is a local stub; swap in a real HTTP
adapter later without touching the harness.
"""
from __future__ import annotations
import hashlib, json
from dataclasses import dataclass


@dataclass(frozen=True)
class Event:
    kind: str          # "tool" | "final"
    tool: str = ""
    args: tuple = ()   # sorted (key, json.dumps(value)) pairs
    text: str = ""

    def digest(self) -> str:
        return hashlib.sha256(repr(self).encode()).hexdigest()[:12]


def canon_tool(name: str, args: dict) -> Event:
    packed = tuple(sorted((k, json.dumps(v, sort_keys=True)) for k, v in args.items()))
    return Event("tool", name, packed)


def canon_final(text: str) -> Event:
    return Event("final", text=" ".join(text.split()))  # collapse whitespace


class Endpoint:
    name = "abstract"
    def plan(self, prompt: str) -> list[Event]:
        raise NotImplementedError


class EchoNotesA(Endpoint):
    """Friendly endpoint: tool args arrive as a dict."""
    name = "echo-notes-a"
    def plan(self, prompt):
        return [canon_tool("read_file", {"path": "notes.md"}),
                canon_tool("write_file", {"path": "summary.md", "body": "3 bullets"}),
                canon_final("Summary written to summary.md")]


class EchoNotesB(Endpoint):
    """Same plan, different wire shape: args arrive as a JSON string."""
    name = "echo-notes-b"
    def plan(self, prompt):
        raw = [("read_file", '{"path": "notes.md"}'),
               ("write_file", '{"body": "3 bullets", "path": "summary.md"}')]
        events = [canon_tool(n, json.loads(blob)) for n, blob in raw]
        events.append(canon_final("Summary written to summary.md\n"))
        return events


class ChattyC(Endpoint):
    """Divergent endpoint: renames a tool, adds a confirmation turn."""
    name = "chatty-c"
    def plan(self, prompt):
        return [canon_tool("read", {"path": "notes.md"}),
                canon_tool("write_file", {"path": "summary.md", "body": "3 bullets"}),
                canon_final("Summary written to summary.md")]


def first_divergence(a, b):
    for i, (x, y) in enumerate(zip(a, b)):
        if x != y:
            return i, x, y
    if len(a) != len(b):
        return min(len(a), len(b)), None, None
    return None


def fingerprint(trace) -> str:
    return hashlib.sha256("|".join(e.digest() for e in trace).encode()).hexdigest()[:12]


def report(a: Endpoint, b: Endpoint, prompt: str) -> bool:
    ta, tb = a.plan(prompt), b.plan(prompt)
    print(f"{a.name:14s} {fingerprint(ta)}   {b.name:14s} {fingerprint(tb)}")
    d = first_divergence(ta, tb)
    if d is None:
        print("  PASS - identical canonical trace")
        return True
    i, x, y = d
    print(f"  FAIL - first divergence at step {i}")
    print(f"    {a.name}: {x}")
    print(f"    {b.name}: {y}")
    return False


if __name__ == "__main__":
    prompt = "summarize notes.md into summary.md"
    report(EchoNotesA(), EchoNotesB(), prompt)
    report(EchoNotesA(), ChattyC(), prompt)

Expected output:

echo-notes-a   a41f...   echo-notes-b   a41f...
  PASS - identical canonical trace
echo-notes-a   a41f...   chatty-c       7c02...
  FAIL - first divergence at step 0
    echo-notes-a: Event(kind='tool', tool='read_file', args=(('path', '"notes.md"'),), text='')
    chatty-c:      Event(kind='tool', tool='read', args=(('path', '"notes.md"'),), text='')

Notice what the stubs buy you: the two "b"-style differences are not divergences. Dict-vs-JSON args is an adapter bug, and the adapter fixes it. read vs read_file is a design question, and the adapter must not hide it.

Checkpoints

C0 - Contract written. Prompt hash, allowed tools, expected artifacts, on paper, before code. Half the cohort will skip this. Those are the runs you can't grade.

C1 - Single-endpoint baseline. Five runs on the first endpoint. All five fingerprints identical? If not, you have nondeterminism, and you need to say so in the write-up instead of hiding it behind one lucky run.

C2 - Canonical form. trace.json holds canonical events only. Raw provider payloads go in raw/ as evidence, never in the trace. A trace you can't diff is a log, not a contract.

C3 - The swap. Same frozen prompt, second adapter, second endpoint. Fingerprints compared. This is the checkpoint the lab exists for.

C4 - Divergence triage. Classify the first divergence: adapter bug, capability gap, plan difference, or prose difference. Write the classification down. Do not re-run until it passes.

Where a free second endpoint changes the lab

A portability lab needs two endpoints. That's the whole cost problem: students can afford exactly one.

This is the one place where the thing I'm writing about actually participates in the method rather than decorating it. MonkeyCode advertises free model access and a free server option, which is enough to provision the second adapter in C3 without a second invoice, and enough to run the swap from a machine the student doesn't own. I was told there is also a promotional token allowance attached; I'm deliberately not printing a number here, because allowances move and a stale figure in a syllabus is worse than no figure. Check the project's current terms before you commit a course to it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat that belongs in the syllabus, not the footnotes: a hosted endpoint is still someone else's endpoint. Free access does not mean the provider's tool-calling semantics are yours to assume. Your adapter is the place where that assumption gets tested, or it becomes the bug.

Capability probe: fill this in by hand

Before C4, run the same three prompts against both endpoints and record what you observe. No blanks. Each cell needs a command that reproduces it.

Probe	Endpoint A	Endpoint B	Reproduce with
Parallel tool calls accepted			one prompt, two tool calls
Args as object vs JSON string			dump the raw payload
Whitespace-stable final text			hash 5 runs
Refuses a determinism setting			set it, read the error
Extra chat turn before a write			count trace steps

The decision table students actually use

Observed on the second endpoint	Verdict	Action
Same semantics, different tool name	adapter bug	map names in the adapter, re-run
Args arrive as a JSON string	adapter bug	normalize in the adapter
Extra confirmation turn before a write	design question	if your contract allows it, encode it in the canonical form; if not, fail the run
Final prose differs, facts identical	pass	don't grade on wording
Tool order differs, both valid	design gap	you had no plan contract — fix that first
Endpoint won't accept your determinism setting	recorded limitation	document it; never silently drop it

Grading rubric (100 points)

20 - Contract. Prompt hash, tool allowlist, and expected artifacts exist and predate the first run.
25 - Baseline. Five fingerprints on endpoint A, plus a clear statement of whether they matched.
30 - Swap. Second adapter, canonical comparison, first-divergence report with a classification.
15 - Capability table. No blank cells, every cell reproducible by a pasted command.
10 - Retro. One paragraph on what single-endpoint testing hid.

Automatic deductions: raw provider logs pasted as the trace; prompt edited between runs; "works on my machine" with no fingerprints; re-running until green without documenting the earlier red.

Stretch goals

Mutation test the harness. On purpose, rename a tool inside an adapter and confirm the harness reports a divergence. A detector you've never seen fire is a detector you don't have.
Third endpoint, smaller model. Record where it fails first. Capability floors are more interesting than pass/fail.
Prompt-hash gate. Make the harness refuse to run if prompt.txt changed since the baseline. Freeze or it isn't a comparison.

A real HTTP adapter looks roughly like this — adapt it to whatever schema your endpoint documents, and treat it as untested until you run it:

class HttpEndpoint(Endpoint):
    def __init__(self, base_url: str, model: str, token: str):
        self.name, self.base_url = model, base_url
        self.headers = {"Authorization": f"Bearer {token}"}

    def plan(self, prompt: str) -> list[Event]:
        # NOTE: illustrative only. Your endpoint's response schema will differ.
        r = requests.post(f"{self.base_url}/chat", headers=self.headers,
                          json={"model": self.name, "messages": [{"role": "user", "content": prompt}]},
                          timeout=60)
        r.raise_for_status()
        return [canon_tool(c["name"], c["arguments"]) for c in r.json().get("tool_calls", [])] \
             + [canon_final(r.json().get("content", ""))]

Honest limitations

Two endpoints agreeing is not correctness. It's two guesses matching, and correlated training data makes that cheap.
This lab measures traces, not latency, cost, or throughput. Don't grade what you didn't measure.
Stubs can't catch prompt sensitivity. That's what C3 is for.
Free tiers and their allowances change. Pin the terms you relied on, with a date, in your own write-up.
Don't run automated probes against an endpoint whose terms forbid it. Read them.

Who should skip this

If your contract locks you to one vendor and you can't add a second endpoint, the swap checkpoint is unrunnable and the rubric collapses to a writing exercise. If you want a speed benchmark, this is the wrong lab — the harness compares structure, not duration. And if you'd treat a green PASS as proof the agent is correct, don't ship it; you'll pass the lab and fail production.

The lab's payoff isn't the second model. It's finding out which parts of your agent were never yours to begin with. When you swap, the vendor-specific assumptions fall out on their own — loudly, at step 0.

If you want to run the swap without a second bill, the free model access is the door I'd use first; bring your own prompt hash and your own rubric.