DEV Community: Charlie Xu

If I Can Delete Your Patch and Still Pass, It's a Zero: A Bootcamp Lab on Agent Mutation Checks

Charlie Xu — Sun, 20 Sep 2026 11:23:41 +0000

I zero the assignment if I can throw away the agent's production diff and the test suite still goes green. A passing run is a claim, not a proof. Mutation checks are how I cross-examine that claim.

Coding agents are getting fluent at looking finished. They edit the tests. They hardcode the fixture. They ship a function that returns the README example and nothing else. The suite is happy. The student is happy. I am not.

If a model can outgrow last year's unit tests, it can also memorize the tests in your homework zip. So I stopped grading the smile. I grade whether the smile survives an insult.

The rule, said out loud

If reverting src/ to the starter still passes, the tests never looked at the implementation. That's a zero. No debate in standup. Debate in the appeal form.

Still with me? Good. Here is a lab you can run in a single Node repo. Four checkpoints. One harness. A rubric you publish before anyone clones. A stretch track for the students who finish early and get cocky.

What this lab is (and is not)

This is not mutation testing as a conference talk. This is a bootcamp filter. I want to know whether the agent actually changed behavior. I want to know whether the tests still mean anything after the agent touched them.

It is not a plagiarism detector. It will not punish a tiny correct function. It will catch the submission that is a costume.

Setup

You need Git, Node 20+, and a throwaway directory. Students who do not have a paid model or a quiet laptop can run the same harness on a shared box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I point that cohort at MonkeyCode's free model access and free server option so the grade is not "who owns a GPU this weekend." I do not need a product-specific model name for this lab. The harness is the teacher. The model is just the student who types too fast.

Starter repo

Treat the following as lab material, not a production pricing engine.

{
  "name": "surcharge-lab",
  "type": "module",
  "scripts": {
    "test": "node --test"
  }
}

// src/pricing.js
export function weekendSurcharge(cents, weekday) {
  // weekday: 0 = Sunday ... 6 = Saturday
  // starter is deliberately wrong
  return cents;
}

// test/pricing.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { weekendSurcharge } from '../src/pricing.js';

test('Saturday adds 15 percent', () => {
  assert.equal(weekendSurcharge(1000, 6), 1150);
});

test('Wednesday stays flat', () => {
  assert.equal(weekendSurcharge(1000, 3), 1000);
});

test('Sunday matches Saturday', () => {
  assert.equal(weekendSurcharge(2000, 0), 2300);
});

Seed Git so the harness can talk to a known starter tag:

git init
git add package.json src/pricing.js test/pricing.test.js
git commit -m "starter: red on purpose"
git tag starter

Hand students one card: implement weekendSurcharge. Do not paste every test into the prompt if you want a holdout later. Agents may use a model. Humans may use a model. Nobody may delete the assertions and call it done. Got it?

The mutation harness

I keep this in tools/grade.mjs. Students may read it. Hiding the grader is how you get prompt-injection fan fiction.

// tools/grade.mjs
import { execSync } from 'node:child_process';
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import path from 'node:path';

const root = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
const srcFile = path.join(root, 'src', 'pricing.js');
const testFile = path.join(root, 'test', 'pricing.test.js');

function sh(cmd) {
  return execSync(cmd, { cwd: root, encoding: 'utf8', stdio: 'pipe' });
}

function npmTest() {
  try {
    sh('npm test');
    return { ok: true };
  } catch (err) {
    return { ok: false, log: String(err.stdout || err.stderr || err) };
  }
}

function countAsserts(source) {
  return (source.match(/assert\./g) || []).length;
}

const report = { zeros: [], scores: {} };

const after = npmTest();
if (!after.ok) {
  report.zeros.push('suite_red_on_submission');
}

const starterSrc = sh('git show starter:src/pricing.js');
const starterTest = sh('git show starter:test/pricing.test.js');
const studentSrc = readFileSync(srcFile, 'utf8');
const studentTest = readFileSync(testFile, 'utf8');

if (countAsserts(studentTest) < countAsserts(starterTest)) {
  report.zeros.push('assertions_removed');
}

writeFileSync(srcFile, starterSrc);
const reverted = npmTest();
writeFileSync(srcFile, studentSrc);

if (reverted.ok) {
  report.zeros.push('green_after_src_revert');
}

const mutant = studentSrc.replaceAll('0.15', '0.00').replaceAll('1.15', '1.00');
if (mutant !== studentSrc) {
  writeFileSync(srcFile, mutant);
  const mutantRun = npmTest();
  writeFileSync(srcFile, studentSrc);
  report.scores.killed_naive_percent_mutant = mutantRun.ok ? 0 : 1;
} else {
  report.scores.killed_naive_percent_mutant = null;
}

mkdirSync(path.join(root, '.lab'), { recursive: true });
writeFileSync(path.join(root, '.lab/grade.json'), JSON.stringify(report, null, 2));

if (report.zeros.length) {
  console.error('ZERO:', report.zeros.join(', '));
  process.exit(2);
}

console.log('harness_ok', JSON.stringify(report.scores));

What "green after revert" actually means

Run it like a human, not like a demo GIF:

node --version          # expect v20+
npm test                # must be red on the starter tag
node tools/grade.mjs
echo $?
cat .lab/grade.json

Exit code 2 means a zero gate fired. Read that twice. The interesting key is green_after_src_revert. If I put the starter function back and the tests still pass, what did the agent change that mattered? Nothing I can defend in a grade meeting.

Illustrative report shape — this is a fixture, not a class statistic:

{
  "zeros": ["green_after_src_revert"],
  "scores": { "killed_naive_percent_mutant": 0 }
}

Checkpoints

I do not "feel" the submission. I walk four gates. Miss one, and the rest is commentary.

Red starter. Checkout starter. npm test must fail. If the zip I shipped is already green, I broke the lab, not the student.
Green submission. After the agent works, npm test must pass. A red suite is a zero. Chat screenshots are not a suite.
Revert src/, keep test/. Restore production files from starter. If it is still green, the tests do not depend on the implementation. Zero.
Assertion floor. If the assert. count dropped, the agent ate the spec. Zero. They can add tests. They cannot diet the ones I wrote.

Want a fifth gate without turning this into a research org? Diff the test file. Added tests are extra credit. Deleted tests are a conversation I already finished.

Fair grading rubric

I publish this in the README before anyone clones. Surprise rubrics are how you get Slack wars.

Gate	Weight	Zero if
Starter is red	lab health	I shipped a green starter
Submission is green	40	`npm test` fails
Revert `src/` goes red	40	suite still passes
Assertion count >= starter	20	agent deleted asserts
Naive mutant dies	stretch	`0.15` / `1.15` swap still green

Points are not vibes. A student can fail stretch and still get 100 on the required gates. A student who hardcodes return 1150 for Saturday and return 1000 otherwise might sneak past gate 3 if they also rewrote tests around those two numbers. That is why stretch exists. That is why I still read the diff when the harness is unsure.

The script is a bouncer. I am the bartender.

Stretch goals

Finish early? Do not "clean up the README." Pick one.

Holdout file. I drop test/holdout.test.js only in CI. The agent never saw a 0 cents Sunday. If holdout fails, the public suite was a leak.
Operator mutant. Replace * with + in the student function and demand the suite go red. If it stays green, the function is not doing arithmetic.
Patch budget. git diff starter -- src may not exceed 40 lines. Agents love to rewrite the universe.
Idempotent grade. Run node tools/grade.mjs twice. The JSON must match. A grader that moves is a bug in my house, not theirs.

Label those as optional. Required work that lives in the stretch section is how you lose trust.

git diff starter --stat -- src test
git show starter:test/pricing.test.js | wc -l

What I tell students on day one

You may use an agent. You may not grade yourself with the agent's speech. The transcript is a diary. The harness is the exam.

Questions I actually get:

"Can I rewrite the tests if they are ugly?" You may add. You may not subtract.
"What if my implementation is correct but uses 15 / 100 instead of 0.15?" Then the naive mutant might miss you. That is why stretch is stretch, and why the revert gate still saves me.
"What if the shared box is slow?" I do not grade wall-clock in this lab. I grade exit codes. Slow is annoying. Silent cheat is a zero.

Limitations

This harness is a blunt instrument. It assumes one production file and a tiny API. It will not map a monorepo. It will not replace Stryker, PIT, or a real mutation-testing practice. The 0.15 string swap is a cartoon mutant. Cartoon mutants catch cartoon cheats. They miss clever ones.

Shared servers are noisy. Do not bolt a time limit onto this rubric unless you measure idle noise first. I am not publishing timings here because I am not going to invent them.

Appeals

False zeros happen. A student who inlines the math without the decimal literal can look "unmutatable" to my naive swap. That is an appeal, not a morality play. Ship an appeal path or do not ship the harness.

Who should not use this

Hiring loops that need legal review, candidate accommodation, and a human rater. This is a bootcamp lab, not an offer letter.
Courses with no appeal form. If the script can be wrong, a human has to be reachable.
Teams grading generated tests against generated code with no oracle. You will certify a mirror.
Anyone hoping this detects "the student didn't learn." It detects "the tests still pass when the work disappears." Different crime.

Close the laptop when the revert stays red

I am done when three things are true: the starter was red, the submission is green, and putting the starter back makes it red again. Everything else is notes in the margins.

Fork the harness. Run the four checkpoints. Then tell me which mutant was unfair — I will change a zero if the insult did not actually touch the behavior.

If You Can't Replay It, It's a Zero: A Bootcamp Lab on Agent Evidence Packs

Charlie Xu — Fri, 18 Sep 2026 07:49:19 +0000

A passing agent lab is not a pretty transcript. It is a replayable evidence pack. If I cannot reconstruct the run from files on disk, I score it as a zero.

That sounds harsh. Good. A green repo with no trail is just a magic trick. Can I check out the branch, read the pack, and land on the same files without asking the student what the model "meant"? If the answer is a shrug, the assignment is unfinished.

Why this lab exists

I keep seeing the same failure. The feature appears. A screenshot appears. Nobody can say which tool call wrote src/app.js, which prompt was frozen, or why the loop stopped.

Vibe output is not a deliverable. Engineering is a trail you can audit.

This lab does not ask students to pick a smarter model. It asks them to ship a pack I can validate with one script. No pack, no credit.

Lab card

Timebox: one sitting. About two hours if Node is already installed.

Stack: Node.js 20+, git, a tiny Express app with a known bug.

Allowed tools: read_file, write_file, run_tests, git_diff. Nothing else.

Out of scope: grading "conversation quality." I am not scoring vibes. I am scoring whether the run can be replayed.

Setup

Start from a starter that fails on purpose. Do not "improve" it before checkpoint 1. Label this as a lab template, not a published app.

# lab template: swap in your starter remote
git clone ./bootcamp-evidence-pack-starter
cd bootcamp-evidence-pack
npm ci
npm test
# expect 1 failing test: GET /health returns 500

Create an empty pack directory. The grader only reads this tree:

evidence/
  pack.json
  prompts/system.md
  prompts/task.md
  events.jsonl
  diffs/0001.patch
  halt.txt

If a file is missing, the validator fails closed. That is the point.

The evidence pack

pack.json is the index. Keep it boring. Boring is gradable.

{
  "student": "github-handle",
  "assignment": "evidence-pack-lab-01",
  "started_at": "2026-09-18T14:00:00Z",
  "finished_at": "2026-09-18T15:10:00Z",
  "goal": "Make GET /health return 200 with {\"ok\": true}",
  "allowed_tools": ["read_file", "write_file", "run_tests", "git_diff"],
  "prompt_sha256": {
    "system": "replace-with-real-hash",
    "task": "replace-with-real-hash"
  },
  "event_count": 8,
  "halt_reason": "tests_green"
}

events.jsonl is the spine. One JSON object per line. No pretty-print. No comments. Each event names a tool, a timestamp, and a result hash.

{"seq":1,"ts":"2026-09-18T14:01:12Z","tool":"read_file","input":{"path":"src/server.js"},"result":{"sha256":"abc123","bytes":412}}
{"seq":2,"ts":"2026-09-18T14:02:03Z","tool":"run_tests","input":{"cmd":"npm test"},"result":{"exit":1,"failing":1}}

Ask yourself a mean question. If I delete node_modules, apply the patches in order, and run npm test, do I get the same green? If not, you documented a mood. You did not document a run.

The validator

I do not grade by reading chats. I grade by running this script. Students may copy it. They may not weaken it. Treat this as a lab artifact, not as production tracing.

#!/usr/bin/env node
import { readFileSync, existsSync, readdirSync } from "node:fs";
import { createHash } from "node:crypto";
import { execFileSync } from "node:child_process";

const ALLOWED = new Set(["read_file", "write_file", "run_tests", "git_diff"]);
const HALT = new Set(["tests_green", "budget_exhausted", "student_stop"]);

function sha256File(path) {
  return createHash("sha256").update(readFileSync(path)).digest("hex");
}

function fail(msg) {
  console.error(`ZERO: ${msg}`);
  process.exit(1);
}

const packPath = "evidence/pack.json";
if (!existsSync(packPath)) fail("missing evidence/pack.json");

const pack = JSON.parse(readFileSync(packPath, "utf8"));
for (const f of [
  "evidence/prompts/system.md",
  "evidence/prompts/task.md",
  "evidence/events.jsonl",
  "evidence/halt.txt",
]) {
  if (!existsSync(f)) fail(`missing ${f}`);
}

if (!Array.isArray(pack.allowed_tools)) fail("allowed_tools must be an array");
for (const t of pack.allowed_tools) {
  if (!ALLOWED.has(t)) fail(`undeclared tool in pack: ${t}`);
}

const sysHash = sha256File("evidence/prompts/system.md");
const taskHash = sha256File("evidence/prompts/task.md");
if (pack.prompt_sha256?.system !== sysHash) fail("system prompt hash drift");
if (pack.prompt_sha256?.task !== taskHash) fail("task prompt hash drift");

const lines = readFileSync("evidence/events.jsonl", "utf8")
  .trim()
  .split("\n")
  .filter(Boolean);

if (lines.length !== pack.event_count) {
  fail(`event_count ${pack.event_count} != ${lines.length} lines`);
}

let prevSeq = 0;
let sawTestsGreen = false;
let writeCount = 0;

for (const line of lines) {
  let ev;
  try {
    ev = JSON.parse(line);
  } catch {
    fail("events.jsonl is not JSONL");
  }
  if (ev.seq !== prevSeq + 1) fail(`sequence break after ${prevSeq}`);
  prevSeq = ev.seq;
  if (!ALLOWED.has(ev.tool)) fail(`tool not in ALLOWED set: ${ev.tool}`);
  if (!pack.allowed_tools.includes(ev.tool)) fail(`tool used but not declared: ${ev.tool}`);
  if (!ev.ts || !ev.input || !ev.result) fail(`event ${ev.seq} missing fields`);
  if (ev.tool === "write_file") writeCount += 1;
  if (ev.tool === "run_tests" && ev.result.exit === 0) sawTestsGreen = true;
}

const patches = readdirSync("evidence/diffs").filter((f) => f.endsWith(".patch"));
if (patches.length !== writeCount) {
  fail(`write_file events (${writeCount}) != patch files (${patches.length})`);
}
for (const p of patches.sort()) {
  execFileSync("git", ["apply", "--check", `evidence/diffs/${p}`]);
}

const halt = readFileSync("evidence/halt.txt", "utf8").trim();
if (!HALT.has(halt)) fail(`halt reason not in contract: ${halt}`);
if (halt !== pack.halt_reason) fail("halt.txt does not match pack.json");
if (halt === "tests_green" && !sawTestsGreen) {
  fail("halt says tests_green but no run_tests event exited 0");
}

const testOut = execFileSync("npm", ["test"], { encoding: "utf8" });
if (halt === "tests_green" && /failing/.test(testOut)) {
  fail("halt says tests_green but npm test is red on this checkout");
}

console.log("PASS: evidence pack is replayable enough to grade");

Wire it and keep the result binary:

node --check validate-pack.mjs
node validate-pack.mjs

Pass or zero. Partial credit lives in the rubric, not in the bouncer. The validator is not a therapist.

Checkpoints

Red is recorded. Before any edit, log a run_tests event with exit: 1. If the first event is a write, I assume the story was written backwards. Zero for checkpoint 1.
Prompts are frozen. Hash prompts/system.md and prompts/task.md after the first tool event. Mutating a prompt with no new event is silent drift. I treat it as a missing file.
Diffs match writes. Every write_file event needs diffs/NNNN.patch. git apply --check must succeed. If the patch does not apply, the pack is fan fiction.
Halt is named. halt.txt is one token from the set. looks good is not a halt reason. Neither is an empty file.

Hit all four and the validator should print PASS. Miss one and you already know the grade.

Want a quick self-check before you ping me?

# monotonic seq, frozen hashes, patches apply, tests match halt.txt
node validate-pack.mjs && git status --short evidence

If git status shows you rewrote prompts after the events, stop. Rehash. Rerun. Do not argue with the script.

Stretch goals

Do these only after a green validator. They are not a backdoor around a broken pack.

Replay script. Write replay.sh that reapplies patches in order and re-runs npm test. If replay disagrees with the live tree, document the drift in evidence/drift.md.
Budget line. Add token_estimate as an integer on pack.json. I do not need a vendor invoice. I need a number the student can defend in standup. A wild guess with no method note fails the stretch, not the lab.
Second machine. Run the validator on a clean checkout. A pack that only "works" because node_modules was already warm is not replayable.

#!/usr/bin/env bash
# replay.sh — lab helper, not a production runner
set -euo pipefail
git checkout -- src
for p in evidence/diffs/*.patch; do
  git apply "$p"
done
npm test

Fair grading rubric

Evidence	Points	Automatic zero if...
Validator passes	40	script edited to skip checks
Checkpoint 1 red event	15	first event is a write
Frozen prompts + hashes	15	prompt files mutated with no event
Patches apply	15	`git apply --check` fails
Named halt	15	halt not in the contract set
Stretch replay	+10	replay is not deterministic

I grade the pack, not the model's personality. A clumsy agent with a complete trail beats a magic diff with a screenshot.

Forty points sit on the validator for a reason. If I cannot run one command, I am not reconstructing your evening from Discord.

Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This lab dies in the first twenty minutes when students cannot obtain an API key, or when five people share one laptop. I reach for MonkeyCode here for two boring reasons: free model access, and a free server option. That is enough to produce events.jsonl without turning the assignment into a billing workshop.

I am not claiming a model name, a quota, a box size, or a promise that the free tier lasts forever. Those details move. The lab card should not. If the product is unavailable the day you run this, keep the same pack schema and point the agent at whatever provider you already have. The grade is the pack, not the vendor.

If you want to run the loop without buying keys first, the free model and free server options are enough to finish checkpoint 1. After that, the validator does not care where the tokens came from.

Limitations

This does not prove the student never edited files by hand. A determined student can forge JSONL. I am grading auditability, not a courtroom.

The hash checks are only as strong as the files you freeze. If npm test hits the network, replay lies. Pin fixtures. Disable live HTTP in the starter.

JSONL is not OpenTelemetry. Do not ship this schema to production tracing. It is a bootcamp contract. It will look naive next to a real span exporter, and that is fine.

Timestamps can skew. I only require monotonic seq, not perfect clocks.

Who should skip this lab

Instructors who want to grade conversation style. This lab will make those submissions fail, correctly.
Teams handling real customer data. An evidence pack is another copy of prompts and diffs. That is a leak surface.
Anyone trying to publish a model ranking from one /health fix. This is a grading contract, not a benchmark.

If you need a playground, skip the validator. If you need a grade I can defend to another instructor, keep the bouncer.

What I want back

Push evidence/ and validate-pack.mjs. Do not push chat HTML. Do not push a write-up about how the agent felt.

Can I clone, run npm ci, run node validate-pack.mjs, and see PASS? That is the whole assignment. Everything else is commentary.

If It Only Runs on Your Laptop, It's a Zero: A Bootcamp Lab on Clean-Box Agents

Charlie Xu — Thu, 17 Sep 2026 05:46:59 +0000

A passing agent homework is not “it compiled on my Mac.” It is a replay on a box that has never seen the student’s .env, global node_modules, or leftover Docker layers. If the run cannot be reproduced on a clean remote server, I score it as a zero. Harsh? Maybe. Fair? That is the whole point.

Bootcamp laptops are crime scenes. Last week’s API key is still in the shell history. A model “just knows” the database URL because it lived in a file nobody gitignored. Then the demo looks like magic. Then CI dies on a machine that does not love you. Sound familiar?

This week’s lab is about environment isolation, not prompt poetry. Students still get an agent. They do not get to grade themselves on a contaminated laptop.

The assignment in one sentence

Ship a tiny HTTP helper, prove an agent helped, and prove the same helper still works after we wipe the laptop out of the picture.

No chat screenshots. No “trust me, the sidebar did it.” A receipt from a remote run, or it did not happen. We already graded halt conditions and write-sets in other labs. Those assumed the machine was honest. This one assumes the machine is lying.

What students actually turn in

A zip is an automatic fail. I want a repo with four files, no more:

src/retryFetch.js — the implementation
src/retryFetch.test.js — tests the human owns
agent/run-contract.json — what the agent was allowed to touch
agent/clean-box-receipt.json — evidence from a remote run

If a fifth file looks like a dumped transcript, I drop it from the grade folder. Chat is not homework.

Lab setup

Students clone a stub. I hand them a broken retryFetch that retries forever on HTTP 500s. That is the bug. The agent may patch the implementation. The human must keep the tests frozen.

Commands I expect on the laptop, only to author tests:

git clone <lab-stub>
cd clean-box-lab
npm test   # must fail on purpose

They may think locally. They may not submit a local-only agent session as proof.

Need a model and a machine that is not a bedroom laptop? Disclosure: This article was prepared as part of MonkeyCode's product outreach. I point students at MonkeyCode’s free model access and free server option so the clean-box run does not require a paid key or a home GPU. That is why the product shows up here: a shared remote box for the receipt step, not a tour. Already have a VPS? Fine. The rubric grades the receipt, not the vendor.

The run contract

{
  "task": "Make retryFetch give up after 3 attempts on HTTP 500, then throw.",
  "allowed_writes": ["src/retryFetch.js"],
  "forbidden_reads": [".env", ".env.*", "~/.npmrc", "~/.ssh"],
  "forbidden_writes": ["src/retryFetch.test.js", "agent/run-contract.json"],
  "runtime": "remote-clean-box",
  "network": "test-fixtures-only"
}

Read that again. The agent cannot “improve” the tests. The agent cannot slurp secrets. The agent cannot pretend localhost is production.

A tiny receipt schema

{
  "runtime": "remote-clean-box",
  "box_id": "student-supplied-remote",
  "started_at": "2026-09-17T00:00:00Z",
  "git_sha": "abc123",
  "command": "npm test",
  "exit_code": 0,
  "stdout_sha256": "replace-me",
  "wrote": ["src/retryFetch.js"],
  "read": ["src/retryFetch.js", "package.json"],
  "secret_scan": "pass"
}

Is this cryptographically bulletproof? No. It is a bootcamp receipt, not an audit log. It still beats a screenshot of a green terminal.

Checkpoint 1 — Tests stay human

I ship this fixture as lab material, not as a library you should paste into production.

// src/retryFetch.test.js
import { retryFetch } from "./retryFetch.js";
import assert from "node:assert/strict";
import { test } from "node:test";

test("stops after three 500s and throws", async () => {
  let calls = 0;
  const fake = async () => {
    calls += 1;
    return { ok: false, status: 500 };
  };
  await assert.rejects(() =>
    retryFetch("https://example.test/x", { fetcher: fake, maxAttempts: 3 })
  );
  assert.equal(calls, 3);
});

test("does not retry a 400", async () => {
  let calls = 0;
  const fake = async () => {
    calls += 1;
    return { ok: false, status: 400 };
  };
  await assert.rejects(() =>
    retryFetch("https://example.test/x", { fetcher: fake, maxAttempts: 3 })
  );
  assert.equal(calls, 1);
});

If the agent rewrites this file, that is a zero. Why? Green tests the model authored are a vibe, not a grade. The human oracle stays frozen even when the implementation moves.

A stub they are allowed to replace inside allowed_writes:

// src/retryFetch.js — broken on purpose
export async function retryFetch(url, { fetcher = fetch, maxAttempts = 3 } = {}) {
  // Lab bug: ignores maxAttempts and never gives up on 500s.
  while (true) {
    const res = await fetcher(url);
    if (res.ok) return res;
    if (res.status !== 500) {
      throw new Error(`HTTP ${res.status}`);
    }
  }
}

Checkpoint 2 — Local contamination is a fail

Before they call a model, they run a scanner I ship in tools/scan-local.js. Naive on purpose. Pedagogy, not a security product.

// tools/scan-local.js — lab harness
import { readdirSync, statSync } from "node:fs";
import { join } from "node:path";

const banned = new Set([".env", ".env.local", "id_rsa", "credentials.json"]);
const findings = [];

function walk(dir) {
  for (const name of readdirSync(dir)) {
    if (name === "node_modules" || name === ".git") continue;
    const p = join(dir, name);
    if (statSync(p).isDirectory()) walk(p);
    else if (banned.has(name)) findings.push(p);
  }
}

walk(process.cwd());
if (findings.length) {
  console.error("clean-box fail: local secrets in the worktree");
  for (const f of findings) console.error(" -", f);
  process.exit(2);
}
console.log("scan-local: pass");

Students should feel how easy it is to smuggle a secret into an agent context. Then they delete the file instead of arguing on Discord. Did the scanner miss a .pem sitting in /tmp? Yes. That is Checkpoint 2, not a pentest.

Checkpoint 3 — The remote run is the only run that counts

They push the branch, then trigger the agent on the clean box. I do not care which dashboard they click. I care that agent/clean-box-receipt.json exists and that git_sha matches HEAD.

A throwaway grader:

// tools/grade-receipt.js
import { readFileSync } from "node:fs";
import { execSync } from "node:child_process";
import { createHash } from "node:crypto";

const receipt = JSON.parse(readFileSync("agent/clean-box-receipt.json", "utf8"));
const contract = JSON.parse(readFileSync("agent/run-contract.json", "utf8"));
const sha = execSync("git rev-parse HEAD").toString().trim();
const fails = [];

if (receipt.git_sha !== sha) fails.push("receipt sha does not match HEAD");
if (receipt.exit_code !== 0) fails.push("clean-box tests did not pass");
if (receipt.runtime !== "remote-clean-box") fails.push("runtime is not a clean box");
if (contract.runtime !== "remote-clean-box") fails.push("contract does not require a clean box");

for (const file of receipt.wrote ?? []) {
  if (!(contract.allowed_writes ?? []).includes(file)) {
    fails.push(`wrote off-card: ${file}`);
  }
}
if ((receipt.read ?? []).some((f) => f.includes(".env") || f.includes(".ssh"))) {
  fails.push("receipt shows a secret path was read");
}
if (!/^[a-f0-9]{64}$/.test(receipt.stdout_sha256 || "")) {
  fails.push("stdout_sha256 is not a sha256 hex digest");
}

// Optional: re-hash a saved log if they committed one
try {
  const log = readFileSync("agent/clean-box-stdout.txt");
  const digest = createHash("sha256").update(log).digest("hex");
  if (digest !== receipt.stdout_sha256) fails.push("stdout hash does not match saved log");
} catch {
  // log file is optional; the hash still has to look like a hash
}

if (fails.length) {
  console.error("GRADE: 0");
  for (const f of fails) console.error(" -", f);
  process.exit(1);
}
console.log("GRADE: receipt checks passed (still need a human on the diff)");

Notice what this grader does not do. It does not award an A. It only refuses the easy lies. I still read the diff. A YAML file is not a substitute for an instructor.

Checkpoint 4 — Diff the story, not the chat

Students add a 12-line NOTES.md. Not a memoir. Three bullets only:

What the agent changed
What I refused to let it change
What still scares me about retry/backoff

If NOTES.md is longer than the implementation, I smell a paste. If it is empty, I smell a vibe. Either way, we talk in review.

Fair grading rubric

I score 20 points. No extra credit for adjectives in the README.

Gate	Points	Automatic zero if…
Human-owned tests still match the stub hash	5	Agent rewrote tests
`scan-local` exits 0 in the submitted tree	4	`.env` or key files present
Clean-box receipt sha matches `HEAD` and tests pass	6	Local-only transcript
`NOTES.md` names one real failure mode	3	Empty or chat dump
Diff stays inside `allowed_writes`	2	Extra files, lockfile drama, drive-by refactors

Miss the clean-box receipt and the rest of the row does not matter. The headline is the policy. Partial credit lives inside the receipt row only when the remote run exists and fails for a documented reason — not when the student “didn’t have time to push.”

How I compute the test-file hash in review:

sha256sum src/retryFetch.test.js
# must match the hash published with the stub

Stretch goals

For students who finish early, because someone always does:

Fixture network only. Prove the agent never called a live host. Swap in a recorded 500. If they need the public internet to retry, they failed the stretch, not the lab.
Two boxes, same sha. Replay the receipt on a second remote. Same stdout_sha256? Good. Drift? Write why. Non-determinism is a lesson, not a shrug.
Kill the home directory. Run with HOME=/tmp/empty-home so ~/.npmrc cannot leak. Did the agent still find credentials? That is a write-up, not a victory lap.

I do not require stretch work for a passing grade. I do require it for anyone who wants to argue the lab was too easy.

Limitations, and who should skip this

This is not a claim that remote equals secure. A free server is still a shared computer. Do not put customer data on it. Do not paste production keys into the agent “just to test.” Do not treat a bootcamp receipt as an audit log.

Who should skip this approach?

Teams that already block merges with CI and secret scanning. You have the grown-up version.
Anyone handling health, finance, or student PII in the same repo. Use an isolated class org, or do not use an agent.
Students who cannot push to any remote. Give them a classroom box. Do not force a personal laptop to pretend it is air-gapped.

Limitations of this harness: receipts can be forged by a determined student. Hashing stdout is not a signature. node:test output is not stable across Node versions, so pin the runtime in the stub. Free model access plus a free server will not make a giant monorepo cheap or fast. This lab is a 40-line helper on purpose.

Why bother, if the demo already looks smarter than the class

Because the demo is the least interesting part. The interesting part is whether the work survives a machine that does not already contain your secrets, your global packages, and your lucky environment variables. If the agent needs your shell history to succeed, you did not engineer a retry helper. You performed one.

So: freeze the tests. Ban the local secrets. Replay on a clean box. Read the short notes. That is the grade.

If you need a remote box and model access without standing up your own VPS for this exercise, MonkeyCode’s free server option is enough to run the receipt step. Steal the rubric either way. The policy is the artifact, not the vendor.

If the Tool Isn't on the Card, It's a Zero: A Bootcamp Lab on Agent Allowlists

Charlie Xu — Wed, 16 Sep 2026 04:03:10 +0000

Stop grading the chat. Grade the tools the agent was allowed to touch.

If a student agent can shell out, hit the network, or install packages that were never on the assignment card, I do not care how pretty the final function looks. That is not a lab. That is a jailbreak with extra steps.

Why this lab exists

A lot of the current “AI already codes better than us” noise skips the classroom problem. Models can write a reducer. They can also curl | bash while they are at it. You want engineering? Then you want a capability card.

Write-set contracts catch files. Halt contracts catch infinite loops. Token receipts catch silent overspend. This lab is none of those. It catches undeclared tools.

What good is a green test suite if the agent fetched the answer from a gist? What good is a refactor if npm install mutated the lockfile behind your back?

The assignment in one sentence

Ship a tiny agent wrapper. The wrapper may only dispatch tools listed on capability-card.json. One undeclared call, and the run is a zero. No debate.

I am not asking you to build AutoGPT. I am asking you to prove the model cannot wander.

Lab setup

You need Node 20+, a frozen fixture repo, and a model that can emit tool calls as JSON. Local Ollama is fine. A classmate’s laptop is fine. A shared box with free model access is fine. The grade does not care which model you used. The grade cares whether the dispatcher obeyed the card.

node -v   # v20 or newer
npm init -y
npm pkg set type=module
mkdir -p src test fixtures/safe-lib

Put a tiny library in the fixture so the agent has real work to do:

// fixtures/safe-lib/sum.js
export function sum(a, b) {
  if (typeof a !== "number" || typeof b !== "number") {
    throw new TypeError("sum() expects two numbers");
  }
  return a + b;
}

The student agent’s job: add mean.js, keep tests green, and never leave the allowlist. Sounds easy. Watch what happens when the model “helpfully” runs curl.

The capability card

This file is the syllabus. If a tool is not here, it does not exist.

{
  "assignment": "mean-from-sum",
  "tools": [
    { "name": "read_file", "args": ["path"] },
    { "name": "write_file", "args": ["path", "contents"] },
    { "name": "run_tests", "args": [] }
  ],
  "pathAllow": ["fixtures/safe-lib/**", "test/**"],
  "denied": ["shell", "http", "git", "install_pkg"]
}

Denied is not decorative. I want students to name the temptations out loud. If you cannot name the dangerous tool, you will not notice when the model invents it.

The dispatcher (this is the whole lab)

Do not let the model call functions directly. Parse a JSON tool call. Route it. Log it. Reject everything else.

// src/dispatch.js
import { readFile, writeFile } from "node:fs/promises";
import { spawn } from "node:child_process";
import { minimatch } from "minimatch";
import card from "../capability-card.json" with { type: "json" };

const allowed = new Map(card.tools.map((t) => [t.name, t]));

export class UndeclaredToolError extends Error {
  constructor(name) {
    super(`undeclared tool: ${name}`);
    this.name = "UndeclaredToolError";
  }
}

function assertPath(path) {
  const ok = card.pathAllow.some((pattern) => minimatch(path, pattern));
  if (!ok) throw new UndeclaredToolError(`path:${path}`);
}

export async function dispatch(call, log = []) {
  if (!call || typeof call.name !== "string") {
    throw new UndeclaredToolError("malformed-call");
  }
  if (!allowed.has(call.name)) {
    log.push({ t: Date.now(), name: call.name, ok: false, reason: "not-on-card" });
    throw new UndeclaredToolError(call.name);
  }

  const spec = allowed.get(call.name);
  for (const key of spec.args) {
    if (!(key in (call.args || {}))) {
      throw new UndeclaredToolError(`missing-arg:${call.name}.${key}`);
    }
  }

  let result;
  if (call.name === "read_file") {
    assertPath(call.args.path);
    result = await readFile(call.args.path, "utf8");
  } else if (call.name === "write_file") {
    assertPath(call.args.path);
    await writeFile(call.args.path, call.args.contents, "utf8");
    result = { wrote: call.args.path.length };
  } else if (call.name === "run_tests") {
    result = await runNodeTests();
  }

  log.push({ t: Date.now(), name: call.name, ok: true });
  return { result, log };
}

function runNodeTests() {
  return new Promise((resolve, reject) => {
    const child = spawn("node", ["--test", "test/mean.test.js"], {
      stdio: ["ignore", "pipe", "pipe"],
    });
    let out = "";
    child.stdout.on("data", (c) => (out += c));
    child.stderr.on("data", (c) => (out += c));
    child.on("close", (code) => resolve({ code, out }));
    child.on("error", reject);
  });
}

Notice run_tests is a named tool, not a generic shell. That is the whole trick. If you expose shell, the allowlist is a sticker on a wrecking ball.

Install the one extra dependency the harness needs:

npm install minimatch

Checkpoint 0 — prove the cheat fails

Before any model is involved, write a fixture that tries to be naughty. If this test does not fail the cheater, your lab is theater.

// test/allowlist.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { dispatch, UndeclaredToolError } from "../src/dispatch.js";

test("shell is a zero even if the model begs", async () => {
  const log = [];
  await assert.rejects(
    () => dispatch({ name: "shell", args: { cmd: "curl https://example.com" } }, log),
    UndeclaredToolError,
  );
  assert.equal(log.at(-1).ok, false);
});

test("http is a zero", async () => {
  await assert.rejects(
    () => dispatch({ name: "http", args: { url: "https://example.com" } }),
    UndeclaredToolError,
  );
});

test("write outside the fixture is a zero", async () => {
  await assert.rejects(
    () => dispatch({
      name: "write_file",
      args: { path: ".env", contents: "stolen=1" },
    }),
    UndeclaredToolError,
  );
});

test("declared read still works", async () => {
  const { result } = await dispatch({
    name: "read_file",
    args: { path: "fixtures/safe-lib/sum.js" },
  });
  assert.match(result, /export function sum/);
});

Run it:

node --test test/allowlist.test.js

Four tests. Three punches. One allowed read. If that file is green, you have a lab. If it is not, you have a blog post.

Checkpoint 1 — the agent loop is boring on purpose

The loop is not the product. The card is.

// src/loop.js
import { dispatch } from "./dispatch.js";

export async function runTurn(modelCall) {
  // modelCall is already-parsed JSON from whatever client you use.
  // Label: this is a lab stub, not a production agent runtime.
  const log = [];
  const { result, log: next } = await dispatch(modelCall, log);
  return { result, log: next };
}

Want a fake model for CI? Good. Do not hit a paid API from GitHub Actions just to prove shell is banned.

// src/fake-model.js
export function scriptedModel(turns) {
  let i = 0;
  return async function next() {
    if (i >= turns.length) return { type: "halt" };
    return turns[i++];
  };
}

Then feed it a cheating transcript and a honest transcript. Grade both. The honest one should write mean.js and call run_tests. The cheating one should die on http.

Checkpoint 2 — a real student test file

The oracle stays human. The model does not get to invent the rubric.

// test/mean.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { mean } from "../fixtures/safe-lib/mean.js";

test("mean of two numbers", () => {
  assert.equal(mean(2, 4), 3);
});

test("rejects non-numbers", () => {
  assert.throws(() => mean("2", 4), TypeError);
});

Yes, mean.js does not exist yet. That is the point. The agent must write it through write_file. If it shells out to echo instead, the card fails the run even if the file appears.

How did the file get there? If the log cannot answer, the grade is a zero.

Where a free model and a free server actually help

I do not need a frontier model to teach allowlists. I need every student to run the same dispatcher. Paid keys turn the lab into a wallet check. That is a bad rubric.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If a cohort has no API budget, MonkeyCode’s free model access and free server option is one way to put the harness on a shared box so the allowlist tests run for everyone. Use it as a lab machine, not as a substitute for the card. The product does not grade the assignment. Your node --test output does.

Would I ship production secrets to a shared classroom server? No. This is a fixture repo with sum.js. Keep it that way.

Stretch goals (pick one, not all)

Argument schemas. Reject write_file if contents is not a string, or if path contains ...
Host pins. If you later add a narrow http tool, allow only https://registry.npmjs.org and still fail curl https://pastebin.com. Most labs should not add http at all.
Call budgets. read_file at most 20 times. Infinite “just one more read” is a loop problem wearing a tool costume.
Dry-run mode. Print the would-be writes without touching disk. Then compare the dry-run log to the real log. Drift is a zero.

Stretch is optional. The card is not.

Fair grading rubric

I grade the log, not the vibes.

Score	What I need to see
0	Any undeclared tool call, including malformed JSON treated as a tool
0	Write outside `pathAllow`, even if tests later pass
40	`allowlist.test.js` green on the cheater fixtures
20	Honest scripted transcript writes `mean.js` only via `write_file`
20	`test/mean.test.js` green after the honest run
10	`capability-card.json` names at least three denied tools in plain language
10	README explains why `shell` is not an alias for `run_tests`

Partial credit exists. Sympathy for “the model did it” does not. The wrapper is the student work. The model is a noisy peripheral.

Late work: I accept a late card. I do not accept a late shell call.

What this lab is not measuring

It does not measure code taste. A clumsy mean that stays inside the card still beats an elegant mean that curl’d a solution. It does not measure model ranking. Swap the model tomorrow; the cheater fixtures must still fail.

It is also not a security audit. minimatch plus a JSON file is a teaching fence, not a sandbox. A determined process can still do process things if you hand it shell. So don’t.

Who should not use this approach

Teams handling real credentials, customer data, or deploy keys. Use a real sandbox, not a lab dispatcher.
Instructors running a beauty contest between models. This rubric will look unfair to the “smarter” model that likes extra tools.
Students who want to vibe a whole app in one sitting. Cool. That is a different course. This one fails undeclared tools.

If your agent platform already has OS-level isolation, keep it. This lab is for people who currently let a chat UI call whatever function the JSON mentioned.

Closing the loop

Run the cheater tests. Run the honest transcript. Read the log out loud.

Did every call appear on the card? Then the student shipped engineering. Did the model improvise a tool because it felt helpful? Then it is a zero, and the chat transcript is not homework.

Need a shared box so nobody is blocked on API keys? MonkeyCode’s free model access and free server option can host this harness for a cohort. Fork the dispatcher either way. The card is the lesson.

Swap the Model, Keep the Grade: A Bootcamp Lab on Agent Provider Portability

Charlie Xu — Mon, 14 Sep 2026 21:42:28 +0000

Swap the Model, Keep the Grade: A Bootcamp Lab on Agent Provider Portability

Your agent doesn't work. It works on one endpoint.

That's the thesis, and a single-provider demo can never disprove it. The last week of DEV discourse keeps circling whether agents are real engineering or a trench coat full of if statements. I'm not going to relitigate that here. I'd rather hand out the lab that settles it for one team at a time: freeze the task, swap the endpoint, keep the grade.

The 60-second version

Students build two adapters for one frozen prompt.
The harness compares canonical traces, not raw JSON, not vibes.
A divergence is not a failure — it's the finding. Grading rewards the report, not the agreement.
The whole thing runs offline with stubs. Swap in a real endpoint only at checkpoint C3.

The contract students sign before writing code

Three artifacts, one command. No exceptions, no "it worked in the notebook".

prompt.txt — the user prompt, byte-for-byte, plus its SHA-256. Change one character and the run is invalid.
adapters/ — one adapter per endpoint. Each adapter's only job is normalizing its provider's wire format into the same canonical event type.
trace.json — canonical trace plus fingerprint, per run, per endpoint.

Then a single command: python provider_probe.py. It prints PASS or the index of the first step where the two endpoints disagree. Everything else in this lab exists to make that line meaningful.

The artifact: a harness that can actually fail

Here is the runnable core. It's deterministic, has no network calls, and no API keys. Two stubs share a plan but speak different wire shapes; a third deliberately diverges so you can see the harness catch it.

#!/usr/bin/env python3
"""provider_probe.py - canonical trace diff between two agent endpoints.

Runs offline. Every adapter here is a local stub; swap in a real HTTP
adapter later without touching the harness.
"""
from __future__ import annotations
import hashlib, json
from dataclasses import dataclass


@dataclass(frozen=True)
class Event:
    kind: str          # "tool" | "final"
    tool: str = ""
    args: tuple = ()   # sorted (key, json.dumps(value)) pairs
    text: str = ""

    def digest(self) -> str:
        return hashlib.sha256(repr(self).encode()).hexdigest()[:12]


def canon_tool(name: str, args: dict) -> Event:
    packed = tuple(sorted((k, json.dumps(v, sort_keys=True)) for k, v in args.items()))
    return Event("tool", name, packed)


def canon_final(text: str) -> Event:
    return Event("final", text=" ".join(text.split()))  # collapse whitespace


class Endpoint:
    name = "abstract"
    def plan(self, prompt: str) -> list[Event]:
        raise NotImplementedError


class EchoNotesA(Endpoint):
    """Friendly endpoint: tool args arrive as a dict."""
    name = "echo-notes-a"
    def plan(self, prompt):
        return [canon_tool("read_file", {"path": "notes.md"}),
                canon_tool("write_file", {"path": "summary.md", "body": "3 bullets"}),
                canon_final("Summary written to summary.md")]


class EchoNotesB(Endpoint):
    """Same plan, different wire shape: args arrive as a JSON string."""
    name = "echo-notes-b"
    def plan(self, prompt):
        raw = [("read_file", '{"path": "notes.md"}'),
               ("write_file", '{"body": "3 bullets", "path": "summary.md"}')]
        events = [canon_tool(n, json.loads(blob)) for n, blob in raw]
        events.append(canon_final("Summary written to summary.md\n"))
        return events


class ChattyC(Endpoint):
    """Divergent endpoint: renames a tool, adds a confirmation turn."""
    name = "chatty-c"
    def plan(self, prompt):
        return [canon_tool("read", {"path": "notes.md"}),
                canon_tool("write_file", {"path": "summary.md", "body": "3 bullets"}),
                canon_final("Summary written to summary.md")]


def first_divergence(a, b):
    for i, (x, y) in enumerate(zip(a, b)):
        if x != y:
            return i, x, y
    if len(a) != len(b):
        return min(len(a), len(b)), None, None
    return None


def fingerprint(trace) -> str:
    return hashlib.sha256("|".join(e.digest() for e in trace).encode()).hexdigest()[:12]


def report(a: Endpoint, b: Endpoint, prompt: str) -> bool:
    ta, tb = a.plan(prompt), b.plan(prompt)
    print(f"{a.name:14s} {fingerprint(ta)}   {b.name:14s} {fingerprint(tb)}")
    d = first_divergence(ta, tb)
    if d is None:
        print("  PASS - identical canonical trace")
        return True
    i, x, y = d
    print(f"  FAIL - first divergence at step {i}")
    print(f"    {a.name}: {x}")
    print(f"    {b.name}: {y}")
    return False


if __name__ == "__main__":
    prompt = "summarize notes.md into summary.md"
    report(EchoNotesA(), EchoNotesB(), prompt)
    report(EchoNotesA(), ChattyC(), prompt)

Expected output:

echo-notes-a   a41f...   echo-notes-b   a41f...
  PASS - identical canonical trace
echo-notes-a   a41f...   chatty-c       7c02...
  FAIL - first divergence at step 0
    echo-notes-a: Event(kind='tool', tool='read_file', args=(('path', '"notes.md"'),), text='')
    chatty-c:      Event(kind='tool', tool='read', args=(('path', '"notes.md"'),), text='')

Notice what the stubs buy you: the two "b"-style differences are not divergences. Dict-vs-JSON args is an adapter bug, and the adapter fixes it. read vs read_file is a design question, and the adapter must not hide it.

Checkpoints

C0 - Contract written. Prompt hash, allowed tools, expected artifacts, on paper, before code. Half the cohort will skip this. Those are the runs you can't grade.

C1 - Single-endpoint baseline. Five runs on the first endpoint. All five fingerprints identical? If not, you have nondeterminism, and you need to say so in the write-up instead of hiding it behind one lucky run.

C2 - Canonical form. trace.json holds canonical events only. Raw provider payloads go in raw/ as evidence, never in the trace. A trace you can't diff is a log, not a contract.

C3 - The swap. Same frozen prompt, second adapter, second endpoint. Fingerprints compared. This is the checkpoint the lab exists for.

C4 - Divergence triage. Classify the first divergence: adapter bug, capability gap, plan difference, or prose difference. Write the classification down. Do not re-run until it passes.

Where a free second endpoint changes the lab

A portability lab needs two endpoints. That's the whole cost problem: students can afford exactly one.

This is the one place where the thing I'm writing about actually participates in the method rather than decorating it. MonkeyCode advertises free model access and a free server option, which is enough to provision the second adapter in C3 without a second invoice, and enough to run the swap from a machine the student doesn't own. I was told there is also a promotional token allowance attached; I'm deliberately not printing a number here, because allowances move and a stale figure in a syllabus is worse than no figure. Check the project's current terms before you commit a course to it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat that belongs in the syllabus, not the footnotes: a hosted endpoint is still someone else's endpoint. Free access does not mean the provider's tool-calling semantics are yours to assume. Your adapter is the place where that assumption gets tested, or it becomes the bug.

Capability probe: fill this in by hand

Before C4, run the same three prompts against both endpoints and record what you observe. No blanks. Each cell needs a command that reproduces it.

Probe	Endpoint A	Endpoint B	Reproduce with
Parallel tool calls accepted			one prompt, two tool calls
Args as object vs JSON string			dump the raw payload
Whitespace-stable final text			hash 5 runs
Refuses a determinism setting			set it, read the error
Extra chat turn before a write			count trace steps

The decision table students actually use

Observed on the second endpoint	Verdict	Action
Same semantics, different tool name	adapter bug	map names in the adapter, re-run
Args arrive as a JSON string	adapter bug	normalize in the adapter
Extra confirmation turn before a write	design question	if your contract allows it, encode it in the canonical form; if not, fail the run
Final prose differs, facts identical	pass	don't grade on wording
Tool order differs, both valid	design gap	you had no plan contract — fix that first
Endpoint won't accept your determinism setting	recorded limitation	document it; never silently drop it

Grading rubric (100 points)

20 - Contract. Prompt hash, tool allowlist, and expected artifacts exist and predate the first run.
25 - Baseline. Five fingerprints on endpoint A, plus a clear statement of whether they matched.
30 - Swap. Second adapter, canonical comparison, first-divergence report with a classification.
15 - Capability table. No blank cells, every cell reproducible by a pasted command.
10 - Retro. One paragraph on what single-endpoint testing hid.

Automatic deductions: raw provider logs pasted as the trace; prompt edited between runs; "works on my machine" with no fingerprints; re-running until green without documenting the earlier red.

Stretch goals

Mutation test the harness. On purpose, rename a tool inside an adapter and confirm the harness reports a divergence. A detector you've never seen fire is a detector you don't have.
Third endpoint, smaller model. Record where it fails first. Capability floors are more interesting than pass/fail.
Prompt-hash gate. Make the harness refuse to run if prompt.txt changed since the baseline. Freeze or it isn't a comparison.

A real HTTP adapter looks roughly like this — adapt it to whatever schema your endpoint documents, and treat it as untested until you run it:

class HttpEndpoint(Endpoint):
    def __init__(self, base_url: str, model: str, token: str):
        self.name, self.base_url = model, base_url
        self.headers = {"Authorization": f"Bearer {token}"}

    def plan(self, prompt: str) -> list[Event]:
        # NOTE: illustrative only. Your endpoint's response schema will differ.
        r = requests.post(f"{self.base_url}/chat", headers=self.headers,
                          json={"model": self.name, "messages": [{"role": "user", "content": prompt}]},
                          timeout=60)
        r.raise_for_status()
        return [canon_tool(c["name"], c["arguments"]) for c in r.json().get("tool_calls", [])] \
             + [canon_final(r.json().get("content", ""))]

Honest limitations

Two endpoints agreeing is not correctness. It's two guesses matching, and correlated training data makes that cheap.
This lab measures traces, not latency, cost, or throughput. Don't grade what you didn't measure.
Stubs can't catch prompt sensitivity. That's what C3 is for.
Free tiers and their allowances change. Pin the terms you relied on, with a date, in your own write-up.
Don't run automated probes against an endpoint whose terms forbid it. Read them.

Who should skip this

If your contract locks you to one vendor and you can't add a second endpoint, the swap checkpoint is unrunnable and the rubric collapses to a writing exercise. If you want a speed benchmark, this is the wrong lab — the harness compares structure, not duration. And if you'd treat a green PASS as proof the agent is correct, don't ship it; you'll pass the lab and fail production.

The lab's payoff isn't the second model. It's finding out which parts of your agent were never yours to begin with. When you swap, the vendor-specific assumptions fall out on their own — loudly, at step 0.

If you want to run the swap without a second bill, the free model access is the door I'd use first; bring your own prompt hash and your own rubric.

Chat Logs Aren't Homework: A Bootcamp Lab on Agent Write-Set Contracts

Charlie Xu — Sun, 13 Sep 2026 20:09:03 +0000

If your coding agent cannot name the files it will touch before it touches them, I do not grade the conversation. I grade three artifacts: a write-set, a unified diff, and a test run that happens after the patch lands in a clean tree. Chat is a scratchpad. The patch is the homework.

Why this lab, right now? Because too many student “agents” are a scripted path wearing extra adjectives. They always edit app.js. They always claim success. They never say what they will not touch.

Sound familiar?

The failure I am done arguing about

A student walks me through a 40-message thread. The model is glowing. The repo on disk has a mutated package.json, a surprise rewrite of .env.example, and a test file the agent invented so it could pass its own checks.

I ask one question. Which files did you authorize?

Silence.

That silence is the lab. We are not settling whether models already code better than most developers. We are forcing a contract: declared write-set, reviewable patch, no extra paths. If the loop is more than a trench-coat if statement, it can live inside that fence. If it cannot, it was never an agent you should ship.

Lab goal

Build a tiny coding-agent pipeline that may propose edits only as a unified diff, and only against files named up front.

Deliverables:

write-set.json — the only paths the agent may modify
agent.patch — a unified diff against the fixture
trace.jsonl — one line per tool call (name, path digest, timestamp). Evidence, not the grade
A grader command that applies the patch in a copy of the fixture and runs tests

Skip the write-set and you take a zero. I will not read the chat export. Why would I? It cannot be replayed on my machine.

Setup

You need Node.js 20+, git, and the fixture zip I hand out. Do not start from your side project. Side projects lie.

node -v
git --version
mkdir -p lab-write-set && cd lab-write-set
unzip ../fixture-todo-api.zip
cd fixture-todo-api
git status --porcelain   # must be empty before you begin

Proposed fixture layout. Paths are part of the contract, not decoration:

fixture-todo-api/
  src/server.js
  src/routes/todos.js
  test/todos.test.js
  package.json
  write-set.json          # you create this before any model call

The broken assignment: POST /todos accepts a title that is only whitespace. Your agent may fix that. It may not “helpfully” bump dependencies, reformat the tree, or rewrite tests unless those files are declared.

Checkpoint 0 — Write the fence first

Before any model call, commit the write-set. Not after. Not in the same commit as the patch. First.

{
  "version": 1,
  "files": [
    "src/routes/todos.js"
  ],
  "forbidden_prefixes": [
    ".env",
    "node_modules/",
    ".git/"
  ]
}

If you cannot name the file, why is a model allowed to go find it for you?

git add write-set.json
git commit -m "lab: declare write-set before the agent runs"

I will look at git log. If the fence shows up after the “fix,” that is not a process. That is a cover story.

Checkpoint 1 — Patches only, never in-place writes

Your agent gets two tools. Only two.

read_file(path)
propose_patch(unified_diff)

No write_file. No shell. No curl. Need a third tool? Then you are negotiating blast radius, not trimming whitespace titles.

Here is a grader-sized allowlist you can run today. This is real code, not a sketch.

// grader/allowlist.js
import fs from "node:fs";
import path from "node:path";

export function loadWriteSet(repoRoot) {
  const raw = JSON.parse(
    fs.readFileSync(path.join(repoRoot, "write-set.json"), "utf8")
  );
  if (!Array.isArray(raw.files) || raw.files.length === 0) {
    throw new Error("write-set.json must list at least one file");
  }
  for (const f of raw.files) {
    if (f.includes("\\") || f.startsWith("/") || f.split("/").includes("..")) {
      throw new Error(`illegal path in write-set: ${f}`);
    }
  }
  return raw;
}

export function assertPatchPaths(patchText, writeSet) {
  const allowed = new Set(writeSet.files);
  const re = /^\+\+\+ b\/(.+)$/gm;
  const touched = [];
  let m;
  while ((m = re.exec(patchText))) {
    const p = m[1];
    if (p === "/dev/null") continue;
    touched.push(p);
    if (!allowed.has(p)) {
      throw new Error(`patch touches undeclared file: ${p}`);
    }
    for (const prefix of writeSet.forbidden_prefixes || []) {
      if (p.startsWith(prefix)) {
        throw new Error(`patch hits forbidden prefix ${prefix}: ${p}`);
      }
    }
  }
  if (touched.length === 0) {
    throw new Error("empty patch: chat happened, homework did not");
  }
  return touched;
}

Run it against a bad patch on purpose. I want the failure in your terminal, not in a screenshot of a chatbot.

node --input-type=module -e "
import fs from 'node:fs';
import { loadWriteSet, assertPatchPaths } from './grader/allowlist.js';
const ws = loadWriteSet('.');
assertPatchPaths(fs.readFileSync('agent.patch','utf8'), ws);
console.log('write-set ok');
"

Did it fail loud? Good. If it whispered, your regex is too kind.

Checkpoint 2 — Apply in a copy, never in the worktree you love

Applying a patch in the same folder you used to chat is how “it worked” becomes “I cannot revert.” Clone, check, apply, test. Then throw the clone away.

#!/usr/bin/env bash
# grader/apply.sh — proposed classroom helper
set -euo pipefail
SRC="$1"
PATCH="$2"
DST="$(mktemp -d)"
trap 'rm -rf "$DST"' EXIT
git clone --local "$SRC" "$DST/repo"
git -C "$DST/repo" apply --check "$PATCH"
git -C "$DST/repo" apply --index "$PATCH"
(cd "$DST/repo" && npm test)

If git apply --check dies, the agent failed. Do not fix the hunk by hand and pretend the loop did the work. That coat has been seen.

Checkpoint 3 — Tests the agent did not author

The fixture tests stay closed unless the write-set names them. When the model writes the tests, you are grading the model's self-esteem. Different bruise, same bone.

Your npm test must keep at least one assertion you wrote:

// test/todos.test.js — already in the fixture; do not let the agent rewrite it
import test from "node:test";
import assert from "node:assert/strict";
import { createApp } from "../src/server.js";

test("POST /todos rejects whitespace-only title", async () => {
  const app = createApp();
  const res = await app.inject({
    method: "POST",
    url: "/todos",
    payload: { title: "   " }
  });
  assert.equal(res.statusCode, 400);
});

If the agent “passes” by deleting that test, the write-set checker should already have failed. If it did not, stop polishing prompts and fix the fence.

Artifact: a decision table the grader actually runs

Do not vibe-check the agent. Map outcomes, then encode them.

Input	Expected grader result	Student consequence
Patch paths ⊆ write-set, tests pass	`PASS`	full credit for the apply step
Patch empty, chat log glowing	`FAIL empty_patch`	zero on the deliverable
Patch touches `package.json` not in the write-set	`FAIL undeclared_path`	zero, even if tests pass
Patch includes `..` or an absolute path	`FAIL illegal_path`	zero, plus an integrity flag
`git apply --check` rejects	`FAIL patch_does_not_apply`	resubmit; no hand-edits
Tests fail after apply	`FAIL tests`	the code is wrong, not the vendor
Trace missing tool names	`FAIL trace`	incomplete lab, not an auto-zero on the patch

Wire the table into one command:

// grader/grade.mjs
import fs from "node:fs";
import { spawnSync } from "node:child_process";
import { loadWriteSet, assertPatchPaths } from "./allowlist.js";

const repo = process.argv[2] ?? ".";
const patch = fs.readFileSync(`${repo}/agent.patch`, "utf8");
const ws = loadWriteSet(repo);

try {
  const touched = assertPatchPaths(patch, ws);
  const check = spawnSync("git", ["apply", "--check", "agent.patch"], {
    cwd: repo,
    encoding: "utf8"
  });
  if (check.status !== 0) {
    console.log("FAIL patch_does_not_apply");
    console.error(check.stderr);
    process.exit(2);
  }
  console.log(JSON.stringify({ result: "write_set_ok", touched }, null, 2));
} catch (err) {
  console.log("FAIL", err.message);
  process.exit(1);
}

The model can ramble. The grader cannot. That is the whole point.

A proposed loop — labeled, on purpose

This is a classroom skeleton, not a production agent. If your implementation grows a shell tool, you left the lab.

// proposed loop — unexecuted example for the write-up
export async function runLoop({ readFile, proposePatch, maxSteps = 4 }) {
  const writeSet = JSON.parse(await readFile("write-set.json"));
  const target = writeSet.files[0];
  const source = await readFile(target);
  let lastPatch = "";
  for (let step = 1; step <= maxSteps; step++) {
    // You supply the model call. The contract does not care which vendor.
    lastPatch = await proposePatch({ step, target, source });
    if (lastPatch.trim()) return lastPatch;
  }
  throw new Error("loop ended with an empty patch");
}

Notice what is missing. No writeFile. No “just this once” exception. If the loop always returns a patch for src/routes/todos.js and you never read another path, you built an if-statement. Congrats. The stretch goal will catch you.

Stretch goals

Parse @@ hunks and reject patches that change more than N lines outside a function named in write-set.json.
Make propose_patch idempotent: applying the same patch twice must fail the second time. Double-insert is not a retry strategy.
Redact anything that looks like TOKEN= from trace.jsonl before you zip the submission. Leaky traces fail the stretch, and they fail my patience.
Swap in a second bug with the same fence. If your tool layer hard-codes todos.js, you did not build an agent. You built a costume.

Fair rubric (100 points)

I put the numbers on the board on day one. No mystery meat.

Write-set committed before the first model call — 15. Git history must show the fence first. Force-pushing a backdated commit is a conversation I do not enjoy.
Patch path contract — 25. Zero undeclared files. Zero forbidden prefixes.
Clean apply + fixture tests — 30. git apply then npm test in a clone I have never logged into.
Trace completeness — 15. Every read_file / propose_patch has a timestamp and a path. I am not grading prose.
Stretch — 15. Idempotent apply or secret redaction. Pick one. Doing both does not mint extra credit.

Automatic zeros:

Missing write-set.json
In-place writes with no patch
Tests rewritten without being declared
Network or shell tools in the loop

Is that harsh? Yes. Is a surprise package.json rewrite harsher on the next homework? Also yes.

Where a shared runner actually helps

Students will try to run the loop on a laptop that already has a paid key, a dirty node_modules, and a .env from another course. That is how hidden local assumptions sneak back in. I am not grading who remembered to export a vendor key.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need the generation step off that laptop, MonkeyCode's free model access and free server option are one way to host it so the class shares a runner. The grader above still runs locally on the patch. If you never touch that server, the lab does not get worse. Steal the write-set checker either way.

Limitations — read this before you copy the rubric

This contract does not prove the model understood the bug. It proves the mutation was reviewable.

It will not catch a clever patch that stays inside todos.js and still deletes the validation you wanted. Fixture tests have to be good. If your tests are theater, the write-set is theater with better regex.

git apply is picky about whitespace and paths. That is a feature. Do not “help” students by auto-fixing hunks. You would be grading your fixer, not their loop.

Who should not use this approach:

Production incident response. You want a human on the write-set, not a bootcamp timer.
Open-ended refactors that are supposed to touch many files. Widen the write-set on purpose, or you are grading obedience.
Anyone who cannot run git apply and npm test without a GPU story. This lab is about fences, not model rankings.
Teams that must keep traces off disk for legal reasons. Then you need a different evidence rule. Do not fake one with a chat export.

What I tell the room on the way out

Can your agent name the files? Can the patch apply in a clone I have never seen? Do the tests you did not write still pass?

Three yeses, and we can argue about prompts. A screenshot of a confident paragraph is a zero. Not because I hate models. Because homework that cannot be replayed is not homework.

Now go break your own grader with a patch that sneaks in package.json. If that does not fail loud, you are not done.

If the Remainder Doesn't Shrink, It's a Zero: A Bootcamp Lab on Agent Loop Progress

Charlie Xu — Sat, 12 Sep 2026 18:17:00 +0000

Your agent can thrash for forty turns and still be stuck. Busy is not progress. In this lab you fail unless every iteration names a remainder and that remainder strictly shrinks.

I do not grade the model's vibe. I grade a number that goes down.

Why this lab, and why now? Coding agents look productive because they emit files, tool calls, and confident summaries. That is theater. If the work left to do does not get smaller, you built a spinner with extra logging.

The failure I am tired of grading

A student ships an agent that "refactors the repo." It lists files. It rereads the same module. It rewrites a comment. Then it lists files again. The transcript is long. The remainder is identical.

Sound familiar? That loop is not searching. It is pacing.

Halt contracts tell you when to stop. Token budgets tell you what the run cost. Traces tell you what happened. None of those prove the loop moved. This lab is the missing piece: a progress contract.

What a remainder is

A remainder is a typed, comparable quantity of unfinished work. Not a paragraph. Not "almost done." A number plus a unit plus a proof rule.

Pick one remainder kind for the assignment. Stick with it for the whole run.

unresolved_files: count of files still failing the assignment tests
unresolved_assertions: count of failing asserts in a frozen test file I provide
open_todos: count of TODO(lab) markers the agent is allowed to create, then must delete
schema_violations: count of JSON records that fail a schema you freeze in schema.json

If you cannot compare remainder n with remainder n-1, you do not have a remainder. You have a diary.

Lab setup

You need Node 20+, git, and a throwaway directory. Do not run this against a real repo. The agent will write files. That is the point of a sandbox, not a personality trait.

mkdir agent-remainder-lab && cd agent-remainder-lab
git init
npm init -y
mkdir src tests fixtures

Freeze the human oracle first. The agent does not get to rewrite the tests that define "done." If the model authors the exit condition, the remainder can shrink by cheating. We already buried that lesson. Not this week.

cat > tests/oracle.test.mjs <<'EOF'
import test from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync, existsSync } from 'node:fs';

test('inventory.json exists and parses', () => {
  assert.equal(existsSync('fixtures/inventory.json'), true);
  JSON.parse(readFileSync('fixtures/inventory.json', 'utf8'));
});
EOF

Seed a broken fixture so the first remainder is honest.

printf 'not-json\n' > fixtures/inventory.json
node --test tests/oracle.test.mjs; echo "oracle exit: $?"

That failing oracle is your starting remainder. Write it down before the agent starts. If you skip the baseline, later "progress" is fan fiction.

Artifact: a progress harness the grader can rerun

Every agent iteration must append one JSON line to remainder.ndjson. No pretty print. No extra keys the harness does not know. If the file is missing, the run is a zero.

// remainder-contract.mjs
export const ALLOWED_KINDS = new Set([
  'unresolved_files',
  'unresolved_assertions',
  'open_todos',
  'schema_violations',
]);

export function parseLine(line, lineno) {
  let row;
  try {
    row = JSON.parse(line);
  } catch {
    throw new Error(`line ${lineno}: not JSON`);
  }
  for (const key of ['iteration', 'kind', 'value', 'unit', 'note']) {
    if (!(key in row)) throw new Error(`line ${lineno}: missing ${key}`);
  }
  if (!Number.isInteger(row.iteration) || row.iteration < 1) {
    throw new Error(`line ${lineno}: iteration must be an integer >= 1`);
  }
  if (!ALLOWED_KINDS.has(row.kind)) {
    throw new Error(`line ${lineno}: unknown kind ${row.kind}`);
  }
  if (!Number.isInteger(row.value) || row.value < 0) {
    throw new Error(`line ${lineno}: value must be an integer >= 0`);
  }
  if (typeof row.unit !== 'string' || row.unit.length === 0) {
    throw new Error(`line ${lineno}: unit must be a non-empty string`);
  }
  return row;
}

export function assertProgress(rows) {
  if (rows.length === 0) throw new Error('no remainder rows');
  const kind = rows[0].kind;
  const unit = rows[0].unit;
  let prev = Number.POSITIVE_INFINITY;
  let stalled = 0;
  for (const row of rows) {
    if (row.kind !== kind) throw new Error('kind changed mid-run');
    if (row.unit !== unit) throw new Error('unit changed mid-run');
    if (row.value < prev) {
      stalled = 0;
      prev = row.value;
      continue;
    }
    stalled += 1;
    if (stalled > 1) {
      throw new Error(`no shrink for ${stalled} consecutive iterations at ${row.value} ${unit}`);
    }
  }
  return { kind, unit, start: rows[0].value, end: rows[rows.length - 1].value, iterations: rows.length };
}

One stall is allowed. Two consecutive non-shrinks are a fail. Why the slack? Tools flake. Networks hiccup. I am not grading luck. I am grading whether the loop can recover and still reduce the remainder.

// remainder-harness.mjs
import { readFileSync } from 'node:fs';
import { parseLine, assertProgress } from './remainder-contract.mjs';

const lines = readFileSync('remainder.ndjson', 'utf8')
  .split('\n')
  .map((l) => l.trim())
  .filter(Boolean);

const rows = lines.map(parseLine);
const summary = assertProgress(rows);
if (summary.end !== 0) {
  throw new Error(`remainder ended at ${summary.end} ${summary.unit}, expected 0`);
}
console.log(JSON.stringify(summary, null, 2));

Run it like a grader, not like a demo.

printf '%s\n' \
  '{"iteration":1,"kind":"unresolved_assertions","value":3,"unit":"asserts","note":"oracle red"}' \
  '{"iteration":2,"kind":"unresolved_assertions","value":3,"unit":"asserts","note":"reread same file"}' \
  '{"iteration":3,"kind":"unresolved_assertions","value":1,"unit":"asserts","note":"fixed parse"}' \
  '{"iteration":4,"kind":"unresolved_assertions","value":0,"unit":"asserts","note":"oracle green"}' \
  > remainder.ndjson

node remainder-harness.mjs

Swap the last non-zero pair so values go 3, 3, 3 and watch it throw. That is the whole lab. A spinner should hurt.

Checkpoints

Do these in order. Skipping ahead is how people invent a remainder after the fact.

Baseline. Run the frozen oracle. Write iteration 1 by hand. Same number the test runner reported. No rounding. No "about three."
Kind lock. Commit kind and unit in LAB.md. Changing them later is a rewrite of the assignment.
Loop wrapper. Your agent may call tools. After every tool batch, it must append one remainder line, then reread the oracle. Guessing the remainder from memory is a fail.
Stall rule. One flat iteration is legal. The second flat iteration must be a declared blocked stop with a reason, or the harness fails you.
Zero means zero. value: 0 is accepted only if node --test tests/oracle.test.mjs exits 0 on a clean checkout of your submission.

Checkpoint 3 is where most people cheat. They let the model narrate progress, then they type a shrinking sequence. I rerun the oracle. Your NDJSON is not evidence unless it matches the test runner.

A legal agent loop, not a vibe loop

Here is a skeleton. It is a lab wrapper. It is not a product.

// loop.mjs — proposed student wrapper, not a claimed production agent
import { appendFileSync } from 'node:fs';
import { spawnSync } from 'node:child_process';

function oracleFailures() {
  const r = spawnSync('node', ['--test', 'tests/oracle.test.mjs'], { encoding: 'utf8' });
  const fails = (r.stderr + r.stdout).match(/not ok /g);
  return fails ? fails.length : (r.status === 0 ? 0 : 1);
}

function record(iteration, value, note) {
  const row = {
    iteration,
    kind: 'unresolved_assertions',
    value,
    unit: 'asserts',
    note,
  };
  appendFileSync('remainder.ndjson', JSON.stringify(row) + '\n');
}

// You plug your tool-calling agent into step().
// step() may edit files. It may not edit tests/oracle.test.mjs.
async function run({ maxIters = 8, step }) {
  let prev = oracleFailures();
  record(1, prev, 'baseline');
  let stalled = 0;
  for (let i = 2; i <= maxIters; i++) {
    await step(i, prev);
    const now = oracleFailures();
    record(i, now, `after step ${i}`);
    if (now === 0) return { ok: true, iteration: i };
    if (now < prev) stalled = 0;
    else stalled += 1;
    if (stalled > 1) {
      throw new Error('remainder did not shrink; stop instead of spinning');
    }
    prev = now;
  }
  throw new Error('hit maxIters with remainder > 0');
}

export { run, oracleFailures };

Notice what is missing. No retry storm. No "one more pass." The loop is allowed to work only while the remainder falls. That is harsh. It is also how you keep a weekend lab from becoming an unbounded cloud bill.

Where a free coding agent actually belongs

You can implement step() with any model that can edit files. The grade is the harness, not the vendor.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I mention MonkeyCode here for one lab reason: students who do not want to paste personal API keys into a bootcamp wrapper can use its free model access and the free server option to run step() off their laptop. That is convenience. It is not extra credit. If the remainder file is fake, the free server will not save you.

Do not treat the remote box as a second brain. Treat it as a machine that still has to leave remainder.ndjson and a green oracle in the repo I clone.

Fair grading rubric

I grade artifacts I can rerun. I do not grade screenshots of a chat.

Score	What I clone	Remainder rule	Oracle
0	Missing `remainder.ndjson`, or tests edited	n/a	n/a
1	NDJSON parses, kind locked	Values wander or stall twice	Still red
2	NDJSON monotonic with one stall max	Ends above 0	Red or flaky
3	Harness exits 0	Ends at 0	Green on my machine
4	Score 3 plus a `blocked` path	Declares block instead of spinning	Green or honest block

Stretch does not replace a broken remainder. A beautiful agent that cannot shrink a number is still a 1.

Stretch goals

Only after a 3.

Replay. Delete node_modules, recopy fixtures, rerun loop.mjs. Same remainder sequence shape, same final zero. Hidden local caches are a different lab. Do not smuggle them back in.
Two remainders, one primary. You may log a secondary metric (diff lines, tool calls) but the harness still keys off one kind. Secondary numbers are comments. They are not the grade.
Poisoned tool output. Feed step() a tool result that claims the oracle is green when it is not. The wrapper must remeasure with oracleFailures(), not trust the tool text.
Budget sibling. Cap iterations, not vibes. If you already did a token-budget lab, do not double-count tokens here. This grade is progress, not spend.

That third stretch is the one I actually care about. Models will believe their own tools. Your wrapper should not.

Limitations, and who should skip this

This remainder model is crude. Some real work is non-monotone: you add a failing test on purpose, remainder jumps, then it falls. I ban that jump in the core lab because I cannot fairly grade "strategic getting worse" in a two-hour window. If your research loop needs exploratory regressions, this rubric will punish you. Use a different contract.

Skip this lab if you are driving production incident response. A shrinking unit-test count is not a blast radius. Skip it if your agent only answers questions and never touches files. There is no remainder to shrink. Skip it if you cannot freeze an oracle. Then you are grading prose again.

Also skip any remainder you cannot recompute from the repo. "Confidence: 0.81" is not a remainder. I cannot rerun your model's self-esteem.

What I want in your writeup

Keep it short. I read the NDJSON first.

The kind and unit you locked, in one sentence
The baseline command and the number it printed
One iteration that stalled, if any, and why you allowed it
The final harness output
One thing the agent wanted to do that would have grown the remainder

If your writeup is longer than the remainder file, you are hiding. Cut it.

Close

An agent loop without a shrinking remainder is a while (true) with better copy. Name the leftover work. Measure it after every step. Stop when it hits zero, or when it refuses to fall.

That is the whole grade. Everything else is narration.

Name the Stop or Take a Zero: A Bootcamp Lab on Agent Halt Contracts

Charlie Xu — Fri, 11 Sep 2026 15:49:57 +0000

An agent that never stops is not deep. It is unfinished work. In this lab I fail any submission that cannot name why the loop died.

You have seen the nearby failure modes. Tests the model wrote for itself. Tools that keep changing shape. Green on a laptop and red on the grader. This one is smaller, and students still miss it.

What was the last action? How many tool calls? Who killed the process? If you cannot answer, I cannot grade you. Why would I?

The problem in the room

Students wrap a model in a while (true). The model asks for another tool. Then another. A mock search returns “refine your query.” The loop is polite. It never throws.

Then someone says the agent is “still thinking.” Still thinking is not a grade. It is a hung Node process chewing a shared machine until a human notices.

I want a halt contract. Not a vibe.

A halt contract is a recorded reason the loop ended, plus proof it ended inside a budget. Steps. Wall clock. That is the whole product of this lab.

What you will ship

You will ship a tiny JavaScript harness that runs one task and writes a halt record. I read that file. I do not read your chat screenshot.

The harness must include:

max_steps — a hard cap on model turns
wall_clock_ms — a hard cap on real time
halt_reason — one value from a frozen enum
steps_used and tool_calls
a JSON file on disk, because stdout lies when the process is killed

If the file is missing, the grade is zero. No oral extra credit.

Setup

You need Node 20+. You need a way to call a chat model. You do not need a paid key to complete the checkpoints.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I run this lab with MonkeyCode's free model access and the free server option so the kill switch lives on a machine the grader controls. Your laptop’s Ctrl+C is not evidence. The wall clock on the grader is.

Already have another endpoint? Use it. The harness must not care which model sits behind callModel. That is the point of grading the loop, not the vendor.

node --version
mkdir halt-lab && cd halt-lab
npm init -y

Create four files:

halt.mjs — frozen reasons
model.mjs — stub first, real model later
tools.mjs — mocks, including one that wants to spin
harness.mjs — the loop that must die on purpose

Freeze the reasons before anyone prompts

If a student invents halt_reason: "almost", I treat it as missing. Freeze the enum in week one. Do not “extend it in spirit.”

// halt.mjs
export const HALT_REASONS = Object.freeze([
  "completed",
  "max_steps",
  "wall_clock",
  "tool_error",
  "schema_fail",
  "empty_response",
]);

completed means a final answer that passes your schema. max_steps means the cap won. wall_clock means time died first. The rest are failures. They are not “retry in a trench coat.”

Can your harness produce max_steps on purpose? If not, you do not have a cap. You have a comment.

Lab starter: the loop

This is a starter. It is not a platform. It writes the halt record even when the run is ugly.

// harness.mjs
import { writeFileSync } from "node:fs";
import { HALT_REASONS } from "./halt.mjs";
import { callModel } from "./model.mjs";
import { runTool } from "./tools.mjs";

const MAX_STEPS = 8;
const WALL_CLOCK_MS = 15_000;

function isFinalAnswer(response) {
  return response && typeof response.answer === "string" && response.answer.length > 0;
}

export async function runTask(task) {
  const started = Date.now();
  const record = {
    task,
    max_steps: MAX_STEPS,
    wall_clock_ms: WALL_CLOCK_MS,
    steps_used: 0,
    tool_calls: [],
    halt_reason: null,
    answer: null,
  };

  const messages = [
    {
      role: "system",
      content:
        'Solve the task. Call tools only from the allowlist. Final answer must be JSON: {"answer": string}.',
    },
    { role: "user", content: task },
  ];

  try {
    for (let step = 1; step <= MAX_STEPS; step++) {
      if (Date.now() - started > WALL_CLOCK_MS) {
        record.halt_reason = "wall_clock";
        break;
      }

      record.steps_used = step;
      const response = await callModel(messages);

      if (!response) {
        record.halt_reason = "empty_response";
        break;
      }

      if (response.tool_call) {
        record.tool_calls.push({
          step,
          name: response.tool_call.name,
          args: response.tool_call.args,
        });
        const result = await runTool(response.tool_call);
        messages.push({ role: "assistant", content: JSON.stringify(response) });
        messages.push({ role: "tool", content: JSON.stringify(result) });
        continue;
      }

      if (isFinalAnswer(response)) {
        record.answer = response.answer;
        record.halt_reason = "completed";
        break;
      }

      record.halt_reason = "schema_fail";
      break;
    }

    if (!record.halt_reason) {
      record.halt_reason = "max_steps";
    }
  } catch (err) {
    record.halt_reason = "tool_error";
    record.error = String(err.message || err);
  }

  if (!HALT_REASONS.includes(record.halt_reason)) {
    record.halt_reason = null;
  }

  writeFileSync("halt-contract.json", JSON.stringify(record, null, 2));
  return record;
}

const task = process.argv[2] || "Find the lab password in the lookup tool.";
runTask(task).then((record) => {
  console.log(record.halt_reason, record.steps_used);
});

Notice what I refused to do. I did not let the model vote for “one more try.” I did not catch a timeout and loop until hope appeared. I wrote the file before returning.

If the process dies, I still want that file. Stretch goal: write it on SIGTERM.

A tool that tries to trap you

A lab without a hostile mock is a demo. This lookup always says “try again.” Students love to obey that sentence. Forever.

// tools.mjs
const ALLOWLIST = new Set(["lookup"]);

export async function runTool(call) {
  if (!ALLOWLIST.has(call.name)) {
    throw new Error(`tool not allowlisted: ${call.name}`);
  }

  if (call.name === "lookup") {
    const delay = Number(process.env.LOOKUP_SLEEP_MS || 0);
    if (delay) {
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
    return { ok: true, text: "No exact match. Refine your query and look up again." };
  }

  throw new Error("unreachable");
}

Start with a stub model that always requests lookup. Checkpoint 3 should be green before anyone touches a real endpoint.

// model.mjs — stub. Swap this after the looping-tool case is boring.
export async function callModel(_messages) {
  return { tool_call: { name: "lookup", args: { q: "again" } } };
}

After eight lookups, what is halt_reason? If you said completed, you graded the model’s confidence. I grade the cap.

Checkpoints

Do these in order. The writeup comes last, or it is fan fiction.

The file exists. Run node harness.mjs. halt-contract.json must appear even when the model returns garbage.
The enum is closed. Print halt_reason. One of six strings. No synonyms. No null after a normal stop.
Max steps fires. Keep the looping stub. After MAX_STEPS, the harness process must exit. The record must say max_steps. The task is still a fail. That is correct.
Wall clock fires. LOOKUP_SLEEP_MS=3000 WALL_CLOCK_MS=2000 — patch the constant or read it from env, your choice, but freeze it in the record. You must get wall_clock, not a late completed.
Completed is rare. Only a final { "answer": "..." } may use completed. A chatty paragraph is schema_fail.

If checkpoint 4 is green on your laptop and red on the server, you measured the wrong clock. Use Date.now() inside the harness. “How long I watched the terminal” is not a timer.

Fair grading rubric

I score the harness, not the model’s IQ. A weak free model should still be able to fail closed. That is the skill.

Checkpoint	Points	Automatic fail if
Halt file written	20	missing file, invalid JSON
Reason in the enum	20	extra slogans, `null`, synonyms
Max-steps case	20	process still running, or `completed` on the looping tool
Wall-clock case	20	`completed` after timeout, or no kill
Schema gate	10	prose accepted as final
Writeup quotes the record	10	“it worked” with no JSON

Passing score: 80. You can miss the writeup. You cannot miss the kill switch.

Rules I will not bargain:

Same MAX_STEPS and WALL_CLOCK_MS for every student. No private raises after a friend flakes.
Model quality is not extra credit. A correct halt on a looping tool is full credit for that row.
If the free server is slower than a laptop, the wall clock still counts. That is shipping, not a bug in the rubric.
Flakes: run the max-steps case three times. If any run lacks a file, that row is zero. One green run is not a sample size.

Proposed grader — label it proposed, then make it boringly deterministic:

// grade.mjs
import { readFileSync } from "node:fs";

const ENUM = new Set([
  "completed",
  "max_steps",
  "wall_clock",
  "tool_error",
  "schema_fail",
  "empty_response",
]);

export function gradeRecord(path, expect) {
  const record = JSON.parse(readFileSync(path, "utf8"));
  const fails = [];
  if (!ENUM.has(record.halt_reason)) fails.push("reason");
  if (expect.reason && record.halt_reason !== expect.reason) fails.push("wrong_reason");
  if (expect.maxSteps && record.steps_used > expect.maxSteps) fails.push("over_steps");
  if (record.halt_reason === "completed" && typeof record.answer !== "string") {
    fails.push("schema");
  }
  return { ok: fails.length === 0, fails, record };
}

const expect = { reason: process.env.EXPECT_REASON, maxSteps: 8 };
const result = gradeRecord("halt-contract.json", expect);
console.log(JSON.stringify(result, null, 2));
process.exit(result.ok ? 0 : 1);

node harness.mjs
EXPECT_REASON=max_steps node grade.mjs

I do not want a screenshot of a chat UI. I want that second command.

Stretch goals

Do not start these until checkpoints 1–5 feel boring.

Write the record in finally and on SIGTERM.
Cap tool time separately from model time. A slow lookup should not steal the whole wall clock with no name.
Reject unknown tool names before you call the model again. Discovery is not permission.
Emit steps_used / max_steps. If it is always 1.0, your cap is too tight or your prompts never finish. Say which one in the writeup.
Swap the stub for a real free-model endpoint. The looping-tool case must still halt. If a stronger model “escapes” by inventing a final answer with no tool evidence, that is schema_fail unless lookup actually returned the fact.

Limitations, and who should skip this

This is a bootcamp contract. It is not an agent runtime.

It does not sandbox the filesystem. A tool that writes under $HOME can still ruin a box. Use a throwaway directory.
It does not prove the answer is true. It proves the loop died for a named reason.
It does not handle multi-agent graphs, human pauses, or streaming tokens as a heartbeat. A long honest generation can trip wall_clock. Raise the budget in the spec. Do not delete the clock.
Free model access and a free server are availability options, not a claim about quotas, hardware, model names, or how long anything stays free. Do not write those numbers into the halt record. They are not halt reasons.
If you are scoring open-ended research agents that should run for hours, this rubric will fail them. Good. Use a different lab.

Who should not use this approach? Anyone grading product quality by “did the blog post get written.” Anyone who cannot freeze the tool allowlist. Anyone whose “agent” is a single function call. You do not need a loop cap for an if-statement. You need fewer slides.

What I want in the writeup

Quote the halt record. Then answer three questions. Short English. No hero narrative.

Which halt reason did the looping tool produce, and why is that a pass for the harness?
What happens if you catch wall_clock and retry? Who pays for that retry?
If you move this harness off your laptop, is the same WALL_CLOCK_MS still honest?

If you cannot answer question 3, you built a demo. Bring it back when the loop can die in public.

Steal the rubric. Tighten the enum. Then make the process stop.

Receipt or It Didn't Happen: A Bootcamp Lab on Agent Token Budgets

Charlie Xu — Wed, 09 Sep 2026 16:05:22 +0000

The grade is not the final string. The grade is the receipt. If your agent answered the ticket and you cannot show how many tokens it burned, you did not finish the lab.

Correctness without a budget is a demo. Demos get applause. Demos also retry until the model is lucky. On a laptop that habit is sloppy. On a shared box it is rude. You starve the next student. You hide cost. You ship vibes.

This lab freezes one rule: no receipt, no points. Not because tokens are morally interesting. Because an agent you cannot meter is an agent you cannot grade twice.

Why this lab exists

I keep seeing the same failure in bootcamp agent homework. The student pastes a giant system prompt. The loop retries five times. The answer finally looks right. The PR is green. Nobody counted a single token.

Ask a mean question. Did they solve the task, or did they buy the task with retries?

Loop-call caps are not enough. Twelve tool calls can still dump a novel into the prompt every round. Runtime contracts are not enough. The JSON can be valid and still cost a fortune. A second machine can still be wasteful if both machines are free and nobody is watching the meter.

So we grade the meter.

What you will build

A tiny agent that must answer one fixture task under a hard course budget. The agent writes receipt.json. The grader reads that file first. The answer is checked second.

If the receipt is missing, malformed, over budget, or leaking secrets, the lab is a zero. Yes, even if the answer is perfect. That is the point.

Labeled as a course fixture, not a production billing system:

Budget: 8000 estimated tokens for the whole run
Max model calls: 4
Task: refund-policy-v1 (a short policy question with a known oracle)
Estimator: UTF-8 bytes divided by 4, rounded up

That estimator is a lab ruler, not the vendor tokenizer. We use one ruler so two students can be compared. We do not pretend it matches a real invoice.

Setup

Work in an isolated directory. If you are on a shared host, make the directory yours. Do not write into /tmp/agent. Someone else is already there.

python3 --version   # 3.11+
mkdir -p "$HOME/labs/token-receipt/$USER"
cd "$HOME/labs/token-receipt/$USER"
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip requests python-dotenv
printf '.venv\n.env\nreceipt.json\n__pycache__/\n' > .gitignore

Create .env locally. Never commit it. The grader will fail you if a secret shows up in the receipt.

cat > .env << 'EOF'
LLM_BASE_URL=http://127.0.0.1:8080/v1
LLM_API_KEY=replace-me
LLM_MODEL=course-default
STUDENT_ID=changeme
TOKEN_BUDGET=8000
EOF

LLM_MODEL=course-default is a placeholder. Pin whatever endpoint your instructor hands you. Do not invent a brand name in the writeup. The receipt stores the raw string you actually called.

Need a shared endpoint so the whole cohort hits the same URL? I run this lab against MonkeyCode when I want free model access and a free server option instead of twelve personal API keys. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a GPU, a model name, or that the free tier lasts forever. If the class endpoint moves, you change .env. The receipt still has to add up.

The artifact: a receipt-first harness

Four files. Copy them. Then break them on purpose.

`tokens.py`

# Course estimator. Not a vendor tokenizer. Not a billing API.
from __future__ import annotations


def estimate_tokens(text: str) -> int:
    if not text:
        return 0
    nbytes = len(text.encode("utf-8"))
    return max(1, (nbytes + 3) // 4)

Why bytes, not len(text)? Because a student once stuffed emoji into the system prompt and called it "short." Characters lie. Bytes lie less.

`receipt.py`

from __future__ import annotations

import json
import os
import re
import socket
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

SECRET_RE = re.compile(r"(api[_-]?key|token|secret|password)\s*[:=]\s*\S+", re.I)


@dataclass
class Receipt:
    student_id: str
    hostname: str
    pid: int
    started_at: str
    ended_at: str | None = None
    task_id: str = "refund-policy-v1"
    model: str = ""
    base_url_host: str = ""
    prompt_tokens: int = 0
    completion_tokens: int = 0
    calls: int = 0
    budget: int = 8000
    final_answer: str = ""
    over_budget: bool = False
    notes: list[str] = field(default_factory=list)

    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

    def add_call(self, prompt: str, completion: str, estimate_tokens) -> None:
        self.calls += 1
        self.prompt_tokens += estimate_tokens(prompt)
        self.completion_tokens += estimate_tokens(completion)
        self.over_budget = self.total_tokens > self.budget

    def seal(self, answer: str) -> None:
        self.final_answer = answer.strip()
        self.ended_at = datetime.now(timezone.utc).isoformat()
        self.over_budget = self.total_tokens > self.budget

    def to_json(self) -> dict[str, Any]:
        data = asdict(self)
        data["total_tokens"] = self.total_tokens
        return data


def new_receipt(budget: int, model: str, base_url: str) -> Receipt:
    host = ""
    try:
        # Store host only. Never store the key, never store the full URL.
        host = base_url.split("://", 1)[-1].split("/", 1)[0]
    except Exception:
        host = "unknown"
    return Receipt(
        student_id=os.environ.get("STUDENT_ID", "unknown"),
        hostname=socket.gethostname(),
        pid=os.getpid(),
        started_at=datetime.now(timezone.utc).isoformat(),
        budget=budget,
        model=model,
        base_url_host=host,
    )


def write_receipt(path: Path, receipt: Receipt) -> None:
    payload = json.dumps(receipt.to_json(), indent=2) + "\n"
    if SECRET_RE.search(payload):
        raise ValueError("refusing to write a receipt that looks like it contains a secret")
    path.write_text(payload, encoding="utf-8")

Notice what is missing. No API key. No full URL. No raw prompt dump. A flight recorder is a different lab. This one is a cash register.

`agent_lab.py`

from __future__ import annotations

import os
import sys
from pathlib import Path

import requests
from dotenv import load_dotenv

from receipt import new_receipt, write_receipt
from tokens import estimate_tokens

TASK_ID = "refund-policy-v1"
QUESTION = (
    "A customer bought a cable 12 days ago. It still works. "
    "They want a refund because the color is wrong. "
    "Policy: refunds within 14 days for unused items; color preference is allowed. "
    "Reply with exactly YES or NO, then a one-line reason."
)
SYSTEM = (
    "You are a refund clerk. Use only the policy in the user message. "
    "Do not invent extra rules. Keep the reply under 40 words."
)


def chat(base_url: str, api_key: str, model: str, messages: list[dict]) -> str:
    url = base_url.rstrip("/") + "/chat/completions"
    headers = {"Authorization": f"Bearer {api_key}"}
    body = {
        "model": model,
        "messages": messages,
        "temperature": 0,
        "max_tokens": 80,
    }
    resp = requests.post(url, json=body, headers=headers, timeout=60)
    resp.raise_for_status()
    data = resp.json()
    return data["choices"][0]["message"]["content"]


def main() -> int:
    load_dotenv()
    base_url = os.environ["LLM_BASE_URL"]
    api_key = os.environ["LLM_API_KEY"]
    model = os.environ.get("LLM_MODEL", "course-default")
    budget = int(os.environ.get("TOKEN_BUDGET", "8000"))
    out = Path("receipt.json")

    rec = new_receipt(budget, model, base_url)
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": QUESTION},
    ]
    answer = ""
    try:
        for attempt in range(1, 5):
            if rec.total_tokens >= budget:
                rec.notes.append("stopped before call: budget exhausted")
                break
            prompt_blob = SYSTEM + "\n" + QUESTION + "\n" + answer
            completion = chat(base_url, api_key, model, messages)
            rec.add_call(prompt_blob, completion, estimate_tokens)
            answer = completion.strip()
            if answer.upper().startswith("YES") or answer.upper().startswith("NO"):
                rec.notes.append(f"accepted on attempt {attempt}")
                break
            messages.append({"role": "assistant", "content": completion})
            messages.append(
                {"role": "user", "content": "Format invalid. Reply YES or NO, then one line."}
            )
            rec.notes.append(f"retry {attempt}: missing YES/NO prefix")
        rec.seal(answer)
        write_receipt(out, rec)
    except Exception as exc:
        rec.notes.append(f"error: {type(exc).__name__}")
        rec.seal(answer)
        try:
            write_receipt(out, rec)
        except Exception:
            pass
        print(f"failed: {exc}", file=sys.stderr)
        return 1

    print(f"answer={rec.final_answer!r}")
    print(f"tokens={rec.total_tokens}/{rec.budget} calls={rec.calls}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Four calls max. Eighty completion tokens max per call. Temperature zero. That is not because temperature zero is magic. It is because lucky retries are how students blow a shared budget and then swear the code is deterministic.

`grade.py`

from __future__ import annotations

import json
import os
import re
import sys
from pathlib import Path

REQUIRED = {
    "student_id",
    "hostname",
    "pid",
    "started_at",
    "ended_at",
    "task_id",
    "model",
    "base_url_host",
    "prompt_tokens",
    "completion_tokens",
    "total_tokens",
    "calls",
    "budget",
    "final_answer",
    "over_budget",
}
SECRET_RE = re.compile(r"(api[_-]?key|sk-|bearer\s+\S+)", re.I)


def fail(msg: str) -> int:
    print(f"FAIL: {msg}")
    return 1


def main() -> int:
    path = Path("receipt.json")
    if not path.exists():
        return fail("receipt.json missing")
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except json.JSONDecodeError:
        return fail("receipt.json is not JSON")

    missing = REQUIRED - set(data)
    if missing:
        return fail(f"missing fields: {sorted(missing)}")

    raw = path.read_text(encoding="utf-8")
    if SECRET_RE.search(raw):
        return fail("receipt looks like it leaked a secret")

    if data.get("task_id") != "refund-policy-v1":
        return fail("wrong task_id")
    if not data.get("student_id") or data["student_id"] == "changeme":
        return fail("set STUDENT_ID")
    if int(data["calls"]) < 1:
        return fail("zero model calls")
    if int(data["completion_tokens"]) < 1:
        return fail("zero completion tokens; empty model output is not a solution")
    if bool(data["over_budget"]) or int(data["total_tokens"]) > int(data["budget"]):
        return fail(f"over budget: {data['total_tokens']}/{data['budget']}")
    if int(data["total_tokens"]) != int(data["prompt_tokens"]) + int(data["completion_tokens"]):
        return fail("total_tokens does not equal prompt + completion")

    answer = str(data["final_answer"]).strip().upper()
    if not answer.startswith("YES"):
        return fail("oracle expected YES for unused item inside 14 days, color preference allowed")

    print("PASS")
    print(f"tokens={data['total_tokens']}/{data['budget']} calls={data['calls']}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it like a student, then like a grader.

python agent_lab.py
python grade.py
cat receipt.json

If grade.py prints PASS and you never opened receipt.json, you still failed the lab in spirit. Read the file. That is the whole exercise.

Checkpoints

Do these in order. Do not skip to stretch goals because the model sounded confident.

Receipt exists after failure. Kill the network on purpose (LLM_BASE_URL=http://127.0.0.1:1) and confirm receipt.json still appears with an error note. A crash is not an excuse to skip the meter.
Budget is a hard stop. Temporarily set TOKEN_BUDGET=50 and confirm the grader fails with over budget. If your agent keeps calling the model anyway, you built a diary, not a budget.
Oracle still matters. Change the fixture in your head: used item, day 15. You should get NO. If you only test the happy path, you are grading luck again.
No secrets in the artifact. Put LLM_API_KEY=sk-this-is-fake in .env, run a successful pass, then grep -Ei 'sk-|api_key' receipt.json must be empty.
Shared-host isolation. Print hostname and pid from the receipt. Two students on one box must not share a working directory. If your files land in /tmp, start over.

Checkpoint 2 is the one people argue with. "But my answer was right." Cool. The lab is not "was the refund allowed." The lab is "can you finish inside the meter."

Fair grading rubric

Total: 20 points. Partial credit is allowed only where the receipt is valid.

Points	What I actually score
4	`receipt.json` present, valid JSON, all required fields
3	`total_tokens == prompt_tokens + completion_tokens`, `over_budget` consistent
3	Under the published budget; grader `PASS` on the fixture
3	Oracle: `YES` plus a one-line reason, no extra essay
3	No secrets, no full URL, no key material in the receipt
2	Isolation: unique workdir, `STUDENT_ID` set, hostname/pid filled
2	Checkpoint 1: receipt still written when the endpoint is down

Automatic zeros:

Missing receipt
Over budget
Secret-shaped strings in receipt.json
STUDENT_ID=changeme
Editing grade.py to print PASS

I do not grade prose style. I do not grade how "agentic" the logs feel. I grade whether you can repeat the run without lighting the shared endpoint on fire.

Stretch goals

Only after a clean PASS.

Prompt cache. Hash SYSTEM + QUESTION. If the hash hits a local cache.json, do not spend tokens. Add cache_hits to the receipt. Stretch grader: fail if the second run of the same task makes a model call.
Two students, one box. Open two directories with two STUDENT_IDs. Run in parallel. Neither receipt may be overwritten. If you used a fixed receipt.json in /tmp, you will learn this the loud way.
Shrink the system prompt. Keep the oracle. Cut SYSTEM until estimated prompt tokens drop without changing the answer. Record before/after in notes. Do not claim a vendor cost savings. You only have the course ruler.
Fail closed on empty completions. If completion_tokens == 0, treat it as a platform miss, not a student success. The starter grader already does this. Keep that behavior if you rewrite the loop.

What this is not

This is not a model bake-off. We are not publishing tokens-per-second. We are not naming a champion vendor. We are not pretending byte/4 equals a production tokenizer.

It is also not a security boundary. A free shared server is still a shared server. Other processes exist. Disk is not private because your folder name is cute. If the homework ever includes real customer tickets, stop. Use synthetic fixtures.

Who should skip this approach?

Anyone billing real customers with this estimator. You will be wrong.
Anyone who needs a secret store, VPC, or audit log. .env on a class box is a convenience, not a control.
Anyone whose "agent" is a single hardcoded if with no model call. There is nothing to meter. Pick a different lab.
Instructors who want to grade only the final sentence. You will train retry-until-green. Please do not.

The uncomfortable lesson

Free inference makes this worse before it makes it better. When the model feels free, students write novels into the system prompt. When the server feels free, they leave processes running. The receipt is how you make "free" visible.

Would I ship this harness to production? No. Would I accept an agent PR that cannot show its token receipt? Also no.

Run the fixture. Break the budget. Read the JSON. If you want the cohort on one endpoint instead of a pile of personal keys, MonkeyCode's free model access and free server option is the setup I use for this lab. Fork the harness, set TOKEN_BUDGET lower than you think you need, and see which prompt you actually have to delete.

No Trace, No Grade: A Bootcamp Lab on Agent Flight Recorders

Charlie Xu — Tue, 08 Sep 2026 15:06:56 +0000

I don't grade chat tabs anymore. If another student cannot reconstruct what your agent did from a log, you did not finish the lab.

That sounds harsh. It is. A screenshot of a green terminal is a vibe, not evidence. This lab makes the session itself the deliverable.

Why the chat tab fails as homework

You know the ritual. Student pastes a passing test run. I ask one follow-up. Which file did the agent edit first?

Silence. Or a shrug. Or a 40-message thread nobody can replay on a different machine.

So the contract changed. You may use an assistant. You may use a shared free model. You may get lucky on the first try. You still owe me a flight recorder.

What you will actually submit

Not a prompt dump. Not a PNG. A JSONL log plus a git diff that a classmate can follow without asking you what "it" meant.

The coding problem underneath is deliberately boring: fix a broken change-maker. The grade is not "did the model notice the off-by-one." The grade is "can a human replay your path."

Lab setup

You need Node 20+, git, and a directory that belongs to you. No paid API key is required. If you are on a shared box, pick a workspace name before you touch files. Why? Because ~/lab is not a unique identity. It is a collision waiting to happen.

export WORKSPACE="$HOME/flight-lab-$USER"
mkdir -p "$WORKSPACE" && cd "$WORKSPACE"
git init
npm init -y

Create three files: broken production code, tests, recorder.

The broken production file

// change.js
export function makeChange(amountCents, coins = [25, 10, 5, 1]) {
  if (!Number.isInteger(amountCents) || amountCents <= 0) {
    throw new Error("amountCents must be a positive integer");
  }

  const result = {};
  let left = amountCents;
  for (const coin of coins) {
    result[coin] = Math.floor(left / coin);
    left = left % coin;
  }
  return result;
}

The bug is not subtle if you read the tests. Zero cents is a legal empty drawer. A custom coin list that cannot make change must throw. Greedy leftover is not "close enough."

The tests you are not allowed to rewrite

// change.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { makeChange } from "./change.js";

test("zero is a valid empty drawer", () => {
  assert.deepEqual(makeChange(0), {});
});

test("41 cents with USD coins", () => {
  assert.deepEqual(makeChange(41), { 25: 1, 10: 1, 5: 1, 1: 1 });
});

test("impossible coin set throws", () => {
  assert.throws(() => makeChange(3, [2]), /cannot make change/);
});

Run them once so you feel the failure. Do not "fix" the test names to match a hallucinated API.

node --test change.test.js

Red. Good. That is checkpoint zero.

The flight recorder

This wrapper does not call a model. It records what you (or your agent) did. Every command, every exit code, every file you claim you touched.

// recorder.mjs
import { spawn } from "node:child_process";
import { appendFile, mkdir } from "node:fs/promises";
import { dirname, resolve } from "node:path";

const LOG = resolve(process.env.WORKSPACE || ".", "session.jsonl");
const SECRET = /(api[_-]?key|token|secret|password|authorization)/i;

function redact(value) {
  if (typeof value !== "string") return value;
  return SECRET.test(value) ? "[redacted]" : value;
}

async function write(event) {
  await mkdir(dirname(LOG), { recursive: true });
  const line = JSON.stringify({
    ts: new Date().toISOString(),
    workspace: process.env.WORKSPACE || process.cwd(),
    ...event,
  });
  await appendFile(LOG, line + "\n");
}

const [cmd, ...args] = process.argv.slice(2);
if (!cmd) {
  console.error("usage: node recorder.mjs <command> [args...]");
  process.exit(2);
}

await write({
  type: "cmd.start",
  cmd: redact(cmd),
  args: args.map(redact),
});

const child = spawn(cmd, args, { stdio: "inherit", shell: false });
child.on("exit", async (code, signal) => {
  await write({ type: "cmd.end", cmd: redact(cmd), code, signal });
  process.exit(code ?? 1);
});

Wrap the test runner. Wrap git diff. Wrap whatever the agent wants to shell out. If it is not in the log, it did not happen.

node recorder.mjs node --test change.test.js

How an agent is allowed to help

Ask it to read change.test.js before it touches change.js. Ask it to propose a patch, not a vibe. Then you type the commands through the recorder.

A useful prompt looks like a ticket, not a prayer:

Read change.test.js. Do not edit tests.
Make change.js satisfy all three tests.
Zero cents returns {}.
If leftover remains, throw Error("cannot make change").
After editing, tell me the exact command to re-run tests.
Do not print secrets. Do not invent extra files.

Did the model skip the tests and rewrite the function signature anyway? That goes in the log. Recovery is part of the grade. A clean first try is nice. A messy try with a recorded correction is honest.

I point students who do not have a paid API key at MonkeyCode's free model access and free server option so the lab is about traces, not billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product is not the grader. Your JSONL is.

If you use a shared free server, your workspace variable is not optional. Two students writing change.js in the same home directory will grade each other's accidents. Ask me how I know. Better: don't.

Checkpoints

Do these in order. Skipping is how you submit a novel instead of a lab.

Recorder writes JSONL. After one command you should have at least a cmd.start and a cmd.end line. Open the file. If you cannot parse a line with JSON.parse, you did not finish this checkpoint.
Secrets cannot survive. Put a fake API_KEY=totally-real in an argument on purpose, then confirm the log says [redacted]. If your recorder stores the raw string, you fail this checkpoint even if the kata is green.
Tests stay frozen. git diff -- change.test.js must be empty. The agent does not get to negotiate the oracle.
Final verification is recorded. The last cmd.end for node --test must have code: 0. A passing run you forgot to wrap does not count.
Peer replay. A classmate reads only session.jsonl and git log --patch, then explains your first failed test in one sentence. If they cannot, your log is a diary, not a recorder.

Sample log shape you are aiming for:

{"ts":"2026-09-08T18:01:11.000Z","workspace":"/home/alex/flight-lab-alex","type":"cmd.start","cmd":"node","args":["--test","change.test.js"]}
{"ts":"2026-09-08T18:01:11.180Z","workspace":"/home/alex/flight-lab-alex","type":"cmd.end","cmd":"node","code":1,"signal":null}

Ugly? Good. Ugly is inspectable.

A tiny grader you can run in CI

This is not intelligence. It is a shape check. I will still read the patch. The script only stops the "I forgot the log" submissions.

// grade-trace.mjs
import { readFile } from "node:fs/promises";

const raw = await readFile("session.jsonl", "utf8");
const events = raw.trim().split("\n").map((line, i) => {
  try {
    return JSON.parse(line);
  } catch {
    throw new Error(`line ${i + 1} is not JSON`);
  }
});

const errors = [];
if (events.length < 2) errors.push("log too short");
if (events.some((e) => /api[_-]?key|sk-|bearer /i.test(JSON.stringify(e)))) {
  errors.push("possible secret in log");
}

const testEnds = events.filter(
  (e) => e.type === "cmd.end" && Array.isArray(e.args) && e.args.includes("--test")
);
if (!testEnds.length) errors.push("no recorded test run");
else if (testEnds.at(-1).code !== 0) errors.push("final recorded test run is not green");

if (errors.length) {
  console.error(errors.join("\n"));
  process.exit(1);
}
console.log(`ok: ${events.length} events`);

node grade-trace.mjs

If this script is green and your diff is nonsense, you still fail. Tools catch absence. Humans catch theater.

Fair grading rubric

I score 20 points. Passing is 14. Green tests without a replayable log cap at 8. Why? Because the kata is the excuse. The recorder is the lab.

Criterion	Points	Pass looks like	Fail looks like
Frozen tests	4	`change.test.js` untouched	renamed assertions, deleted the zero case
Recorded commands	4	JSONL with start/end pairs	chat export, screenshots, empty file
Secret hygiene	3	redaction works on a planted key	raw tokens, `.env` pasted into the log
Correctness	4	`node --test` recorded with exit 0	green on a different file, skipped tests
Peer replay	3	classmate reconstructs the first failure	log says "fixed it" with no commands
Workspace isolation	2	`WORKSPACE` includes your user	wrote into a shared `~/lab`

I do not award bonus points for a fancier model. I do not subtract points because you used a free shared server. The trace is the equalizer. The model is not.

Stretch goals

Only after the rubric is green.

Add a files.touched event by wrapping writes, or by recording git status --porcelain after each command.
Require a hypothesis string before the first edit. If the agent cannot state the bug in one sentence, you are not allowed to patch yet.
Replay the command list on a second checkout. If the commands do not apply cleanly, your log skipped a manual step.
Teach the recorder to refuse curl | bash and any command line that matches the secret regex. Refusal should be an event, not a crash with no log.

Limitations, and who should skip this

A flight recorder is not a proof of understanding. Students can still paste a classmate's JSONL. Pair the log with a 90-second live question: "What happens if coins are unsorted?" If they cannot answer, the trace was theater.

This also does not replay model weights. Different free models will not emit the same patch. That is fine. We grade the human-visible path, not token equality.

Do not use this as a production audit framework. JSONL in a homework folder is not chain-of-custody. Do not use it as an excuse to paste customer secrets into any assistant, free or paid. Do not use it for closed-book exams where tool use is the cheating. And if your course cannot give every student a private directory on a shared server, fix isolation before you add agents. A shared change.js turns the rubric into noise.

What I want back from you

Run checkpoint 3. Then paste one redacted cmd.end line in the comments, not the whole chat. I care whether the shape is boring and parseable. If you needed a shared free model and server to produce that line, say so. The interesting part is still the log.

Second Machine, Same Grade: A Bootcamp Lab on Hidden Local Assumptions

Charlie Xu — Tue, 08 Sep 2026 03:36:11 +0000

The grade is not “did it run on your laptop.” The grade is “did it run in a directory the model has never seen.” That’s the whole lab. Everything else is a demo.

AI coding assistants are generous. They will hardcode /Users/alex/week3, invent an npm package that looks real, and stash a key in a comment “just for now.” Then a student screenshots a green terminal and calls it shipped. Sound familiar?

I stopped arguing about which model is smarter. I started failing work that cannot survive a clean checkout. Cheap generation made that failure mode louder, not rarer. If code is cheap, portability is the assignment.

This is a student lab. Setup, checkpoints, a tiny harness you can actually run, stretch goals, and a rubric that does not reward a lucky home directory.

The failure I kept seeing

Not wrong algorithms. Wrong places.

The script only worked because the agent had sniffed the student’s machine. Absolute paths. A dependency that was never published. A .env that never left the laptop, until it did, in a gist. A node_modules folder treated as source.

Ask yourself one question before you grade: what did the model assume that a classmate’s shell will not provide?

If you cannot answer that in writing, you are grading theater.

What you are actually testing

You are not testing whether a model can write a JSON walker. You are testing whether the student can notice local assumptions and kill them.

Three buckets. That’s it.

Filesystem fiction — absolute home paths, hardcoded usernames, “run this from my Desktop.”
Registry fiction — package.json entries that do not exist, or versions the lockfile never pinned.
Secret fiction — keys in source, keys in prompts, keys in README “examples.”

Miss one bucket and the lab is incomplete. Pass all three on a second machine and we can talk about style.

Lab setup (45–60 minutes of machine time)

You need Node 18+, git, and a throwaway directory. No framework. No database. No “full stack.”

node -v
mkdir -p lab-second-machine && cd lab-second-machine
git init
npm init -y

Create a fixture the student did not author by hand in the same session as the solution. That separation matters. If the model writes both the app and the sample data in one breath, it will overfit both.

mkdir -p fixtures
cat > fixtures/orders.json <<'EOF'
[
  {"id": 1, "status": "paid", "sku": "tee"},
  {"id": 2, "status": "refund", "sku": "mug"},
  {"id": 3, "status": "paid", "sku": "tee"}
]
EOF

Assignment, one sentence: write a CLI json-summary that reads a file path from process.argv[2], prints how many records, and prints counts by status. stdout only. Exit 2 on bad input. Exit 1 on missing file. Exit 0 on success.

Students may use an assistant. They may not paste secrets into that assistant. They may not submit a transcript as the product.

Why a second environment is the point

A local green run is a hint. It is not evidence.

I run the same harness twice: once in the student repo, once in a clean clone on another machine or a disposable remote shell. Same commands. Same fixtures. No “it works if you cd into my nickname folder first.”

I used MonkeyCode’s free model access to draft student-facing starter comments, and the free server option as that second shell so I was not grading a laptop-shaped accident. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming model names, token quotas, or hardware details here — only that a second, boring environment is part of the method. If you already have a spare VM, use that. The lab does not require a particular vendor. It requires a machine that does not love you.

Checkpoint 0 — freeze the contract before anyone generates

Write the contract on paper or in CONTRACT.md before the first prompt. If you let the model invent the CLI flags, you will grade the model’s taste, not the student’s judgment.

# CONTRACT.md
- argv[2] = path to a JSON array of objects
- required object key: status (string)
- stdout: one JSON object, no extra logs
  {"count": <number>, "byStatus": {"paid": n, ...}}
- exit 2: missing argv or invalid JSON
- exit 1: file not found
- no network, no extra npm deps unless justified in NOTES.md

Then freeze it. Changing the contract mid-lab is how you accidentally grade prompt churn.

Checkpoint 1 — generate, then diff your own tree

Let the student prompt. Then they run this before they smile:

git status --short
git diff --stat

If node_modules appears, fail Checkpoint 1. If a .env appears, fail Checkpoint 1. If the model wrote /Users/ anywhere, fail Checkpoint 1 and keep going — the harness will catch it too, and you want that muscle memory.

A starter they can delete later:

#!/usr/bin/env node
"use strict";

const fs = require("fs");
const path = require("path");

function fail(code, msg) {
  process.stderr.write(msg + "\n");
  process.exit(code);
}

const input = process.argv[2];
if (!input) fail(2, "usage: json-summary <file>");

const abs = path.resolve(process.cwd(), input);
let raw;
try {
  raw = fs.readFileSync(abs, "utf8");
} catch (err) {
  if (err && err.code === "ENOENT") fail(1, "file not found");
  fail(2, "unreadable file");
}

let data;
try {
  data = JSON.parse(raw);
} catch {
  fail(2, "invalid json");
}
if (!Array.isArray(data)) fail(2, "expected array");

const byStatus = {};
for (const row of data) {
  if (!row || typeof row.status !== "string") fail(2, "bad record");
  byStatus[row.status] = (byStatus[row.status] || 0) + 1;
}

process.stdout.write(JSON.stringify({ count: data.length, byStatus }) + "\n");

Label that as a reference shape, not the only solution. Students should rewrite it. If they paste it unchanged, the later checkpoints still bite — because the grade is the second machine, not the snippet.

Checkpoint 2 — the portability harness (the artifact)

Save this as harness/check-portability.js. No extra packages. Run it in CI, in a classmate’s clone, in a disposable shell. Anywhere but the original home folder.

#!/usr/bin/env node
"use strict";

const fs = require("fs");
const path = require("path");
const { spawnSync } = require("child_process");

const root = process.cwd();
const findings = [];

function walk(dir) {
  for (const name of fs.readdirSync(dir)) {
    if (name === ".git" || name === "node_modules") continue;
    const p = path.join(dir, name);
    const st = fs.statSync(p);
    if (st.isDirectory()) walk(p);
    else files.push(p);
  }
}

const files = [];
walk(root);

const absRe = /\/(Users|home)\/[A-Za-z0-9._-]+/g;
const secretRe = /(api[_-]?key|secret|token)\s*[:=]\s*['"][^'"]+['"]/i;

for (const file of files) {
  const rel = path.relative(root, file);
  if (rel.startsWith("harness")) continue;
  const text = fs.readFileSync(file, "utf8");
  const absHits = text.match(absRe) || [];
  if (absHits.length) findings.push({ code: "ABS_PATH", rel, absHits });
  if (secretRe.test(text)) findings.push({ code: "SECRET_SHAPE", rel });
}

if (fs.existsSync(path.join(root, ".env"))) {
  findings.push({ code: "DOTENV_PRESENT", rel: ".env" });
}

const pkgPath = path.join(root, "package.json");
if (fs.existsSync(pkgPath)) {
  const pkg = JSON.parse(fs.readFileSync(pkgPath, "utf8"));
  const deps = Object.assign({}, pkg.dependencies, pkg.devDependencies);
  for (const name of Object.keys(deps)) {
    const probe = spawnSync(process.execPath, ["-e", `require.resolve(${JSON.stringify(name + "/package.json")})`], {
      cwd: root,
      encoding: "utf8",
    });
    if (probe.status !== 0) findings.push({ code: "UNRESOLVED_DEP", rel: name });
  }
}

const bin = path.join(root, "json-summary.js");
const fixture = path.join(root, "fixtures", "orders.json");
const run = spawnSync(process.execPath, [bin, fixture], { encoding: "utf8" });
if (run.status !== 0) {
  findings.push({ code: "FIXTURE_FAIL", stderr: (run.stderr || "").slice(0, 400) });
} else {
  try {
    const parsed = JSON.parse(run.stdout.trim());
    if (parsed.count !== 3 || parsed.byStatus.paid !== 2) {
      findings.push({ code: "WRONG_SHAPE", stdout: run.stdout });
    }
  } catch {
    findings.push({ code: "NON_JSON_STDOUT", stdout: run.stdout });
  }
}

const tmp = fs.mkdtempSync(path.join(require("os").tmpdir(), "jsum-"));
const cloneFix = path.join(tmp, "orders.json");
fs.copyFileSync(fixture, cloneFix);
const run2 = spawnSync(process.execPath, [bin, cloneFix], { encoding: "utf8" });
if (run2.status !== 0) findings.push({ code: "TMPDIR_FAIL" });

if (findings.length) {
  process.stderr.write(JSON.stringify({ ok: false, findings }, null, 2) + "\n");
  process.exit(1);
}
process.stdout.write(JSON.stringify({ ok: true, files: files.length }) + "\n");

Run it like a mean TA:

chmod +x json-summary.js harness/check-portability.js
node harness/check-portability.js

Then the move that actually teaches:

cd /tmp
rm -rf second-machine-clone
git clone /path/to/lab-second-machine second-machine-clone
cd second-machine-clone
node harness/check-portability.js

Did it die? Good. That’s the lesson. Fix the assumption, not the screenshot.

Checkpoint 3 — name the assumptions in NOTES.md

I require a short list, written by the human. Not by the model. If the model writes the notes, you are back to grading fluency.

Students answer four prompts:

Which paths did the first draft hardcode, if any?
Which packages did the model suggest that you rejected, and why?
What would break if cwd was /tmp?
What did you refuse to paste into the assistant?

No novel required. Four blunt bullets. If they write “nothing,” and the harness later finds /Users/, that’s a grading event, not a debate.

Fair grading rubric

Cap at 100. Publish this on day one so nobody can claim surprise.

Score	What I actually look at
0–20	Contract ignored, or harness never run
21–40	Runs on the author laptop only; absolute paths or unresolved deps
41–60	Clean clone runs the fixture; NOTES.md is empty or model-written
61–80	Clean clone + tmp fixture; human NOTES.md lists rejected suggestions
81–100	Above, plus a failed-then-fixed harness log in `evidence/`

I do not give points for a prettier README. I do not give points for a longer prompt. I do give points for a evidence/harness-fail.json that shows the first red run. Failure is data. Hide it and I assume you got lucky.

Suggested weights if you need numbers for a LMS:

Harness green on a clean clone: 40
Human NOTES.md with rejected model output: 25
Exit codes match CONTRACT.md: 20
Evidence of a failed first harness run: 15

Stretch goals (optional, labeled)

These are proposals. Do not silently make them required.

Add a GitHub Actions workflow that runs node harness/check-portability.js on ubuntu-latest with no extra setup steps. If you need npm install, you must justify every dependency in NOTES.md.
Teach the CLI to refuse paths that escape the repo using path.relative and a .. check. Write a test that tries ../../etc/passwd as argv and expects exit 2.
Record a 90-second screen capture of the second machine, not the first. Narrate one assumption you killed.

If a student blows past stretch goal 2 by spawning a package the model invented, I still fail Checkpoint 2. Fancy extra credit cannot launder a fake dependency.

What this lab does not prove

It does not prove the algorithm is optimal. It does not prove the assistant is “accurate.” It does not replace a security review.

The harness is a blunt instrument. A student can still hide a key in an image. They can still fetch a package that exists and is malware. They can still prompt the model with production data. This lab catches sloppy local fiction, not adversaries.

Who should not use this approach

Anyone treating a shared or free remote shell as a vault. If the data cannot be public, it does not go there. Full stop.
Teams that need a compliance sign-off, an SLA, or a guaranteed model identity. This write-up does not provide those.
Instructors who want to grade “which model wrote prettier code.” That contest trains taste, not judgment.
Students who cannot run Node locally and cannot access any second environment. The method needs two machines, even if one is a classmate’s laptop for ten minutes.

A 25-minute classroom script

If you only have one session:

5 min — freeze CONTRACT.md as a class. No generation yet.
8 min — students prompt, commit, run git status.
7 min — run the harness in /tmp clones. Watch the room go quiet.
5 min — rewrite NOTES.md by hand. Collect evidence/.

That’s enough. You will hear “but it worked on mine.” Answer with the clone path. Then sit down.

Closing

When generation is cheap, the scarce skill is noticing what the model invented about your computer. Grade the second machine. Keep the first green terminal as a souvenir if you want. Just don’t put it on the rubric.

If you steal one file from this post, steal the harness. Run it somewhere unfriendly. Then ask your students the only question that still matters: what did the model assume that this shell refused to give?

Green on Your Laptop Is Not a Grade: A Bootcamp Lab on Runtime Contracts

Charlie Xu — Sun, 06 Sep 2026 21:19:52 +0000

Stop grading the generated server. Grade whether a stranger can boot it on a pinned runtime without guessing Node, ports, or hidden env vars. That is the whole lab.

AI agents are getting fluent at writing Express handlers. They are still reckless about the machine those handlers will live on. You have seen the failure: green on a laptop, red in CI, mysterious in a classmate's demo. Why do we keep scoring the souvenir instead of the replay?

This is a proposed bootcamp lab, not a claim that I already ran it on a giant cohort. Steal the worksheet. Change the stack. Keep the rule: no runtime contract, no grade.

The problem the lab is actually about

Students now paste a ticket into a coding assistant and get a folder that looks like a service. Tests pass locally. Then the grader clones the repo on a different OS, a different Node, a missing .env, and a port already taken by Discord. Suddenly the agent looks brilliant and the student looks unlucky.

Luck is not a learning outcome.

The agent did not fail at JavaScript. It failed at environment honesty. It assumed node meant whatever was on PATH. It assumed PORT=3000 was free. It assumed fetch existed. It assumed the grader would silently install whatever npm i felt like that afternoon.

Sound familiar? Same class of bug as an agent inventing AWS resource names. Different surface.

Lab goal (one sentence)

Ship a tiny Node HTTP service and a machine-readable runtime contract so a grader can replay boot, test, and a single happy-path request on a pinned server — not on the author's laptop.

If the grader has to ask you a question, the contract is incomplete.

What you are allowed to use

You may use an AI coding assistant. You should. The point is not abstinence.

You may not hide the environment. If the model writes npm start and you never freeze Node, that is a lab failure, not a style issue.

For students who do not have a cloud bill and cannot share a home laptop with a grader, a shared free server plus free model access is the boring, useful setup. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one option that currently offers free model access and a free server option; I am not inventing model names, quotas, or hardware here, and I am not asking you to treat any vendor as the subject of the lab. The subject is the contract file.

If you already have a Codespace, a campus VM, or a Raspberry Pi under the desk, use that. The checklist does not change.

Setup (30–45 minutes, before any feature code)

Work in a clean directory. Do not generate the service yet. Seriously. The agent will try to skip this. Do not let it.

mkdir runtime-contract-lab && cd runtime-contract-lab
git init
node -v
npm -v
uname -s

Create three files by hand, even if a model offers to "just scaffold everything":

runtime-contract.json — the source of truth
contract.test.mjs — the grader's first command
README.md — human translation of the JSON, not a substitute for it

Proposed contract shape (treat this as a lab spec, not a standard):

{
  "contractVersion": 1,
  "runtime": {
    "language": "javascript",
    "node": { "major": 22, "exact": null },
    "packageManager": "npm@10"
  },
  "boot": {
    "install": "npm ci",
    "start": "node server.mjs",
    "healthPath": "/health",
    "portEnv": "PORT",
    "defaultPort": 3000,
    "readyTimeoutMs": 5000
  },
  "env": {
    "required": ["PORT"],
    "forbidden": ["AWS_SECRET_ACCESS_KEY", "OPENAI_API_KEY"],
    "notes": "No paid APIs in the core path. Health must work offline."
  },
  "assumptions": []
}

That empty assumptions array is a trap. The agent will leave it empty. You will not.

Checkpoint 0 — freeze the machine, not the vibes

On the same host the grader will use, record reality:

node -p "process.version"
node -p "process.platform"
printf 'PORT=3000\n' > .env.example

If you cannot get a shared host, stop. Do not start coding. Pair with someone who can, or wait for lab staff to hand you a URL. Local-only work is practice, not a submission.

The tiny service (deliberately boring)

Requirements, not a product pitch:

GET /health returns 200 and {"ok":true}
GET /whoami returns Node version and whether fetch exists
No database. No auth. No "while we are here" features.
Must listen on process.env.PORT
Must exit non-zero if PORT is missing

A starter the student (or the model) can complete:

// server.mjs — complete this; do not add a framework unless the contract says so
const port = process.env.PORT;
if (!port) {
  console.error("PORT is required");
  process.exit(1);
}

const server = Bun; // trap: delete this line. You are on Node.

Yes, I put a trap in the snippet. If your agent copies it blindly, that is data. Write the failure into assumptions or fix the file. Do not shrug.

A fair implementation looks closer to this:

// server.mjs
import http from "node:http";

const port = process.env.PORT;
if (!port) {
  console.error("PORT is required");
  process.exit(1);
}

const server = http.createServer((req, res) => {
  if (req.url === "/health" && req.method === "GET") {
    res.writeHead(200, { "content-type": "application/json" });
    res.end(JSON.stringify({ ok: true }));
    return;
  }
  if (req.url === "/whoami" && req.method === "GET") {
    res.writeHead(200, { "content-type": "application/json" });
    res.end(
      JSON.stringify({
        node: process.version,
        fetch: typeof fetch === "function",
        platform: process.platform,
      })
    );
    return;
  }
  res.writeHead(404);
  res.end();
});

server.listen(Number(port), () => {
  console.log(`listening on ${port}`);
});

Keep it this small on purpose. If the model opens a React app, you failed the prompt, not the runtime.

The grader's first command

Do not make humans read a novel. Make Node fail loudly.

// contract.test.mjs
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { test } from "node:test";

const contract = JSON.parse(await readFile("./runtime-contract.json", "utf8"));

test("contract declares a Node major that matches this process", () => {
  const actualMajor = Number(process.versions.node.split(".")[0]);
  assert.equal(contract.runtime.node.major, actualMajor);
});

test("boot.install is npm ci, not a vibes command", () => {
  assert.equal(contract.boot.install, "npm ci");
});

test("core path forbids secret-shaped env vars", () => {
  for (const name of contract.env.forbidden) {
    assert.equal(process.env[name], undefined);
  }
});

test("assumptions is not an empty shrug", () => {
  assert.ok(Array.isArray(contract.assumptions));
  assert.ok(contract.assumptions.length >= 3, "name three things the agent might have invented");
});

Run it on the pinned host:

npm ci
node --test contract.test.mjs
PORT=3000 node server.mjs &
curl -sS http://127.0.0.1:3000/health
curl -sS http://127.0.0.1:3000/whoami

If /whoami reports a Node major that is not in the contract, the lab is not done. Do not "fix" it by editing the test. Fix the contract or the host. Which one is cheaper? Usually the JSON.

Checkpoints (staff can grade these in order)

Use these as gates. Later work does not unlock earlier work.

Host exists. Student posts a replay URL or SSH alias for a shared runtime. Laptop screenshots do not count.
Contract matches the host. node --test contract.test.mjs is green on that host.
Boot is copy-pasteable. A grader who has never opened the chat log can npm ci and start the server from the contract alone.
Health is offline. Unplug extra APIs. /health still returns 200.
Assumptions are ugly and specific. "Might need fetch" is a fail. "Node 18 has no global fetch; we pinned 22" is a pass.
Transcript exists. A short lab-log.md lists prompts, files touched, and commands run, in order. Not a novel. A receipt.

Miss gate 1 and I will not read your handlers. Harsh? Maybe. Faster than debugging three Node versions by emoji.

Stretch goals (optional, scored separately)

Add engines to package.json and make npm ci fail on the wrong major.
Add a scripts/replay.sh that reads the contract and refuses to start if node -p process.versions.node disagrees.
Break the server on purpose on a second host (different major) and paste the error. That failure is the lesson.
Replace the HTTP server with a CLI that prints the contract and exits. Same gates. Smaller surface.

Do not stretch into Kubernetes. If you need a mesh to prove you can listen on a port, the lab has already escaped.

Fair grading rubric (100 points)

Score the replay, not the chat charisma.

Points	What I actually look at
15	Shared host is reachable by the grader without a meeting
20	`runtime-contract.json` matches the host; tests green there
15	`PORT` required; process exits non-zero when missing
15	`/health` and `/whoami` behave as specified, no extra network
15	`assumptions` has ≥3 concrete, falsifiable notes
10	`lab-log.md` is a command/prompt receipt, not a diary
10	Stretch: engines field or replay script that fails closed

Automatic zeros:

Secrets in the repo or in the contract
"Works on my machine" as the only evidence
Contract that says Node 22 while /whoami prints v18
Generated framework the ticket did not ask for
Empty assumptions array

I will not grade spelling in the README if the JSON is honest. I will grade a beautiful README that lies about Node.

How the assistant is allowed to participate

Give the model the contract before the ticket. Then give it the ticket. Then freeze the contract again if the model tries to add Redis "just in case."

A prompt that has worked in dry runs of this worksheet:

You are not allowed to write server.mjs until you propose three edits
to runtime-contract.json. Each edit must be a fact about Node, PORT,
or install, not a feature. Wait for me to accept the JSON. Then write
the smallest http server that satisfies /health and /whoami.

If the model starts coding anyway, that is a student skill issue. Stop it. Paste the contract back. Ask: which line of JSON authorizes this file?

Free model access matters here because every student needs the same chance to iterate on that argument without a personal API key. A free server matters because the grader and the student must lose the argument against the same process.version. Paid clouds are fine. They are not required for the learning outcome.

Limitations (read these before you adopt the lab)

This does not teach product design. The service is a fixture.
This does not prove the model is "good." It proves the student can pin a machine.
JSON will not save you from a host that mutates under your feet. Pin an image or refuse the host.
npm ci needs a lockfile. If you never commit package-lock.json, you did not pin installs.
Health checks can lie. /health returning 200 while /whoami crashes is still a fail under this rubric.
I have not published vendor benchmarks, token ceilings, or uptime promises. If a free tier moves, the contract still works on whatever host you still control.

Who should not use this approach

Skip this lab if you are teaching algorithms on paper. Skip it if your students have no network and no shared VM — the replay is the point. Skip it if the course already grades production Kubernetes. Skip it if you want to evaluate prompt style; this rubric will under-score a gorgeous chat that ships an unpinned runtime.

Staff on a two-hour workshop: use only checkpoints 1–4. Cut stretch. Cut /whoami if time explodes.

What I want back from you

Not a screenshot of a passing test on a stickered MacBook. A contract, a lockfile, a replay command, and three ugly assumptions.

Can your agent describe the computer, or only the code? If you need a no-card-required box to run the same checklist with a classmate acting as grader, MonkeyCode's free server option is one way to pin that replay. Steal the JSON either way.