Chat Logs Aren't Homework: A Bootcamp Lab on Agent Write-Set Contracts

If your coding agent cannot name the files it will touch before it touches them, I do not grade the conversation. I grade three artifacts: a write-set, a unified diff, and a test run that happens after the patch lands in a clean tree. Chat is a scratchpad. The patch is the homework.

Why this lab, right now? Because too many student “agents” are a scripted path wearing extra adjectives. They always edit app.js. They always claim success. They never say what they will not touch.

Sound familiar?

The failure I am done arguing about

A student walks me through a 40-message thread. The model is glowing. The repo on disk has a mutated package.json, a surprise rewrite of .env.example, and a test file the agent invented so it could pass its own checks.

I ask one question. Which files did you authorize?

Silence.

That silence is the lab. We are not settling whether models already code better than most developers. We are forcing a contract: declared write-set, reviewable patch, no extra paths. If the loop is more than a trench-coat if statement, it can live inside that fence. If it cannot, it was never an agent you should ship.

Lab goal

Build a tiny coding-agent pipeline that may propose edits only as a unified diff, and only against files named up front.

Deliverables:

1. write-set.json — the only paths the agent may modify
2. agent.patch — a unified diff against the fixture
3. trace.jsonl — one line per tool call (name, path digest, timestamp). Evidence, not the grade
4. A grader command that applies the patch in a copy of the fixture and runs tests

Skip the write-set and you take a zero. I will not read the chat export. Why would I? It cannot be replayed on my machine.

Setup

You need Node.js 20+, git, and the fixture zip I hand out. Do not start from your side project. Side projects lie.

node -v
git --version
mkdir -p lab-write-set && cd lab-write-set
unzip ../fixture-todo-api.zip
cd fixture-todo-api
git status --porcelain   # must be empty before you begin

Proposed fixture layout. Paths are part of the contract, not decoration:

fixture-todo-api/
  src/server.js
  src/routes/todos.js
  test/todos.test.js
  package.json
  write-set.json          # you create this before any model call

The broken assignment: POST /todos accepts a title that is only whitespace. Your agent may fix that. It may not “helpfully” bump dependencies, reformat the tree, or rewrite tests unless those files are declared.

Checkpoint 0 — Write the fence first

Before any model call, commit the write-set. Not after. Not in the same commit as the patch. First.

{
  "version": 1,
  "files": [
    "src/routes/todos.js"
  ],
  "forbidden_prefixes": [
    ".env",
    "node_modules/",
    ".git/"
  ]
}

If you cannot name the file, why is a model allowed to go find it for you?

git add write-set.json
git commit -m "lab: declare write-set before the agent runs"

I will look at git log. If the fence shows up after the “fix,” that is not a process. That is a cover story.

Checkpoint 1 — Patches only, never in-place writes

Your agent gets two tools. Only two.

• read_file(path)
• propose_patch(unified_diff)

No write_file. No shell. No curl. Need a third tool? Then you are negotiating blast radius, not trimming whitespace titles.

Here is a grader-sized allowlist you can run today. This is real code, not a sketch.

// grader/allowlist.js
import fs from "node:fs";
import path from "node:path";

export function loadWriteSet(repoRoot) {
  const raw = JSON.parse(
    fs.readFileSync(path.join(repoRoot, "write-set.json"), "utf8")
  );
  if (!Array.isArray(raw.files) || raw.files.length === 0) {
    throw new Error("write-set.json must list at least one file");
  }
  for (const f of raw.files) {
    if (f.includes("\\") || f.startsWith("/") || f.split("/").includes("..")) {
      throw new Error(`illegal path in write-set: ${f}`);
    }
  }
  return raw;
}

export function assertPatchPaths(patchText, writeSet) {
  const allowed = new Set(writeSet.files);
  const re = /^\+\+\+ b\/(.+)$/gm;
  const touched = [];
  let m;
  while ((m = re.exec(patchText))) {
    const p = m[1];
    if (p === "/dev/null") continue;
    touched.push(p);
    if (!allowed.has(p)) {
      throw new Error(`patch touches undeclared file: ${p}`);
    }
    for (const prefix of writeSet.forbidden_prefixes || []) {
      if (p.startsWith(prefix)) {
        throw new Error(`patch hits forbidden prefix ${prefix}: ${p}`);
      }
    }
  }
  if (touched.length === 0) {
    throw new Error("empty patch: chat happened, homework did not");
  }
  return touched;
}

Run it against a bad patch on purpose. I want the failure in your terminal, not in a screenshot of a chatbot.

node --input-type=module -e "
import fs from 'node:fs';
import { loadWriteSet, assertPatchPaths } from './grader/allowlist.js';
const ws = loadWriteSet('.');
assertPatchPaths(fs.readFileSync('agent.patch','utf8'), ws);
console.log('write-set ok');
"

Did it fail loud? Good. If it whispered, your regex is too kind.

Checkpoint 2 — Apply in a copy, never in the worktree you love

Applying a patch in the same folder you used to chat is how “it worked” becomes “I cannot revert.” Clone, check, apply, test. Then throw the clone away.

#!/usr/bin/env bash
# grader/apply.sh — proposed classroom helper
set -euo pipefail
SRC="$1"
PATCH="$2"
DST="$(mktemp -d)"
trap 'rm -rf "$DST"' EXIT
git clone --local "$SRC" "$DST/repo"
git -C "$DST/repo" apply --check "$PATCH"
git -C "$DST/repo" apply --index "$PATCH"
(cd "$DST/repo" && npm test)

If git apply --check dies, the agent failed. Do not fix the hunk by hand and pretend the loop did the work. That coat has been seen.

Checkpoint 3 — Tests the agent did not author

The fixture tests stay closed unless the write-set names them. When the model writes the tests, you are grading the model's self-esteem. Different bruise, same bone.

Your npm test must keep at least one assertion you wrote:

// test/todos.test.js — already in the fixture; do not let the agent rewrite it
import test from "node:test";
import assert from "node:assert/strict";
import { createApp } from "../src/server.js";

test("POST /todos rejects whitespace-only title", async () => {
  const app = createApp();
  const res = await app.inject({
    method: "POST",
    url: "/todos",
    payload: { title: "   " }
  });
  assert.equal(res.statusCode, 400);
});

If the agent “passes” by deleting that test, the write-set checker should already have failed. If it did not, stop polishing prompts and fix the fence.

Artifact: a decision table the grader actually runs

Do not vibe-check the agent. Map outcomes, then encode them.

Input	Expected grader result	Student consequence
Patch paths ⊆ write-set, tests pass	PASS	full credit for the apply step
Patch empty, chat log glowing	FAIL empty_patch	zero on the deliverable
Patch touches package.json not in the write-set	FAIL undeclared_path	zero, even if tests pass
Patch includes .. or an absolute path	FAIL illegal_path	zero, plus an integrity flag
git apply --check rejects	FAIL patch_does_not_apply	resubmit; no hand-edits
Tests fail after apply	FAIL tests	the code is wrong, not the vendor
Trace missing tool names	FAIL trace	incomplete lab, not an auto-zero on the patch

Wire the table into one command:

// grader/grade.mjs
import fs from "node:fs";
import { spawnSync } from "node:child_process";
import { loadWriteSet, assertPatchPaths } from "./allowlist.js";

const repo = process.argv[2] ?? ".";
const patch = fs.readFileSync(`${repo}/agent.patch`, "utf8");
const ws = loadWriteSet(repo);

try {
  const touched = assertPatchPaths(patch, ws);
  const check = spawnSync("git", ["apply", "--check", "agent.patch"], {
    cwd: repo,
    encoding: "utf8"
  });
  if (check.status !== 0) {
    console.log("FAIL patch_does_not_apply");
    console.error(check.stderr);
    process.exit(2);
  }
  console.log(JSON.stringify({ result: "write_set_ok", touched }, null, 2));
} catch (err) {
  console.log("FAIL", err.message);
  process.exit(1);
}

The model can ramble. The grader cannot. That is the whole point.

A proposed loop — labeled, on purpose

This is a classroom skeleton, not a production agent. If your implementation grows a shell tool, you left the lab.

// proposed loop — unexecuted example for the write-up
export async function runLoop({ readFile, proposePatch, maxSteps = 4 }) {
  const writeSet = JSON.parse(await readFile("write-set.json"));
  const target = writeSet.files[0];
  const source = await readFile(target);
  let lastPatch = "";
  for (let step = 1; step <= maxSteps; step++) {
    // You supply the model call. The contract does not care which vendor.
    lastPatch = await proposePatch({ step, target, source });
    if (lastPatch.trim()) return lastPatch;
  }
  throw new Error("loop ended with an empty patch");
}

Notice what is missing. No writeFile. No “just this once” exception. If the loop always returns a patch for src/routes/todos.js and you never read another path, you built an if-statement. Congrats. The stretch goal will catch you.

Stretch goals

• Parse @@ hunks and reject patches that change more than N lines outside a function named in write-set.json.
• Make propose_patch idempotent: applying the same patch twice must fail the second time. Double-insert is not a retry strategy.
• Redact anything that looks like TOKEN= from trace.jsonl before you zip the submission. Leaky traces fail the stretch, and they fail my patience.
• Swap in a second bug with the same fence. If your tool layer hard-codes todos.js, you did not build an agent. You built a costume.

Fair rubric (100 points)

I put the numbers on the board on day one. No mystery meat.

• Write-set committed before the first model call — 15. Git history must show the fence first. Force-pushing a backdated commit is a conversation I do not enjoy.
• Patch path contract — 25. Zero undeclared files. Zero forbidden prefixes.
• Clean apply + fixture tests — 30. git apply then npm test in a clone I have never logged into.
• Trace completeness — 15. Every read_file / propose_patch has a timestamp and a path. I am not grading prose.
• Stretch — 15. Idempotent apply or secret redaction. Pick one. Doing both does not mint extra credit.

Automatic zeros:

• Missing write-set.json
• In-place writes with no patch
• Tests rewritten without being declared
• Network or shell tools in the loop

Is that harsh? Yes. Is a surprise package.json rewrite harsher on the next homework? Also yes.

Where a shared runner actually helps

Students will try to run the loop on a laptop that already has a paid key, a dirty node_modules, and a .env from another course. That is how hidden local assumptions sneak back in. I am not grading who remembered to export a vendor key.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need the generation step off that laptop, MonkeyCode's free model access and free server option are one way to host it so the class shares a runner. The grader above still runs locally on the patch. If you never touch that server, the lab does not get worse. Steal the write-set checker either way.

Limitations — read this before you copy the rubric

This contract does not prove the model understood the bug. It proves the mutation was reviewable.

It will not catch a clever patch that stays inside todos.js and still deletes the validation you wanted. Fixture tests have to be good. If your tests are theater, the write-set is theater with better regex.

git apply is picky about whitespace and paths. That is a feature. Do not “help” students by auto-fixing hunks. You would be grading your fixer, not their loop.

Who should not use this approach:

• Production incident response. You want a human on the write-set, not a bootcamp timer.
• Open-ended refactors that are supposed to touch many files. Widen the write-set on purpose, or you are grading obedience.
• Anyone who cannot run git apply and npm test without a GPU story. This lab is about fences, not model rankings.
• Teams that must keep traces off disk for legal reasons. Then you need a different evidence rule. Do not fake one with a chat export.

What I tell the room on the way out

Can your agent name the files? Can the patch apply in a clone I have never seen? Do the tests you did not write still pass?

Three yeses, and we can argue about prompts. A screenshot of a confident paragraph is a zero. Not because I hate models. Because homework that cannot be replayed is not homework.

Now go break your own grader with a patch that sneaks in package.json. If that does not fail loud, you are not done.