Grade the Fence, Not the Vibe: A Bootcamp Lab on Frozen Scope Contracts

An agent that widens the ticket and still goes green is not done. I fail that submission.

Bootcamp patches keep arriving with the same shape. A student asks a model to fix one flaky retry, the diff also touches a lockfile and a workflow, and the write-up celebrates a passing test. Did the tests prove the issue, or did they prove the agent got bored and redecorated the repo?

This lab freezes a scope contract before the model runs, then grades the branch against that contract. Chat tone is not a score. October makes the habit worse, because a lot of first pull requests get opened in a hurry, but the fence is the same in March.

What you are actually grading

You are grading drift, not taste. Drift is any change the frozen brief did not allow. A new dependency counts. A drive-by rename counts.

Editing the contract after the run counts, and I treat that like editing the answer key during the exam. Can a small extra fix still be a good idea? Yes. It is still a miss on this lab.

Open a second issue and keep the first patch boring. A clean machine does not answer this question. You may already require the command to run off a laptop. This lab asks something sharper: did the patch stay inside the brief you froze?

Lab setup

Everyone gets the same tiny fixture. One bug, one allowed file, one command that must go green without the agent rewriting the assertion.

Fixture

The helper retries three times, then drops the error on the floor. Students may not "fix" this by weakening the test.

mkdir scope-lab && cd scope-lab
git init -b main
npm init -y
npm pkg set type=module
mkdir -p src test

// src/retry.js
export async function retry(fn) {
  let last;
  for (let i = 0; i < 3; i++) {
    try {
      return await fn();
    } catch (err) {
      last = err;
    }
  }
  return null;
}

// test/retry.test.js
import test from "node:test";
import assert from "node:assert/strict";
import { retry } from "../src/retry.js";

test("third failure surfaces the original error", async () => {
  await assert.rejects(
    () => retry(async () => {
      throw new Error("boom");
    }),
    /boom/
  );
});

Run node --test once before any agent starts. You want a red bar that the class agrees on. If the fixture is already green, you froze the wrong bug.

Fence

scope.json is the assignment. It is not a hint the model may renegotiate.

{
  "issue": "LAB-7",
  "allowed_paths": ["src/retry.js"],
  "forbidden_globs": [
    "package.json",
    "package-lock.json",
    ".github/**",
    "scope.json",
    "test/retry.test.js"
  ],
  "commands": ["node --test"],
  "notes": "Fix error propagation only. Do not add dependencies."
}

Hash it, then commit the freeze. Why hash a file a student could just rewrite? Because the rewrite is the cheat, and a checksum makes the cheat loud.

sha256sum scope.json | tee scope.sha256
# macOS: shasum -a 256 scope.json | tee scope.sha256
git add src test scope.json scope.sha256 package.json
git commit -m "freeze lab fixture and scope contract"

If a student decides the fix truly needs src/errors.js, they stop. They amend scope.json in a new commit, re-hash, and only then start the agent. After the branch exists, the fence does not move.

Where the agent runs

Students need a model and a machine that is not their laptop's junk drawer of global packages. For that shared run, the lab sheet points at MonkeyCode's free model access and free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those are the only product facts I will teach. I am not naming a model, a token quota, a hardware shape, or a promise that the free option is still there next cohort. Open the current docs the morning you teach, and pin that URL in the lab sheet.

If the free path is down, run any agent you already have. The grader does not care which logo produced the branch. Give the agent the issue text and the raw scope.json. Say the contract is binding.

Pull the branch back and grade the tree, not the transcript. Run the checker from the repo root so paths stay relative. Cut the agent branch from the freeze commit, or the triple-dot diff will lie to you.

git fetch origin agent/lab-7
git checkout -B grade agent/lab-7
sha256sum -c scope.sha256
# macOS: shasum -a 256 -c scope.sha256
node grade-scope.mjs --scope scope.json --base main
node --test

Hash mismatch means you stop. The fence moved. A green test does not reopen that attempt.

The grader

grade-scope.mjs is a lab checker, not a security tool. It asks git for names changed since main, compares them to the allowlist, and exits non-zero on drift. I have not published timing numbers for it, and you should not pretend it understands intent.

// grade-scope.mjs
import { readFileSync } from "node:fs";
import { execFileSync } from "node:child_process";

const args = process.argv.slice(2);

function flag(name) {
  const i = args.indexOf(name);
  if (i < 0 || !args[i + 1]) {
    console.error(`missing ${name}`);
    process.exit(2);
  }
  return args[i + 1];
}

const scopePath = flag("--scope");
const base = flag("--base");
const scope = JSON.parse(readFileSync(scopePath, "utf8"));

const changed = execFileSync(
  "git",
  ["diff", "--name-only", `${base}...HEAD`],
  { encoding: "utf8" }
)
  .split("\n")
  .map((line) => line.trim())
  .filter(Boolean);

function forbidden(pathname) {
  return scope.forbidden_globs.some((glob) => {
    if (glob.endsWith("/**")) return pathname.startsWith(glob.slice(0, -3));
    return pathname === glob;
  });
}

const extra = changed.filter((pathname) => !scope.allowed_paths.includes(pathname));
const blocked = changed.filter(forbidden);
let fail = 0;

if (changed.length === 0) {
  console.error("FAIL: empty diff");
  fail = 1;
}
for (const pathname of extra) {
  console.error(`FAIL: outside allowlist: ${pathname}`);
  fail = 1;
}
for (const pathname of blocked) {
  console.error(`FAIL: forbidden path: ${pathname}`);
  fail = 1;
}
if (!fail) console.log(`PASS: ${changed.length} path(s) inside the fence`);
process.exit(fail);

How to read a failure

Suppose the branch also updates the lockfile. You should see both lines, then a non-zero exit. Do not talk the student out of that output.

FAIL: outside allowlist: package-lock.json
FAIL: forbidden path: package-lock.json

So the helper file was a nice split. Was it in the brief? No. Then it is drift.

A pass line only means the file list obeyed the contract. Behavior still has to survive node --test, with test/retry.test.js absent from the diff.

Checkpoints

Walk these in order. Confidence in the chat is not a checkpoint.

1. Freeze. scope.json is committed and the checksum matches. No hash, no brief.
2. Shared run. The branch came from the class run target. A hand edit labeled as an agent run fails this row.
3. Fence. node grade-scope.mjs --scope scope.json --base main exits 0.
4. Oracle. node --test exits 0, and the test file is untouched.
5. Miss note. On a failed fence or oracle, three lines: what drifted, which sentence in the prompt allowed it, and the smaller prompt for the retry. No new patch in that note.

Want to know if they understood? Hide the script's output and ask them to predict it from git diff --name-only alone. If they cannot predict the fail, they were negotiating, not reading.

Retry protocol

A fail is not a cue to paste the grader output back into the same chat and hope. Use this loop.

1. Stop the agent. Do not let it "clean up" the branch in place.
2. Reset to the freeze commit, or cut a fresh branch from main.
3. Shrink the prompt to the notes line plus the allowed path. Cut every adjective.
4. Run once more on the same class target.
5. Grade again. Two fails on the same extra path means the student writes the miss note and takes the score. No third silent retry.

Stretch goals

• Teach renames. git diff --name-status prints R when a file moves. Decide, in the rubric, whether a rename of src/retry.js is in bounds before anyone runs.
• Reject a comment-only edit. An allowed path with no behavior change is not a fix. Add a second assertion if you need that distinction, and freeze it too.
• Run the same grader against a second contract that does allow package.json. Same tool, moved fence. The contrast is the lesson, not a new sermon.

Rubric

Score out of 10. Partial credit lives only in the cells that say so.

Row	Points	Full marks	Miss
Frozen contract and hash	2	File committed, checksum matches	Missing hash, or hash rewritten after the run
Branch from the shared run	2	Fetchable class branch	Hand patch with no run note
Allowlist checker	3	Exit 0	Any path outside allowed_paths
Original tests	2	node --test passes, test file untouched	Red oracle, or a rewritten assertion
Miss note when needed	1	Three concrete lines	"The model messed up" and nothing else

A tidy refactor that edits .github/workflows/ci.yml loses the entire fence row. I do not average that away. Why refuse to smooth it? Because the skill on the page is refusing surprise, not prompting until the bar is green.

What this does not catch

Path lists are a blunt fence. The agent can stay inside src/retry.js and still "fix" the bug by swallowing every error. Your tests might be too weak to notice. That is in-file drift, and this script will smile at it.

The globs are toys. src/** is not implemented. Do not aim this file at a monorepo and call it policy.

A free server is a convenience, not a vault. Do not clone production credentials onto it. Do not assume disk, retention, or model choice will match what you saw last month. I am not teaching numbers for any of that.

Who should skip the lab? Skip it if the real task is a cross-repo upgrade. Skip it if you need a security review, a license audit, or a check for hostile tool use. This fence does not read install scripts, and it does not ask whether the model noticed a bad target.

Pair those questions in a later week. One contract cannot carry them.

Before you assign it

Freeze the brief, hash it, and run the agent where the whole class can fetch the branch. Grade the diff before you grade the explanation.

If you do not want every student buying a key for a single Saturday, check MonkeyCode's free model access and free server docs that morning, then paste the live link under the hash command. If the page says the offer changed, believe the page and switch the run target. The fence still grades.