No Trace, No Grade: A Bootcamp Lab on Agent Flight Recorders

I don't grade chat tabs anymore. If another student cannot reconstruct what your agent did from a log, you did not finish the lab.

That sounds harsh. It is. A screenshot of a green terminal is a vibe, not evidence. This lab makes the session itself the deliverable.

Why the chat tab fails as homework

You know the ritual. Student pastes a passing test run. I ask one follow-up. Which file did the agent edit first?

Silence. Or a shrug. Or a 40-message thread nobody can replay on a different machine.

So the contract changed. You may use an assistant. You may use a shared free model. You may get lucky on the first try. You still owe me a flight recorder.

What you will actually submit

Not a prompt dump. Not a PNG. A JSONL log plus a git diff that a classmate can follow without asking you what "it" meant.

The coding problem underneath is deliberately boring: fix a broken change-maker. The grade is not "did the model notice the off-by-one." The grade is "can a human replay your path."

Lab setup

You need Node 20+, git, and a directory that belongs to you. No paid API key is required. If you are on a shared box, pick a workspace name before you touch files. Why? Because ~/lab is not a unique identity. It is a collision waiting to happen.

export WORKSPACE="$HOME/flight-lab-$USER"
mkdir -p "$WORKSPACE" && cd "$WORKSPACE"
git init
npm init -y

Create three files: broken production code, tests, recorder.

The broken production file

// change.js
export function makeChange(amountCents, coins = [25, 10, 5, 1]) {
  if (!Number.isInteger(amountCents) || amountCents <= 0) {
    throw new Error("amountCents must be a positive integer");
  }

  const result = {};
  let left = amountCents;
  for (const coin of coins) {
    result[coin] = Math.floor(left / coin);
    left = left % coin;
  }
  return result;
}

The bug is not subtle if you read the tests. Zero cents is a legal empty drawer. A custom coin list that cannot make change must throw. Greedy leftover is not "close enough."

The tests you are not allowed to rewrite

// change.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { makeChange } from "./change.js";

test("zero is a valid empty drawer", () => {
  assert.deepEqual(makeChange(0), {});
});

test("41 cents with USD coins", () => {
  assert.deepEqual(makeChange(41), { 25: 1, 10: 1, 5: 1, 1: 1 });
});

test("impossible coin set throws", () => {
  assert.throws(() => makeChange(3, [2]), /cannot make change/);
});

Run them once so you feel the failure. Do not "fix" the test names to match a hallucinated API.

node --test change.test.js

Red. Good. That is checkpoint zero.

The flight recorder

This wrapper does not call a model. It records what you (or your agent) did. Every command, every exit code, every file you claim you touched.

// recorder.mjs
import { spawn } from "node:child_process";
import { appendFile, mkdir } from "node:fs/promises";
import { dirname, resolve } from "node:path";

const LOG = resolve(process.env.WORKSPACE || ".", "session.jsonl");
const SECRET = /(api[_-]?key|token|secret|password|authorization)/i;

function redact(value) {
  if (typeof value !== "string") return value;
  return SECRET.test(value) ? "[redacted]" : value;
}

async function write(event) {
  await mkdir(dirname(LOG), { recursive: true });
  const line = JSON.stringify({
    ts: new Date().toISOString(),
    workspace: process.env.WORKSPACE || process.cwd(),
    ...event,
  });
  await appendFile(LOG, line + "\n");
}

const [cmd, ...args] = process.argv.slice(2);
if (!cmd) {
  console.error("usage: node recorder.mjs <command> [args...]");
  process.exit(2);
}

await write({
  type: "cmd.start",
  cmd: redact(cmd),
  args: args.map(redact),
});

const child = spawn(cmd, args, { stdio: "inherit", shell: false });
child.on("exit", async (code, signal) => {
  await write({ type: "cmd.end", cmd: redact(cmd), code, signal });
  process.exit(code ?? 1);
});

Wrap the test runner. Wrap git diff. Wrap whatever the agent wants to shell out. If it is not in the log, it did not happen.

node recorder.mjs node --test change.test.js

How an agent is allowed to help

Ask it to read change.test.js before it touches change.js. Ask it to propose a patch, not a vibe. Then you type the commands through the recorder.

A useful prompt looks like a ticket, not a prayer:

Read change.test.js. Do not edit tests.
Make change.js satisfy all three tests.
Zero cents returns {}.
If leftover remains, throw Error("cannot make change").
After editing, tell me the exact command to re-run tests.
Do not print secrets. Do not invent extra files.

Did the model skip the tests and rewrite the function signature anyway? That goes in the log. Recovery is part of the grade. A clean first try is nice. A messy try with a recorded correction is honest.

I point students who do not have a paid API key at MonkeyCode's free model access and free server option so the lab is about traces, not billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product is not the grader. Your JSONL is.

If you use a shared free server, your workspace variable is not optional. Two students writing change.js in the same home directory will grade each other's accidents. Ask me how I know. Better: don't.

Checkpoints

Do these in order. Skipping is how you submit a novel instead of a lab.

1. Recorder writes JSONL. After one command you should have at least a cmd.start and a cmd.end line. Open the file. If you cannot parse a line with JSON.parse, you did not finish this checkpoint.
2. Secrets cannot survive. Put a fake API_KEY=totally-real in an argument on purpose, then confirm the log says [redacted]. If your recorder stores the raw string, you fail this checkpoint even if the kata is green.
3. Tests stay frozen. git diff -- change.test.js must be empty. The agent does not get to negotiate the oracle.
4. Final verification is recorded. The last cmd.end for node --test must have code: 0. A passing run you forgot to wrap does not count.
5. Peer replay. A classmate reads only session.jsonl and git log --patch, then explains your first failed test in one sentence. If they cannot, your log is a diary, not a recorder.

Sample log shape you are aiming for:

{"ts":"2026-09-08T18:01:11.000Z","workspace":"/home/alex/flight-lab-alex","type":"cmd.start","cmd":"node","args":["--test","change.test.js"]}
{"ts":"2026-09-08T18:01:11.180Z","workspace":"/home/alex/flight-lab-alex","type":"cmd.end","cmd":"node","code":1,"signal":null}

Ugly? Good. Ugly is inspectable.

A tiny grader you can run in CI

This is not intelligence. It is a shape check. I will still read the patch. The script only stops the "I forgot the log" submissions.

// grade-trace.mjs
import { readFile } from "node:fs/promises";

const raw = await readFile("session.jsonl", "utf8");
const events = raw.trim().split("\n").map((line, i) => {
  try {
    return JSON.parse(line);
  } catch {
    throw new Error(`line ${i + 1} is not JSON`);
  }
});

const errors = [];
if (events.length < 2) errors.push("log too short");
if (events.some((e) => /api[_-]?key|sk-|bearer /i.test(JSON.stringify(e)))) {
  errors.push("possible secret in log");
}

const testEnds = events.filter(
  (e) => e.type === "cmd.end" && Array.isArray(e.args) && e.args.includes("--test")
);
if (!testEnds.length) errors.push("no recorded test run");
else if (testEnds.at(-1).code !== 0) errors.push("final recorded test run is not green");

if (errors.length) {
  console.error(errors.join("\n"));
  process.exit(1);
}
console.log(`ok: ${events.length} events`);

node grade-trace.mjs

If this script is green and your diff is nonsense, you still fail. Tools catch absence. Humans catch theater.

Fair grading rubric

I score 20 points. Passing is 14. Green tests without a replayable log cap at 8. Why? Because the kata is the excuse. The recorder is the lab.

Criterion	Points	Pass looks like	Fail looks like
Frozen tests	4	change.test.js untouched	renamed assertions, deleted the zero case
Recorded commands	4	JSONL with start/end pairs	chat export, screenshots, empty file
Secret hygiene	3	redaction works on a planted key	raw tokens, .env pasted into the log
Correctness	4	node --test recorded with exit 0	green on a different file, skipped tests
Peer replay	3	classmate reconstructs the first failure	log says "fixed it" with no commands
Workspace isolation	2	WORKSPACE includes your user	wrote into a shared ~/lab

I do not award bonus points for a fancier model. I do not subtract points because you used a free shared server. The trace is the equalizer. The model is not.

Stretch goals

Only after the rubric is green.

• Add a files.touched event by wrapping writes, or by recording git status --porcelain after each command.
• Require a hypothesis string before the first edit. If the agent cannot state the bug in one sentence, you are not allowed to patch yet.
• Replay the command list on a second checkout. If the commands do not apply cleanly, your log skipped a manual step.
• Teach the recorder to refuse curl | bash and any command line that matches the secret regex. Refusal should be an event, not a crash with no log.

Limitations, and who should skip this

A flight recorder is not a proof of understanding. Students can still paste a classmate's JSONL. Pair the log with a 90-second live question: "What happens if coins are unsorted?" If they cannot answer, the trace was theater.

This also does not replay model weights. Different free models will not emit the same patch. That is fine. We grade the human-visible path, not token equality.

Do not use this as a production audit framework. JSONL in a homework folder is not chain-of-custody. Do not use it as an excuse to paste customer secrets into any assistant, free or paid. Do not use it for closed-book exams where tool use is the cheating. And if your course cannot give every student a private directory on a shared server, fix isolation before you add agents. A shared change.js turns the rubric into noise.

What I want back from you

Run checkpoint 3. Then paste one redacted cmd.end line in the comments, not the whole chat. I care whether the shape is boring and parseable. If you needed a shared free model and server to produce that line, say so. The interesting part is still the log.