DEV Community

Charlie Xu
Charlie Xu

Posted on

If You Can't Replay It, It's a Zero: A Bootcamp Lab on Agent Evidence Packs

#javascript #ai #beginners #testing

A passing agent lab is not a pretty transcript. It is a replayable evidence pack. If I cannot reconstruct the run from files on disk, I score it as a zero.

That sounds harsh. Good. A green repo with no trail is just a magic trick. Can I check out the branch, read the pack, and land on the same files without asking the student what the model "meant"? If the answer is a shrug, the assignment is unfinished.

Why this lab exists

I keep seeing the same failure. The feature appears. A screenshot appears. Nobody can say which tool call wrote src/app.js, which prompt was frozen, or why the loop stopped.

Vibe output is not a deliverable. Engineering is a trail you can audit.

This lab does not ask students to pick a smarter model. It asks them to ship a pack I can validate with one script. No pack, no credit.

Lab card

Timebox: one sitting. About two hours if Node is already installed.

Stack: Node.js 20+, git, a tiny Express app with a known bug.

Allowed tools: read_file, write_file, run_tests, git_diff. Nothing else.

Out of scope: grading "conversation quality." I am not scoring vibes. I am scoring whether the run can be replayed.

Setup

Start from a starter that fails on purpose. Do not "improve" it before checkpoint 1. Label this as a lab template, not a published app.

# lab template: swap in your starter remote
git clone ./bootcamp-evidence-pack-starter
cd bootcamp-evidence-pack
npm ci
npm test
# expect 1 failing test: GET /health returns 500
Enter fullscreen mode Exit fullscreen mode

Create an empty pack directory. The grader only reads this tree:

evidence/
  pack.json
  prompts/system.md
  prompts/task.md
  events.jsonl
  diffs/0001.patch
  halt.txt
Enter fullscreen mode Exit fullscreen mode

If a file is missing, the validator fails closed. That is the point.

The evidence pack

pack.json is the index. Keep it boring. Boring is gradable.

{
  "student": "github-handle",
  "assignment": "evidence-pack-lab-01",
  "started_at": "2026-09-18T14:00:00Z",
  "finished_at": "2026-09-18T15:10:00Z",
  "goal": "Make GET /health return 200 with {\"ok\": true}",
  "allowed_tools": ["read_file", "write_file", "run_tests", "git_diff"],
  "prompt_sha256": {
    "system": "replace-with-real-hash",
    "task": "replace-with-real-hash"
  },
  "event_count": 8,
  "halt_reason": "tests_green"
}
Enter fullscreen mode Exit fullscreen mode

events.jsonl is the spine. One JSON object per line. No pretty-print. No comments. Each event names a tool, a timestamp, and a result hash.

{"seq":1,"ts":"2026-09-18T14:01:12Z","tool":"read_file","input":{"path":"src/server.js"},"result":{"sha256":"abc123","bytes":412}}
{"seq":2,"ts":"2026-09-18T14:02:03Z","tool":"run_tests","input":{"cmd":"npm test"},"result":{"exit":1,"failing":1}}
Enter fullscreen mode Exit fullscreen mode

Ask yourself a mean question. If I delete node_modules, apply the patches in order, and run npm test, do I get the same green? If not, you documented a mood. You did not document a run.

The validator

I do not grade by reading chats. I grade by running this script. Students may copy it. They may not weaken it. Treat this as a lab artifact, not as production tracing.

#!/usr/bin/env node
import { readFileSync, existsSync, readdirSync } from "node:fs";
import { createHash } from "node:crypto";
import { execFileSync } from "node:child_process";

const ALLOWED = new Set(["read_file", "write_file", "run_tests", "git_diff"]);
const HALT = new Set(["tests_green", "budget_exhausted", "student_stop"]);

function sha256File(path) {
  return createHash("sha256").update(readFileSync(path)).digest("hex");
}

function fail(msg) {
  console.error(`ZERO: ${msg}`);
  process.exit(1);
}

const packPath = "evidence/pack.json";
if (!existsSync(packPath)) fail("missing evidence/pack.json");

const pack = JSON.parse(readFileSync(packPath, "utf8"));
for (const f of [
  "evidence/prompts/system.md",
  "evidence/prompts/task.md",
  "evidence/events.jsonl",
  "evidence/halt.txt",
]) {
  if (!existsSync(f)) fail(`missing ${f}`);
}

if (!Array.isArray(pack.allowed_tools)) fail("allowed_tools must be an array");
for (const t of pack.allowed_tools) {
  if (!ALLOWED.has(t)) fail(`undeclared tool in pack: ${t}`);
}

const sysHash = sha256File("evidence/prompts/system.md");
const taskHash = sha256File("evidence/prompts/task.md");
if (pack.prompt_sha256?.system !== sysHash) fail("system prompt hash drift");
if (pack.prompt_sha256?.task !== taskHash) fail("task prompt hash drift");

const lines = readFileSync("evidence/events.jsonl", "utf8")
  .trim()
  .split("\n")
  .filter(Boolean);

if (lines.length !== pack.event_count) {
  fail(`event_count ${pack.event_count} != ${lines.length} lines`);
}

let prevSeq = 0;
let sawTestsGreen = false;
let writeCount = 0;

for (const line of lines) {
  let ev;
  try {
    ev = JSON.parse(line);
  } catch {
    fail("events.jsonl is not JSONL");
  }
  if (ev.seq !== prevSeq + 1) fail(`sequence break after ${prevSeq}`);
  prevSeq = ev.seq;
  if (!ALLOWED.has(ev.tool)) fail(`tool not in ALLOWED set: ${ev.tool}`);
  if (!pack.allowed_tools.includes(ev.tool)) fail(`tool used but not declared: ${ev.tool}`);
  if (!ev.ts || !ev.input || !ev.result) fail(`event ${ev.seq} missing fields`);
  if (ev.tool === "write_file") writeCount += 1;
  if (ev.tool === "run_tests" && ev.result.exit === 0) sawTestsGreen = true;
}

const patches = readdirSync("evidence/diffs").filter((f) => f.endsWith(".patch"));
if (patches.length !== writeCount) {
  fail(`write_file events (${writeCount}) != patch files (${patches.length})`);
}
for (const p of patches.sort()) {
  execFileSync("git", ["apply", "--check", `evidence/diffs/${p}`]);
}

const halt = readFileSync("evidence/halt.txt", "utf8").trim();
if (!HALT.has(halt)) fail(`halt reason not in contract: ${halt}`);
if (halt !== pack.halt_reason) fail("halt.txt does not match pack.json");
if (halt === "tests_green" && !sawTestsGreen) {
  fail("halt says tests_green but no run_tests event exited 0");
}

const testOut = execFileSync("npm", ["test"], { encoding: "utf8" });
if (halt === "tests_green" && /failing/.test(testOut)) {
  fail("halt says tests_green but npm test is red on this checkout");
}

console.log("PASS: evidence pack is replayable enough to grade");
Enter fullscreen mode Exit fullscreen mode

Wire it and keep the result binary:

node --check validate-pack.mjs
node validate-pack.mjs
Enter fullscreen mode Exit fullscreen mode

Pass or zero. Partial credit lives in the rubric, not in the bouncer. The validator is not a therapist.

Checkpoints

  1. Red is recorded. Before any edit, log a run_tests event with exit: 1. If the first event is a write, I assume the story was written backwards. Zero for checkpoint 1.
  2. Prompts are frozen. Hash prompts/system.md and prompts/task.md after the first tool event. Mutating a prompt with no new event is silent drift. I treat it as a missing file.
  3. Diffs match writes. Every write_file event needs diffs/NNNN.patch. git apply --check must succeed. If the patch does not apply, the pack is fan fiction.
  4. Halt is named. halt.txt is one token from the set. looks good is not a halt reason. Neither is an empty file.

Hit all four and the validator should print PASS. Miss one and you already know the grade.

Want a quick self-check before you ping me?

# monotonic seq, frozen hashes, patches apply, tests match halt.txt
node validate-pack.mjs && git status --short evidence
Enter fullscreen mode Exit fullscreen mode

If git status shows you rewrote prompts after the events, stop. Rehash. Rerun. Do not argue with the script.

Stretch goals

Do these only after a green validator. They are not a backdoor around a broken pack.

  • Replay script. Write replay.sh that reapplies patches in order and re-runs npm test. If replay disagrees with the live tree, document the drift in evidence/drift.md.
  • Budget line. Add token_estimate as an integer on pack.json. I do not need a vendor invoice. I need a number the student can defend in standup. A wild guess with no method note fails the stretch, not the lab.
  • Second machine. Run the validator on a clean checkout. A pack that only "works" because node_modules was already warm is not replayable. 
#!/usr/bin/env bash
# replay.sh — lab helper, not a production runner
set -euo pipefail
git checkout -- src
for p in evidence/diffs/*.patch; do
  git apply "$p"
done
npm test
Enter fullscreen mode Exit fullscreen mode

Fair grading rubric

Evidence Points Automatic zero if...
Validator passes 40 script edited to skip checks
Checkpoint 1 red event 15 first event is a write
Frozen prompts + hashes 15 prompt files mutated with no event
Patches apply 15 git apply --check fails
Named halt 15 halt not in the contract set
Stretch replay +10 replay is not deterministic

I grade the pack, not the model's personality. A clumsy agent with a complete trail beats a magic diff with a screenshot.

Forty points sit on the validator for a reason. If I cannot run one command, I am not reconstructing your evening from Discord.

Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This lab dies in the first twenty minutes when students cannot obtain an API key, or when five people share one laptop. I reach for MonkeyCode here for two boring reasons: free model access, and a free server option. That is enough to produce events.jsonl without turning the assignment into a billing workshop.

I am not claiming a model name, a quota, a box size, or a promise that the free tier lasts forever. Those details move. The lab card should not. If the product is unavailable the day you run this, keep the same pack schema and point the agent at whatever provider you already have. The grade is the pack, not the vendor.

If you want to run the loop without buying keys first, the free model and free server options are enough to finish checkpoint 1. After that, the validator does not care where the tokens came from.

Limitations

This does not prove the student never edited files by hand. A determined student can forge JSONL. I am grading auditability, not a courtroom.

The hash checks are only as strong as the files you freeze. If npm test hits the network, replay lies. Pin fixtures. Disable live HTTP in the starter.

JSONL is not OpenTelemetry. Do not ship this schema to production tracing. It is a bootcamp contract. It will look naive next to a real span exporter, and that is fine.

Timestamps can skew. I only require monotonic seq, not perfect clocks.

Who should skip this lab

  • Instructors who want to grade conversation style. This lab will make those submissions fail, correctly.
  • Teams handling real customer data. An evidence pack is another copy of prompts and diffs. That is a leak surface.
  • Anyone trying to publish a model ranking from one /health fix. This is a grading contract, not a benchmark.

If you need a playground, skip the validator. If you need a grade I can defend to another instructor, keep the bouncer.

What I want back

Push evidence/ and validate-pack.mjs. Do not push chat HTML. Do not push a write-up about how the agent felt.

Can I clone, run npm ci, run node validate-pack.mjs, and see PASS? That is the whole assignment. Everything else is commentary.

Top comments (0)