Don't Grade the Agent Until the Tool Schema Is Frozen

If a bootcamp agent can invent, rename, or silently drop tool fields mid-demo, you are not grading an agent. You are grading theater. Freeze a JSON contract first. Reject drift in CI. Only then let a model write the handler behind that lock.

That is the whole lab. Not a glossary. Not a vibe check. A fail-closed gate you can run on a laptop in one sitting.

Why start here? Because students will clap at a tool call that printed JSON. They will not notice user_id became userId on the fourth hop. Have you ever watched a demo succeed while the fixture file quietly rot? I have. It is an ugly way to learn.

What this lab is (and is not)

This is a four-hour checkpointed exercise for a JS-friendly bootcamp. Students pin one tool contract, write a validator that fails closed, lock a hash of that file, and only then generate an implementation. The model is optional labor. The schema is the product.

It is not an MCP tour. It is not a production permission system. JSON Schema does not know who the user is. If your students still struggle to return a JSON object from a function, this lab is too early. Send them back to JSON.parse drills.

Trend talk this week keeps circling “agentic” vocabulary and cheap generated code. Fine. Vocabulary does not catch field drift. A frozen file does.

Lab setup

Timebox: 4 hours plus a 90-minute stretch. Pairing is allowed. Copy-pasting a schema from a chatbot before checkpoint 1 is not.

You need:

• Node.js 20+
• Git
• A text editor
• No cloud account required for checkpoints 1–4

Scaffold:

mkdir tool-contract-lab && cd tool-contract-lab
git init
mkdir -p tools tests fixtures handlers
printf '{"name":"tool-contract-lab","type":"module"}\n' > package.json

One tool only: create_note. Ticket-shaped. Boring on purpose. If the domain is exciting, students will argue product instead of contracts. We want arguments about additionalProperties.

Checkpoint 0 — write the freeze, out loud

Before any model runs, students write tools/create_note.schema.json by hand. No generator. If they cannot name the required fields, they are not ready to “use tools.”

{
  "$id": "create_note/v1",
  "title": "create_note",
  "type": "object",
  "additionalProperties": false,
  "required": ["ticket_id", "body", "visibility"],
  "properties": {
    "ticket_id": {
      "type": "string",
      "pattern": "^tkt_[a-z0-9]{8}$"
    },
    "body": {
      "type": "string",
      "minLength": 1,
      "maxLength": 500
    },
    "visibility": {
      "enum": ["private", "public"]
    },
    "idempotency_key": {
      "type": "string",
      "minLength": 8,
      "maxLength": 64
    }
  }
}

Notice additionalProperties: false. That is the whole punchline. Models love bonus keys. Bonus keys are how demos lie.

Also notice idempotency_key is optional in required but typed if present. Why optional? Because week-one students will forget it. We will make it required in a stretch goal, not in the first hour.

Commit this file. Do not touch it again without a version bump. That rule is the lab.

Checkpoint 1 — a fail-closed validator (Node built-ins only)

Full JSON Schema engines are great. They are also a rabbit hole. This lab ships a subset checker so the gate is readable. Label it as a subset. Do not pretend it is Draft 2020-12.

validate.mjs:

import { readFileSync } from 'node:fs';

export function loadSchema(path) {
  return JSON.parse(readFileSync(path, 'utf8'));
}

export function validate(schema, payload) {
  const errors = [];
  if (payload === null || typeof payload !== 'object' || Array.isArray(payload)) {
    return { ok: false, errors: ['payload must be a plain object'] };
  }

  if (schema.additionalProperties === false) {
    for (const key of Object.keys(payload)) {
      if (!schema.properties[key]) errors.push(`unknown field: ${key}`);
    }
  }

  for (const key of schema.required ?? []) {
    if (!(key in payload)) errors.push(`missing field: ${key}`);
  }

  for (const [key, value] of Object.entries(payload)) {
    const rule = schema.properties[key];
    if (!rule) continue;
    if (rule.type && typeof value !== rule.type) {
      errors.push(`${key} should be ${rule.type}`);
    }
    if (rule.enum && !rule.enum.includes(value)) {
      errors.push(`${key} not in enum`);
    }
    if (rule.minLength && String(value).length < rule.minLength) {
      errors.push(`${key} too short`);
    }
    if (rule.maxLength && String(value).length > rule.maxLength) {
      errors.push(`${key} too long`);
    }
    if (rule.pattern && !new RegExp(rule.pattern).test(String(value))) {
      errors.push(`${key} fails pattern`);
    }
  }

  return { ok: errors.length === 0, errors };
}

Is this complete? No. Nested objects, $ref, and anyOf are out of scope. That is a feature. Students can see every branch. If they need $ref they are in a different course.

Checkpoint 2 — two fixtures, then a third that must fail

fixtures/create_note.valid.json:

{
  "ticket_id": "tkt_ab12cd34",
  "body": "Customer asked for a refund window.",
  "visibility": "private"
}

fixtures/create_note.invalid-unknown-field.json:

{
  "ticket_id": "tkt_ab12cd34",
  "body": "Customer asked for a refund window.",
  "visibility": "private",
  "userId": "oops"
}

tests/validate.test.mjs:

import assert from 'node:assert/strict';
import { loadSchema, validate } from '../validate.mjs';

const schema = loadSchema(new URL('../tools/create_note.schema.json', import.meta.url));

const valid = JSON.parse(
  await (await import('node:fs/promises')).readFile(new URL('../fixtures/create_note.valid.json', import.meta.url), 'utf8')
);
const invalid = JSON.parse(
  await (await import('node:fs/promises')).readFile(new URL('../fixtures/create_note.invalid-unknown-field.json', import.meta.url), 'utf8')
);

assert.equal(validate(schema, valid).ok, true);
assert.equal(validate(schema, invalid).ok, false);
assert.ok(validate(schema, invalid).errors.some((e) => e.includes('unknown field')));
console.log('fixtures ok');

Run it:

node tests/validate.test.mjs

If this test is skipped, the rest of the lab is fan fiction. I will not grade a README screenshot of a chat transcript. Will you?

Checkpoint 3 — lock the bytes, not the intention

Schemas drift the same way comments drift. Students “just add a field for the demo.” So we hash the file.

scripts/freeze.mjs:

import { readFileSync, writeFileSync } from 'node:fs';
import { createHash } from 'node:crypto';

const path = 'tools/create_note.schema.json';
const bytes = readFileSync(path);
const digest = createHash('sha256').update(bytes).digest('hex');
writeFileSync('tools/create_note.schema.sha256', `${digest}  ${path}\n`);
console.log(digest);

scripts/check-freeze.mjs:

import { readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';

const path = 'tools/create_note.schema.json';
const expected = readFileSync('tools/create_note.schema.sha256', 'utf8').trim();
const digest = createHash('sha256').update(readFileSync(path)).digest('hex');
const line = `${digest}  ${path}`;
if (line !== expected) {
  console.error('schema drifted without a freeze update');
  console.error('expected:', expected);
  console.error('actual:  ', line);
  process.exit(1);
}
console.log('schema freeze intact');

Workflow:

node scripts/freeze.mjs
git add tools/create_note.schema.json tools/create_note.schema.sha256
git commit -m "freeze create_note v1"
node scripts/check-freeze.mjs

Now mutate the schema by adding "priority": { "type": "number" } and run the check. It must die. If it does not die, you built a diary, not a contract.

Version bump rule, written on the board: change the schema → change $id to create_note/v2 → new hash → new fixtures. No silent edits. Ever.

Checkpoint 4 — a stub that refuses bad calls

The handler does not exist yet. The door does.

server.mjs:

import http from 'node:http';
import { loadSchema, validate } from './validate.mjs';

const schema = loadSchema('tools/create_note.schema.json');

const server = http.createServer(async (req, res) => {
  if (req.method !== 'POST' || req.url !== '/tools/create_note') {
    res.writeHead(404); res.end('{"error":"not found"}'); return;
  }
  const chunks = [];
  for await (const c of req) chunks.push(c);
  let payload;
  try {
    payload = JSON.parse(Buffer.concat(chunks).toString('utf8') || '{}');
  } catch {
    res.writeHead(400); res.end('{"error":"invalid json"}'); return;
  }
  const result = validate(schema, payload);
  if (!result.ok) {
    res.writeHead(422);
    res.end(JSON.stringify({ error: 'contract_violation', details: result.errors }));
    return;
  }
  res.writeHead(204); res.end();
});

server.listen(8787, () => console.log('stub on :8787'));

Probe it:

node server.mjs &
curl -sS -D - -o /tmp/body -X POST http://127.0.0.1:8787/tools/create_note \
  -H 'content-type: application/json' \
  -d '{"ticket_id":"tkt_ab12cd34","body":"hi","visibility":"public","userId":1}'

You want 422. A 204 here means the gate is decorative. Decorative gates are how bootcamp agents become production incidents with extra steps.

Checkpoint 5 — now the model may speak

Only after 204-on-valid and 422-on-drift do students generate handlers/create_note.mjs. The handler must import the same validate function. It must not parse fields the schema does not name.

Proposed shape (students fill the body; this is a sketch, not a claimed production service):

import { validate } from '../validate.mjs';

export function createNote(schema, payload, store) {
  const result = validate(schema, payload);
  if (!result.ok) return { status: 422, body: { error: 'contract_violation', details: result.errors } };
  const id = `note_${store.size + 1}`;
  store.set(id, { id, ...payload, created_at: 'lab-fixed-timestamp' });
  return { status: 201, body: { id } };
}

Where does a model enter? After the freeze. Not before. Students may paste the schema and the stub into any coding assistant and ask for a handler that cannot add fields. If the assistant emits userId, the validator still wins. That is the lesson.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a $0 generate step for checkpoint 5, I point the lab at MonkeyCode’s free model access and free server option — after the schema hash exists, not as a substitute for it. Strip that sentence out and the lab still runs on any editor.

Hard rule for the generate step: the model may edit handlers/ only. tools/*.schema.json is read-only in the assignment. If a student “lets the agent tidy the schema,” they lose the freeze points. No debate.

Stretch goals

Do these in order. Skipping ahead is a tell.

1. Promote idempotency_key to required. Same $id is illegal. Ship create_note/v2 and a second freeze file.
2. Record a golden trace: one JSONL file of {request, status, errors}. Re-run it in CI. If status codes move, the trace fails.
3. Add a second tool, list_notes, with its own schema. Share zero fields by accident. If ticket_id patterns diverge, that is a design smell — write it down, do not “fix it in the prompt.”
4. Replace the subset validator with a real library without relaxing additionalProperties. If tests get greener by deleting assertions, that is a failing stretch, not a passing one.

Fair rubric (20 points)

Grade the lock. Do not grade the chat.

Evidence	Points	Automatic zero if…
Hand-written v1 schema with additionalProperties: false	4	Schema was generated before checkpoint 0
Valid + invalid fixtures and node tests/validate.test.mjs passes	4	Unknown-field fixture is missing
check-freeze.mjs fails on a one-character schema edit	4	Hash file is re-frozen in the same commit as a “demo fix”
Stub returns 422 with error names, 204/201 on valid	4	Server accepts extra keys
Handler lives behind validate() and does not invent fields	4	Model-only demo with no stub

Stretch work can add up to 6 bonus points. Bonus cannot rescue a missing freeze. A flashy multi-tool demo with a mutable schema is a 0 with extra animation.

I will answer “but it worked in the screenshot” with the hash mismatch. That is the pedagogy.

What I am not claiming

This subset validator is not JSON Schema. It will not save you from confused types inside nested arrays. It will not do auth. It will not cap token spend. It will not stop a model from lying in body text. It only stops shape drift at the tool boundary.

Who should skip this lab?

• Folks who already ship OpenAPI or protobuf with breaking-change CI. You already have the grown-up version.
• Production platform teams looking for an agent runtime. This is a teaching stub on port 8787.
• Students who have never written a unit test. Teach assert first.
• Anyone hoping a free model will invent a stable contract for them. It will not. That is why the freeze is checkpoint zero.

Closing the loop

Agents do not fail because students forgot a buzzword. They fail because the interface moved while nobody was looking. Pin the bytes. Fail closed. Generate later.

If your cohort only remembers one sentence, make it this one: no live tools until the schema diffs empty. Everything else is costume jewelry on an untyped POST.