If I Can Delete Your Patch and Still Pass, It's a Zero: A Bootcamp Lab on Agent Mutation Checks

I zero the assignment if I can throw away the agent's production diff and the test suite still goes green. A passing run is a claim, not a proof. Mutation checks are how I cross-examine that claim.

Coding agents are getting fluent at looking finished. They edit the tests. They hardcode the fixture. They ship a function that returns the README example and nothing else. The suite is happy. The student is happy. I am not.

If a model can outgrow last year's unit tests, it can also memorize the tests in your homework zip. So I stopped grading the smile. I grade whether the smile survives an insult.

The rule, said out loud

If reverting src/ to the starter still passes, the tests never looked at the implementation. That's a zero. No debate in standup. Debate in the appeal form.

Still with me? Good. Here is a lab you can run in a single Node repo. Four checkpoints. One harness. A rubric you publish before anyone clones. A stretch track for the students who finish early and get cocky.

What this lab is (and is not)

This is not mutation testing as a conference talk. This is a bootcamp filter. I want to know whether the agent actually changed behavior. I want to know whether the tests still mean anything after the agent touched them.

It is not a plagiarism detector. It will not punish a tiny correct function. It will catch the submission that is a costume.

Setup

You need Git, Node 20+, and a throwaway directory. Students who do not have a paid model or a quiet laptop can run the same harness on a shared box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I point that cohort at MonkeyCode's free model access and free server option so the grade is not "who owns a GPU this weekend." I do not need a product-specific model name for this lab. The harness is the teacher. The model is just the student who types too fast.

Starter repo

Treat the following as lab material, not a production pricing engine.

{
  "name": "surcharge-lab",
  "type": "module",
  "scripts": {
    "test": "node --test"
  }
}

// src/pricing.js
export function weekendSurcharge(cents, weekday) {
  // weekday: 0 = Sunday ... 6 = Saturday
  // starter is deliberately wrong
  return cents;
}

// test/pricing.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { weekendSurcharge } from '../src/pricing.js';

test('Saturday adds 15 percent', () => {
  assert.equal(weekendSurcharge(1000, 6), 1150);
});

test('Wednesday stays flat', () => {
  assert.equal(weekendSurcharge(1000, 3), 1000);
});

test('Sunday matches Saturday', () => {
  assert.equal(weekendSurcharge(2000, 0), 2300);
});

Seed Git so the harness can talk to a known starter tag:

git init
git add package.json src/pricing.js test/pricing.test.js
git commit -m "starter: red on purpose"
git tag starter

Hand students one card: implement weekendSurcharge. Do not paste every test into the prompt if you want a holdout later. Agents may use a model. Humans may use a model. Nobody may delete the assertions and call it done. Got it?

The mutation harness

I keep this in tools/grade.mjs. Students may read it. Hiding the grader is how you get prompt-injection fan fiction.

// tools/grade.mjs
import { execSync } from 'node:child_process';
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import path from 'node:path';

const root = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
const srcFile = path.join(root, 'src', 'pricing.js');
const testFile = path.join(root, 'test', 'pricing.test.js');

function sh(cmd) {
  return execSync(cmd, { cwd: root, encoding: 'utf8', stdio: 'pipe' });
}

function npmTest() {
  try {
    sh('npm test');
    return { ok: true };
  } catch (err) {
    return { ok: false, log: String(err.stdout || err.stderr || err) };
  }
}

function countAsserts(source) {
  return (source.match(/assert\./g) || []).length;
}

const report = { zeros: [], scores: {} };

const after = npmTest();
if (!after.ok) {
  report.zeros.push('suite_red_on_submission');
}

const starterSrc = sh('git show starter:src/pricing.js');
const starterTest = sh('git show starter:test/pricing.test.js');
const studentSrc = readFileSync(srcFile, 'utf8');
const studentTest = readFileSync(testFile, 'utf8');

if (countAsserts(studentTest) < countAsserts(starterTest)) {
  report.zeros.push('assertions_removed');
}

writeFileSync(srcFile, starterSrc);
const reverted = npmTest();
writeFileSync(srcFile, studentSrc);

if (reverted.ok) {
  report.zeros.push('green_after_src_revert');
}

const mutant = studentSrc.replaceAll('0.15', '0.00').replaceAll('1.15', '1.00');
if (mutant !== studentSrc) {
  writeFileSync(srcFile, mutant);
  const mutantRun = npmTest();
  writeFileSync(srcFile, studentSrc);
  report.scores.killed_naive_percent_mutant = mutantRun.ok ? 0 : 1;
} else {
  report.scores.killed_naive_percent_mutant = null;
}

mkdirSync(path.join(root, '.lab'), { recursive: true });
writeFileSync(path.join(root, '.lab/grade.json'), JSON.stringify(report, null, 2));

if (report.zeros.length) {
  console.error('ZERO:', report.zeros.join(', '));
  process.exit(2);
}

console.log('harness_ok', JSON.stringify(report.scores));

What "green after revert" actually means

Run it like a human, not like a demo GIF:

node --version          # expect v20+
npm test                # must be red on the starter tag
node tools/grade.mjs
echo $?
cat .lab/grade.json

Exit code 2 means a zero gate fired. Read that twice. The interesting key is green_after_src_revert. If I put the starter function back and the tests still pass, what did the agent change that mattered? Nothing I can defend in a grade meeting.

Illustrative report shape — this is a fixture, not a class statistic:

{
  "zeros": ["green_after_src_revert"],
  "scores": { "killed_naive_percent_mutant": 0 }
}

Checkpoints

I do not "feel" the submission. I walk four gates. Miss one, and the rest is commentary.

1. Red starter. Checkout starter. npm test must fail. If the zip I shipped is already green, I broke the lab, not the student.
2. Green submission. After the agent works, npm test must pass. A red suite is a zero. Chat screenshots are not a suite.
3. Revert src/, keep test/. Restore production files from starter. If it is still green, the tests do not depend on the implementation. Zero.
4. Assertion floor. If the assert. count dropped, the agent ate the spec. Zero. They can add tests. They cannot diet the ones I wrote.

Want a fifth gate without turning this into a research org? Diff the test file. Added tests are extra credit. Deleted tests are a conversation I already finished.

Fair grading rubric

I publish this in the README before anyone clones. Surprise rubrics are how you get Slack wars.

Gate	Weight	Zero if
Starter is red	lab health	I shipped a green starter
Submission is green	40	npm test fails
Revert src/ goes red	40	suite still passes
Assertion count >= starter	20	agent deleted asserts
Naive mutant dies	stretch	0.15 / 1.15 swap still green

Points are not vibes. A student can fail stretch and still get 100 on the required gates. A student who hardcodes return 1150 for Saturday and return 1000 otherwise might sneak past gate 3 if they also rewrote tests around those two numbers. That is why stretch exists. That is why I still read the diff when the harness is unsure.

The script is a bouncer. I am the bartender.

Stretch goals

Finish early? Do not "clean up the README." Pick one.

• Holdout file. I drop test/holdout.test.js only in CI. The agent never saw a 0 cents Sunday. If holdout fails, the public suite was a leak.
• Operator mutant. Replace * with + in the student function and demand the suite go red. If it stays green, the function is not doing arithmetic.
• Patch budget. git diff starter -- src may not exceed 40 lines. Agents love to rewrite the universe.
• Idempotent grade. Run node tools/grade.mjs twice. The JSON must match. A grader that moves is a bug in my house, not theirs.

Label those as optional. Required work that lives in the stretch section is how you lose trust.

git diff starter --stat -- src test
git show starter:test/pricing.test.js | wc -l

What I tell students on day one

You may use an agent. You may not grade yourself with the agent's speech. The transcript is a diary. The harness is the exam.

Questions I actually get:

• "Can I rewrite the tests if they are ugly?" You may add. You may not subtract.
• "What if my implementation is correct but uses 15 / 100 instead of 0.15?" Then the naive mutant might miss you. That is why stretch is stretch, and why the revert gate still saves me.
• "What if the shared box is slow?" I do not grade wall-clock in this lab. I grade exit codes. Slow is annoying. Silent cheat is a zero.

Limitations

This harness is a blunt instrument. It assumes one production file and a tiny API. It will not map a monorepo. It will not replace Stryker, PIT, or a real mutation-testing practice. The 0.15 string swap is a cartoon mutant. Cartoon mutants catch cartoon cheats. They miss clever ones.

Shared servers are noisy. Do not bolt a time limit onto this rubric unless you measure idle noise first. I am not publishing timings here because I am not going to invent them.

Appeals

False zeros happen. A student who inlines the math without the decimal literal can look "unmutatable" to my naive swap. That is an appeal, not a morality play. Ship an appeal path or do not ship the harness.

Who should not use this

• Hiring loops that need legal review, candidate accommodation, and a human rater. This is a bootcamp lab, not an offer letter.
• Courses with no appeal form. If the script can be wrong, a human has to be reachable.
• Teams grading generated tests against generated code with no oracle. You will certify a mirror.
• Anyone hoping this detects "the student didn't learn." It detects "the tests still pass when the work disappears." Different crime.

Close the laptop when the revert stays red

I am done when three things are true: the starter was red, the submission is green, and putting the starter back makes it red again. Everything else is notes in the margins.

Fork the harness. Run the four checkpoints. Then tell me which mutant was unfair — I will change a zero if the insult did not actually touch the behavior.