Workshop: Map Agent Claims to a Failing Command Ledger in 90 Minutes

An agent-assisted lab only counts as engineering when each claim maps to a command that fails first. Natural-language success reports from a coding model are not evidence, because they can pass by deleting tests. This workshop gives students a ledger format, a Makefile contract, and a mutation check they can rerun. The same method still works if the lab later moves onto another provider or endpoint.

Who this session is for

The outline targets instructors and self-taught developers who already run short coding labs with language models. Students should be comfortable with Python 3.11 or newer, pytest, and a local POSIX shell. Teams that need production SLAs, private data handling, or multi-service orchestration should pick another control plane. This session is not a model bake-off, and it does not rank vendors or quote unpublished benchmarks.

Use the ledger when the problem has a small oracle that students can encode as JSON fixtures. Skip it when the task has no deterministic expected output, or when claim commands would execute untrusted strings. The decision table below is a teaching filter, not a capacity plan.

Situation	Use this ledger	Do not use it
Classroom lab with a known oracle	Yes	—
Agent output is prose without a fixture	—	Yes
Commands in the ledger are student-authored and allowlisted	Yes	—
Commands could include untrusted shell fragments	—	Yes until argv is allowlisted
Production incident work with private payloads	—	Yes

Timing

Treat the schedule as a hard budget rather than a loose suggestion for discussion.

1. 0–10 min — State the rule: no claim enters the ledger without a currently failing command.
2. 10–25 min — Write three claims and fixtures for a small interval-merge problem.
3. 25–50 min — Request an implementation from a free coding endpoint, then run make prove.
4. 50–70 min — Inject a mutation that drops one assertion and confirm the prove target goes red.
5. 70–85 min — Review which ledger fields students actually filled, and which they left blank.
6. 85–90 min — Recap limitations and name the claims this ledger cannot prove.

The core artifact

Students keep one JSON ledger next to the code under test, and they treat every row as an untrusted claim. Each row binds a claim id, a shell command, an expected initial exit code, and a fixture path. The Makefile refuses prove until fail-first has been recorded in the same working tree. That stamp is session memory, not a gradebook.

{
  "problem": "merge_intervals",
  "endpoint": "free-lab",
  "claims": [
    {
      "id": "C1",
      "statement": "Overlapping closed intervals collapse into a single span.",
      "command": "pytest -q tests/test_c1_overlap.py",
      "fixture": "fixtures/c1_overlap.json",
      "fail_first_exit": 1
    },
    {
      "id": "C2",
      "statement": "Touching intervals that share an endpoint also merge.",
      "command": "pytest -q tests/test_c2_touch.py",
      "fixture": "fixtures/c2_touch.json",
      "fail_first_exit": 1
    },
    {
      "id": "C3",
      "statement": "An empty input list returns an empty list without raising.",
      "command": "pytest -q tests/test_c3_empty.py",
      "fixture": "fixtures/c3_empty.json",
      "fail_first_exit": 1
    }
  ]
}

Label the JSON above as a teaching fixture, not as production telemetry from a live classroom. Students copy it into claims.json and must not edit fail_first_exit after the first red run. If a stub accidentally returns a plausible list, the claim is too weak, and the fixture must be tightened before the generate step starts.

Makefile contract

PYTHON ?= python3
LEDGER := claims.json
STAMP  := .fail-first.ok

.PHONY: fail-first prove mutate clean

fail-first:
    $(PYTHON) tools/check_ledger.py --mode fail-first --ledger $(LEDGER)
    touch $(STAMP)

prove: $(STAMP)
    $(PYTHON) tools/check_ledger.py --mode prove --ledger $(LEDGER)

mutate:
    $(PYTHON) tools/mutate_drop_assert.py tests/test_c1_overlap.py
    -$(PYTHON) tools/check_ledger.py --mode prove --ledger $(LEDGER); \
        status=$$?; \
        $(PYTHON) tools/mutate_drop_assert.py --restore tests/test_c1_overlap.py; \
        exit $$status

clean:
    rm -f $(STAMP)

The stamp file is the only session memory the lab needs to keep honest. If a student skips fail-first, prove does not run, which blocks the shortcut of writing tests after a green implementation. Instructors should collect the stamp and the ledger, not a chat export, when they grade the lab.

Exercise 1 — Write claims that already fail (15 minutes)

Create src/intervals.py with a stub that raises NotImplementedError so every first run is red for a known reason. Then add one pytest module per claim, each loading JSON rather than inlined literals that drift during copy. Keep the assertion count at one behavioral check per file so later mutations stay obvious to a reader. The empty-input claim exists to catch models that return None instead of [].

# tests/test_c1_overlap.py
import json
from pathlib import Path
from src.intervals import merge_intervals

def test_overlapping_closed_intervals_collapse():
    payload = json.loads(Path("fixtures/c1_overlap.json").read_text())
    assert merge_intervals(payload["input"]) == payload["expected"]

{
  "input": [[1, 3], [2, 6], [8, 10], [15, 18]],
  "expected": [[1, 6], [8, 10], [15, 18]]
}

{
  "input": [[1, 4], [4, 5]],
  "expected": [[1, 5]]
}

{
  "input": [],
  "expected": []
}

Run make fail-first and keep the checker exit at zero while pytest exits non-zero for every claim. If C2 is omitted, many generated functions will treat touching ranges as disjoint and still look correct on C1. That gap is a ledger defect, not a model defect, and it should be fixed before anyone pastes a prompt.

Exercise 2 — Generate an implementation on a shared free endpoint (25 minutes)

This is the point where a shared free coding setup matters, because a classroom cannot grade work that only ran inside one paid chat window. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which is enough to host the same prompt and the same runner for every student laptop in the room.

Treat that product as a lab backend, not as the subject of the lesson or as a scored vendor. Students paste a prompt that forbids extra files and forbids editing tests, fixtures, or the ledger. The prompt below is a proposal for this outline, not a measured template with claimed pass rates.

Implement merge_intervals(intervals) in src/intervals.py only.
Do not edit tests/, fixtures/, claims.json, or the Makefile.
Intervals are inclusive integer pairs [start, end] with start <= end.
Return merged spans sorted by start. Empty input returns [].

After the model writes the function, students run make prove against the unchanged tests. The ledger checker should load each claim, execute the command, and require exit code 0. Any extra file the model created is a lab failure even if tests pass, because the contract was file-scoped rather than chat-scoped.

Ledger checker (teaching code)

# tools/check_ledger.py
from __future__ import annotations

import argparse, json, subprocess, sys
from pathlib import Path

def load_claims(path: Path) -> list[dict]:
    data = json.loads(path.read_text())
    claims = data.get("claims") or []
    if not claims:
        raise SystemExit("ledger has no claims")
    return claims

def run_cmd(command: str) -> int:
    completed = subprocess.run(command, shell=True)
    return int(completed.returncode)

def fail_first(claims: list[dict]) -> int:
    bad = 0
    for claim in claims:
        code = run_cmd(claim["command"])
        expected = int(claim["fail_first_exit"])
        if code == 0:
            print(f"{claim['id']} already passes; claim is not a test")
            bad += 1
        elif code != expected:
            print(f"{claim['id']} exited {code}, expected {expected}")
            bad += 1
        else:
            print(f"{claim['id']} fails as required")
    return 1 if bad else 0

def prove(claims: list[dict]) -> int:
    bad = 0
    for claim in claims:
        code = run_cmd(claim["command"])
        if code != 0:
            print(f"{claim['id']} still failing with exit {code}")
            bad += 1
        else:
            print(f"{claim['id']} proved")
    return 1 if bad else 0

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--mode", choices=("fail-first", "prove"), required=True)
    parser.add_argument("--ledger", required=True)
    args = parser.parse_args()
    claims = load_claims(Path(args.ledger))
    status = fail_first(claims) if args.mode == "fail-first" else prove(claims)
    sys.exit(status)

if __name__ == "__main__":
    main()

This checker is intentionally small so students can read it in one sitting without a framework tour. It does not sandbox the shell, so untrusted commands do not belong in claims.json during a first offering. Instructors grading a second cohort should replace shell=True with an allowlisted argv map before they collect stamps.

Exercise 3 — Mutation that drops an assertion (20 minutes)

Green tests after an agent edit are still weak if the agent can delete the assertion and keep the file importable. The mutate target comments out the first assert in tests/test_c1_overlap.py and reruns prove without touching src/intervals.py. The expected teaching result is a non-zero exit from the prove path. If prove stays green after the drop, the claim never tested behavior and must be rewritten.

# tools/mutate_drop_assert.py
from pathlib import Path
import sys

MARKER = "# mutated-assert: "

def drop(path: Path) -> None:
    lines = path.read_text().splitlines(True)
    for i, line in enumerate(lines):
        stripped = line.lstrip()
        if stripped.startswith("assert "):
            indent = line[: len(line) - len(stripped)]
            lines[i] = f"{indent}{MARKER}{stripped}"
            path.write_text("".join(lines))
            return
    raise SystemExit(f"no assert found in {path}")

def restore(path: Path) -> None:
    text = path.read_text().replace(MARKER, "")
    path.write_text(text)

if __name__ == "__main__":
    restore_mode = "--restore" in sys.argv
    target = Path(sys.argv[-1])
    restore(target) if restore_mode else drop(target)

Students should log three numbers only: fail-first checker status, prove exit, and mutate exit. Do not collect latency, token counts, or model quality scores in this lab, because those numbers change by endpoint and distract from the ledger rule. A useful board note is a three-column table with claim ids, not a screenshot of a chat transcript.

Worked example students can rerun

Use this sequence on a clean tree after the files above exist. The commands assume a POSIX shell and a virtualenv that already has pytest.

python3 -m venv .venv
. .venv/bin/activate
pip install pytest
mkdir -p src tests fixtures tools
# copy the ledger, tests, fixtures, Makefile, and tools from this article
rm -f .fail-first.ok
make fail-first    # pytest red per claim; checker status 0
# paste the prompt into the shared free lab endpoint, write src/intervals.py only
make prove         # expect 0
make mutate        # expect non-zero from prove, then restored tests

A reference implementation that satisfies the three fixtures is shown next. Label it as a teaching solution, not as the only correct merge, and do not paste it into the prompt.

# src/intervals.py
def merge_intervals(intervals):
    if not intervals:
        return []
    ordered = sorted(intervals, key=lambda pair: (pair[0], pair[1]))
    merged = [list(ordered[0])]
    for start, end in ordered[1:]:
        last = merged[-1]
        if start <= last[1]:
            last[1] = max(last[1], end)
        else:
            merged.append([start, end])
    return merged

If a student's model returns a version that drops touching intervals, claim C2 stays red and the ledger remains honest. That failure is the lesson, not a reason to loosen the fixture or to delete C2 from claims.json. Students who “fix” the lab by editing expected JSON have left the engineering rule, even when the chat window still looks successful.

What the ledger does not prove

The method does not prove algorithmic optimality, concurrency safety, or prompt robustness across vendors. It also does not prove that a free server will remain available, because availability is an operational choice rather than a test assertion. Students who need property-based coverage should add Hypothesis after this lab, not during the ninety-minute window, because shrinking counterexamples will blow the timing budget.

Do not use this workshop when the task has no oracle, when fixtures would leak private data, or when shelling out from check_ledger.py would execute untrusted strings. Do not use it as a substitute for code review on security-sensitive diffs, because a passing ledger only shows that named fixtures stayed green. Instructors running large classes should allowlist commands before the second session, then keep the fail-first stamp as the gradeable artifact.

Closing

The transferable output is the ledger plus the fail-first stamp, not a chat transcript and not a vendor comparison. If a class needs one shared free endpoint for the generate step, MonkeyCode's free model access and free server option can host that lab runner without changing the Makefile contract.