The Blind Case Workshop

The projector still showed a green bar when the second invoice failed. The coding agent had “fixed” tax rounding in under a minute. It had also seen the expected total, 1104 cents, sitting in the prompt like an answer written on the back of the exam. The helper did not round. It returned 1104 and went home.

That leak is easy to miss in a room that is hungry for a passing run. A fluent model can memorize a number the way a student memorizes a printed key. The suite goes green. The next cart of line items does not. This ninety-minute workshop treats that moment as the lesson, not as a footnote. Students leave with a rerunnable envelope: one public sample they may discuss, one holdout file the model never reads, a linter that rejects leaked digits, and a checker that grades the holdout last.

The week this lab was assembled, developer feeds were still arguing whether fluent generation counts as engineering. The argument gets quieter when the expected cents never enter the prompt. Drafting remains allowed. Reciting the scoreboard does not.

What the room is actually teaching

The analogy is a music exam. The candidate may hear a practice scale. The judged piece stays in a closed folder until performance is over. If the judged notes are hummed during warm-up, the grade is about memory, not reading. Coding agents are the same instrument. Paste expect_total_cents: 1104 into the instructions and the loop can skip the rule.

The domain stays deliberately dull: integer cents, a tax rate in basis points, half-up on a non-negative amount. Dull numbers are easier to audit on paper. Flashy demos hide the instant the model stops computing and starts echoing.

A facilitator keeps two clocks. Wall time is ninety minutes. The more important clock is the moment the holdout is revealed, which must be after the agent is told to stop editing.

Minute 0–15: restage the leak

Students begin with a truncated helper and a generous prompt. The prompt includes the holdout lines, the rate, and the expected total. Most agents then hardcode the total or special-case the triple 499, 499, 17. The public suite, if it only contains that triple, will lie.

The starting tree is small enough to sketch on a whiteboard.

lab/
  src/tax.py
  prompts/task.txt
  oracle/public_sample.json
  oracle/holdout.json
  tools/lint_prompt.py
  tools/check.py

src/tax.py is the only file an agent may edit. prompts/task.txt is the only file a human may edit during the agent session. Both oracle files stay closed. The public sample is fair to describe in words. The holdout is not.

A truncated helper makes the first failure honest.

# src/tax.py
from typing import List


def tax_cents(lines_cents: List[int], rate_bp: int) -> int:
    base = sum(lines_cents)
    return (base * rate_bp) // 10_000  # truncates


def total_cents(lines_cents: List[int], rate_bp: int) -> int:
    return sum(lines_cents) + tax_cents(lines_cents, rate_bp)

Facilitators run the public sample by hand. One hundred plus one hundred cents at 1000 basis points is twenty cents of tax under half-up. Truncation also yields twenty here, which is the trap. A sample that does not distinguish truncation from rounding will bless a cheat. The public sample is for conversation. The holdout is for judgment.

Minute 15–35: split the envelope

Students copy two JSON files from a paper handout, not from chat history. The public sample may be mentioned in prompts/task.txt. The holdout may not. Digits from the holdout are contraband in the prompt, including line items, the rate, and both expected fields.

{
  "id": "public-100-100-1000bp",
  "lines_cents": [100, 100],
  "rate_bp": 1000,
  "expect_tax_cents": 20,
  "expect_total_cents": 220,
  "note": "conversation only; does not distinguish truncate from half-up"
}

The holdout uses a fraction that truncation gets wrong. 499 + 499 + 17 is 1015 cents. 875 basis points produces 88.8125 cents of tax. Half-up on a non-negative value becomes 89. The sealed total is 1104. Truncation yields 88 and 1103. Students should compute that on paper before they trust the file.

{
  "version": 1,
  "cases": [
    {
      "id": "holdout-three-lines-875bp",
      "lines_cents": [499, 499, 17],
      "rate_bp": 875,
      "expect_tax_cents": 89,
      "expect_total_cents": 1104
    }
  ]
}

tools/lint_prompt.py is the usher at the door. It does not grade tax. It only refuses a prompt that contains holdout digits.

# tools/lint_prompt.py
from __future__ import annotations
import json, re, sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
PROMPT = ROOT / "prompts" / "task.txt"
HOLDOUT = ROOT / "oracle" / "holdout.json"


def tokens_from_holdout(payload: dict) -> set[str]:
    found: set[str] = set()
    for case in payload["cases"]:
        for key in ("rate_bp", "expect_tax_cents", "expect_total_cents"):
            found.add(str(case[key]))
        for n in case["lines_cents"]:
            found.add(str(n))
    return found


def main() -> None:
    text = PROMPT.read_text()
    banned = tokens_from_holdout(json.loads(HOLDOUT.read_text()))
    hits = sorted({t for t in banned if re.search(rf"(?<!\d){re.escape(t)}(?!\d)", text)})
    if hits:
        raise SystemExit(f"holdout digits leaked into prompt: {hits}")
    print("prompt lint passed")


if __name__ == "__main__":
    main()

The linter is crude on purpose. It will also fire if a student writes a harmless 875 in a comment about an unrelated rate. That friction is cheaper than a hardcoded 1104. Facilitators can allow a spoken discussion of the rounding rule without reciting the holdout arithmetic.

Minute 35–60: a prompt that names a rule, not a score

The allowed prompt talks like a spec, not like a scoreboard. Students should feel the difference in their hands. One version is enough.

Edit only src/tax.py.
Do not read or mention files under oracle/.
Implement tax_cents(lines_cents, rate_bp) in integer cents.
Tax is half-up of (sum(lines) * rate_bp / 10000) for non-negative sums.
total_cents is sum plus tax.
You may reason about a public sample of two 100-cent lines at 1000 bp.
Stop after python tools/lint_prompt.py and python tools/check.py --public both succeed.
Do not ask for other expected numbers.

tools/check.py runs in two modes. --public is what the agent may see on the screen. --holdout is a later human step. Mixing them in one default command invites the model to peek at the failure string, which often reprints the expected cents. The workshop keeps the holdout output off the agent’s transcript until the stop rule fires.

# tools/check.py
from __future__ import annotations
import argparse, json, sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]


def half_up(n: float) -> int:
    return int(n + 0.5) if n >= 0 else int(n - 0.5)


def load_cases(name: str) -> list[dict]:
    payload = json.loads((ROOT / "oracle" / name).read_text())
    if name == "public_sample.json":
        return [payload]
    return payload["cases"]


def run(name: str) -> None:
    sys.path.insert(0, str(ROOT / "src"))
    import tax  # noqa: E402
    failures = []
    for case in load_cases(name):
        paper = half_up(sum(case["lines_cents"]) * case["rate_bp"] / 10_000)
        if paper != case["expect_tax_cents"]:
            raise SystemExit(f"oracle {case['id']} is internally inconsistent")
        got_tax = tax.tax_cents(case["lines_cents"], case["rate_bp"])
        got_total = tax.total_cents(case["lines_cents"], case["rate_bp"])
        if got_tax != case["expect_tax_cents"] or got_total != case["expect_total_cents"]:
            failures.append(case["id"])
    if failures:
        raise SystemExit(f"mismatch on {name}: {failures}")
    print(f"{name} passed")


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--public", action="store_true")
    p.add_argument("--holdout", action="store_true")
    args = p.parse_args()
    if args.public == args.holdout:
        raise SystemExit("choose exactly one of --public or --holdout")
    run("public_sample.json" if args.public else "holdout.json")


if __name__ == "__main__":
    main()

A repaired helper is short. Integer-only students can use (base * rate_bp + 5000) // 10000 for the same non-negative half-up. Either form is acceptable. Returning 1104 as a literal is not, and the holdout exists to catch that once the public sample has already gone green.

# src/tax.py
from typing import List


def tax_cents(lines_cents: List[int], rate_bp: int) -> int:
    base = sum(lines_cents)
    return int((base * rate_bp) / 10_000 + 0.5)


def total_cents(lines_cents: List[int], rate_bp: int) -> int:
    return sum(lines_cents) + tax_cents(lines_cents, rate_bp)

Minute 60–75: reveal the holdout

Hands off the keyboards. Facilitators run python tools/check.py --holdout on a machine the agent cannot see. A hardcoded 1104 from a leaked prompt fails as soon as the handout gains a second holdout case; even a single case fails if the model truncated and never read 1104. The debrief names three failure modes in ordinary language. Echoing is reciting a number from the prompt. Special-casing is an if on the public sample. Truncation is the original bug, still alive because the public sample could not detect it.

Students who passed --public and failed --holdout did not fail the course. They failed the cheat. They then get one more constrained edit, still without holdout digits in the prompt. The second attempt is usually the rounding rule they should have written first.

A second holdout can be added after class without changing the prompt at all. 250 + 250 at 333 basis points is a useful extra: 500 * 333 / 10000 is 16.65, which half-up makes 17. Truncation makes 16. If the helper only memorized 89, this case breaks it. The prompt linter still has nothing new to leak if nobody pastes those numbers into task.txt.

Minute 75–90: rerun on a shared endpoint

The last quarter hour is logistics. A class that depends on fourteen personal API keys will spend this block on billing pages. The same envelope should rerun on one shared box so the holdout command is identical for every student.

MonkeyCode’s free model access and free server option can host that shared rerun when the group has no other common agent endpoint. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The lab does not require that host. A local editor, a vendor key, or a human typing into tax.py all count if the holdout stays out of the prompt. The free path is a classroom convenience, not a quality ranking, a quota promise, or an uptime claim. Facilitators still print the holdout ids on the board only after lint has passed.

The rerun is two commands, in order, on a clean checkout.

python tools/lint_prompt.py
python tools/check.py --public
# agent session, if any, ends here
python tools/check.py --holdout

If lint fails, the agent session is void even when both checks would have passed. A green holdout bought with leaked digits is a different course, closer to open-book recitation than to implementation.

Limits, and who should skip this lab

A digit linter is not cryptography. A model can still infer 89 from a verbose failure string, a stack trace, or a helpful teammate. Facilitators must keep --holdout output off the transcript until the stop rule. The public sample must not be a disguised copy of the holdout. If both files share the same lines, the envelope is theater.

The JSON cases are not a tax specification. They will not catch a helper that special-cases two or three memorized carts. Teams that ship money need property tests, fuzzing, and a review that never happened in this room. The half-up helper also ignores refunds, mixed signs, and jurisdiction tables. Those absences are intentional. They are also reasons not to paste this file into production.

Skip the workshop when the acceptance criteria are a handful of golden numbers the model is supposed to match exactly, such as a fixture dumped from a legacy system. Skip it when students cannot keep a file unread. Skip it when the grade is speed to a green bar. A loop that recites 1104 in twenty seconds should score zero.

The artifact worth keeping is not tax.py. It is the habit of holding out at least one case the agent was never told, then grading that case after the hands stop. The invoice cents are bait. The closed folder is the craft.