The Receiving Inspection Workshop: A Gate for AI-Written Code

Three agent-written patches landed on a Friday afternoon. Two were nine-line diffs, one touched a helper nobody remembered owning, and all three were reviewed in the ten minutes between a standup and a release. On Monday the nightly seed job died, and nobody could say which candidate produced the line that killed it.

That team did not have a review problem. It had a receiving problem. A candidate sat in a pull request, looked plausible, and crossed into the trunk without anything recording what had actually been checked.

The last few weeks of DEV discussion keep circling a familiar split: model capability climbs, and the software wrapped around it stays about as fragile as the day it was written. That gap is not a model problem. It is a testing problem, and it is teachable in ninety minutes.

This workshop builds a receiving dock. Students leave with a gate that parses a candidate, runs it against a fixture, writes a verdict to a ledger, and refuses to be the last word on anything.

The invariant

The whole workshop hangs on one sentence, written by the students in the first ten minutes and never edited afterward: nothing reaches the trunk without a receipt.

A receipt is one JSON line. It names the task, the model that produced the candidate, the verdict, a machine-readable reason, and a timestamp. It is boring on purpose. When the nightly job breaks three days later, the receipt is what lets someone say this candidate passed a three-case fixture and nothing else, instead of reconstructing the week from chat scrollback.

The run sheet

Clock	Segment	Artifact at the end
00–10	The Monday failure story, told once	The written invariant
10–25	Fixture before gate	tasks/03_slugify/test_solution.py, three cases
25–45	Build the gate	gate.py, first accept receipt
45–60	Break it deliberately	Two quarantines, one false accept
60–75	Boundary sweep	A property test that catches the fake
75–90	Yield accounting and transport swap	receipts.jsonl and a yield number

Every segment ends with a file. If a segment runs long, the artifact gets cut down, never skipped, because a workshop that finishes with a feeling instead of a file teaches nothing rerunnable.

Fixture before gate

Students almost always want to write the gate first. That order produces a gate that grades whatever it happens to be shown, which is the exact failure mode of human review under deadline pressure.

The fixture comes first. A task is a directory with a prompt and a test file, and the candidate is expected to expose one function.

{
  "id": "03_slugify",
  "model": "fill-from-current-provider-docs",
  "prompt": "Write slugify(text: str) -> str"
}

# tasks/03_slugify/test_solution.py
from solution import slugify


def test_basic():
    assert slugify("Hello World") == "hello-world"


def test_collapses_runs():
    assert slugify("a   b---c") == "a-b-c"


def test_empty_stays_empty():
    assert slugify("") == ""

Three cases is the right number for the first pass. A fixture large enough to be impressive is a fixture nobody in the room can explain, and an unexplained fixture fails silently the day it is edited.

The gate

The gate is deliberately small: it parses, it runs, it records, it stops.

#!/usr/bin/env python3
"""gate.py - receiving inspection for model-written candidates."""
import ast, json, pathlib, subprocess, sys, time

ROOT = pathlib.Path(__file__).parent
MAX_BYTES = 8192


def parse_ok(path):
    try:
        ast.parse(path.read_text())
    except SyntaxError as exc:
        return False, f"syntax_error_line_{exc.lineno}"
    return True, "parsed"


def behavior(test_path, scratch):
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", str(test_path)],
        cwd=scratch, capture_output=True, text=True, timeout=60,
    )
    lines = proc.stdout.strip().splitlines()
    return proc.returncode == 0, (lines[-1] if lines else "no_output")


def main(task_dir):
    task = json.loads(pathlib.Path(task_dir, "task.json").read_text())
    scratch = ROOT / "scratch" / task["id"]
    scratch.mkdir(parents=True, exist_ok=True)
    candidate = ROOT / "candidates" / f'{task["id"]}.py'
    verdict, reason = "quarantine", "no_candidate"

    if candidate.exists() and candidate.stat().st_size <= MAX_BYTES:
        ok, why = parse_ok(candidate)
        if ok:
            (scratch / "solution.py").write_text(candidate.read_text())
            (scratch / "test_solution.py").write_text(
                pathlib.Path(task_dir, "test_solution.py").read_text()
            )
            ok, why = behavior(scratch / "test_solution.py", scratch)
        verdict, reason = ("accept" if ok else "quarantine"), why
    elif candidate.exists():
        reason = f"oversize_{candidate.stat().st_size}b"

    receipt = {
        "task": task["id"],
        "model": task.get("model", "unspecified"),
        "verdict": verdict,
        "reason": reason,
        "ts": int(time.time()),
    }
    with open(ROOT / "receipts.jsonl", "a") as fh:
        fh.write(json.dumps(receipt) + "\n")
    print(json.dumps(receipt))


if __name__ == "__main__":
    main(sys.argv[1])

A candidate for the slug task, dropped at candidates/03_slugify.py, looks like this:

import re


def slugify(text):
    return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")

$ pip install pytest
$ python gate.py tasks/03_slugify
{"task": "03_slugify", "model": "fill-from-current-provider-docs", "verdict": "accept", "reason": "3 passed", "ts": 1792000000}

If pytest reports an ImportError on solution, the instructor has hit the classic path problem; run the gate with PYTHONPATH=. set and the scratch directory as the working directory, and the failure disappears. That detour is worth five minutes, because everyone meets it eventually in their own repository.

Break it deliberately

This is the segment that changes how students think about green checks. The instructor supplies a candidate that hardcodes the fixture's answers:

def slugify(text):
    return {
        "Hello World": "hello-world",
        "a   b---c": "a-b-c",
        "": "",
    }.get(text, "")

The gate accepts it. Three tests, three passes, one receipt that says accept, and a function that will return an empty string for every input the product actually receives. The fixture graded the fixture.

The correction is a property check that ties output back to input instead of to remembered examples:

# tasks/03_slugify/test_boundary.py
import random, string
from solution import slugify


def test_alnum_digest_matches():
    rng = random.Random(7)
    for _ in range(200):
        raw = "".join(
            rng.choice(string.printable) for _ in range(rng.randint(1, 40))
        )
        digest = "".join(ch for ch in raw.lower() if ch.isalnum())
        assert slugify(raw).replace("-", "") == digest

The hardcoded candidate now fails within a handful of iterations, while the real implementation passes all two hundred. Students see the difference between evidence and a memory test, and they see it in code rather than in a slide.

Where the compute comes from

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two hundred generated candidates per cohort is small by industry standards and painful in a classroom, because it forces every student to hold a paid key before they have learned anything worth paying for. MonkeyCode, the project behind this article's operator, describes a free tier and a free server option; the operator currently reports the free allowance at ten million tokens. Those terms live on the project's own page and can change, so this workshop treats them as a starting point to check rather than constants to quote in a syllabus, and it asserts nothing about which models are included, how large the server is, or how long either offer lasts.

The harness itself assumes no vendor at all. The transport is one function, and the shape below is what students fill in from whatever endpoint they are actually using:

import json, os, urllib.request


def ask(prompt):
    request = urllib.request.Request(
        os.environ["BASE_URL"] + "/chat/completions",
        data=json.dumps({
            "model": os.environ["MODEL"],
            "messages": [{"role": "user", "content": prompt}],
        }).encode(),
        headers={
            "Authorization": f'Bearer {os.environ["API_KEY"]}',
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        return json.load(response)["choices"][0]["message"]["content"]

The free server option earns its place in the boundary-sweep segment, where a loop of two hundred candidates would otherwise compete with whatever else is on the student's laptop. It is a convenience for the exercise, not a claim about production throughput.

Yield accounting

The last artifact is a number students compute themselves:

import collections, json

rows = [json.loads(line) for line in open("receipts.jsonl")]
counts = collections.Counter(row["verdict"] for row in rows)
total = sum(counts.values())
print(counts, "yield=", round(counts["accept"] / total, 3))

Yield is acceptance rate per task, and it belongs to the task, not to the model. A slug function with a clear contract collects accepts; a task whose prompt hides its error handling collects quarantines and tells the room nothing about model quality. That distinction is worth more than any leaderboard.

The verdict table

The gate produces four outcomes, and each maps to one action.

Verdict	Signal	Action
Accept	Parses, fixture green, sweep green	Merge with a link to the receipt
Repair	Parses, one named test fails with a local cause	One bounded revision, re-run the gate, two attempts maximum
Quarantine	Syntax error, oversize candidate, timeout, or missing file	Move the candidate to quarantine/ and stop
Escalate	Sweep green but the diff touches files, network, or credentials	Human review of record, no exceptions

What this gate cannot see

The gate checks exactly what the fixture encodes and nothing more. A candidate that passes one afternoon is not guaranteed to pass tomorrow, because sampling variance means the same prompt can produce a different artifact; that is why the candidate bytes get stored next to the receipt and re-run on demand. The byte limit is arbitrary and exists to keep reviews short. The sixty-second timeout catches hangs, not deadlocks that resolve in ninety seconds. Concurrency, license headers, and secret handling are invisible to it.

Teams in regulated pipelines should not treat this gate as a control. It is a teaching device and a first filter, and a named human reviewer of record is still required wherever the rules say so. Teams with no fixture-writing habit should fix that habit before building the gate, or they will automate the grading of nothing.

A cohort that wants to see the sweep finish before deciding whether the gate belongs in its pipeline can start on the free tier and the free server option; the current terms page is the only place worth trusting for what is included today.

MonkeyCode provides free models that can run this workflow.