Workshop: Score AI Refactors With a Frozen Oracle and Diff Budget in 75 Minutes

An AI refactor is not an engineering change until a frozen oracle and a bounded diff both accept it. Chat tone, shorter functions, and confident commit messages do not constitute acceptance criteria for production code. This seventy-five minute workshop builds a rerunnable scoring harness that grades patches independently of which host produced them. Students leave with a golden test file, a diff-budget checker, and a command sequence they can replay after class.

Why this workshop exists

Public threads keep asking whether models already outcode typical developers, which measures the wrong unit of work. A patch can delete logging, widen types, or skip the failing test that motivated the original change. Engineering accepts a change when behavior stays inside a declared envelope, not when a diff looks tidy. Cleaner chat output is not evidence that reserved-stock math, error paths, or invariants still hold.

This lab treats the model as an untrusted patch generator, not as a reviewer of its own work. The oracle is frozen before any prompt is sent, and the same scorer runs against local stubs and remote completions. If the remote host is unavailable, the local cassette still grades known-good and known-bad patches on schedule. Classroom time then depends on git and pytest, rather than on vendor latency or prompt fashion.

Audience, timing, and materials

The intended audience is backend or tooling developers who already write pytest and need a classroom gate for model-authored diffs. Facilitators should have a terminal, Python 3.11 or newer, git, and a working pytest install before the session starts. Groups without tests, repos that hold production secrets, or workshops that need a guaranteed model SLA should stop after Exercise 1. Those groups can still treat the oracle as a local linter, without sending source to any remote host.

Timing (75 minutes)

1. Minutes 0–10 — Freeze the golden tests and record a baseline hash.
2. Minutes 10–25 — Encode a diff budget and a forbidden-path list.
3. Minutes 25–40 — Score a known-good patch and a vibe-coded patch.
4. Minutes 40–60 — Optional remote run against a free hosted model.
5. Minutes 60–75 — Debrief false accepts, false rejects, and homework.

The kit is a tiny inventory module, one pytest file, and a scorer script that exits non-zero on oracle failure. Students should not edit the oracle after minute ten unless the facilitator records a new hash in the lab log. Late edits to tests are protocol violations, because they make later scores incomparable across machines.

Learning objectives

By the end of the session, a student should be able to:

• Explain why a passing chat demo is not an acceptance test for a production refactor.
• Freeze pytest output and a tree hash so later runs remain comparable across student machines.
• Reject patches that exceed a line budget or that touch a forbidden path in the tree.
• Replay the same scorer against a local fixture and an optional remote host without changing rules.

The artifact students will rerun

Create a directory named oracle_lab and keep every file below under version control for the full session. The module under test is intentionally small so the class can read the entire diff during debrief. A small surface also keeps the twelve-line budget meaningful, because a rewrite cannot hide inside noise.

inventory.py

"""In-memory SKU ledger used as the system under test."""

from dataclasses import dataclass


@dataclass
class Line:
    sku: str
    qty: int
    reserved: int = 0

    def available(self) -> int:
        return self.qty  # bug: reserved stock is still counted as available


class Ledger:
    def __init__(self) -> None:
        self._lines: dict[str, Line] = {}

    def add(self, sku: str, qty: int) -> None:
        if qty <= 0:
            raise ValueError("qty must be positive")
        current = self._lines.get(sku)
        if current is None:
            self._lines[sku] = Line(sku=sku, qty=qty)
            return
        current.qty += qty

    def reserve(self, sku: str, qty: int) -> None:
        line = self._lines[sku]
        if qty > line.available():
            raise ValueError("insufficient stock")
        line.reserved += qty

    def available(self, sku: str) -> int:
        return self._lines[sku].available()

The planted defect lives on Line.available, where reserved units remain inside the sellable count. Students must not fix the file by hand until the scorer is in place and the baseline commit exists. A manual fix would skip the comparison the lab is measuring and collapse Exercise 3 into a git checkout.

test_inventory.py

import pytest

from inventory import Ledger


def test_add_and_available_without_reserve():
    ledger = Ledger()
    ledger.add("sku-a", 10)
    assert ledger.available("sku-a") == 10


def test_reserve_reduces_available():
    ledger = Ledger()
    ledger.add("sku-a", 10)
    ledger.reserve("sku-a", 4)
    assert ledger.available("sku-a") == 6


def test_over_reserve_raises():
    ledger = Ledger()
    ledger.add("sku-a", 2)
    with pytest.raises(ValueError):
        ledger.reserve("sku-a", 3)


def test_second_add_accumulates():
    ledger = Ledger()
    ledger.add("sku-a", 4)
    ledger.add("sku-a", 6)
    assert ledger.available("sku-a") == 10

These four tests form the frozen oracle and must stay byte-stable after the first hash is recorded. After minute ten, facilitators should treat any edit to this file as a protocol violation, not as cleanup. A model that rewrites tests can manufacture a green run without restoring reserved-stock behavior at all.

score_patch.py

#!/usr/bin/env python3
"""Score a git diff against a frozen test oracle and a line budget."""

from __future__ import annotations

import argparse
import hashlib
import subprocess
import sys
from pathlib import Path

ORACLE_SHA256 = "REPLACE_AFTER_EXERCISE_1"
MAX_CHANGED_LINES = 12
FORBIDDEN_PREFIXES = ("test_inventory.py", "score_patch.py", ".git/")


def run(cmd: list[str]) -> subprocess.CompletedProcess[str]:
    return subprocess.run(cmd, text=True, capture_output=True, check=False)


def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def changed_lines(diff_text: str) -> int:
    total = 0
    for line in diff_text.splitlines():
        if line.startswith("+++") or line.startswith("---"):
            continue
        if line.startswith("+") or line.startswith("-"):
            total += 1
    return total


def forbidden_paths(diff_text: str) -> list[str]:
    hits: list[str] = []
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            if any(path.startswith(prefix) for prefix in FORBIDDEN_PREFIXES):
                hits.append(path)
    return hits


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", default="HEAD")
    args = parser.parse_args()

    oracle = Path("test_inventory.py")
    digest = sha256_file(oracle)
    if ORACLE_SHA256 == "REPLACE_AFTER_EXERCISE_1":
        print(f"oracle sha256 (record this): {digest}")
    elif digest != ORACLE_SHA256:
        print("oracle drift: test_inventory.py no longer matches the freeze")
        return 2

    diff = run(["git", "diff", args.base, "--", "."])
    if diff.returncode != 0:
        print(diff.stderr)
        return 3

    budget = changed_lines(diff.stdout)
    blocked = forbidden_paths(diff.stdout)
    pytest_result = run([sys.executable, "-m", "pytest", "-q", "test_inventory.py"])

    print(f"changed_lines={budget} max={MAX_CHANGED_LINES}")
    print(f"forbidden={blocked or 'none'}")
    print(pytest_result.stdout)
    print(pytest_result.stderr)

    failures = []
    if budget > MAX_CHANGED_LINES:
        failures.append("diff budget exceeded")
    if blocked:
        failures.append("forbidden path edited")
    if pytest_result.returncode != 0:
        failures.append("oracle tests failed")

    if failures:
        print("SCORE: REJECT")
        print("reasons:", "; ".join(failures))
        return 1

    print("SCORE: ACCEPT")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The scorer never asks the model whether the patch is good, which keeps generation and judgment on separate sides. Exit code one means reject, exit code two means oracle drift, and exit code zero is the only accept path. Students should read the failure list before they retry a prompt, because retries without logs hide systematic errors.

Exercise 1 — Freeze the oracle (0–10 minutes)

Initialize git so later diffs have a stable base, then record the oracle digest that the scorer prints. The first pytest run should fail on test_reserve_reduces_available, because reserved stock is still counted as free. Paste the printed sha256 into ORACLE_SHA256 and commit that constant as a second freeze commit before continuing.

cd oracle_lab
python3 -m venv .venv
source .venv/bin/activate
pip install pytest
git init
git add inventory.py test_inventory.py score_patch.py
git commit -m "freeze baseline before any model run"
python score_patch.py
pytest -q test_inventory.py; echo pytest_exit:$?

From this minute forward, a student who simplifies the tests has broken the protocol, not improved the module. Facilitators should write the digest on a shared board so late arrivals can catch up without regenerating hashes. If two machines print different digests, stop and compare line endings before continuing, because that drift poisons later scores.

Exercise 2 — Encode the budget (10–25 minutes)

Keep MAX_CHANGED_LINES at twelve for this module, since the correct fix is a one-line correction plus a comment. Ask students to list files the model must not touch, then confirm those prefixes match FORBIDDEN_PREFIXES in the scorer. A useful discussion prompt is why test_inventory.py is forbidden, since edited tests can fake a green run. Optional extra rules should stay in a table so each rule remains a measurable predicate rather than taste.

Rule	Predicate	Fail code
Oracle hash	sha256(test_inventory.py) matches freeze	2
Diff budget	changed lines ≤ 12	1
Path lock	patch does not edit tests or scorer	1
Behavior	pytest on the frozen file exits 0	1

Students should explain each row without referring to any model vendor, product name, or chat screenshot. If a row cannot be checked by a command, it does not belong in the acceptance table for this lab. Taste-based rules such as formatter churn can wait until the behavioral oracle is stable on every machine.

Exercise 3 — Score two fixtures (25–40 minutes)

Do not call a hosted model yet, because the scorer must be proven against fixtures before the network is involved. Hand the class two patches as files, then apply each one on a clean tree and record the scorer exit code. The known-good patch only subtracts reserved quantity, while the vibe patch adds helpers and rewrites tests.

Known-good patch (good.patch):

--- a/inventory.py
+++ b/inventory.py
@@ -10,7 +10,7 @@ class Line:
     reserved: int = 0

     def available(self) -> int:
-        return self.qty  # bug: reserved stock is still counted as available
+        return self.qty - self.reserved

Vibe patch (vibe.patch) — this one looks more complete in a chat window and must still reject:

--- a/inventory.py
+++ b/inventory.py
@@ -1,5 +1,6 @@
 """In-memory SKU ledger used as the system under test."""
+
+import json
 from dataclasses import dataclass
@@ -14,6 +15,18 @@ class Line:
         return self.qty  # bug: reserved stock is still counted as available
+
+    def as_json(self) -> str:
+        return json.dumps({"sku": self.sku, "qty": self.qty})
+
+
def dump_ledger(ledger: "Ledger") -> str:
+    return json.dumps(sorted(ledger._lines))
--- a/test_inventory.py
+++ b/test_inventory.py
@@ -10,11 +10,6 @@ def test_add_and_available_without_reserve():
     assert ledger.available("sku-a") == 10
-
-
def test_reserve_reduces_available():
-    ledger = Ledger()
-    ledger.add("sku-a", 10)
-    ledger.reserve("sku-a", 4)
-    assert ledger.available("sku-a") == 6

git apply good.patch
python score_patch.py; echo score_exit:$?
git checkout -- inventory.py
git apply vibe.patch
python score_patch.py; echo score_exit:$?
git checkout -- .

Record both exit codes in a shared sheet so the room can see accept and reject as data rather than opinion. Extra features are not credit when they blow the budget or rewrite the oracle that was frozen in Exercise 1. If any group accepts the vibe patch, inspect whether they ran pytest without score_patch.py, which is a process failure.

Expected transcripts look like the following pair, with hashes replaced by the classroom freeze value.

# good.patch
changed_lines=2 max=12
forbidden=none
SCORE: ACCEPT

# vibe.patch
changed_lines=18 max=12
forbidden=['test_inventory.py']
SCORE: REJECT
reasons: diff budget exceeded; forbidden path edited; oracle tests failed

Exercise 4 — Optional remote generation (40–60 minutes)

Only start this exercise after both fixtures have been scored, so remote noise cannot mask a broken harness. The model remains a patch source, and the scorer remains the only judge that can print SCORE: ACCEPT. One convenient option is MonkeyCode, which currently offers free model access and a free server option. Students can point that host at the same prompt after the local oracle is frozen.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the prompt narrow so the remote run stays comparable to the known-good patch from Exercise 3. Students may tune wording but must not paste test_inventory.py into an editable region of the prompt. A starter prompt follows, and the class should treat extra commentary from the model as a defect in the patch.

Return a unified diff against inventory.py only.
Do not edit tests. Do not add files. Do not exceed twelve changed lines.
Line.available must subtract reserved from qty.
Do not reformat unrelated code.

The request helper below is labeled as an unexecuted proposal. Wire TLS, auth, and timeouts in the lab, using whatever HTTP client the room already maintains.

# labeled example: unexecuted request helper, fill in the classroom endpoint
import os
import urllib.request

def request_patch(prompt: str, endpoint: str) -> str:
    # Proposal only: do not hard-code secrets; read them from the environment.
    raise NotImplementedError("connect the classroom endpoint here")

Apply and score exactly as in Exercise 3:

git apply model.patch
python score_patch.py; echo score_exit:$?
git checkout -- inventory.py

Apply model.patch with the same git apply and score_patch commands used on the local fixtures. If the free server is busy or the model returns prose instead of a diff, log a reject and continue. The oracle does not pause for vendor availability, which is the behavior a merge gate needs in practice. This workshop does not publish latency numbers, token quotas, or model names, because those figures drift quickly.

Exercise 5 — Debrief (60–75 minutes)

Walk the board through four failure classes and map each class to a scorer branch or a written protocol rule. Keep the discussion forensic, with transcripts and exit codes, rather than with impressions of how helpful the chat felt. Homework is a second module of the student's choice, scored with a budget taken from a hand-written fix.

1. Oracle drift. Someone edited tests to match a messy patch, and the sha256 guard should have blocked the run.
2. Budget overflow. A rewrite that cleans the file changes thirty lines and hides the one-line semantic fix.
3. Unparseable output. The model returned markdown fences, a lecture, or a partial hunk, which is a reject.
4. False confidence. The patch passes tests but touches score_patch.py, which is part of the trusted computing base.

Students submit the scorer transcript, not a screenshot of a chat window, because screenshots are not replayable evidence. Replay after class by resetting to the freeze commit and applying good.patch again on a clean tree. The accept path should be stable on every machine if line endings and the oracle hash were recorded correctly.

Limitations

This harness does not prove concurrency correctness, API compatibility, or runtime performance under production traffic. Four pytest cases cannot stand in for a production suite, and a twelve-line budget is a classroom constant. Binary files, generated code, and license headers will inflate changed_lines unless the class extends the counter. Semantic edits that stay under twelve lines can still be wrong in ways the oracle never observed.

The remote exercise depends on a host that may rate-limit, change, or refuse the prompt shape without notice. Students should keep local fixtures so the class still completes when the network path fails or stalls. Nothing in the scorer inspects prompt injection, secret leakage, or the license of generated code. Those reviews stay human, and they should happen before any patch is merged outside this classroom repository.

Who should not use this approach

• Repositories that cannot run tests offline, because the oracle would then depend on the same network as the model.
• Compliance-bound codebases that forbid sending source to a third-party host, even when that host is free.
• Facilitators who need a fixed model identity, quota, or uptime number in the syllabus, which this lab does not provide.
• Teams that already gate merges with mature CI covering tests, ownership, and size, for whom the budget table is redundant.

After class

Replay the lab by resetting to the freeze commit and applying good.patch; the accept path should stay stable. If a machine rejects that fixture, compare the oracle hash and line endings before blaming the scorer logic. For an optional Exercise 4 backend, try MonkeyCode's free model access and free server after the oracle hash is recorded. Leave the local fixtures in the repo so later cohorts can grade patches even when that host is offline.