Pin the Contract, Not the Model: A Bootcamp Lab on Provider-Swappable Agent Runs

A lab that grades an agent run should never care which model produced that run. If your rubric silently changes when the free tier changes, you were grading the vendor, not the student.

That is the whole thesis of this lab, and it fits in one sentence: record once, replay forever, grade offline.

The failure I keep watching students repeat

Every bootcamp cohort hits the same wall around week two. A student builds an agent loop against whatever model was free that morning. It works. They submit. Two days later the grader re-runs the submission and gets a different answer, a different tool call, a timeout, or a rate-limit page.

So what did the rubric actually measure? Not the design. It measured how lucky the re-run was.

And honestly — is that the student's fault? Partly. But mostly it's a lab-design bug. If the grader has to call a live model to decide a grade, the grader is non-deterministic by construction, and no rubric can survive that.

The rule: pin the contract, not the model

Here's what I now require from every submission:

1. Prompts are hashed. Same prompt string, plus or minus whitespace, maps to the same key.
2. Responses are recorded to a plain JSON cassette, committed to the repo.
3. Grading runs on replay only. No network, no API key, no clock dependency.
4. One live run at the end, as a demo, never as the source of truth.

That's it. Four rules, and the whole class stops fighting flaky re-runs.

Setup: three files, fifteen minutes

python -m venv .venv && . .venv/bin/activate
pip install httpx pytest
mkdir -p cassettes lab && touch lab/__init__.py

The contract is a Protocol. Not an abstract base class, not a framework — a protocol, so any object with a complete method qualifies.

# lab/providers.py
import hashlib, json
from pathlib import Path
from typing import Protocol

def prompt_key(prompt: str) -> str:
    # normalize whitespace so "hi" and "hi " are the same recording
    return hashlib.sha256(prompt.strip().encode("utf-8")).hexdigest()[:16]

class Provider(Protocol):
    name: str
    def complete(self, prompt: str) -> str: ...

class CassetteMiss(KeyError):
    """Raised when the replay provider has no recording for a prompt."""

class Replay:
    name = "replay"

    def __init__(self, path: Path):
        self.path = path
        self.data = json.loads(path.read_text()) if path.exists() else {}

    def complete(self, prompt: str) -> str:
        k = prompt_key(prompt)
        if k not in self.data:
            raise CassetteMiss(f"no recording for {k}: re-record, do not blind-retry")
        return self.data[k]

The recorder wraps any live provider and appends what it sees. Note the ordering: it checks the cassette before calling out, so re-running a half-finished session doesn't burn calls.

class Recorder:
    name = "recorder"

    def __init__(self, inner: Provider, path: Path):
        self.inner, self.path = inner, path
        self.data = json.loads(path.read_text()) if path.exists() else {}

    def complete(self, prompt: str) -> str:
        k = prompt_key(prompt)
        if k in self.data:
            return self.data[k]
        out = self.inner.complete(prompt)
        self.data[k] = out
        self.path.write_text(json.dumps(self.data, indent=2, sort_keys=True))
        return out

The live adapter below is a sketch, not verified code — endpoints, auth headers, and response shapes differ per provider, so copy field names from that provider's own docs.

# lab/live.py — sketch, adapt before use
import httpx

class Live:
    name = "live"

    def __init__(self, base_url: str, api_key: str, model: str):
        self.base_url, self.api_key, self.model = base_url, api_key, model

    def complete(self, prompt: str) -> str:
        r = httpx.post(
            f"{self.base_url}/chat",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"model": self.model, "prompt": prompt},
            timeout=60,
        )
        r.raise_for_status()
        return r.json()["text"]

Checkpoints: three, each pass/fail in under a minute

Checkpoint 1 — whitespace is not a new prompt. Two calls, different padding, one recording.

def test_whitespace_normalized(tmp_path):
    c = tmp_path / "c.json"
    c.write_text(json.dumps({prompt_key("summarize this"): "ok"}))
    assert Replay(c).complete("summarize this") == "ok"
    assert Replay(c).complete("  summarize this\n") == "ok"

Checkpoint 2 — a miss is loud. No empty strings, no None, no swallowed exception. The CLI must exit non-zero.

def test_miss_raises(tmp_path):
    p = Replay(tmp_path / "empty.json")
    with pytest.raises(CassetteMiss):
        p.complete("never recorded")

Checkpoint 3 — two providers, one grade. Feed the same cassette through the grader twice; the output hash must match. This is the checkpoint that proves the rubric is model-independent, and it's the one most students fail on the first attempt because they hid a model_name inside their scoring logic.

Stretch goals, for the students who finish early

• Drift detector: hash the prompt template file itself; if the template changes, warn that cassettes are stale.
• Cassette hygiene: flag recordings older than the last template change, and refuse to grade on them.
• Structured output guard: if the agent is supposed to return JSON, validate the replayed string against a schema and fail the checkpoint on malformed output — a replay should still be parsed, not trusted.

Pick one. Document the failure mode you found. Three half-finished stretch goals score worse than one finished one, and I say that in the syllabus.

The rubric I actually use

Criterion	Weight	Full marks looks like	Typical loss
Contract stability	30	identical grade across two providers	only passes on the model they happened to use
Loud failure	25	CassetteMiss propagates, exit code ≠ 0	returns "" and looks successful
Trace	20	log line per call: key, provider, elapsed ms	only the final answer is logged
Write-up	15	one honest paragraph on what broke	“it worked”
Stretch	10	one goal, documented, with the failure it surfaced	three started, none finished

Notice what's absent: no points for model choice, no points for tokens spent, no points for speed. Those are environment variables, not skills.

Running the lab when the budget is zero

A room full of students on personal laptops needs two things a lab budget usually can't buy: model calls, and somewhere to run the grader that isn't a sleeping laptop on hostel Wi-Fi.

This is where I point students at MonkeyCode's open-source project, which offers free model access and a free server option — enough to record a cassette on day one and replay it for the rest of the term. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat I'd give any cohort: free allowances move. Check the current allowance in the console on the day you record, and don't copy a number from a blog post — including this one. The recorded cassette is the durable artifact; the quota is weather.

When this lab is the wrong tool

Replay hides variance, and sometimes variance is the subject.

• Don't use it for tool-calling semantics. Providers disagree on how function arguments are serialized; a replayed call can hide a real incompatibility.
• Don't use it for latency or quality benchmarks. You'd be benchmarking a JSON file.
• Don't use it as a production eval harness. Production needs stochastic sampling; this is a grading device, not an evaluation platform.

Boundaries are part of the deliverable. If a student can't say what their lab can't measure, they haven't finished the write-up.

What the students actually learn

The lesson isn't about providers. It's that anything you can't re-run identically, you can't grade fairly — and that includes your own assumptions about the model. Record once, replay forever, keep the rubric boring.

If you run this lab with a cohort, start a FAILURES.md in the repo on day one. The second entry is always more interesting than the first.