Workshop: Pin Agent Labs to Replay Fixtures in 85 Minutes

A classroom agent lab usually fails from live-model variance, not from the student's control flow. Pin request and response fixtures before anyone calls a shared endpoint, then treat live inference as a rare capture step. This workshop walks through an 85-minute lab that records one canonical trace and grades every later run against it. Students still write real tool-calling code; they simply stop depending on a moving model for the pass/fail signal.

What you will ship

By the end of the session, each pair should have three artifacts in a small git repo. The first artifact is a thin client wrapper that can record or replay chat completions from disk. The second artifact is a gold fixture captured from a single live call with secrets already redacted. The third artifact is a pytest file that fails closed when the prompt hash, tool name, or JSON shape drifts.

You will not tune prompts for cleverness in this block, and you will not compare model vendors. You also will not quote latency percentiles or claim a quota that your classroom does not control. Keep the scope on reproducibility, because extra vendor talk steals time from the two graded exercises. Write the wrapper first so later HTTP details cannot leak into the student-facing assertions.

Who should skip this workshop

Skip this outline when the course is about sampling, temperature, or creative variation across completions. Skip it when students must observe live refusal behavior, rate-limit headers, or provider-specific safety filters. Skip it when you cannot redact API keys, emails, and file paths before fixtures land in git.

Do not use this cassette pattern as a production billing fabric. Production multi-tenant routers need identity, quotas, and audit trails that a teaching file cannot provide. A research seminar that studies sampling distributions also needs live draws, not a frozen assistant turn.

Timing (85 minutes)

Block	Minutes	Outcome
0. Failure demo	10	Class sees two live runs disagree
1. Wrapper contract	15	Record/replay API sketched
2. Worked capture	20	One gold fixture on disk
3. Exercise A	15	Hash mismatch fails the test
4. Exercise B	15	Tool-name drift fails closed
5. Debrief	10	Decision table and limits

Keep a visible timer on the projector so pairs do not overfit the capture block. If the live capture hangs, switch the room to a pre-baked fixture rather than burning remaining exercises. Publish the backup cassette in the repo before class so the fallback does not become a side channel.

The failure mode, stated as variables

Live teaching endpoints introduce four independent variables into a lab that staff still treat as deterministic. Completion text can change across retries even with temperature set to zero on some stacks. Tool-argument JSON can add or drop optional keys when the schema exists only as prose. Shared classrooms contend for one process, so a noisy student loop can starve everyone else's wall clock.

Logs then mix prompt text, secrets, and partial tool results until staff cannot reconstruct one failing request. A replay cassette collapses those variables into one content-addressed file that every laptop can open. Student code still builds messages and selects tools against that frozen turn sequence, not against the wall clock. The grade then measures control flow, JSON shape, and tool choice instead of provider luck.

Wrapper contract

Define one function the rest of the lab may call, and hide HTTP details behind it. Everything else, including retries and provider SDKs, stays behind that function so tests can intercept I/O. Unknown modes must raise, because a silent fallthrough to the network breaks the grading contract.

# cassette.py — teaching sketch, not a production SDK
from __future__ import annotations

import hashlib
import json
import os
from pathlib import Path
from typing import Any, Callable, Mapping

CASSETTE_DIR = Path(os.environ.get("CASSETTE_DIR", "cassettes"))
MODE = os.environ.get("LLM_MODE", "replay")  # replay | record | live


def canonical_request(payload: Mapping[str, Any]) -> str:
    body = {
        "model": payload.get("model"),
        "messages": payload.get("messages"),
        "tools": payload.get("tools") or [],
        "tool_choice": payload.get("tool_choice"),
        "temperature": payload.get("temperature", 0),
    }
    return json.dumps(body, sort_keys=True, separators=(",", ":"))


def request_id(payload: Mapping[str, Any]) -> str:
    digest = hashlib.sha256(canonical_request(payload).encode("utf-8")).hexdigest()
    return digest[:16]


def _path_for(payload: Mapping[str, Any]) -> Path:
    CASSETTE_DIR.mkdir(parents=True, exist_ok=True)
    return CASSETTE_DIR / f"{request_id(payload)}.json"


def redact(obj: Any) -> Any:
    if isinstance(obj, Mapping):
        hidden = {"authorization", "api_key", "token", "email", "file_path"}
        return {
            k: ("***" if k.lower() in hidden else redact(v))
            for k, v in obj.items()
        }
    if isinstance(obj, list):
        return [redact(v) for v in obj]
    return obj


def complete(
    payload: Mapping[str, Any],
    live_call: Callable[[Mapping[str, Any]], Mapping[str, Any]],
) -> Mapping[str, Any]:
    path = _path_for(payload)
    if MODE == "replay":
        if not path.exists():
            raise FileNotFoundError(
                f"missing cassette {path.name}; recapture with LLM_MODE=record"
            )
        return json.loads(path.read_text(encoding="utf-8"))
    if MODE not in {"record", "live"}:
        raise ValueError(f"unsupported LLM_MODE={MODE!r}")
    response = redact(live_call(payload))
    if MODE == "record":
        path.write_text(
            json.dumps(response, indent=2, sort_keys=True),
            encoding="utf-8",
        )
    return response

Label the wrapper as a teaching sketch rather than a production SDK with retries and metrics. It deliberately refuses unknown modes instead of falling through to a live network call. Students who add extra kwargs to the payload will change the hash and must recapture on purpose.

Field exclusions in canonical_request

Authorization headers, wall-clock timestamps, and request ids are excluded because they change without a semantic prompt change. Including them would force a recapture on every retry and would store credentials beside the gold output. Temperature stays inside the hash so a student cannot silently move from zero to a sampling run. If you later add top_p or max_tokens, fold those fields into the canonical body on the same day you recapture.

Why the request hash is short on purpose

The sixteen hex characters are a teaching aid, not a cryptographic recommendation for production cassettes. A short prefix makes missing-file errors readable on a projector, which matters more than collision resistance in a 30-file lab. If two prompts collide, lengthen the prefix in cassette.py and recapture once; do not add timestamps to the key. Timestamps would make every retry a new identity, which recreates the original flake the workshop is trying to remove.

Worked example students can rerun

The lab agent exposes one tool named lookup_invoice and must return a closed JSON object. Students implement message construction only, while the cassette supplies both model turns in order. The first recorded turn must request the tool, and the second turn must format the invoice summary.

# agent_lab.py — teaching sketch
from __future__ import annotations

import json
from typing import Any, Mapping

from cassette import complete

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "lookup_invoice",
            "parameters": {
                "type": "object",
                "properties": {"invoice_id": {"type": "string"}},
                "required": ["invoice_id"],
            },
        },
    }
]

INVOICES = {"INV-1042": {"status": "open", "cents": 12900}}


def lookup_invoice(invoice_id: str) -> Mapping[str, Any]:
    row = INVOICES.get(invoice_id)
    if row is None:
        return {"error": "not_found", "invoice_id": invoice_id}
    return {"invoice_id": invoice_id, **row}


def live_call(payload: Mapping[str, Any]) -> Mapping[str, Any]:
    raise RuntimeError("live_call is only used when LLM_MODE=record")


def run_agent(invoice_id: str) -> Mapping[str, Any]:
    messages: list[dict[str, Any]] = [
        {
            "role": "system",
            "content": "Return JSON with keys invoice_id, status, cents. Use lookup_invoice.",
        },
        {"role": "user", "content": f"Summarize invoice {invoice_id}"},
    ]
    first = complete(
        {
            "model": "classroom-free",
            "temperature": 0,
            "tools": TOOLS,
            "messages": messages,
        },
        live_call,
    )
    tool_calls = first["choices"][0]["message"].get("tool_calls") or []
    if not tool_calls:
        raise ValueError("model skipped the required tool")
    name = tool_calls[0]["function"]["name"]
    if name != "lookup_invoice":
        raise ValueError(f"unexpected tool {name}")
    args = json.loads(tool_calls[0]["function"]["arguments"])
    result = lookup_invoice(args["invoice_id"])
    messages.append(first["choices"][0]["message"])
    messages.append(
        {
            "role": "tool",
            "tool_call_id": tool_calls[0]["id"],
            "content": json.dumps(result, sort_keys=True),
        }
    )
    second = complete(
        {
            "model": "classroom-free",
            "temperature": 0,
            "tools": TOOLS,
            "messages": messages,
        },
        live_call,
    )
    return json.loads(second["choices"][0]["message"]["content"])

Provide a checked-in cassette so the replay path can run without any classroom network. Keep the filename equal to the truncated request hash, or print that hash from a tiny setup script. Students then rename a captured file once instead of editing hashes by hand during the lab.

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "tool_calls": [
          {
            "id": "call_1",
            "type": "function",
            "function": {
              "name": "lookup_invoice",
              "arguments": "{\"invoice_id\":\"INV-1042\"}"
            }
          }
        ]
      }
    }
  ]
}

# test_agent_lab.py
import os

import pytest

os.environ["LLM_MODE"] = "replay"

from agent_lab import run_agent


def test_invoice_summary_is_closed_json():
    payload = run_agent("INV-1042")
    assert payload == {
        "invoice_id": "INV-1042",
        "status": "open",
        "cents": 12900,
    }


def test_missing_cassette_fails_closed():
    with pytest.raises(FileNotFoundError):
        run_agent("INV-9999")

Commands the room can paste without guessing flags:

export LLM_MODE=replay
export CASSETTE_DIR=cassettes
python -m pytest test_agent_lab.py -q

Instructor capture pass, after a redaction review, not during student time:

export LLM_MODE=record
python -c "from agent_lab import run_agent; print(run_agent('INV-1042'))"
git add cassettes

Treat the model string classroom-free as a fixture label rather than a vendor product name. Swap live_call for whatever HTTP client your room already uses during the instructor capture. Do not bake a provider SDK into the exercises, because SDK defaults will leak into the canonical hash.

Exercise A — break the prompt hash (15 minutes)

Change one word in the system prompt and rerun pytest on the same cassette directory. Students should see FileNotFoundError because the canonical request changed and the truncated hash missed the gold file. Restore the prompt, confirm a green run, then record in the test that hash drift is a failed lab. Optional extension: print request_id inside the assertion so staff can match a missing file without a debugger.

Exercise B — fail closed on tool-name drift (15 minutes)

Edit the gold cassette so the tool name becomes lookupInvoice, then rerun the agent test. Require a ValueError that names the unexpected tool rather than a JSON decode error from later steps. Restore the cassette from git so later pairs do not inherit a poisoned file for Exercise A. Tool-name drift is how incomplete plans leak into side effects once a lab later grows write tools.

Instructor checklist before the room arrives

Run this checklist on the instructor laptop the night before class, not during the first ten minutes.

1. Redact secrets in every pre-baked cassette, including nested tool payloads.
2. Confirm pytest is green with LLM_MODE=replay from a clean clone.
3. Document LLM_MODE=record as instructor-only in the lab README.
4. Keep a backup cassette on the default branch for capture failure.
5. Leave live_call raising on the student branch so homework cannot silently hit the network.

Confirm that pytest stays green under replay mode before any student clone lands on a laptop. Keep the record mode export out of .env.example so a copy-paste does not recapture during class. Put the backup cassette on the default branch so a hung capture cannot strand the later exercises.

Decision table

Choose a mode with the table below before anyone exports LLM_MODE on a shared laptop.

Situation	Mode	Why
First green path of a new lab	record, once	Need a canonical trace
Student homework and CI	replay	Deterministic grade
Investigating a new provider error	live, off git	Do not commit raw errors
Prompt text changed on purpose	record again	Old hashes are now lies
Fixture contains emails or paths	do not commit	Redact or drop the file

If two modes seem tempting at once, choose replay and schedule a separate recapture later. Mixed mode in a classroom produces cassettes that only one laptop can replay during office hours. Recapture is a contract change, and it should land as its own commit with a one-line lab note.

Where a free classroom endpoint still helps

You still need one honest capture before a cassette can exist in the student repository. That capture should happen on a throwaway server with a shared model endpoint and instructor-only credentials. Set LLM_MODE=record on the instructor machine only, then turn the network off for the remaining blocks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option for that single instructor capture without a billed cluster. After the JSON files exist, the rest of the 85 minutes should not touch the network at all. If you run the capture pass there, commit the redacted cassettes and keep the endpoint out of student scripts.

Limitations

The wrapper hashes prompts and tools, not semantic equivalence, so a harmless wording change looks like a missing fixture. Temperature exploration is impossible in replay mode by design, which is correct for grading and wrong for sampling research. Cassettes can encode a bad tool call forever if the instructor records a hallucinated argument and never inspects the file. Large binary tool results will bloat git, so store hashes or truncated JSON instead of raw blobs.

This pattern also does not prove the live model still behaves like the fixture on a later date. Schedule a separate, infrequent recapture, and treat diffs as contract changes rather than as student bugs. If recapture day shows new optional keys, update assertions explicitly instead of widening JSON parsers silently.

What not to automate next

Do not auto-record from student machines, because those runs will mix secrets, half prompts, and one-off tools. Do not replay fixtures that still contain authorization headers, even when the class repo is private. Do not hide live_call failures behind retries inside the 15-minute exercises, because retries destroy the teaching signal. Do not grade latency, token counts, or prose style from a cassette that was captured on a different week.

After those limits are on the projector, the core conclusion does not change for the lab. Pin the model I/O first, then let students debug their own control flow against a file they can open. A shared free endpoint is useful for one capture, and harmful as the source of truth for grades. Ship the cassette, ship the hash function, and only then invite the model back into the room.