DEV Community

Finley Zhu
Finley Zhu

Posted on

Workshop: Pin a Dual-Backend Tool Receipt in 80 Minutes

#testing #python #ai #agents

A live agent demo is not a lab until both backends emit the same canonical tool receipt. Students often treat a successful chat reply as evidence, even when the tool JSON cannot be replayed. This workshop isolates that failure with a dual-run harness, a fixture server, and a sorted-key canonicalizer. The core result is a failing or passing receipt file, not a screenshot of a fluent answer.

Recent discussion around vibe coding versus engineering is useful here only as a measurement problem. A fluent model turn can hide argument drift, extra keys, and type coercion that later labs cannot rerun. The fix is not a longer prompt. The fix is a receipt that two backends must both satisfy before the exercise is marked complete.

What this lab measures

The unit of evidence is a tool call, not assistant prose. A student prompt may produce a correct English summary while the tool payload silently changes shape. That split is why classroom reruns collapse after a backend swap. This outline treats that collapse as the bug under study.

You will leave with four artifacts:

  • A local fixture server that returns a frozen tool payload.
  • A canonicalizer that sorts keys and strips non-contract fields.
  • A dual-run script that hits fixture and live backends with one prompt hash.
  • A pass or fail receipt that classmates can rerun without the original chat UI.

Timing box

Total scheduled time is eighty minutes, including a short debrief. Keep a visible timer so discussion does not eat the dual-run.

  1. Minutes 0–10 — Frame the failure. Show two JSON payloads that look equal in a chat log and fail after canonicalization.
  2. Minutes 10–25 — Exercise A: capture. Record one tool name plus raw arguments from a single agent turn.
  3. Minutes 25–45 — Exercise B: canonicalize. Implement sorted-key JSON and a deny list of extra fields.
  4. Minutes 45–70 — Exercise C: dual-run. Compare fixture output against one live backend using the same prompt hash.
  5. Minutes 70–80 — Decision table and debrief. Classify misses as shape drift, value drift, or transport noise.

Lab layout students can rerun

Keep the repository to three files so the receipt stays reviewable in a pull request. Extra notebooks hide the contract. Label the snippets below as a teaching harness, not as production agent infrastructure.

lab_tool_receipt/
  fixture_server.py
  canonicalize.py
  dual_run.py
  receipts/
    prompt_a.fixture.json
    prompt_a.live.json
Enter fullscreen mode Exit fullscreen mode

Exercise A — Capture a raw tool turn

Start from a single intended tool, not a multi-step agent loop. Multi-step traces mix planning noise with contract noise. The worked example uses create_ticket with three fields that classrooms actually grade: title, priority, and project_id.

# Label: teaching example, not a live production client.
RAW_TURN = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "high",
        "project_id": "ops-214",
        "confidence": 0.81,          # not in the contract
        "Priority": "high",          # duplicate key after lowercasing
    },
}
Enter fullscreen mode Exit fullscreen mode

Write the raw turn to disk before you clean it. Students who canonicalize in their head cannot later prove which field the model invented. A receipt that starts after cleanup is not replayable.

Exercise B — Canonicalize before you compare

Canonical form is the only equality that survives pretty-printers and key reordering. The function below lowercases object keys, drops unknown fields, and encodes with sorted keys. That trio catches the three classroom failures that look like “the model changed its mind.”

# canonicalize.py — teaching harness
import json
from typing import Any

CONTRACT_FIELDS = {
    "create_ticket": ("title", "priority", "project_id"),
}

ALLOWED_PRIORITY = {"low", "medium", "high"}


def canonicalize_args(tool: str, args: dict[str, Any]) -> dict[str, Any]:
    lowered = {str(k).lower(): v for k, v in args.items()}
    allowed = CONTRACT_FIELDS[tool]
    clipped = {k: lowered[k] for k in allowed if k in lowered}
    missing = [k for k in allowed if k not in clipped]
    if missing:
        raise ValueError(f"missing contract fields: {missing}")
    if clipped["priority"] not in ALLOWED_PRIORITY:
        raise ValueError("priority outside contract enum")
    if not isinstance(clipped["title"], str) or not clipped["title"].strip():
        raise ValueError("title must be a non-empty string")
    if not isinstance(clipped["project_id"], str):
        raise ValueError("project_id must be a string")
    return {k: clipped[k] for k in allowed}


def receipt_bytes(tool: str, args: dict[str, Any], backend: str, prompt_sha: str) -> bytes:
    body = {
        "prompt_sha256": prompt_sha,
        "tool": tool,
        "backend": backend,
        "args": canonicalize_args(tool, args),
    }
    return json.dumps(body, sort_keys=True, separators=(",", ":")).encode("utf-8")
Enter fullscreen mode Exit fullscreen mode

Run a local self-check before the dual-run so canonicalizer bugs are not blamed on the model.

python - <<'PY'
from canonicalize import receipt_bytes, RAW_TURN
raw = {
  "title": "Pager duty runbook is stale",
  "priority": "high",
  "project_id": "ops-214",
  "confidence": 0.81,
}
print(receipt_bytes("create_ticket", raw, "fixture", "abc123").decode())
PY
Enter fullscreen mode Exit fullscreen mode

Expected canonical object, ignoring hash, is exactly three keys in sorted order. Extra model fields must not appear. If confidence survives, the deny list is incomplete and later diffs will be noisy.

Exercise C — Dual-run fixture versus live

The fixture server is the frozen half of the experiment. It should not call a model. It exists so students can fail the harness on purpose and still rerun the passing side after class.

# fixture_server.py — stdlib only, teaching example
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

FIXTURE = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "high",
        "project_id": "ops-214",
    },
}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(length)
        payload = json.dumps(FIXTURE).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Start the fixture in one terminal, then run the comparator in another. The live client is intentionally tiny so transport code does not become the lesson.

python fixture_server.py
Enter fullscreen mode Exit fullscreen mode 
# dual_run.py — teaching example; swap LIVE_URL for your classroom endpoint
import hashlib, json, urllib.request
from pathlib import Path
from canonicalize import receipt_bytes

PROMPT = "Create a high-priority ops ticket about the stale pager runbook in ops-214."
PROMPT_SHA = hashlib.sha256(PROMPT.encode()).hexdigest()
LIVE_URL = "http://127.0.0.1:8765"  # replace with the live classroom backend

def post_json(url: str) -> dict:
    req = urllib.request.Request(
        url,
        data=json.dumps({"prompt": PROMPT}).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode("utf-8"))

def write_receipt(backend: str, turn: dict) -> Path:
    blob = receipt_bytes(turn["tool"], turn["arguments"], backend, PROMPT_SHA)
    path = Path("receipts") / f"prompt_a.{backend}.json"
    path.parent.mkdir(exist_ok=True)
    path.write_bytes(blob)
    return path

fixture_turn = post_json("http://127.0.0.1:8765")
live_turn = post_json(LIVE_URL)
f_path = write_receipt("fixture", fixture_turn)
l_path = write_receipt("live", live_turn)
print("match" if f_path.read_bytes() == l_path.read_bytes() else "drift")
print(f_path.read_text())
print(l_path.read_text())
Enter fullscreen mode Exit fullscreen mode

If you need a no-cost live backend for Exercise C, MonkeyCode’s free model access and free server option can occupy that slot without inventing a paid key workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The receipt still has to match the fixture; availability of a free path does not relax the contract.

Worked miss that students should rerun

Seed a deliberate live payload that would pass a casual eyeball check. Then watch the canonicalizer fail it. This is the teaching moment that screenshots never capture.

LIVE_DRIFT = {
    "tool": "create_ticket",
    "arguments": {
        "title": "Pager duty runbook is stale",
        "priority": "High",      # enum case drift
        "project_id": 214,       # type drift
        "assignee": "oncall",    # extra field
    },
}
Enter fullscreen mode Exit fullscreen mode

Expected classification after Exercise C:

Symptom Canonicalizer result Grade as
Key order changed, values identical Pass after sort Transport noise
Extra confidence or assignee Dropped, then pass if core equal Shape noise
priority case or unknown enum ValueError Contract miss
project_id became an integer ValueError Type drift
title paraphrased Pass shape, fail bytes Value drift
Tool name changed Do not canonicalize Wrong action

Value drift is the only row that should reopen prompt design. Shape and type misses belong to the contract, not to more adjectives in the system prompt. Students who respond to every miss by editing prose will not finish Exercise C inside the remaining twenty-five minutes.

Commands for a cold rerun

A classmate who was absent should reach the same pass or fail without the original chat transcript. That constraint is the point of the receipt.

python -m py_compile fixture_server.py canonicalize.py dual_run.py
mkdir -p receipts
python fixture_server.py &
python dual_run.py
sha256sum receipts/prompt_a.fixture.json receipts/prompt_a.live.json
Enter fullscreen mode Exit fullscreen mode

If the hashes differ, print a field-level diff of the decoded objects. Do not paste both files into a chat model and ask which is better. The lab already defined equality.

import json
from pathlib import Path

a = json.loads(Path("receipts/prompt_a.fixture.json").read_text())
b = json.loads(Path("receipts/prompt_a.live.json").read_text())
for key in sorted(set(a) | set(b)):
    if a.get(key) != b.get(key):
        print(key, a.get(key), b.get(key))
Enter fullscreen mode Exit fullscreen mode

Limitations

Canonical byte equality is stricter than human agreement and weaker than a full behavioral specification. It will fail useful paraphrases of title and it will pass a wrong ticket that still uses the three required keys. The fixture server also cannot represent rate limits, auth failures, or streaming token events. Treat those as out of scope for an eighty-minute session.

Free live backends can still change outputs between class meetings. The receipt documents that drift; it does not freeze a third-party endpoint. If the live path moves, keep the fixture receipt as the gradeable object and record the live miss as a dated lab note. Do not silently retune the fixture to match a new live payload, because that erases the teaching signal.

Who should not use this outline

Skip this workshop if your tools have no stable argument schema, or if you are grading writing quality rather than tool effects. Skip it for production incident work that needs authentication, PII controls, and real service-level objectives. Skip it when the assignment is open-ended generation with no tool layer, because there is then no receipt to pin. Teams that already freeze OpenAPI contracts and run consumer-driven pact tests will find this lab redundant except as a student on-ramp.

The conclusion stays narrow. A classroom can use a free live path, but only the dual-run receipt turns that path into engineering evidence. Fluent text remains optional commentary beside the JSON, not the thing you grade.

Top comments (0)