Workshop: Dual-Run Agent Fixtures Until Tool-Call JSON Stays Stable in 90 Minutes

Silent tool-argument drift is a more common agent failure than a hard crash, because the run still returns a fluent answer. This workshop treats that drift as a testable contract rather than a prompt-writing exercise. Two independent runs of the same fixture must emit JSON that matches a frozen schema and a frozen set of assumption keys. Students leave with a rerunnable checker, a timed exercise sequence, and a stop rule for labs that should not dual-run at all.

Recent developer threads often dwell on agent vocabulary and on agents that invent missing facts. Those threads are useful as topic signals, not as a glossary to reproduce in class. The failure that actually burns a lab hour is narrower and measurable: a tool call that adds a field, drops a required id, or fills a default the user never supplied. Fluent prose after that call is not evidence of stability.

Who should take this workshop

This session is for people who already have a small agent loop with tools, fixtures, and logs. It is not an introduction to agents, tool protocols, or prompt style. Instructors can run it as one 90-minute studio or as three 30-minute blocks on consecutive days.

Bring the following materials to the session, and do not start Lab 0 until every student has them:

• One frozen user task in a JSON file, never in an unsaved chat window
• One tool schema with required fields and explicit additionalProperties: false
• Two runnable executions that accept the same request body
• Python 3.11+ with pytest and jsonschema installed in a clean virtualenv

If a student only has one network endpoint, they can still complete Lab 0 and Lab 1 on captured traces. Dual-run comparison in Lab 2 needs two isolated executions, which may be two local processes, two sequential calls after a memory reset, or two hosted endpoints with identical request bodies.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Students who need a no-key host for the second execution can park the fixture runner on MonkeyCode's free model access and free server option, then keep every assertion on their own machine so grading never depends on a vendor UI.

Timing board

Block	Minutes	Outcome students must show
Briefing	10	Shared definition of drift versus crash
Lab 0	15	Frozen tool-argument schema that rejects one extra field
Lab 1	20	Assumption-key extractor with a short allow list
Lab 2	30	Dual-run JSON fingerprint that fails on a field change
Lab 3	15	Fail-closed pytest log from a deliberately broken fixture
Debrief	10	Decision table filled with one real trace from class

Instructors should refuse to extend the clock for extra prompt tuning. The graded artifact is the checker and the failing test, not a prettier final answer.

The contract students will enforce

Define three artifacts before any model call leaves the laptop:

1. task.json — the user request, known ids, and hard budgets
2. tool_args.schema.json — allowed arguments for every tool the fixture may call
3. assumptions.allowed.txt — keys the agent may infer, one per line, eight or fewer

A run counts as stable only when both executions produce tool-argument JSON that:

• Validates against the frozen schema with additionalProperties disabled
• Contains no assumption key outside the allow list
• Differs from the sibling run only in fields marked volatile, such as timestamps or request ids

Everything else is drift, even when the natural-language summary looks careful and complete.

Lab 0: freeze the tool-argument schema (15 minutes)

Start from a tool the class already uses in a prior lab, and do not invent a second product surface for this exercise. The example below is a ticketing lookup that students commonly over-parameterize when a model tries to be helpful.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "LookupTicketArgs",
  "type": "object",
  "additionalProperties": false,
  "required": ["ticket_id", "fields"],
  "properties": {
    "ticket_id": { "type": "string", "pattern": "^TCK-[0-9]{6}$" },
    "fields": {
      "type": "array",
      "minItems": 1,
      "maxItems": 8,
      "items": { "enum": ["status", "assignee", "priority", "updated_at"] }
    },
    "volatile": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "request_id": { "type": "string" }
      }
    }
  }
}

Exercise 0a: take one captured tool call from last week's lab and validate it against this schema. Exercise 0b: add one extra field that models commonly invent, such as include_comments, and confirm the validator rejects it. Students who cannot produce a failing extra field have not frozen the schema; they have only restated the happy path.

python -m venv .venv
. .venv/bin/activate
pip install jsonschema pytest
python -m jsonschema -i captures/run_a.args.json schemas/tool_args.schema.json

Lab 1: extract assumption keys (20 minutes)

Agents fill gaps with polite defaults: timezone, page size, current user, and priority. Those defaults are assumptions, not facts, and they belong in an allow list or they belong in task.json. The extractor below is labeled lab code; it reads a local trace and does not call a network.

# lab_assumption_keys.py — workshop extractor, not production telemetry
import json
import re
from pathlib import Path

FACT_KEYS = {"ticket_id", "fields", "user_id", "org_id"}
ASSUMPTION_PATTERNS = (
    re.compile(r"assume[d]?", re.I),
    re.compile(r"default(?:ing)? to", re.I),
    re.compile(r"current user", re.I),
    re.compile(r"probably", re.I),
)

def extract_keys(trace_path: Path) -> set[str]:
    blob = json.loads(trace_path.read_text())
    found: set[str] = set()
    for event in blob.get("events", []):
        text = str(event.get("content", ""))
        for pat in ASSUMPTION_PATTERNS:
            if pat.search(text):
                found.add(pat.pattern)
        for key in event.get("tool_args", {}):
            if key not in FACT_KEYS:
                found.add(key)
    return found

def assert_allowed(found: set[str], allow_path: Path) -> None:
    allowed = {
        line.strip()
        for line in allow_path.read_text().splitlines()
        if line.strip() and not line.startswith("#")
    }
    extra = found - allowed
    if extra:
        raise AssertionError(f"undeclared assumptions: {sorted(extra)}")

Exercise 1a: run the extractor on a fixture that never mentions defaults and record the empty set. Exercise 1b: inject the sentence “defaulting to page size 50” into the trace and watch the assertion fire. Keep the allow list shorter than eight keys; a longer list usually means the task file is missing facts the agent should not have to guess.

Lab 2: dual-run and diff (30 minutes)

Dual-run means two executions of the same task.json with isolated memory and no shared scratchpad. Do not compare prose, because two different argument lists can still produce similar sentences. Compare tool-argument JSON after stripping volatile fields, then fingerprint the remainder.

# lab_dual_run.py — rerunnable checker for the 90-minute workshop
import hashlib
import json
from pathlib import Path

from jsonschema import Draft202012Validator

VOLATILE_KEYS = {"request_id", "ts", "latency_ms"}

def strip_volatile(obj):
    if isinstance(obj, dict):
        return {
            k: strip_volatile(v)
            for k, v in obj.items()
            if k not in VOLATILE_KEYS
        }
    if isinstance(obj, list):
        return [strip_volatile(x) for x in obj]
    return obj

def load_tool_calls(path: Path) -> list:
    payload = json.loads(path.read_text())
    return payload["tool_calls"]

def schema_ok(args: dict, schema: dict) -> None:
    Draft202012Validator(schema).validate(args)

def fingerprint(tool_calls: list) -> str:
    stable = strip_volatile(tool_calls)
    blob = json.dumps(stable, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()

def assert_stable(run_a: Path, run_b: Path, schema_path: Path) -> None:
    schema = json.loads(schema_path.read_text())
    a = load_tool_calls(run_a)
    b = load_tool_calls(run_b)
    for call in a + b:
        schema_ok(call["args"], schema)
    fa, fb = fingerprint(a), fingerprint(b)
    if fa != fb:
        raise AssertionError(f"tool-call drift: {fa} != {fb}")

Exercise 2a: duplicate a fixture, change only request_id, and confirm the checker passes. Exercise 2b: change fields from ["status"] to ["status", "priority"] and confirm it fails. Exercise 2c: send the same body to two isolated runners and store traces as run_a.json and run_b.json without editing either file by hand.

A local loop that students can rerun without a dashboard looks like this:

python agent_runner.py --task fixtures/task.json --out traces/run_a.json
python agent_runner.py --task fixtures/task.json --out traces/run_b.json
python -c "from pathlib import Path; from lab_dual_run import assert_stable; assert_stable(Path('traces/run_a.json'), Path('traces/run_b.json'), Path('schemas/tool_args.schema.json')); print('stable')"

If a hosted second endpoint is required, point agent_runner.py at the class host chosen in the briefing and keep the same task.json. The grade still comes from assert_stable, not from a screenshot of a chat transcript.

Lab 3: fail closed in pytest (15 minutes)

Wrap the checker so continuous integration, not a teaching assistant, rejects drift. The broken-fixture trial is mandatory; a green run with no negative control does not count as completion.

# test_dual_run_stability.py
import json
from pathlib import Path

import pytest
from jsonschema import Draft202012Validator, ValidationError

from lab_assumption_keys import assert_allowed, extract_keys
from lab_dual_run import assert_stable

FIXTURES = Path("fixtures")
TRACES = Path("traces")
SCHEMAS = Path("schemas")

def test_schema_rejects_extra_field():
    schema = json.loads((SCHEMAS / "tool_args.schema.json").read_text())
    bad = {
        "ticket_id": "TCK-000042",
        "fields": ["status"],
        "include_comments": True,
    }
    with pytest.raises(ValidationError):
        Draft202012Validator(schema).validate(bad)

def test_dual_run_fingerprints_match():
    assert_stable(
        TRACES / "run_a.json",
        TRACES / "run_b.json",
        SCHEMAS / "tool_args.schema.json",
    )

def test_assumption_allow_list():
    found = extract_keys(TRACES / "run_a.json") | extract_keys(TRACES / "run_b.json")
    assert_allowed(found, FIXTURES / "assumptions.allowed.txt")

Exercise 3: break one fixture on purpose, run pytest -q, and paste the assertion message into the lab notes. If the class cannot show a failing log, the schema is not yet a gate.

Worked example students can rerun

Use this minimal task.json as the shared fixture so dual-run cost stays visible and comparable across laptops. The budgets are part of the contract, not comments for humans.

{
  "task_id": "lab-dual-run-001",
  "user": "Return status and assignee for ticket TCK-000042. Do not add fields.",
  "facts": {
    "ticket_id": "TCK-000042",
    "fields": ["status", "assignee"]
  },
  "budgets": {
    "max_tool_calls": 2,
    "max_retries": 0
  }
}

Expected stable tool call after both runs:

{
  "name": "lookup_ticket",
  "args": {
    "ticket_id": "TCK-000042",
    "fields": ["status", "assignee"]
  }
}

Unstable variants the checker must reject, even when the final sentence still names the ticket:

• fields expanded to include priority after a reorder that looks harmless
• ticket_id rewritten as an integer or as a bare six-digit string
• a second tool call such as list_comments that the task never requested
• prose that says “I assumed the current user is the assignee” while emitting extra arguments

Students rerun by deleting traces/, executing the two runner commands, and calling pytest. If results change after a prompt edit, the schema was not the source of truth and the lab is incomplete.

Decision table

Observation in the traces	Treat as	Next action in the remaining minutes
Schema fail on an extra key	Drift	Shrink the prompt; do not widen the schema
Fingerprints differ only in volatile keys	Stable	Keep the strip list explicit in code
Assumption key missing from the allow list	Hidden default	Move the fact into task.json or fail the run
Answers match while tool args differ	Silent drift	Fail the lab; do not grade the prose
One endpoint unavailable	Incomplete dual-run	Use two local processes; do not skip Lab 2
Two identical wrong tool calls	False stable	Add a domain assertion outside this workshop

Limitations and who should not use this approach

Dual-run comparison is a cheap consistency gate, not a correctness proof, and two wrong tool calls with the same JSON will pass. Non-deterministic tools such as clocks, live queues, or sampling with temperature above zero will fail the fingerprint even when operators consider the behavior acceptable. Teams that need creative variation should mark those fields volatile, lower temperature for the lab, or skip this workshop entirely.

Do not use this method when any of the following constraints are true:

• The tool has required side effects that cannot be sandboxed for a second run
• The class cannot freeze a schema because the upstream API is unbounded
• The latency or token cost of a second run exceeds the published lab budget
• Graders want to score writing quality rather than tool-call contracts

The checker also ignores natural-language caveats on purpose. If the agent writes “I might be wrong” but still emits extra arguments, the test fails, and that severity is the intended teaching signal.

What instructors should collect

Before dismissing the class, collect four files per student: task.json, tool_args.schema.json, assumptions.allowed.txt, and the pytest log. Those files are the evidence that the contract exists outside a chat window. Screenshots of a vendor UI are not substitutes, because they cannot be rerun after the session ends.

This workshop stays useful if any hosted model is removed, because the schema, extractor, and fingerprint live in the repository. Instructors should treat dual-run stability as a gate that happens before memory writes or side effects, not as a substitute for domain tests that check ticket state in a real system.