Workshop: Fail Closed on Hallucinated Tool Arguments in 65 Minutes

Agent pipelines usually break after the model has already emitted a tool call that looks well-formed. A typed JSON object can still name the wrong order, invent a currency, or scale a refund by a factor of one hundred. This workshop treats that failure as a contract problem: validate arguments with a deterministic schema gate, then a cheap semantic gate, and only then allow the tool to run. Students leave with a fixture pack, a fail-closed runner, and a scorecard they can rerun on any OpenAI-compatible endpoint, including a free hosted model if they do not want a local GPU.

The method does not require a production agent framework. It requires one tool definition, twelve frozen calls, and a runner that returns allow, repair, or block. If the semantic model is unavailable, the schema gate still fails closed. That property is the point of the exercise, not a leaderboard of model quality.

Who this workshop is for

Use this outline with a small backend or platform group that already ships function calling, MCP tools, or homegrown agent loops. The useful audience is people who have seen a tool mutate state from a plausible hallucination. Skip it if your tools are already invoked only through generated clients with server-side authorization that ignores model-supplied identifiers.

Time box: 65 minutes, including a ten-minute readout.

Prerequisites: Python 3.11+, jsonschema, and httpx. No GPU is required. Set MODEL_BASE_URL, MODEL_API_KEY, and MODEL_NAME if you want the optional semantic pass.

The failure you will freeze

Consider a single refund tool. The model is asked to refund order ord_1842 for 19.00 USD because of a damaged shipment. A typical bad completion still satisfies a loose “call a function” prompt:

{
  "name": "refund_order",
  "arguments": {
    "order_id": "ord_9999",
    "amount": 1900,
    "currency": "US",
    "reason": "customer asked nicely"
  }
}

Three independent defects sit in that object. The identifier is not in the allowlist. The amount is two orders of magnitude too large because the model dropped the decimal. The currency is not an ISO code. A JSON parser will accept all three. A payment tool that trusts the model will accept them too.

The workshop freezes that shape as fixtures/tool_calls.jsonl so later model swaps cannot silently change the lesson.

Workshop clock

Follow the clock even if the semantic endpoint is slow. The deterministic gate is the pass/fail backbone; the model pass is extra coverage, not the grade.

1. 0–8 min — Frame the contract. Write the tool schema and the three fail-closed outcomes on a whiteboard or in CONTRACT.md.
2. 8–20 min — Build the fixture pack. Encode twelve calls: four valid, four schema-invalid, four schema-valid but semantically wrong.
3. 20–35 min — Schema gate. Reject unknown fields, wrong types, missing required keys, and non-enum currencies.
4. 35–50 min — Semantic gate. Check allowlisted IDs, amount bounds from the source order, and unit consistency with a cheap model only after JSON is valid.
5. 50–65 min — Runner and scorecard. Fail closed on transport errors. Print precision, recall, and a confusion table against the fixture labels.

Exercise 1: write the contract, not the prompt

Create tools/refund_order.schema.json before any model call. The schema is the source of truth. Prompts are comments on that schema, not a replacement for it.

{
  "$id": "refund_order.schema.json",
  "type": "object",
  "additionalProperties": false,
  "required": ["order_id", "amount", "currency", "reason"],
  "properties": {
    "order_id": { "type": "string", "pattern": "^ord_[0-9]{4}$" },
    "amount": { "type": "number", "exclusiveMinimum": 0, "maximum": 500 },
    "currency": { "type": "string", "enum": ["USD", "EUR", "GBP"] },
    "reason": { "type": "string", "minLength": 8, "maxLength": 240 }
  }
}

Also freeze a tiny order store. Students should not query a live warehouse during the workshop. A file keeps the allowlist reviewable in pull requests.

{
  "ord_1842": { "status": "delivered", "currency": "USD", "max_refund": 19.00 },
  "ord_1843": { "status": "cancelled", "currency": "EUR", "max_refund": 0.00 }
}

Pass rule: a call is schema-valid only if jsonschema accepts it. Do not “fix” extra keys in this exercise. Extra keys are a block, because agents often smuggle user_id or notify_slack into trusted tools.

Exercise 2: twelve frozen calls

Put one JSON object per line in fixtures/tool_calls.jsonl. Each line needs id, label (allow or block), arguments, and note. Example rows students can rerun:

{"id":"t01","label":"allow","arguments":{"order_id":"ord_1842","amount":19.0,"currency":"USD","reason":"damaged shipment on delivery"},"note":"happy path"}
{"id":"t05","label":"block","arguments":{"order_id":"ord_1842","amount":19.0,"currency":"US","reason":"damaged shipment on delivery"},"note":"currency not in enum"}
{"id":"t09","label":"block","arguments":{"order_id":"ord_9999","amount":19.0,"currency":"USD","reason":"damaged shipment on delivery"},"note":"unknown order id"}
{"id":"t10","label":"block","arguments":{"order_id":"ord_1842","amount":1900,"currency":"USD","reason":"damaged shipment on delivery"},"note":"amount exceeds max_refund"}
{"id":"t11","label":"block","arguments":{"order_id":"ord_1843","amount":5.0,"currency":"EUR","reason":"customer changed their mind"},"note":"cancelled order, max_refund is 0"}

Keep the remaining seven rows in the same file so the scorecard is not a toy. Mix missing reason, a string amount, an extra notify field, a GBP call against a USD order, and a valid partial refund of 5.00 on ord_1842.

Pass rule: labels are assigned by humans before any model sees the file. If the model later disagrees, the fixture wins.

Exercise 3: the deterministic gate

The first gate must run with no network. That constraint keeps CI honest when a free endpoint is rate-limited or cold.

# example runner fragment — students should execute this locally
import json
from pathlib import Path
from jsonschema import Draft202012Validator

schema = json.loads(Path("tools/refund_order.schema.json").read_text())
orders = json.loads(Path("data/orders.json").read_text())
validator = Draft202012Validator(schema)

def schema_gate(arguments: dict) -> tuple[str, list[str]]:
    errors = [e.message for e in validator.iter_errors(arguments)]
    if errors:
        return "block", errors
    return "allow", []

def store_gate(arguments: dict) -> tuple[str, list[str]]:
    order = orders.get(arguments["order_id"])
    reasons = []
    if order is None:
        return "block", ["order_id not in allowlist"]
    if arguments["currency"] != order["currency"]:
        reasons.append("currency mismatch vs source order")
    if arguments["amount"] > order["max_refund"]:
        reasons.append("amount exceeds max_refund")
    if order["status"] != "delivered":
        reasons.append("order is not refundable in current status")
    return ("block", reasons) if reasons else ("allow", [])

Run both gates in order. If schema_gate blocks, skip store_gate. Logging the skipped stage is useful later when you debug a flood of enum failures and think the allowlist is broken.

python -m pytest tests/test_gates.py -q
python scripts/scorecard.py --gates schema,store --fixtures fixtures/tool_calls.jsonl

A minimal tests/test_gates.py should assert t05 blocks on currency and t01 allows. Do not assert on model text in this file. Deterministic tests that parse generated prose become flaky the first time the endpoint changes.

Exercise 4: the cheap semantic gate

The store gate catches ID and bound errors you already encoded. It will not catch a well-typed reason that contradicts policy, such as refunding for “customer asked nicely” when policy requires a documented defect. That is the only place a model belongs in this workshop.

Label the next function as a proposal if you cannot reach an endpoint during the session. The control flow still matters: call the model only after JSON and store gates allow, then fail closed on timeouts.

# proposed semantic gate; requires MODEL_* env vars
import os, json, httpx

POLICY = """Refund reasons must cite damage, loss, or a documented billing error.
Refuse goodwill, threats, or unspecified dissatisfaction. Reply with JSON only:
{"decision":"allow"|"block","rationale":"..."}"""

def semantic_gate(arguments: dict, timeout_s: float = 12.0) -> tuple[str, list[str]]:
    payload = {
        "model": os.environ["MODEL_NAME"],
        "temperature": 0,
        "messages": [
            {"role": "system", "content": POLICY},
            {"role": "user", "content": json.dumps(arguments)},
        ],
    }
    try:
        response = httpx.post(
            os.environ["MODEL_BASE_URL"].rstrip("/") + "/chat/completions",
            headers={"Authorization": f"Bearer {os.environ['MODEL_API_KEY']}"},
            json=payload,
            timeout=timeout_s,
        )
        response.raise_for_status()
        content = response.json()["choices"][0]["message"]["content"]
        parsed = json.loads(content)
    except Exception as exc:
        return "block", [f"semantic gate failed closed: {exc.__class__.__name__}"]
    if parsed.get("decision") not in {"allow", "block"}:
        return "block", ["semantic gate returned an unknown decision"]
    return parsed["decision"], [parsed.get("rationale", "")]

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If the group has no GPU and no paid key, MonkeyCode’s free model access and free server option can host this semantic pass; the same runner still works against any compatible base URL you already operate.

Pass rule: transport errors, non-JSON completions, and unknown decisions are block. Never default to allow because the model “probably meant yes.”

Exercise 5: scorecard the whole pipeline

Compose gates so a single fixture row produces one final decision. Suggested precedence: schema, store, semantic. Print a table, not a narrative summary.

id   human  machine  stage_blocked
t01  allow  allow    -
t05  block  block    schema
t09  block  block    store
t10  block  block    store
t12  block  block    semantic

Compute four counts only: true allow, true block, false allow, false block. False allows are the only paging-worthy metric in this workshop. A false block is annoying; a false allow refunds the wrong order.

# example scoring fragment
def tally(rows):
    ta = sum(1 for r in rows if r["human"] == "allow" and r["machine"] == "allow")
    tb = sum(1 for r in rows if r["human"] == "block" and r["machine"] == "block")
    fa = sum(1 for r in rows if r["human"] == "block" and r["machine"] == "allow")
    fb = sum(1 for r in rows if r["human"] == "allow" and r["machine"] == "block")
    return {"true_allow": ta, "true_block": tb, "false_allow": fa, "false_block": fb}

Pass rule for the session: zero false allows on the twelve fixtures. False blocks may be discussed, not excused into production.

What this workshop does not prove

The scorecard is a regression harness, not a capability claim about any vendor model. Twelve rows cannot estimate production hallucination rates. Amount bounds in orders.json are teaching data, not a finance policy. Latency, token cost, and uptime are intentionally unmeasured here because a 65-minute class cannot produce a durable benchmark.

Cheap models can rubber-stamp a policy paragraph if the system prompt is vague. If students observe that behavior, tighten the policy and keep fail-closed transport handling; do not add a second model “just in case.” Two weak judges in a row still fail open if either returns allow by default.

Who should not use this approach

Do not use this gate as the only control on money movement, identity changes, or destructive infrastructure tools. Server-side authorization, idempotency keys, and human review remain mandatory for those classes. Teams with generated, typed tool clients and strict allowlists inside the tool implementation will get little value from a second JSON schema sitting in the agent loop.

Also skip the semantic pass when you cannot define a written policy in fewer than ten lines. If reviewers cannot agree on why a reason is invalid, a model will not settle the argument during a workshop.

Close the loop in the last five minutes

Ask each pair to name one tool in their own stack that currently trusts model-supplied identifiers. Then ask whether that tool has an allowlist as small as orders.json. If it does not, the next homework is not a larger model. It is a frozen fixture file and a schema that fails closed before the first byte of side effect.

The runner above is enough to start that homework on a laptop. A hosted free model is optional coverage for the policy paragraph, not a substitute for the allowlist.