Workshop: Stamp Agent Assumptions With Source Tags in 65 Minutes

Agents fail less from missing tools than from silent guesses that later look like retrieved facts. This workshop stamps every pre-tool claim with a source tag, then blocks calls when required fields remain assumed. Students leave with a rerunnable ledger, a pytest gate, and a timing plan that fits a single lab block. The tests stay deterministic even when a later extraction pass uses free inference.

The failure mode this lab isolates

Tool-calling agents often invent hostnames, ticket identifiers, file paths, and time ranges because the prompt asked them to be helpful. Those invented values then enter JSON arguments, after which logs treat them as ordinary parameters. Downstream tools succeed against the wrong object, or they fail with an error that never names the invented field. The lab treats that gap as a contract problem, not as a model-quality complaint.

A source tag is a small enum attached to each claim the agent wants to use. Allowed values in this lab are observed, retrieved, user, derived, and assumed. Tool schemas then declare which argument names must not arrive with assumed. That rule is boring on purpose, because boring rules survive a noisy classroom and a later code review.

Trend pieces about agents that “just assume things” are useful as a symptom list, not as a design spec. This workshop does not classify agent architectures or catalog vendor terms. It only asks whether a required argument still carries an assumed tag when the process is about to execute a side effect.

Lab goals and timing

By minute 65, each pair should be able to rerun the fixture and explain a rejected call in one sentence. The session is split so instructors can cut the optional extraction pass if the room has no network.

• 0–8 min — Frame the bug. Show one transcript where a missing ticket id becomes a plausible id, then a write tool runs.
• 8–18 min — Ledger schema. Students copy the typed ledger below and label three claims by hand.
• 18–38 min — Gate and tests. They implement assert_tool_args_sourced and run the pytest file twice.
• 38–52 min — Exercise A and B. One broken fixture, one derived-field fixture, both checked into the lab repo.
• 52–65 min — Debrief. Compare which fields were allowed to stay assumed, and name two fields that must never be.

If the room is large, keep the debrief on a whiteboard table with four columns: field, tag, source pointer, allowed for writes. Do not spend the remaining minutes adding another tool. Extra tools hide the contract instead of proving it.

Prerequisites

Students need Python 3.11+, pytest, and a text editor. No GPU is required for the core path, because the ledger is ordinary JSON. Optional extraction later can call a remote model, but the grade depends on the tests, not on the model.

Clone or copy the files into an empty directory named assumption-lab. The commands below assume a Unix shell. Windows students can use the same pytest invocation inside the project folder.

python -m venv .venv
source .venv/bin/activate
pip install pytest
mkdir -p lab tests fixtures

Worked example: a ledger the tests can own

The ledger is the original artifact for this session. It is not a prompt. It is a document the gate reads before any tool process starts. Students should treat a missing source_pointer on retrieved or observed tags as a test failure, because a tag without a pointer is another form of guessing.

{
  "run_id": "lab-65-ticket-reply",
  "claims": [
    {
      "name": "ticket_id",
      "value": "TCK-10419",
      "tag": "user",
      "source_pointer": "user_message:0",
      "required_for": ["ticket.update"]
    },
    {
      "name": "assignee_email",
      "value": "[e-mail]",
      "tag": "assumed",
      "source_pointer": null,
      "required_for": ["ticket.update"]
    },
    {
      "name": "last_status",
      "value": "waiting_on_customer",
      "tag": "retrieved",
      "source_pointer": "tools.ticket.get:result.status",
      "required_for": ["ticket.update"]
    }
  ],
  "pending_tool": {
    "name": "ticket.update",
    "args": {
      "ticket_id": "TCK-10419",
      "assignee_email": "[e-mail]",
      "last_status": "waiting_on_customer"
    }
  }
}

Save that document as fixtures/ticket_update_assumed_assignee.json. The second claim is the intended failure. The first and third claims show tags the gate should accept for a write.

Gate module

The gate is deliberately small. It does not call models, networks, or ticket APIs. It only joins pending tool arguments to claims by name, then applies the tag policy. Label this as a teaching implementation, not as production middleware.

# lab/assumption_gate.py
from __future__ import annotations

from typing import Any

WRITE_TOOLS = {"ticket.update", "ticket.close", "fs.write", "shell.run"}
FORBIDDEN_TAGS_FOR_WRITES = {"assumed"}
POINTER_REQUIRED = {"observed", "retrieved", "user", "derived"}


class AssumptionGateError(ValueError):
    pass


def _claim_map(ledger: dict[str, Any]) -> dict[str, dict[str, Any]]:
    claims = ledger.get("claims") or []
    out: dict[str, dict[str, Any]] = {}
    for claim in claims:
        name = claim.get("name")
        if not name:
            raise AssumptionGateError("claim missing name")
        if name in out:
            raise AssumptionGateError(f"duplicate claim name: {name}")
        out[name] = claim
    return out


def assert_tool_args_sourced(ledger: dict[str, Any]) -> None:
    pending = ledger.get("pending_tool") or {}
    tool_name = pending.get("name")
    args = pending.get("args") or {}
    if not tool_name:
        raise AssumptionGateError("pending_tool.name is required")

    claims = _claim_map(ledger)
    for arg_name, arg_value in args.items():
        claim = claims.get(arg_name)
        if claim is None:
            raise AssumptionGateError(
                f"{tool_name} arg {arg_name!r} has no claim in the ledger"
            )
        if claim.get("value") != arg_value:
            raise AssumptionGateError(
                f"{arg_name} value does not match ledger claim"
            )
        tag = claim.get("tag")
        pointer = claim.get("source_pointer")
        if tag in POINTER_REQUIRED and not pointer:
            raise AssumptionGateError(
                f"{arg_name} tag {tag!r} requires source_pointer"
            )
        if tool_name in WRITE_TOOLS and tag in FORBIDDEN_TAGS_FOR_WRITES:
            raise AssumptionGateError(
                f"refusing {tool_name}: {arg_name} is tagged {tag!r}"
            )
        required_for = set(claim.get("required_for") or [])
        if required_for and tool_name not in required_for:
            raise AssumptionGateError(
                f"{arg_name} is not listed for {tool_name}"
            )

Tests students rerun

The first test encodes the worked example. The second test shows a legal derived field, which is the usual student question after the first failure. Derived values are allowed when the pointer names the inputs, not when the model shrugs.

# tests/test_assumption_gate.py
import json
from pathlib import Path

import pytest

from lab.assumption_gate import AssumptionGateError, assert_tool_args_sourced

FIXTURES = Path(__file__).resolve().parents[1] / "fixtures"


def load(name: str) -> dict:
    return json.loads((FIXTURES / name).read_text(encoding="utf-8"))


def test_write_rejected_when_required_arg_is_assumed() -> None:
    ledger = load("ticket_update_assumed_assignee.json")
    with pytest.raises(AssumptionGateError, match="assignee_email"):
        assert_tool_args_sourced(ledger)


def test_derived_sla_bucket_is_allowed_with_pointer() -> None:
    ledger = {
        "claims": [
            {
                "name": "ticket_id",
                "value": "TCK-10419",
                "tag": "user",
                "source_pointer": "user_message:0",
                "required_for": ["ticket.update"],
            },
            {
                "name": "sla_bucket",
                "value": "t24h",
                "tag": "derived",
                "source_pointer": "derive.sla_bucket(last_status, opened_at)",
                "required_for": ["ticket.update"],
            },
        ],
        "pending_tool": {
            "name": "ticket.update",
            "args": {"ticket_id": "TCK-10419", "sla_bucket": "t24h"},
        },
    }
    assert_tool_args_sourced(ledger)  # no exception

Rerun from the project root. The first command should fail until students understand the fixture, then pass after they change assignee_email or drop it from the write. Keep both outcomes in git so later groups can diff them.

export PYTHONPATH=.
pytest -q tests/test_assumption_gate.py

Expected teaching sequence: first run fails on assignee_email, students edit the fixture, second run passes, then they add a third test that omits source_pointer on a retrieved field. That third test is Exercise A.

Exercise A — missing pointers are guesses

Duplicate the retrieved last_status claim, set source_pointer to null, and keep the tag as retrieved. The gate must raise. If a pair makes the test pass by weakening POINTER_REQUIRED, stop them and restore the set. The point is not coverage count. The point is that a tag without a pointer cannot be audited after class.

Suggested assertion text: last_status tag 'retrieved' requires source_pointer. Students may write a more specific matcher. Instructors should reject matchers that only look for Error, because those pass for the wrong reason.

Exercise B — read tools may keep unknowns, writes may not

Add a second pending tool named ticket.get with argument ticket_id tagged user. Reads can proceed with unknowns in other fields, as long as those fields are not in args. Writes cannot. Have students encode that distinction as a table, then as one extra test, not as a paragraph in the prompt.

Field	Tag	In write args?	Gate result
ticket_id	user	yes	allow
last_status	retrieved	yes	allow if pointer present
assignee_email	assumed	yes	reject
customer_region	assumed	no	allow, remains an open question
sla_bucket	derived	yes	allow if pointer names inputs

The table is the decision artifact. If a pair cannot fill the last column without looking at the code, the schema is still too vague for the room.

Optional extraction pass on free inference

The core lab does not need a model. A later station can feed a messy transcript into a constrained extractor that only emits claims, never tool calls. That split keeps pytest stable while still giving students a realistic upstream.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that extractor when the classroom has no paid inference budget. The product is optional here: any endpoint that returns JSON can fill the same role, and the gate still refuses assumed write args.

Keep the extractor prompt tiny. Ask only for claims with name, value, tag, and source_pointer. Discard any extra keys. If the model omits a pointer, store tag as assumed rather than inventing retrieved. Students should commit the extractor output as a fixture, then run the same tests they already have.

# lab/extract_claims.py  — teaching sketch, not a vendor SDK
EXTRACTOR_INSTRUCTIONS = """
Return JSON with key claims. Each claim needs name, value, tag,
and source_pointer. Tags: observed, retrieved, user, derived, assumed.
If the transcript does not ground a value, tag it assumed and set
source_pointer to null. Do not emit tool calls.
"""

Do not grade model fluency. Grade whether the written fixture still fails when assignee_email is assumed. If the extractor marks a user-supplied ticket id as assumed, that is a prompt bug, and Exercise A already taught the fix: correct the tag and pointer, then rerun pytest.

Limitations and who should skip this lab

The gate does not prove that retrieved values are true. It only proves that the run recorded a pointer. Poisoned retrieval, stale tickets, and lying tools remain out of scope. The enum also collapses several real-world source types, including human operators and batch jobs, into user or observed.

Skip this design when the agent cannot name arguments, when tools accept free-form blobs, or when a human already confirms every write. Also skip it as a production authorization layer. Source tags are an audit aid sitting in front of existing authz, not a replacement for it. Teams that need cryptographic provenance want signed tool results, which this 65-minute lab does not build.

Classroom risk is over-fitting the ticket example. If every fixture uses ticket.update, students will treat the policy as product-specific. Add one fs.write fixture before the debrief so the same tag rules apply to paths. If time is short, discuss that fixture instead of coding it.

What to keep after the session

Keep the ledger schema, the write-tool set, and the pytest file. Delete the extractor if it started to dominate discussion. The durable lesson is that assumptions need a field, a tag, and a pointer before they can approach a side effect. Models can propose those fields. Tests still decide whether the process may continue.

Pairs that finish early can add a JSON Schema file for the ledger and validate fixtures in CI. That extra file is optional homework, not a requirement for the 65-minute block. The minimum passing artifact is one rejected write, one accepted derived field, and a table the next lab group can read without the instructor.