Workshop: Validate Tool Results Before They Enter Agent Memory in 75 Minutes

Agent memory becomes unreliable the moment an unchecked tool payload is stored as if it were a fact. Extra fields, wrong types, and silently defaulted keys survive into later planning steps and look like evidence. This workshop treats every tool result as untrusted input and refuses to append it until a typed gate accepts the object. Students leave with a replayable Python harness, a 75-minute exercise plan, and tests that fail closed on invented keys.

The core conclusion is operational rather than rhetorical: planner text can be sloppy, but memory writes must be narrow. A model may narrate a discount code, a retry token, or a warehouse that the tool never returned. If those keys land in the transcript, the next loop treats them as retrieved state instead of model residue. The gate below is intentionally boring, because boring checks are the ones you can rerun in CI.

Who this workshop is for

This session is for developers who already have an agent loop that calls tools and then feeds the raw JSON back into context. It is also for instructors who need a lab that fails on purpose before it teaches a fix. You do not need a paid model, a cluster, or a production MCP server. You need Python 3.11+, a terminal, and 75 focused minutes.

Skip this outline if your tools already return signed, schema-versioned envelopes that a policy engine verifies outside the agent. Also skip it if the student environment cannot buffer a complete JSON object before the next model call. Streaming token UIs can still use the idea, but the exercises assume a full payload.

Timing board

Use this schedule as a hard cap, not a suggestion. If an exercise overruns, cut discussion rather than the tests.

1. 0–10 min — Reproduce the leak. Run a tool stub that returns extra keys and watch memory absorb them.
2. 10–25 min — Freeze the contract. Write the allowlist, required fields, enums, and additionalProperties: false.
3. 25–45 min — Fail closed. Implement the gate so unknown tools, bad types, and extras raise.
4. 45–60 min — Isolate memory writes. Route every append through the gate; keep the raw payload out of the transcript.
5. 60–75 min — Replay fixtures. Store pass and fail cases as JSON files and assert the same decisions twice.

The failure you will reproduce

Beginner agent tutorials often concatenate tool JSON into the next prompt with no type check. That pattern is easy to demo and easy to poison. Consider an inventory lookup that is specified to return sku, quantity, and warehouse. A noisy implementation, or a model pretending to be the tool, might also emit reorder_hint or coerce quantity into a string.

Once those extras sit in memory, a later planning step can “see” a reorder hint that no warehouse system produced. The agent then calls a purchase tool with an argument that never existed in the business contract. The bug is not the purchase call itself. The bug is an untyped append that turned speculation into state.

Artifact: a fail-closed result gate

The artifact is a tiny Python module plus tests. It does not call a network. Students can rerun it on a laptop after the workshop without credentials. Label this code as a teaching harness, not a production policy engine.

Create result_gate.py:

from __future__ import annotations

from typing import Any, Mapping

ALLOWED_TOOLS: dict[str, dict[str, Any]] = {
    "inventory.lookup": {
        "required": ["sku", "quantity", "warehouse"],
        "properties": {
            "sku": {"type": "string", "min": 1, "max": 32},
            "quantity": {"type": "int", "min": 0, "max": 1_000_000},
            "warehouse": {"type": "enum", "values": ["sfo", "ams", "nrt"]},
        },
        "additional_properties": False,
    }
}


class ResultGateError(ValueError):
    """Raised when a tool payload must not enter agent memory."""


def validate_tool_result(tool_name: str, payload: Any) -> dict[str, Any]:
    if tool_name not in ALLOWED_TOOLS:
        raise ResultGateError(f"unknown tool: {tool_name}")
    if not isinstance(payload, dict):
        raise ResultGateError("payload must be an object")

    schema = ALLOWED_TOOLS[tool_name]
    props: Mapping[str, Any] = schema["properties"]

    if schema.get("additional_properties") is False:
        extra = sorted(set(payload) - set(props))
        if extra:
            raise ResultGateError(f"unexpected fields: {extra}")

    missing = [key for key in schema["required"] if key not in payload]
    if missing:
        raise ResultGateError(f"missing fields: {missing}")

    cleaned: dict[str, Any] = {}
    for key, rule in props.items():
        if key not in payload:
            continue
        value = payload[key]
        cleaned[key] = _check_field(key, value, rule)
    return cleaned


def _check_field(key: str, value: Any, rule: Mapping[str, Any]) -> Any:
    kind = rule["type"]
    if kind == "string":
        if not isinstance(value, str):
            raise ResultGateError(f"{key} must be a string")
        if not (rule["min"] <= len(value) <= rule["max"]):
            raise ResultGateError(f"{key} length out of range")
        return value
    if kind == "int":
        if isinstance(value, bool) or not isinstance(value, int):
            raise ResultGateError(f"{key} must be an integer")
        if not (rule["min"] <= value <= rule["max"]):
            raise ResultGateError(f"{key} out of range")
        return value
    if kind == "enum":
        if value not in rule["values"]:
            raise ResultGateError(f"{key} not in allowlist")
        return value
    raise ResultGateError(f"unsupported rule for {key}")


class AgentMemory:
    def __init__(self) -> None:
        self.events: list[dict[str, Any]] = []

    def append_tool_result(self, tool_name: str, payload: Any) -> dict[str, Any]:
        clean = validate_tool_result(tool_name, payload)
        self.events.append({"role": "tool", "name": tool_name, "content": clean})
        return clean

Create test_result_gate.py next to it:

import unittest

from result_gate import AgentMemory, ResultGateError, validate_tool_result

VALID = {"sku": "SKU-19", "quantity": 4, "warehouse": "sfo"}


class ResultGateTests(unittest.TestCase):
    def test_accepts_contract_payload(self) -> None:
        self.assertEqual(validate_tool_result("inventory.lookup", VALID), VALID)

    def test_rejects_extra_fields(self) -> None:
        poisoned = {**VALID, "reorder_hint": "buy-now"}
        with self.assertRaises(ResultGateError):
            validate_tool_result("inventory.lookup", poisoned)

    def test_rejects_string_quantity(self) -> None:
        bad = {**VALID, "quantity": "4"}
        with self.assertRaises(ResultGateError):
            validate_tool_result("inventory.lookup", bad)

    def test_memory_never_stores_raw_poison(self) -> None:
        memory = AgentMemory()
        with self.assertRaises(ResultGateError):
            memory.append_tool_result(
                "inventory.lookup",
                {**VALID, "discount_code": "STAFF"},
            )
        self.assertEqual(memory.events, [])

    def test_unknown_tool_is_closed(self) -> None:
        with self.assertRaises(ResultGateError):
            validate_tool_result("inventory.delete_all", VALID)


if __name__ == "__main__":
    unittest.main()

Run the fixture from the same directory:

python -m unittest test_result_gate.py -v

Expected teaching result: four failures if students append first and validate later, and five passes once memory writes go through the gate. Do not treat the pass count as a product benchmark. It is only a classroom invariant for this schema.

Decision table students should fill

Have each pair complete this table before they change production code. The point is to make accept and reject rules visible, not to debate model quality.

• Valid object with sku, integer quantity, and warehouse=sfo → accept and store only those three keys.
• Extra key such as reorder_hint or discount_code → reject; memory stays empty.
• Missing warehouse → reject; do not default to the first enum value.
• quantity as "4" → reject; JSON parsing is not type validation.
• Unknown tool name → reject; a fluent name is not an allowlist entry.
• warehouse=lon → reject; new sites need a schema change, not a silent store.

If two students disagree on a row, freeze the table and change the schema in version control. Do not “fix” the disagreement by letting the model explain the field in prose. Prose is not a contract.

Exercise notes that keep the lab honest

During minutes 0–10, force the leak on purpose. Temporarily append payload before calling validate_tool_result, print memory.events, and read the invented key out loud. The embarrassment is the lesson. Then restore the gate and confirm the list is empty.

During minutes 25–45, refuse helper defaults. A function that inserts warehouse="sfo" when the key is missing will pass tests and still lie in production. Missing required fields must raise. Optional fields, if you add them later, need an explicit schema flag rather than a Python default buried in the adapter.

During minutes 60–75, save at least three JSON fixtures under fixtures/: one valid, one extra-field, one wrong-type. Reload them in the test file so the workshop is not a live-coding memory test. Replay is the difference between a demo and a lab students can rerun on Monday.

Where free model access actually fits

The gate does not require a hosted model. If you later wrap the same loop around a live planner, keep the validator on the tool-result path, not inside the prompt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is enough to drive the planner side of this lab without turning the exercises into a billing subplot.

Use that compute only after the unittest file is green on stubs. A free endpoint that emits extra keys should still bounce off the same ResultGateError. If you already teach with this stack, run the harness against one planner call and paste the rejected payload into fixtures/; do not expand the allowlist just to make the demo look smooth.

Limitations

A schema gate stops structural invention. It does not stop a tool that returns a schema-valid lie, such as quantity: 4 when the shelf is empty. It also does not authenticate the tool, pin a schema version across deploys, or bound token spend. Integer ranges here are classroom constants, not measured warehouse limits.

additional_properties: false will break legitimate additive API changes until you version the contract. That breakage is desirable in a lab and painful in an unversioned production agent. Binary tools, streamed partial JSON, and multi-result pages need a different envelope than this object gate.

Who should not use this approach

Do not ship this module as your only control for a multi-tenant agent that can send mail, move money, or delete cloud resources. Do not use it as a substitute for OAuth scopes, idempotency keys, or human approval on irreversible tools. Do not apply it to exploratory notebooks where the point is to inspect raw payloads rather than to keep a durable memory.

Instructors should also avoid grading speed. The useful outcome is a red test on extra fields, not a student who “got the agent working” by widening the schema. If the table and the tests disagree, the tests win until the table is revised in the same commit.

Close

Unchecked tool JSON is a memory injection bug with a friendly syntax. Freeze the result contract, reject extras, and keep the raw payload out of the transcript. Rerun the unittest file whenever you add a tool, and treat every new key as a schema change rather than a convenience.