Workshop: Reject Incomplete Agent Plans Before Side Effects in 80 Minutes

Incomplete agent plans fail more cheaply than half-executed tool chains that already produced irreversible side effects in production. This workshop treats a model-produced plan as untrusted structured data, then rejects missing fields before any tool function runs. Students leave with a JSON schema, a small Python validator, and a pytest suite they can rerun against later agents.

Who should run this lab

You already wire small agents to HTTP APIs, local files, or internal CLIs driven by model output. You do not need a full orchestration platform, a GPU cluster, or a paid evaluation vendor for the exercises. You do need a repeatable check that catches invented record identifiers, missing idempotency keys, and silent default values.

Skip the lab if your agents only draft text that a human must approve before anything executes. Also skip it if every tool already sits behind a true compensating transaction that operators trust in failure. Those environments already have a human or transactional gate, so a plan schema adds ceremony without new information.

The rule on the whiteboard

A plan is executable only when every required field is present, typed, and sourced from a non-model origin. Missing source is classified as an assumption, and assumptions fail closed in this teaching design. The model may propose an ordered list of steps; the validator alone decides whether those steps may leave the dry-run sandbox.

Learning objectives

Write these four outcomes where the group can see them during the session.

1. Represent an agent plan as JSON that a draft-2020-12 schema can reject without calling tools.
2. Distinguish user-supplied or system-supplied facts from values the model invented during planning.
3. Run a dry-run logger that records intended tool names without performing any network I/O.
4. Add pytest cases that fail when a required field is inferred instead of referenced from facts.

Timing box (80 minutes)

Block	Minutes	Facilitation outcome
0. Inventory one real tool	10	A written list of required fields and side effects
1. Freeze the plan contract	15	Schema file committed beside the client
2. Implement the validator	15	Reject path returns structured errors
3. Lock behavior with tests	15	Two failing fixtures turned green
4. Break the fixture on purpose	15	Students show which assumption leaked
5. Debrief limits	10	List of cases this method will not catch

Keep a visible timer on the projector so schema debate cannot consume the implementation block. If the group is still arguing about property names at minute twenty, freeze the draft below and move on. Aesthetic disagreement is cheaper after the reject path is demonstrated with a real fixture.

Exercise 0 — inventory one real tool (10 minutes)

Pick a single tool the agent is allowed to call, not a catalog of twenty integrations. Write four lines on paper or in a gist before any model is invoked. The inventory is the lab's ground truth, and later JSON must not invent fields the inventory never named.

• Tool name as the runtime will see it, for example create_support_credit.
• Required arguments that must never be guessed, such as account_id and idempotency_key.
• Side effects that survive a process crash, such as an emailed credit or a ledger write.
• The human or system source that can legally supply each required argument.

If you cannot name the source for an argument, that argument is already an assumption. Do not proceed to schema work until the source column is filled or the tool is removed from the lab. Removing a tool is a successful inventory outcome, not a facilitation failure.

The plan contract students will reuse

Save the following document as agent_plan.schema.json. It is intentionally small so a class can read every keyword aloud. Larger catalogs belong in a later session after this gate produces a red test on invented identifiers.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AgentPlan",
  "type": "object",
  "additionalProperties": false,
  "required": ["goal", "facts", "steps", "dry_run"],
  "properties": {
    "goal": { "type": "string", "minLength": 8 },
    "dry_run": { "type": "boolean" },
    "facts": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["id", "key", "value", "source"],
        "properties": {
          "id": { "type": "string", "pattern": "^fact_[a-z0-9_]+$" },
          "key": { "type": "string", "minLength": 1 },
          "value": { "type": ["string", "number", "boolean"] },
          "source": { "enum": ["user", "system", "retrieved"] }
        }
      }
    },
    "steps": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["id", "tool", "args", "fact_refs", "on_failure"],
        "properties": {
          "id": { "type": "string", "pattern": "^step_[0-9]+$" },
          "tool": { "type": "string", "minLength": 1 },
          "args": { "type": "object" },
          "fact_refs": {
            "type": "array",
            "minItems": 1,
            "items": { "type": "string" }
          },
          "on_failure": { "enum": ["abort"] }
        }
      }
    }
  }
}

Three design choices are load-bearing and should be spoken before anyone edits the file. source cannot be model, because invented facts must not enter the table that later binds arguments. on_failure is only abort in the teaching subset, which prevents students from encoding retry poetry inside the plan. dry_run is a required boolean so a missing flag cannot default to live execution during class.

Worked example students can rerun

The scenario is a support credit that looks like a refund but never talks to a payment network. The user supplied an account identifier, and the billing system supplied an idempotency key. The model must not invent either value, including look-alike identifiers that merely resemble the sourced ones.

Save this as fixtures/valid_plan.json.

{
  "goal": "Issue a $15 support credit for a documented billing defect",
  "dry_run": true,
  "facts": [
    {
      "id": "fact_account",
      "key": "account_id",
      "value": "acct_1044",
      "source": "user"
    },
    {
      "id": "fact_key",
      "key": "idempotency_key",
      "value": "cred-1044-20260905",
      "source": "system"
    },
    {
      "id": "fact_amount",
      "key": "amount_usd",
      "value": 15,
      "source": "retrieved"
    }
  ],
  "steps": [
    {
      "id": "step_1",
      "tool": "create_support_credit",
      "args": {
        "account_id": "acct_1044",
        "idempotency_key": "cred-1044-20260905",
        "amount_usd": 15
      },
      "fact_refs": ["fact_account", "fact_key", "fact_amount"],
      "on_failure": "abort"
    }
  ]
}

Save fixtures/assumed_plan.json next. Keep the facts identical, keep dry_run true, and change only args.account_id to acct_0000 so the binding checker has a definite mismatch.

{
  "goal": "Issue a $15 support credit for a documented billing defect",
  "dry_run": true,
  "facts": [
    {
      "id": "fact_account",
      "key": "account_id",
      "value": "acct_1044",
      "source": "user"
    },
    {
      "id": "fact_key",
      "key": "idempotency_key",
      "value": "cred-1044-20260905",
      "source": "system"
    },
    {
      "id": "fact_amount",
      "key": "amount_usd",
      "value": 15,
      "source": "retrieved"
    }
  ],
  "steps": [
    {
      "id": "step_1",
      "tool": "create_support_credit",
      "args": {
        "account_id": "acct_0000",
        "idempotency_key": "cred-1044-20260905",
        "amount_usd": 15
      },
      "fact_refs": ["fact_account", "fact_key", "fact_amount"],
      "on_failure": "abort"
    }
  ]
}

That second fixture is the teaching failure the room should be able to rerun after the session. Students should see a structured reject that names account_id, not a tool log line that implies the credit was staged.

Validator (copy into plan_gate.py)

Schema validity is necessary and still insufficient, because argument values must match sourced facts. Run schema errors first so a missing facts array cannot crash the binding loop with a KeyError during class.

from __future__ import annotations

import json
from pathlib import Path
from typing import Any

from jsonschema import Draft202012Validator

SCHEMA = json.loads(Path("agent_plan.schema.json").read_text())


class PlanRejected(ValueError):
    pass


def load_plan(path: str | Path) -> dict[str, Any]:
    return json.loads(Path(path).read_text())


def validate_schema(plan: dict[str, Any]) -> list[str]:
    validator = Draft202012Validator(SCHEMA)
    return sorted(
        f"{list(err.path)}: {err.message}" for err in validator.iter_errors(plan)
    )


def validate_bindings(plan: dict[str, Any]) -> list[str]:
    facts = {item["id"]: item for item in plan["facts"]}
    errors: list[str] = []
    for step in plan["steps"]:
        for ref in step["fact_refs"]:
            if ref not in facts:
                errors.append(f"{step['id']} references missing fact {ref}")
        bound = {
            facts[ref]["key"]: facts[ref]["value"]
            for ref in step["fact_refs"]
            if ref in facts
        }
        for key, value in step["args"].items():
            if key not in bound:
                errors.append(f"{step['id']} arg {key} has no sourced fact")
            elif bound[key] != value:
                errors.append(
                    f"{step['id']} arg {key}={value!r} does not match sourced {bound[key]!r}"
                )
        extra_keys = set(bound) - set(step["args"])
        if extra_keys:
            errors.append(
                f"{step['id']} fact keys unused in args: {sorted(extra_keys)}"
            )
    return errors


def gate_plan(plan: dict[str, Any]) -> dict[str, Any]:
    schema_problems = validate_schema(plan)
    if schema_problems:
        raise PlanRejected("; ".join(schema_problems))
    binding_problems = validate_bindings(plan)
    if binding_problems:
        raise PlanRejected("; ".join(binding_problems))
    if plan["dry_run"] is not True:
        raise PlanRejected("live execution is out of scope for this workshop")
    return {
        "status": "dry_run_accepted",
        "tools": [step["tool"] for step in plan["steps"]],
        "fact_ids": [fact["id"] for fact in plan["facts"]],
    }


if __name__ == "__main__":
    import sys

    target = load_plan(sys.argv[1])
    print(json.dumps(gate_plan(target), indent=2))

Commands the room can run together after the files exist:

python -m pip install 'jsonschema>=4.18' pytest
python plan_gate.py fixtures/valid_plan.json
python plan_gate.py fixtures/assumed_plan.json

The first command should print dry_run_accepted and the tool name only. The second should raise PlanRejected and mention the mismatched account_id. If both commands succeed, the binding check is not wired, and the lab is not complete.

Tests that lock the teaching point

Save the following as test_plan_gate.py. The third test exists because a sourced plan with dry_run flipped to false is still out of scope for an 80-minute room.

import pytest
from plan_gate import PlanRejected, gate_plan, load_plan


def test_sourced_plan_is_accepted_for_dry_run():
    result = gate_plan(load_plan("fixtures/valid_plan.json"))
    assert result["status"] == "dry_run_accepted"
    assert result["tools"] == ["create_support_credit"]


def test_invented_account_id_is_rejected():
    plan = load_plan("fixtures/assumed_plan.json")
    with pytest.raises(PlanRejected) as err:
        gate_plan(plan)
    assert "account_id" in str(err.value)


def test_live_flag_is_rejected_even_when_facts_match():
    plan = load_plan("fixtures/valid_plan.json")
    plan["dry_run"] = False
    with pytest.raises(PlanRejected, match="live execution"):
        gate_plan(plan)

pytest -q test_plan_gate.py

Expect three passing tests after assumed_plan.json contains the conflicting account identifier. If the invented identifier accidentally matches a fact value, the second test will fail open and teach the wrong lesson. Check the fixture before debugging pytest itself.

Exercise 1 — generate, then gate (15 minutes)

Point any model client at a prompt that asks for JSON matching the schema. Do not paste the schema into a system prompt and hope for compliance as the only control. Persist the raw model output as fixtures/model_plan.json, then run python plan_gate.py fixtures/model_plan.json before anyone discusses prompt wording.

Typical reject classes in a first pass:

• source set to a string such as "assistant" or omitted entirely from a fact object
• fact_refs empty while args still contain identifiers the model copied from the prompt
• dry_run missing, which the schema must fail before the Python binding checks run
• extra keys such as "confidence" that additionalProperties: false should block

Record the reject string in the lab notes before rewriting the prompt. Prompt iteration comes after the gate is red, not before, because otherwise the class cannot tell prompt luck from contract enforcement.

Exercise 2 — break one sourced fact (10 minutes)

Change fact_amount from 15 to 20 while leaving args.amount_usd at 15. Rerun pytest and confirm the binding error names step_1. This is the regression students should keep when product language later asks the model to round the credit up. A sourced number that no longer matches the argument is still an assumption, even when both values are plausible.

Exercise 3 — add a second tool without expanding permissions (10 minutes)

Add notify_account_email as a second step that needs account_id and a retrieved email. If the email is not in facts with source of retrieved or user, the gate must reject the whole plan. Do not special-case notifications as harmless, because email is a side effect with an audience outside the lab process.

Where a shared free model and server fit

The gate is local Python and does not require a vendor to be useful in a single-laptop lab. When a class needs one place to host the files, run pytest, and call a model for Exercise 1, a shared lab server reduces setup drift across student machines.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option that can host this lab workspace. Treat both as convenience for the workshop, not as evidence that the gate is unnecessary. Keep the schema and tests in git even if the model client later moves to another host.

If you rerun the fixtures on that stack, keep agent_plan.schema.json identical so reject strings stay comparable across students.

What this method does not catch

Read this list before you export the lab as a team standard. A passing dry-run is not an authorization decision, and it is not a semantic proof that the sourced values were the right customer.

• Semantically wrong but well-sourced values, such as a user-supplied account that belongs to a different customer
• Tools that are allowed by name but unsafe in combination, which needs an allow-list beyond this schema
• Prompt injection that arrives inside a retrieved fact; retrieval still requires its own trust policy
• Non-JSON model output, which should fail at parse time before gate_plan runs
• Performance, cost, or latency budgets; this workshop does not meter tokens or queue time
• Authorization checks: a sourced account_id is not the same as a checked permission

Teams that already have signed webhooks, two-phase commits, and human approval on every mutating tool will find little new here. Teams that let a model choose SQL, shell, or credit amounts inline should not treat this schema as a complete control plane. The artifact is a cheap preflight, not a substitute for tool sandboxing.

Facilitation notes

Keep live execution disabled for the whole 80 minutes, including the last ten-minute debrief. The dry_run flag is a teaching brake, not a production safety certification, and students will try to flip it once the tests are green. If someone wants to hit a real credit API, pause and replay the inventory from Exercise 0, because the side-effect column is the point of the session.

The original artifact is the triad of schema, binding checker, and the invented-identifier fixture. If you change the domain from support credits to file writes, keep that triad intact. Replace tool names, keep source as an enum without model, and keep pytest as the graduation criteria for the workshop.