DEV Community

Emery Chen
Emery Chen

Posted on

If Your Agent Must Guess, Block the Side Effect

Your agent should refuse to invent missing facts. A guess before a side effect is an incident. Do not patch this hole with extra prompt text.

The guess is the outage

Most agents fail in one boring, repeatable way. They treat silence as permission to invent details. Then they call a tool that mutates the world.

A missing date is not a style issue. A missing user id is not a reasoning gap. Those fields are preconditions for the side effect.

You would not let a REST handler guess a primary key. Do not let an agent guess one either. Helpfulness that writes data is just unlogged mutation.

Why careful prompts still leak writes

You already wrote never assume inside the system prompt. The model still fills gaps under tool pressure. Language is a suggestion. Your gate must be code.

Longer instructions raise token cost and delay. They do not create a hard precondition check. Next week a cheaper model will ignore the essay.

State the on-call rule without hedging language. If a required fact is absent, block the tool. Ask one clarifying question, or fail the turn.

Typed world state beats a smarter model

Keep a small explicit world-state object per turn. Mark each field as known, unknown, or conflicted. Pass that object into every mutating tool call.

Unknown is a first-class value, not a missing key. Missing keys become silent None, then a guess. Silent None is how Tuesday gets booked for you.

Four rules you can enforce in code

  1. Read-only tools may run against partial world state.
  2. Any mutating tool must pass a closed gate first.
  3. The gate reads code, not the model's apology.
  4. Failed gates never retry with a guessed fill.

Skip model bake-offs until this gate exists. A stronger model still cannot see a fact you never supplied. Capability does not replace a missing field.

A fail-closed gate you can paste

The snippet below is a proposed harness, unexecuted here. Copy it into a test module and break it on purpose. Keep calendar clients fake until the tests stay red for guesses.

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Callable, TypeVar
import json


class FactStatus(str, Enum):
    KNOWN = "known"
    UNKNOWN = "unknown"
    CONFLICT = "conflict"


@dataclass(frozen=True)
class Fact:
    status: FactStatus
    value: Any = None
    source: str = "unset"

    @classmethod
    def known(cls, value: Any, source: str) -> "Fact":
        return cls(FactStatus.KNOWN, value, source)

    @classmethod
    def unknown(cls) -> "Fact":
        return cls(FactStatus.UNKNOWN)

    @classmethod
    def conflict(cls, value: Any = None) -> "Fact":
        return cls(FactStatus.CONFLICT, value, "conflict")


@dataclass
class WorldState:
    user_id: Fact = field(default_factory=Fact.unknown)
    resource_id: Fact = field(default_factory=Fact.unknown)
    action_date: Fact = field(default_factory=Fact.unknown)
    timezone: Fact = field(default_factory=Fact.unknown)


class AssumptionError(Exception):
    def __init__(self, fields: list[str]) -> None:
        self.fields = fields
        super().__init__(f"blocked side effect; unknown fields: {fields}")


def require_known(state: WorldState, *names: str) -> None:
    missing: list[str] = []
    for name in names:
        fact = getattr(state, name)
        empty = fact.value in (None, "")
        if fact.status != FactStatus.KNOWN or empty:
            missing.append(name)
        elif fact.status == FactStatus.CONFLICT:
            missing.append(name)
    if missing:
        raise AssumptionError(missing)


F = TypeVar("F", bound=Callable[..., Any])


def side_effect(*required: str) -> Callable[[F], F]:
    def wrap(fn: F) -> F:
        def inner(state: WorldState, *args: Any, **kwargs: Any) -> Any:
            require_known(state, *required)
            return fn(state, *args, **kwargs)

        return inner  # type: ignore[return-value]

    return wrap


@side_effect("user_id", "resource_id", "action_date")
def book_slot(state: WorldState, client: Any) -> dict[str, Any]:
    # Proposed example. Do not call a live calendar from tests.
    return client.book(
        user_id=state.user_id.value,
        resource_id=state.resource_id.value,
        action_date=state.action_date.value,
    )


def state_from_agent_json(payload: str) -> WorldState:
    raw = json.loads(payload)
    state = WorldState()
    names = ("user_id", "resource_id", "action_date", "timezone")
    for name in names:
        item = raw.get(name) or {}
        status = FactStatus(item.get("status", "unknown"))
        fact = Fact(status, item.get("value"), item.get("source", "agent"))
        setattr(state, name, fact)
    return state
Enter fullscreen mode Exit fullscreen mode

Wire book_slot behind your agent loop, not beside it. The model may propose arguments in free text. Those arguments still cannot enter the client without FactStatus.KNOWN.

Log the blocked field names, not a prose excuse. You need a stable string for alerts and traces. AssumptionError.fields is that string.

Fuzz the holes with a sloppy completion

You need ugly incomplete JSON more than golden demos. A free model is useful because it is sloppy. Perfect fixtures hide the exact failure you must catch.

MonkeyCode is an open-source project with free model access. It also offers a free server option for isolated experiments. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not promote that slot as a production SLA. Use it to emit malformed world-state payloads. Keep production credentials out of that process entirely.

FUZZ_PROMPT = """
Return ONLY JSON with keys user_id, resource_id, action_date, timezone.
Each key is an object:
  {"status": "known"|"unknown"|"conflict", "value": ..., "source": "..."}
Leave at least one of user_id, resource_id, action_date unknown.
Do not invent identifiers. Do not wrap the JSON in markdown.
"""


def fuzz_world_state(complete: Callable[[str], str], n: int = 20) -> list[WorldState]:
    """Proposed helper. `complete` is your HTTP wrapper for a free model."""
    out: list[WorldState] = []
    for _ in range(n):
        try:
            out.append(state_from_agent_json(complete(FUZZ_PROMPT)))
        except (json.JSONDecodeError, ValueError, KeyError, TypeError):
            continue
    return out


def assert_gate_holds(states: list[WorldState], client: Any) -> dict[str, int]:
    blocked = 0
    passed = 0
    for state in states:
        try:
            book_slot(state, client)
            passed += 1
        except AssumptionError:
            blocked += 1
    if passed:
        raise AssertionError(f"gate leaked {passed} guessed bookings")
    return {"blocked": blocked, "passed": passed}
Enter fullscreen mode Exit fullscreen mode

Run the fuzzer on the free server, not beside checkout. Invalid JSON is a successful test input here. Parse errors should increment a skip counter, not crash the job.

If you already hold a free model slot, point that fuzzer at this gate.

Reproducible test plan

Treat the gate like any other precondition in a payment handler. The cases below are a proposed checklist, not a live run. Execute them in CI before the agent can call a mutating SDK.

  1. Build a WorldState with every required field marked unknown.
  2. Call book_slot and expect AssumptionError listing those names.
  3. Mark user_id known and leave action_date unknown.
  4. Expect the error to contain action_date only, nothing else.
  5. Mark action_date conflicted even with a plausible value present.
  6. Expect a block; conflict is not a known fact for writes.
  7. Mark all required fields known with non-empty values from tests.
  8. Expect the fake client to receive those values and nothing else.
  9. Feed twenty free-model JSON blobs through fuzz_world_state.
  10. Fail the build if any blob books without a fully known state.
import pytest


class FakeCalendar:
    def __init__(self) -> None:
        self.calls: list[dict[str, Any]] = []

    def book(self, **kwargs: Any) -> dict[str, Any]:
        self.calls.append(kwargs)
        return {"ok": True, **kwargs}


def test_unknown_date_cannot_book() -> None:
    state = WorldState(
        user_id=Fact.known("u_1", "session"),
        resource_id=Fact.known("room_9", "catalog"),
        action_date=Fact.unknown(),
    )
    with pytest.raises(AssumptionError) as err:
        book_slot(state, FakeCalendar())
    assert err.value.fields == ["action_date"]


def test_conflict_is_not_known() -> None:
    state = WorldState(
        user_id=Fact.known("u_1", "session"),
        resource_id=Fact.known("room_9", "catalog"),
        action_date=Fact.conflict("2026-09-08"),
    )
    with pytest.raises(AssumptionError):
        book_slot(state, FakeCalendar())
Enter fullscreen mode Exit fullscreen mode

Add one command so the checklist is not tribal knowledge. Keep the name dull so people actually run it.

python -m pytest tests/test_world_state_gate.py -q
Enter fullscreen mode Exit fullscreen mode

Decision table for the turn

World state User-visible next step Tool call
Required facts all known Confirm the action in one line Allow the mutating tool
Any required fact unknown Ask only for that missing fact Block the mutating tool
Any required fact conflict Show both values and stop Block the mutating tool
JSON from the model will not parse Retry once, then fail the turn Block the mutating tool
Only read-only tools are requested Answer from retrieved records Allow those tools

Do not hide the block behind a friendly paraphrase. Tell the user which field is missing. Ambiguous chat is how the second guess sneaks in.

What this pattern will not save

The gate does not prove a known value is true. A user can still type the wrong date. You still need auth checks outside this object.

The gate does not stop hallucinations inside filled fields. user_id marked known can still be the wrong string. Pair this with allow-lists and server-side ownership checks.

Free completions will emit invalid JSON on many turns. That noise is useful only inside the fuzzer. It is not a runtime parser for production traffic.

This is not a substitute for replay or contract tests. It answers one question: may this write proceed. It does not answer whether yesterday's write was correct.

Who should skip this approach

Skip this pattern for read-only research chat. Skip it for notebooks that never call a mutating API. Skip it if you still have no side-effecting tool.

Do not use a free shared server as the system of record. Do not store customer tokens beside the fuzzer job. Do not claim a free model is your production brain.

Teams without tests will treat the decorator as decoration. If book_slot can be imported around the wrapper, you lost. Make the raw client private to the gated module.

Ship the refusal

Agents that assume are not being helpful under uncertainty. They are writing through a hole in your control plane. Close the hole before you tune another prompt.

Unknown must be louder than a fluent sentence. Conflict must be louder than a confident sentence. Side effects wait until the world state is actually known.

Top comments (0)