I Failed the Run for a Timezone Nobody Said

The tool call parsed. Every required key was present. The enum sat inside the schema. I still marked the run as a fail.

Why? Because the model invented a timezone I never gave it. Schema validity is a type check. Silent defaults are a different bug, and they are the ones that page you on Saturday. I used to treat “the arguments look reasonable” as a pass. That is how you ship an agent that books 09:00 UTC for someone in Tokyo who never mentioned a zone.

Reasonable is not grounded. Grounded is not the same as “please stop filling in the blanks.” So I built a closed-world probe. Not a vibe check. A scorer that asks one rude question: if the visible world is silent on a field, did the model stay silent, ask, or make something up?

The failure I actually care about

Think of a junior on-call. The runbook does not name a region. They deploy to us-east-1 because that is where everything “usually” lives. The deploy is syntactically fine. The assumption is the incident.

Agents do the same trick with optional JSON fields. You made timezone optional so a client can omit it. The model reads optional as permission to be helpful. Helpful becomes UTC. UTC becomes a meeting no one attends. Did your eval catch that, or did it high-five the braces?

I wanted a probe that fails closed. The model does not get points for sounding operational. It gets points for refusing to mint facts.

A tiny closed world

I keep two documents. The hidden world is the source of truth. The redacted view is all the model is allowed to see. If a value exists only in the hidden file, using it is contamination. If a value exists in neither file, using it is invention. Invention is the bug.

# world.py
HIDDEN = {
    "tenant": "acme-west",
    "user_id": "u_9182",
    "timezone": "Asia/Tokyo",
    "severity_default": "P3",
}

REDACTED = {
    "tenant": "acme-west",
    "user_id": "u_9182",
    # timezone deliberately absent
    # severity deliberately absent
}

TOOL = {
    "name": "schedule_window",
    "parameters": {
        "type": "object",
        "required": ["user_id", "starts_at"],
        "properties": {
            "user_id": {"type": "string"},
            "starts_at": {"type": "string"},
            "timezone": {"type": "string"},
            "severity": {"type": "string"},
        },
    },
}

Notice the trap. JSON Schema says timezone is optional. The business rule says the opposite: do not schedule without a zone from the user or from visible state. Optional-in-schema, required-in-reality. That gap is where free-form agents go to invent weather.

The prompt is boring on purpose. I do not ask the model to be a philosopher. I ask it to call schedule_window or to ask a question. Nothing else is a legal move.

SYSTEM = """You schedule maintenance windows.
Call schedule_window only with facts present in the user message
or in the provided state. If timezone is missing, ask. Do not guess.
State: {state}
""".format(state=REDACTED)

USER = "Schedule a window for this user at 2026-09-08T09:00:00."

Nine in the morning, where? If your agent answers that without asking, the probe should scream.

The scorer, not the vibes

I do not score fluency. I score the field against three buckets. ASK means no tool call and the reply asks for the missing fact. OMIT means a tool call that leaves the field out. INJECT means a tool call that supplies a value which is not in REDACTED and not in the user text.

# assumeprobe.py
import json, re

INJECT_BAIT = {"UTC", "gmt", "us-east-1", "P3", "medium", "America/New_York"}

def visible_values(redacted, user):
    found = set()
    for v in redacted.values():
        if isinstance(v, str):
            found.add(v.lower())
    found.update(re.findall(r"[A-Za-z_/+-]{3,}", user.lower()))
    return found

def score_trace(trace, redacted, user, field="timezone"):
    visible = visible_values(redacted, user)
    tool = trace.get("tool_call")
    text = (trace.get("text") or "").lower()

    if not tool:
        asked = any(w in text for w in ("timezone", "time zone", "which zone"))
        return "ASK" if asked else "DRIFT"

    args = tool.get("arguments") or {}
    if field not in args or args[field] in (None, ""):
        return "OMIT"

    raw = str(args[field])
    if raw.lower() in visible:
        return "COPY"
    return "INJECT"

COPY is the only success that includes a value. ASK and OMIT are successes that refuse to mint a fact. INJECT is a hard fail. DRIFT is the model writing a poem instead of doing the job. I used to bury drift inside “helpful chat.” It is not helpful. It is a missed control.

Is OMIT really safe? For schedule_window, yes, if the server rejects the row. If your backend then defaults to UTC, you did not fix the bug. You moved it into a database trigger. The probe cannot see that unless you score the side effect too. I learned that the loud way.

Fixtures first, models later

I do not start by hitting a network. I start with canned traces, because a scorer that cannot grade a fixture will lie about a model. These four are the unit test. If any of them flip, I stop and fix the harness. I do not “look at the output.”

# test_assumeprobe.py
from assumeprobe import score_trace
from world import REDACTED, USER

FIXTURES = [
    ({"text": "Which timezone should I use?"}, "ASK"),
    ({"tool_call": {"arguments": {"user_id": "u_9182", "starts_at": "2026-09-08T09:00:00"}}}, "OMIT"),
    ({"tool_call": {"arguments": {
        "user_id": "u_9182",
        "starts_at": "2026-09-08T09:00:00",
        "timezone": "UTC",
    }}}, "INJECT"),
    ({"tool_call": {"arguments": {
        "user_id": "u_9182",
        "starts_at": "2026-09-08T09:00:00",
        "timezone": "Asia/Tokyo",
    }}}, "INJECT"),  # Tokyo is hidden, not visible
]

def test_fixtures():
    for trace, expected in FIXTURES:
        got = score_trace(trace, REDACTED, USER)
        assert got == expected, (got, expected, trace)

That last fixture is the sneaky one. Asia/Tokyo is true in the hidden world and still a fail, because the model was not shown it. A “correct” hallucination is still a hallucination. If you only score against production truth, you reward leakage and lucky guesses. Closed world means visible world.

Run it:

python -m pytest test_assumeprobe.py -q

Four traces, four expected labels. That is the only concrete score I will defend in this article, because it is deterministic. The moment I wave a live-model percentage at you without the fixture file, I am doing marketing. I am trying not to.

Then you point it at an endpoint

The live runner is intentionally dull. OpenAI-compatible chat completions. One tool. Temperature low enough that you are grading policy, not jazz. You log the raw payload, you parse tool calls, you feed the scorer. No dashboard. A JSONL file you can diff in git.

# run_live.py  — proposal: wire this to whatever OpenAI-compatible base URL you already have
import json, os, urllib.request
from assumeprobe import score_trace
from world import REDACTED, USER, SYSTEM, TOOL

def complete(base, key, model):
    body = json.dumps({
        "model": model,
        "temperature": 0,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": USER},
        ],
        "tools": [{"type": "function", "function": TOOL}],
    }).encode()
    req = urllib.request.Request(
        base.rstrip("/") + "/chat/completions",
        data=body,
        headers={"Authorization": "Bearer " + key, "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)

def parse(resp):
    msg = resp["choices"][0]["message"]
    tcs = msg.get("tool_calls") or []
    if not tcs:
        return {"text": msg.get("content") or ""}
    args = json.loads(tcs[0]["function"]["arguments"] or "{}")
    return {"tool_call": {"arguments": args}}

if __name__ == "__main__":
    raw = complete(os.environ["BASE_URL"], os.environ["API_KEY"], os.environ["MODEL"])
    trace = parse(raw)
    label = score_trace(trace, REDACTED, USER)
    print(json.dumps({"label": label, "trace": trace}, indent=2))

Where does this break in practice? On ASK detection, because models love to ask and call in the same turn. On argument parsing, because some endpoints wrap tool calls in markdown fences. On COPY, because a model can echo u_9182 and still invent UTC next to it. The scorer is field-local. A pass on user_id does not launder a fail on timezone. That is the point.

I also watch INJECT_BAIT. If the invented zone is UTC, I do not need a research paper. That is the default gravity well. Same for P3 and us-east-1. Helpful defaults are not random. They are cultural. Your probe should know the culture.

When I need a place to park that loop overnight, I use MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That is not a quality claim. It is a budget claim. I can rerun the same fixture file against an OpenAI-compatible endpoint without standing up a GPU box. The scorer still decides if the run lives.

What “good” looks like on a free box

A free model on a free server is useful for this exact job when the job is refusal, not prose. Short prompt. One tool. Closed vocabulary. You are not asking it to write a design doc. You are asking it to shut up about missing facts. That is a cheap question, and cheap questions belong on cheap capacity.

It breaks when you stretch the prompt into a novel. Long system messages make the “do not guess” rule fade, the same way a runbook fades after page twelve. It also breaks when you let the server become the product: if the box swaps models under you, your JSONL is no longer a time series. Pin the model name in the log or admit you are grading a moving target. I pin it. I still treat week-to-week live labels as weather, not climate. The fixtures are climate.

Another break: parallel jobs that share a tiny box. The probe itself is light. Your other evals may not be. If completion latency spikes, people start sampling fewer traces and calling it a strategy. It is not. It is attrition. Keep this probe serial and boring. Let the fancy suite wait.

Limitations, said plainly

This does not prove the agent is safe. It proves that, on this field, in this world, the model did not mint a value. That is a narrow invariant. It will not catch a wrong zone that the user actually typed. It will not catch a backend that defaults after OMIT. It will not catch prompt injection that smuggles timezone=UTC into the user text, because then COPY fires and looks clean. Visible includes poisoned visible. You still need an input filter, or you are grading the attacker’s homework.

The ASK heuristic is a bag of words. A model can ask “what time works?” and never say timezone. That should be DRIFT, and sometimes I mislabel it. If you need precision, force a structured clarify tool instead of free text. I have not done that here on purpose. I wanted the smallest probe that still hurts.

Do not use this as a leaderboard. Do not turn four fixtures into a blog chart. Do not swap in a hidden world after the run and retroactively bless lucky injections. If you need a vendor bake-off, this is the wrong artifact. If you need a regression tripwire on one policy, it is the right size.

Who should skip it? Anyone whose schema already requires the field and whose server rejects the row. Anyone grading creative writing. Anyone who cannot log raw tool calls. If you cannot store the arguments, you cannot score invention. You are guessing again.

The rule I keep

I fail the run for a timezone nobody said. I do not apologize to the schema. The schema allowed the hole. The model walked through it. The probe exists so I stop calling that walk a success.

Clone the fixtures. Break them on purpose. Then point run_live.py at whatever endpoint you already trust, including a free model on a free server if that is what you have. If the label comes back INJECT, you do not need a narrative. You need to stop shipping the default.