Postmortem: A Free Route Replayed a Paid Tool Grant

A free model route must never inherit a paid tool grant. This reconstruction shows how a missing tier field certified a forbidden shell call. The durable fix binds every grant to model class, server class, and task id.

Scope of this write-up

No production log from a named customer supports this timeline. The sequence is a lab fixture for reviewers who gate AI tool calls. Treat every timestamp below as fixture time, not an outage clock.

What broke

The orchestrator kept a tool grant after a route change. The first hop used a paid-class model on a paid-class server. A capacity miss then moved the same task to a free model route.

That route also selected the free server option. The stored grant still allowed a shell tool. Policy for the free route forbade shell tools.

The runner trusted the grant and started the command. The command was still the approved dry-run text. The policy break was the route, not a new argument string.

Fixture timeline

The fixture clock starts at 09:00 local lab time. Each line below is a lab event, not a status page.

• At 09:00 a task asks for a migration plan only.
• At 09:01 the planner binds a paid model and paid server.
• At 09:02 a human approves a shell grant for a dry-run.
• At 09:03 the grant stores tool name, expiry, and task id.
• At 09:04 the grant omits model class and server class.
• At 09:07 the paid route returns a capacity miss.
• At 09:08 failover selects free model access and the free server.
• At 09:09 the runner loads the prior grant by task id.
• At 09:10 the free route executes the shell tool.
• At 09:11 the dry-run command changes no stored data.
• At 09:12 the audit log marks the call policy-clean.
• At 09:14 a reviewer notices the server class on the span.
• At 09:20 the route halts and the grant issuer freezes.

Impact

No customer database was migrated in the fixture. The real impact is a false certification of the call. A forbidden tool call looked approved by policy.

A later real migrate could reuse the same hole. That is why the audit mark matters more than the dry-run result. Severity follows the tool class, not the luck of a dry-run.

Contributing factors

Several small omissions stacked into one false allow. None of those omissions alone was the whole bug.

• The grant lookup key used the task id only.
• Model class was display text, not a verified field.
• Server class was chosen only after grant issue.
• Failover treated a capacity miss as a pure transport retry.
• The policy engine checked tool name, not route class.
• The free route and the paid route shared one runner pool.
• Expiry was twenty minutes, so the grant was still fresh.
• The dry-run success hid the underlying policy break.
• Tests covered grant expiry, not a tier change.
• Logs stored the tool name without the server class.

What did not cause it

A stale cache entry did not certify the call. A foreign eval receipt was not replayed here. A warm workspace did not pin an old commit.

A lockfile from another volume was not reused. The bug was privilege scope, not artifact freshness. Refreshing files would not have closed this hole.

Durable fix

Bind the grant to four fields before any tool call. Reject the tool call when any bound field differs. Do not refresh the grant in place during failover.

Issue a new grant on the new route, or stop. Expiry remains in the record as a time check. Expiry alone is not a valid scope check.

Bound fields

1. Task id must match the live route.
2. Model class must match the live route.
3. Server class must match the live route.
4. Tool name plus argument hash must match.

Decision table

Use this table at the runner trust boundary. A single failed row denies the tool call.

Check	Match	Action
Task id	No	Deny and alert
Model class	No	Deny and reissue
Server class	No	Deny and reissue
Tool name	No	Deny
Argument hash	No	Deny
Expiry	Past	Deny
All fields	Yes	Allow once

A reissue still needs a fresh human approval for shell tools. Automatic reissue is only for read-only tools. That split belongs in policy, not in the runner.

Lab fixture

The following Python is a labeled, unexecuted proposal. It was not run against a production control plane. It encodes the reject rule for local review.

from hashlib import sha256
import time

def arg_hash(args: dict) -> str:
    blob = repr(sorted(args.items())).encode()
    return sha256(blob).hexdigest()

def accept_grant(grant: dict, route: dict, now: float) -> str:
    if now > grant["exp"]:
        return "deny_expired"
    if grant["task_id"] != route["task_id"]:
        return "deny_task"
    if grant["model_class"] != route["model_class"]:
        return "deny_model_class"
    if grant["server_class"] != route["server_class"]:
        return "deny_server_class"
    if grant["tool"] != route["tool"]:
        return "deny_tool"
    if grant["arg_hash"] != arg_hash(route["args"]):
        return "deny_args"
    return "allow"

def test_free_route_rejects_paid_grant() -> None:
    now = time.time()
    args = {"cmd": "migrate --dry-run"}
    grant = {
        "task_id": "t-17",
        "model_class": "paid",
        "server_class": "paid",
        "tool": "shell",
        "arg_hash": arg_hash(args),
        "exp": now + 600,
    }
    route = {
        "task_id": "t-17",
        "model_class": "free",
        "server_class": "free",
        "tool": "shell",
        "args": args,
    }
    assert accept_grant(grant, route, now) == "deny_model_class"

def test_server_class_mismatch_is_denied() -> None:
    now = time.time()
    args = {"cmd": "migrate --dry-run"}
    grant = {
        "task_id": "t-17",
        "model_class": "free",
        "server_class": "paid",
        "tool": "shell",
        "arg_hash": arg_hash(args),
        "exp": now + 600,
    }
    route = {
        "task_id": "t-17",
        "model_class": "free",
        "server_class": "free",
        "tool": "shell",
        "args": args,
    }
    assert accept_grant(grant, route, now) == "deny_server_class"

Save the test under test_grant_scope.py before running it. Run the file with pytest in quiet mode from that folder. The expected result is two passing local tests.

python -m pytest -q test_grant_scope.py

A pass means the class mismatch is denied. A pass does not measure model output quality. Wire the same function into the runner before trusting CI.

Workflow after the fix

Use this order on every capacity failover event. Skip no step because the prior hop succeeded.

1. Stop the tool runner for that task first.
2. Read model class and server class from the new route.
3. Compare those two fields with the stored grant.
4. Deny the tool call on any field mismatch.
5. Drop the old grant without extending its expiry.
6. If the tool is read-only, request a new grant.
7. If the tool writes or shells, require human approval again.
8. Log both class pairs on the deny span.
9. Add the mismatch case to the nightly fixture.
10. Resume planning only after those checks all pass.

Where the free path fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Either path can appear during a capacity failover.

This note claims no token quota and no hardware shape. It also claims no duration and no benchmark. Current product docs remain the source for limits.

Use the free path as the mismatch target in the fixture. Point a paid-class grant at a free-class route. Confirm the runner returns a model-class or server-class deny.

Readers with that free path can replay the fixture before the next drill. Keep the written result beside the failover runbook. Do not treat a blog note as a quota promise.

Detection

Alert when a tool span's server class differs from the grant. Alert when a shell tool runs after failover. Keep each alert local to the single task.

Do not page on every free-route completion event. Page only when a denied shell tool was nearly allowed. Sample spans if the free route volume is high.

Fields to keep

• Record the task id on every tool span.
• Record grant model class beside route model class.
• Record grant server class beside route server class.
• Record the tool name and the decision code.

A dashboard that shows only tool name will miss this bug. Include both class pairs on every tool span.

Approval record

Shell grants need a separate approval record too. Store approver id, time, tool, and argument hash. Do not copy an approval onto a new route.

A new route needs a new approval row. Read-only tools can skip that human row. Document that skip inside the team policy file.

Canonical arguments

Sort object keys before hashing the tool arguments. Reject tool arguments that contain non-finite numeric values. Reject tool arguments that include raw secret values.

Hash the canonical bytes, not a Python repr, in production. Keep the repr hash only inside this lab fixture. Document that gap next to the test name.

Rollout

Ship the binder beside the old checker for one release. Log deny decisions without blocking for two lab days. Compare those deny counts with known failover drills.

Block only after the mismatch test is green in CI. Remove the old task-id-only lookup in the same change. Do not leave both lookups active after rollout.

Rollback

If the binder rejects every call, check label sources. A bad scheduler label can deny all free routes. Roll back the enforce flag, not the log field.

Keep writing both class pairs while the flag is off. Fix the class labels before enabling enforcement again.

Limitations

This fixture uses two class labels, free and paid. Real fleets may use more than two classes. The argument hash uses repr on sorted items.

That choice is stable only for simple values. Nested objects may need canonical JSON instead of repr. The local test does not start a server.

The local test does not call a model. A green test does not prove the live runner is wired. Human approval can still be rubber-stamped in review.

The dry-run in the story did not touch data. A write tool would raise the incident severity. No latency or cost numbers are claimed here.

Who should skip this

Skip this pattern when any item below is true.

• The team has no written tool policy yet.
• Grants are not stored, so nothing can be bound.
• The goal is a quality score for a free model.
• The write-up must cite a real outage with vendor logs.
• Legal review forbids even dry-run shell grants.

Residual risk

A correct binder still fails if the route labels are lies. If the free server reports class paid, the check passes. Class labels must come from the scheduler, not model text.

Rotate the grant signing key after the fix ships. Old unsigned grants should expire without any renewal. Review the signer during the same change window.

Closing

Tool privilege must die when the route changes. Freshness of a grant is not fitness for a route. Bind class fields, deny mismatches, and reissue only under policy.

Replay the fixture whenever failover rules change again. Keep the free route in that drill as the mismatch case. Stop the drill if class labels come from model prose.