DEV Community: Alexey Spinov

You Can't Patch Prompt Injection. Gate the Lethal Trifecta Before the Agent Runs.

Alexey Spinov — Tue, 30 Jun 2026 07:39:00 +0000

The lethal trifecta closes when one agent can read untrusted input, read private data, and send it out on a shared context. You can't patch that in the model, so gate it before the agent runs. trifecta_gate.py checks reachability on the manifest: the vulnerable fixture returns 2 paths (exit 1); the safe one, same capabilities, returns 0 (exit 0).

AI disclosure: I wrote trifecta_gate.py with an AI assistant and ran it myself, offline, before publishing. Every number in the output blocks below is pasted from a real local run on Python 3.13.5, stdlib only, on the synthetic manifests included in this post. I checked the exit codes (0 / 1 / 2), hashed the STDOUT twice to confirm it is byte-for-byte deterministic, and edited every line. The external figures (Tenet Security's Agentjacking numbers, Simon Willison's term) are theirs, not mine, and I link the primary sources. I label which numbers are theirs and which are mine.

In short:

Prompt injection is not a bug you patch in the model. The model cannot reliably tell trusted instructions from untrusted data, so the fix is not "detect the injection." The fix is to stop the dangerous capability combination from being reachable in the first place.
The lethal trifecta (a term coined by Simon Willison) is the combination: in one session the agent can read untrusted input, read private data, and send data outside. When all three reach each other, injected text can read your secrets and mail them out.
trifecta_gate.py reads a static tool manifest and asks one question by graph reachability: is there a path where untrusted input reaches a private read, and that read reaches an egress sink?
The key result: a safe manifest and a vulnerable one declare the same three capabilities. The vulnerable one returns 2 paths and exit 1. The safe one isolates egress off the shared context and returns 0 paths and exit 0. The gate decides on the data-flow graph, not on a checklist of flags.
Stdlib only (json, sys, collections.deque). No network, no model, no subprocess. The run is byte-for-byte deterministic. The tool and all three manifests are in this post.

The incident that makes this concrete

In June 2026, Tenet Security disclosed an attack they call Agentjacking. The mechanics are almost insultingly simple. An attacker pushes a fake error event into a Sentry project using a public DSN. The Sentry MCP server feeds that error to an AI coding agent as a real bug to fix, with malicious instructions hidden in the message body as a fake ## Resolution section. The agent reads it, believes it, and runs attacker text against the developer's machine.

Tenet's figures: passive recon found 2,388 organizations with valid injectable DSNs, 71 of them in the Tranco top one million. In controlled testing against more than a hundred consenting organizations, 100+ agents acted on the injected errors with an 85% exploitation success rate, against Claude Code, Cursor, and Codex among others. What got pulled in their tests: AWS secret keys, GitHub OAuth tokens, SSH agent sockets, Kubernetes tokens, ~/.aws/config, ~/.npmrc. Those are Tenet's numbers, from their writeup, not mine (Tenet Security, Agentjacking).

The part I keep rereading is Sentry's response. They acknowledged it and declined to fix it at the root, calling it "technically not defensible" and noting that model vendors run middleware against it. Read that again. The platform holding the untrusted input is telling you, correctly, that they cannot fix this for you. The model vendor's middleware catches some of it. Neither owns the actual hole, which is the shape of your agent: untrusted in, private read, egress out, all on one bus.

That is the lethal trifecta, and Agentjacking is just one well-documented instance of it.

Why you can't patch this in the model

Simon Willison named the lethal trifecta on 16 June 2025: access to private data, exposure to untrusted content, and the ability to communicate externally. His point about why prompt injection resists a model-level fix is the foundation here. In his words, "LLMs are unable to reliably distinguish the importance of instructions based on where they came from," and "we still don't know how to 100% reliably prevent this from happening." His conclusion is to avoid the combination rather than trust a guardrail to catch every attack. Those positions are his, and I am building on them.

I will say the uncomfortable version plainly. A guardrail that catches 95% of injection attempts sounds great until you remember that an attacker retries. In security, 95% is a failure rate, not a pass rate. If the only thing standing between untrusted text and your AWS keys is a classifier that is wrong one time in twenty, you do not have a control. You have a coin that an attacker gets to flip until it lands their way.

So the lever moves. Not "detect the injection better." That is tracking, and tracking is the thing I keep arguing against in this series. The lever is to gate the capability composition before the agent runs, so that even a successful injection has nowhere to send what it steals.

The claim, sharp enough to argue with

Here is the falsifiable version: the danger is not the presence of three capabilities, it is their reachability across one shared context, and you can decide that statically from the manifest before the agent starts.

If that claim is wrong, two things would have to be true. First, isolating one leg (taking egress off the shared bus) would not reduce the risk. Second, "all three capabilities present" would always mean "exploitable," regardless of how data flows between the tools. The safe fixture below is a single counter-example to both: three capabilities present, zero reachable paths, and a concrete reason why.

The mechanism worth internalizing is the shared context bus. In a default agent, every tool's output lands in the same LLM context, and that context steers the next tool call. So untrusted text read by one tool can influence any later tool. That is why the three capabilities are not three independent checkboxes. They are nodes on a graph, wired together by the context they share. Willison's own mitigation is to remove a leg from the shared context, for instance by running egress in a separate sandboxed sub-agent that never sees the tainted context. The gate is just a way to check, mechanically, whether you actually did that.

What the lethal trifecta gate checks, and how

The input is one JSON file: an agent manifest. Each tool carries a list of capabilities drawn from exactly three classes (ingests_untrusted, reads_private, can_egress), an optional isolated flag, and the manifest declares a data_flow mode.

In shared_context mode (the realistic default), every non-isolated tool both feeds and is steered by a virtual <shared-context> node. An isolated tool is taken off that bus, modeling a sandboxed sub-agent that receives a structurally fixed input and never sees the shared context. In explicit mode, only the data-flow edges you declare carry taint, which is the honest mode if you actually wire tools point to point.

Then it builds a directed graph and runs breadth-first reachability, visiting neighbours in sorted order so the path it reports is deterministic. The trifecta is reachable if there exist an untrusted tool u, a private tool p, and an egress tool e such that p is reachable from u and e is reachable from p. Those can be the same tool: one tool with all three capabilities is a trifecta by itself. Here is the whole thing.

#!/usr/bin/env python3
"""trifecta_gate.py - a static PRE-RUN gate for the "lethal trifecta".

You cannot patch prompt injection inside the model. The model cannot reliably
tell trusted instructions from untrusted data, so an attacker's text that lands
in the context is treated like a command. The lever is not "detect the injection
better" (that is still tracking). The lever is to gate the *capability
composition* of an agent's tool manifest BEFORE the agent ever runs.

The lethal trifecta (a term coined by Simon Willison) names the dangerous
combination: in one session an agent can (1) read UNTRUSTED input, (2) read
PRIVATE data, and (3) send data to the OUTSIDE (egress). When all three can
reach each other through one shared context, injected text can read your
secrets and mail them out. This script intercepts NOTHING at runtime. It reads
a static tool manifest (JSON) where each tool is tagged with capabilities and
(optionally) data-flow edges, then answers ONE question by graph reachability:

    Is there a path, inside a single agent/session, where UNTRUSTED input can
    reach a PRIVATE read AND that data can then reach an EGRESS sink?

The point the fixtures prove: "all three capabilities are present" is NOT the
same as "the trifecta is reachable". A safe manifest and a vulnerable one can
declare the SAME three capabilities; what differs is whether the egress tool
sits on the shared context bus. The gate decides on the GRAPH, not a checklist.

exit 0 = trifecta NOT reachable (safe to start)
exit 1 = trifecta reachable (print the data-flow path(s) that close it)
exit 2 = bad input

Offline / keyless / read-only / zero-network. Stdlib only (json, sys, deque).
It reads a manifest file and prints a verdict. No network, no child process, no
model load, no install, and it never launches the agent itself.
"""
import json
import sys
from collections import deque

CAPS = {"ingests_untrusted", "reads_private", "can_egress"}
CTX = "<shared-context>"  # virtual node: the LLM context bus all tools share


def die(msg):
    print("trifecta-gate: ERROR: " + msg)
    sys.exit(2)


def load_manifest(path):
    try:
        with open(path, "r") as fh:
            raw = fh.read()
    except OSError:
        die("cannot read manifest file: " + path)
    try:
        data = json.loads(raw)
    except ValueError:
        die("manifest is not valid JSON")
    if not isinstance(data, dict):
        die("manifest must be a JSON object")
    if not isinstance(data.get("tools"), list) or not data["tools"]:
        die("manifest.tools must be a non-empty list")
    mode = data.get("data_flow", "shared_context")
    if mode not in ("shared_context", "explicit"):
        die("data_flow must be 'shared_context' or 'explicit'")
    tools = {}
    for t in data["tools"]:
        if not isinstance(t, dict) or "id" not in t:
            die("each tool must be an object with an 'id'")
        tid = t["id"]
        if not isinstance(tid, str) or not tid:
            die("tool id must be a non-empty string")
        if tid in tools:
            die("duplicate tool id: " + tid)
        caps = t.get("capabilities", [])
        if not isinstance(caps, list):
            die("capabilities of " + tid + " must be a list")
        for c in caps:
            if c not in CAPS:
                die("unknown capability '" + str(c) + "' on tool " + tid)
        tools[tid] = {"caps": set(caps), "isolated": bool(t.get("isolated", False))}
    return data.get("agent", "(unnamed-agent)"), mode, tools, data.get("flows", [])


def build_graph(mode, tools, flows):
    """Directed adjacency. In shared_context mode every non-isolated tool both
    feeds and is steered by the shared LLM context bus; isolated tools are off
    the bus (sandboxed). In explicit mode only declared flows carry taint."""
    adj = {}

    def edge(a, b):
        adj.setdefault(a, set()).add(b)

    if mode == "shared_context":
        for tid, t in tools.items():
            if not t["isolated"]:
                edge(tid, CTX)   # tool output enters the shared context
                edge(CTX, tid)   # context steers this tool's next input
    for f in flows:  # explicit flows are honored in BOTH modes
        if not isinstance(f, dict) or "from" not in f or "to" not in f:
            die("each flow must be an object with 'from' and 'to'")
        if f["from"] not in tools or f["to"] not in tools:
            die("flow references unknown tool: " + json.dumps(f, sort_keys=True))
        edge(f["from"], f["to"])
    # deterministic neighbour order
    return {k: sorted(v) for k, v in adj.items()}


def bfs(adj, src):
    """Shortest path tree from src. Returns (reachable_set, predecessor_map).
    Neighbours visited in sorted order -> path is deterministic."""
    seen = {src}
    pred = {}
    q = deque([src])
    while q:
        cur = q.popleft()
        for nxt in adj.get(cur, ()):  # already sorted
            if nxt not in seen:
                seen.add(nxt)
                pred[nxt] = cur
                q.append(nxt)
    return seen, pred


def path_of(pred, src, dst):
    out = [dst]
    while out[-1] != src:
        out.append(pred[out[-1]])
    out.reverse()
    return out


def find_trifecta(adj, tools):
    U = sorted(t for t, v in tools.items() if "ingests_untrusted" in v["caps"])
    P = sorted(t for t, v in tools.items() if "reads_private" in v["caps"])
    E = sorted(t for t, v in tools.items() if "can_egress" in v["caps"])
    found = []
    for u in U:
        reach_u, pred_u = bfs(adj, u)
        for p in P:
            if p not in reach_u:
                continue
            reach_p, pred_p = bfs(adj, p)
            for e in E:
                if e not in reach_p:
                    continue
                full = path_of(pred_u, u, p) + path_of(pred_p, p, e)[1:]
                found.append((u, p, e, full))
    found.sort(key=lambda x: (x[0], x[1], x[2]))
    return U, P, E, found


def main(argv):
    if len(argv) != 2:
        print("usage: trifecta_gate.py <agent_manifest.json>")
        sys.exit(2)
    agent, mode, tools, flows = load_manifest(argv[1])
    adj = build_graph(mode, tools, flows)
    U, P, E, found = find_trifecta(adj, tools)
    isolated = sorted(t for t, v in tools.items() if v["isolated"])

    print("trifecta-gate: agent=%s mode=%s tools=%d" % (agent, mode, len(tools)))
    print("capabilities: ingests_untrusted=%d reads_private=%d can_egress=%d"
          % (len(U), len(P), len(E)))
    print("isolated (off shared context): " + (", ".join(isolated) if isolated else "(none)"))
    print("")

    if found:
        print("VERDICT: LETHAL TRIFECTA REACHABLE - %d path(s)" % len(found))
        for i, (u, p, e, full) in enumerate(found, 1):
            print("  [%d] untrusted=%s -> private=%s -> egress=%s" % (i, u, p, e))
            print("      flow: " + " -> ".join(full))
        print("")
        print("ACTION: do NOT start agent. Break one leg (isolate the egress or the")
        print("        private read into a separate context/sub-agent) before launch.")
        sys.exit(1)

    print("VERDICT: trifecta NOT reachable (0 paths) - safe to start")
    if U and P and E:
        print("note: all three capability classes are present but no untrusted->private")
        print("      ->egress data-flow path exists (isolation/edges break the chain).")
    else:
        missing = sorted(CAPS - set(
            (["ingests_untrusted"] if U else []) +
            (["reads_private"] if P else []) +
            (["can_egress"] if E else [])))
        print("note: trifecta cannot close - missing capability class(es): " + ", ".join(missing))
    sys.exit(0)


if __name__ == "__main__":
    main(sys.argv)

The vulnerable manifest: a normal inbox agent

The first manifest is a LangGraph-style inbox assistant, the kind of thing people ship in a weekend. Four tools. read_email pulls message bodies, which an attacker controls, so it ingests untrusted input. search_inbox and read_contacts touch private data. send_email can mail anyone, so it can egress. Nothing is isolated, and everything shares one context. Run it:

$ python3 trifecta_gate.py fixtures/vulnerable_manifest.json
trifecta-gate: agent=langgraph-inbox-assistant mode=shared_context tools=4
capabilities: ingests_untrusted=1 reads_private=2 can_egress=1
isolated (off shared context): (none)

VERDICT: LETHAL TRIFECTA REACHABLE - 2 path(s)
  [1] untrusted=read_email -> private=read_contacts -> egress=send_email
      flow: read_email -> <shared-context> -> read_contacts -> <shared-context> -> send_email
  [2] untrusted=read_email -> private=search_inbox -> egress=send_email
      flow: read_email -> <shared-context> -> search_inbox -> <shared-context> -> send_email

ACTION: do NOT start agent. Break one leg (isolate the egress or the
        private read into a separate context/sub-agent) before launch.

Exit 1. Two paths, both running through the shared context node. To be precise about what "2 paths" means: that is two distinct trifecta closures in this manifest, one through read_contacts and one through search_inbox. It is not a count of attacks in the wild, and it is not a measurement of anything beyond this file. The gate is telling you the shape is exploitable and pointing at exactly where. An attacker who plants instructions in an email body can, in principle, get the agent to read your contacts and forward them. The defense the gate suggests is structural: do not start the agent in this shape.

The safe manifest: same three capabilities, one is moved off the bus

Now the manifest that makes the point. It declares the same four tools and the same three capability classes. The only change: send_email is marked isolated, modeling an egress step that runs as a sandboxed sub-agent receiving a structurally fixed recipient and a templated body, never the shared context.

$ python3 trifecta_gate.py fixtures/safe_manifest.json
trifecta-gate: agent=langgraph-inbox-assistant-isolated-send mode=shared_context tools=4
capabilities: ingests_untrusted=1 reads_private=2 can_egress=1
isolated (off shared context): send_email

VERDICT: trifecta NOT reachable (0 paths) - safe to start
note: all three capability classes are present but no untrusted->private
      ->egress data-flow path exists (isolation/edges break the chain).

Exit 0. Zero paths. Look at the capabilities line: it is identical to the vulnerable run, ingests_untrusted=1 reads_private=2 can_egress=1. If this gate were a checklist, both manifests would score three out of three and both would fail. They do not. The vulnerable one fails, the safe one passes, and the only difference is whether egress sits on the shared bus. That is the entire argument for doing this by reachability instead of by counting flags. A checklist cannot tell these two apart. The graph can.

This is also why a crypto agent is the expensive version of the same picture. Swap the labels: read_pool_data ingests untrusted on-chain or web input, read_wallet reads a private key or balance, sign_and_send_tx egresses, except here egress is money leaving the wallet. Same graph, same closing path, except a successful injection moves funds instead of contacts. I did not ship a crypto manifest in this post, but the structure is identical, and so is the fix: take the signing step off the shared context.

The exit code is the gate

The point of returning 0, 1, or 2 is that CI can read it without reading prose. Wire it before you launch the agent or merge a manifest change:

if python3 trifecta_gate.py agent_manifest.json; then
    echo "trifecta not reachable - safe to start the agent"
else
    echo "trifecta reachable or bad manifest - hold the launch"
fi

Exit 0 lets the launch proceed. Exit 1 holds it and prints the paths to break. Exit 2 means the manifest was malformed, and a malformed manifest must never read as "safe." Feed it the broken fixture, where tools is a string instead of a list, and it exits 2 with trifecta-gate: ERROR: manifest.tools must be a non-empty list. Run it with no arguments and it prints usage and exits 2. A gate that cannot tell "all clear" from "I could not check" is not a gate.

Deterministic, so it can live in CI

The output carries no timestamps, no map iteration order, no floating point. Neighbours are sorted before traversal, so the path it prints is stable. Hash the STDOUT of each fixture twice and it is identical both times: the vulnerable run is 7718cefc5ffce46aee99111e595262803698d2fab3a741a7eb3137e95a8b11aa, the safe run is a561c84a8f2022f1742fa888268255d00d15221c801065f8ecd206f8e2eeeadc, the bad run is 9ed47c5847aaf49f8d2f9733a1c4565959b775bfce945dfcd65a369a2e624557. That matters because a check you cannot reproduce is a second opinion, not a gate. This one you can pin in a test and diff on every manifest change.

What this is NOT

I would rather you know the edges than hit them in production.

It does not catch the injection. It never sees a prompt, a model, or a runtime call. It reads a static manifest and reasons about reachability. It assumes the injection can happen (because at the model level, it can) and asks whether a successful one would have a path to exfiltrate. If you want to know whether a specific payload gets through, this is the wrong tool.
It does not replace sandboxing or human approval. Marking a tool isolated is a claim that you actually sandboxed it. The gate checks that your declared architecture breaks the chain; it does not enforce the sandbox. The runtime isolation and the human-in-the-loop on high-risk actions are still your job. This tells you whether the design, as declared, is sound.
It trusts your capability tags. If you label send_email as not able to egress, or forget that a "read-only" tool can leak data through an error message or a logged URL, the graph is wrong and so is the verdict. The honesty of the tags is the whole foundation. Garbage tags, garbage gate.
It is a per-session, single-agent model. It does not reason about data that leaves one agent, gets stored, and returns to another later. Cross-agent and persisted-memory flows are a harder graph than this one draws.
The shared_context model is an assumption, not a law. It encodes the common default where everything shares one bus. If your orchestrator wires tools point to point, use explicit mode and declare the real edges. The contract is "reason about reachability on the data flow you actually have," not "every agent looks like my default."

Where this sits next to the other gates

This is one more pre-execution check in a series, and it helps to say how it differs from its closest neighbors so you reach for the right one.

It is the manifest-shaped sibling of the pre-execution gate for AI agents: a check that has to pass before an action, applied here to the whole capability graph instead of a single call.

It is not the blast radius of a leaked agent API key. That one measures the scope of damage from one leaked credential. This one measures the composition of many tools, whether their capabilities can reach each other at all. Different object: one is the reach of a key, this is the reach of a graph.

It is not redacting credential leaks at the boundary either. That tool scrubs the secret out of what egresses, the content leaving the wire. This one asks an earlier question: can untrusted input reach the egress sink in the first place? One cleans the payload, the other removes the path.

And it is a different axis from pinning and verifying MCP tools. That guards against the manifest drifting or a tool being swapped under you, a question of version integrity. This reads the same manifest and asks whether its capabilities, as declared, compose into the trifecta. Drift versus reachability.

Finally, it shares a family resemblance with your agent returns 200 and lies: both refuse to trust a surface signal. There it is a clean status code over a wrong effect; here it is "all three capabilities, looks scary" versus the actual reachable paths. Different artifact, same suspicion of the easy read.

The question I am still chewing on

The model assumes one shared context per agent. Real platform-MCP setups are messier. You connect five servers, each adding tools, and some of them quietly add a leg to the trifecta on a context they all share. The honest hard part is explicit mode: drawing the true data-flow edges of a real orchestrator is work, and if you draw them wrong the gate is confidently wrong with you.

So here is the real open question for anyone running an MCP composition or a multi-server agent: how many of your connected servers sit on the same context bus, and which one of them opens egress? If you cannot answer that quickly, the trifecta might already be reachable in your agent and nobody drew the graph. I have a partial answer (tag every tool's three capabilities at registration, fail the build on a reachable path) and no clean way to keep the explicit edges honest as the agent grows. Tell me how you model the context bus in your setup. I read every comment.

Follow for the next runnable gate in this series on controlling agents before you trust them.

Written by Alexey Spinov. AI-assisted, human-verified: the tool, all three manifests, and every number above come from a real local run on 2026-06-30 (Python 3.13.5, stdlib only, offline). I ran it, checked the exit codes (0 / 1 / 2), hashed the STDOUT twice to confirm determinism, and edited every line. The Agentjacking figures (2,388 organizations, 100+ agents at an 85% success rate in controlled testing, the "technically not defensible" quote) are Tenet Security's, from their June 2026 writeup, not my measurements. The term "lethal trifecta" and the position that LLMs cannot reliably distinguish trusted instructions from untrusted data are Simon Willison's, from his 16 June 2025 post. I label which numbers are theirs and which are mine.

A 58% Win-Rate Over Zero Closed Trades: Recompute Agent Scorecard

Alexey Spinov — Mon, 29 Jun 2026 07:30:13 +0000

Recompute the agent scorecard from its primary event journal before you trust it: a self-reported metric is the actor grading itself. scorecard_reconcile.py re-derives every metric independently and flags divergence. On the divergent fixture a claimed 58% win-rate sits over zero closed trades, yielding 5 DIVERGENT and 1 UNSUPPORTED metric and exit 1, blocking the add-capital decision.

AI disclosure: I wrote scorecard_reconcile.py with an AI assistant and ran it myself, offline, before publishing. Every number in the output blocks below is pasted from a real local run on Python 3.13.5, stdlib only, on the synthetic fixtures included in this post. I checked the exit codes, hashed the STDOUT twice to confirm it is byte-for-byte deterministic, and edited every line. The external figures (the SEC's $12.3M case) are the SEC's numbers, not mine, and I link the primary source. I label which numbers are theirs and which are mine.

In short:

A self-reported metric and a real one look identical on a dashboard. The agent prints both.
scorecard_reconcile.py reads the scorecard the agent claimed plus the journal of events it logged, then recomputes each metric from the journal and flags anything that does not reconcile.
On the clean fixture, all 6 metrics MATCH, 0 flagged, exit 0. On the divergent fixture, a claimed 12-trade, 58.3% win-rate scorecard sits over a journal with zero closing events: 5 metrics DIVERGENT, 1 UNSUPPORTED, exit 1.
The 58.3% is the dangerous one. A win-rate over zero closed trades is not 58%. It is undefined, and a number you cannot disprove is worse than a number that is merely wrong.
Stdlib only (json, sys, hashlib). No network, no model, no exec. The run is byte-for-byte deterministic. The code and both fixtures are in this post.

The case that makes this concrete

On 28 May 2026 the SEC charged Nathan Fuller, founder of Privvy Investments, over a crypto scheme it valued at about $12.3 million raised from roughly 150 investors (SEC litigation release LR-26558, reported by CoinDesk on 30 May 2026). The pitch was proprietary AI trading bots running high-frequency arbitrage. According to the complaint, only about $380,000, roughly 3% of the money, ever bought crypto at all, those trades ran without the advertised bots, and they made no profit. Investors were kept calm with fake account statements and fabricated correspondence. Those are the SEC's figures, from their filing, not mine.

Strip out the fraud and a quieter version of the same shape shows up in honest setups every week. A trading agent prints a dashboard. The dashboard says 4 active days, 12 trades, 58.3% win-rate. Nobody re-derives those numbers from the exchange fills. The statement and the reality were never reconciled. Fuller's was a deliberate fabrication; most are just an agent confidently reporting work that the journal does not back. The defense is the same in both: do not accept the scorecard the actor printed about itself. Recompute it from the primary events first.

The claim, sharp enough to argue with

Here is the falsifiable version: a self-reported metric is the actor grading itself, and a scorecard you did not recompute is not evidence. Win-rate, trade count, "ran for N days," realized PnL: every one of those is emitted by the agent, from the agent's own view of what it did. If that view is wrong, optimistic, or invented, the metric inherits the error and presents it as a clean number on a chart.

If this claim were false, the recomputed scorecard would always equal the claimed one, and a tool like this would be pointless. You could read the dashboard and move on. The tool earns its place precisely when the two disagree, and the divergent fixture below is one keystroke of proof that they can disagree completely.

The control is not "log the metrics and watch them." That is tracking. Tracking tells you what the agent said about itself. Control is recomputing the metric from the primary journal and gating the next decision, keep it running, scale it, add capital, on whether the recomputation agrees. This is the post-hoc sibling of the pre-execution gate for AI agents: same idea, a check that has to pass before an action, applied to the aggregate report instead of a single call.

What the tool recomputes, and how

The input is two small JSON files. The first is claimed.json, the scorecard the agent printed about itself. The second is evidence.json, the primary journal: an events list of open and close records, each with a timestamp and, for closes, a PnL. Think exchange fills, not the agent's summary of them.

The tool ignores the claimed numbers and rebuilds each metric from the journal with one defensible definition per metric:

A trade is a closed position, a close event. Open attempts that never fill are not trades.
A win is a close with PnL above zero, a loss is a close below zero.
Active days are the count of distinct ISO dates across all events, taken as the first ten characters of the timestamp. No clock, no date library, no timezone math.
Win-rate is wins over decided trades. Over zero decided trades it is None, undefined, not zero.
Realized PnL is the sum of PnL over closes, rounded to a cent.

Tolerances are explicit so the verdict is not a matter of taste: counts must match exactly, ratios within 0.005 (half a percentage point), money within one cent. Here is the whole tool.

#!/usr/bin/env python3
"""scorecard_reconcile.py - recompute an agent's self-reported scorecard from its
primary event journal BEFORE you trust the report (or add capital / scale / keep it on).

A self-reported metric is the actor grading itself: win-rate, "ran for N days",
trade count, realized PnL - printed by the same agent whose work they describe.
This tool ignores the claimed numbers and INDEPENDENTLY recomputes each one from
the primary event journal (the fills / closes / activity the agent actually
logged), then flags every metric that does not reconcile.

Two failure shapes, not one:
  DIVERGENT   - claimed value != value recomputed from evidence.
  UNSUPPORTED - claimed value cannot be derived from the journal at all
                (a win-rate over zero closed trades is not 58%, it is
                undefined; the number is unfalsifiable, which is worse).

offline / keyless / read-only / zero-network. stdlib only (json, sys, hashlib).
The journal is read, never run. Nothing is fetched, nothing is sent.

Exit 0 = every claimed metric is supported by evidence AND matches   -> report trustworthy
Exit 1 = at least one metric DIVERGENT or UNSUPPORTED                 -> do NOT trust the report
Exit 2 = bad input

Usage:
  python3 scorecard_reconcile.py <claimed.json> <evidence.json>
"""

import json
import sys
import hashlib

RATIO_TOL = 0.005   # 0.5 percentage points
MONEY_TOL = 0.01    # one cent


class BadInput(Exception):
    pass


def load_json(path):
    try:
        with open(path, "r") as fh:
            return json.load(fh)
    except FileNotFoundError:
        raise BadInput("file not found: %s" % path)
    except json.JSONDecodeError as exc:
        raise BadInput("invalid json in %s: %s" % (path, exc))


def as_number(value, field):
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise BadInput("metric %r is not numeric: %r" % (field, value))
    return float(value)


def recompute(events):
    """Independently rebuild every metric from the primary journal."""
    if not isinstance(events, list):
        raise BadInput("evidence.events must be a list")
    closes, days = [], set()
    for ev in events:
        if not isinstance(ev, dict):
            raise BadInput("event is not an object: %r" % (ev,))
        ts = ev.get("ts")
        if ts:
            days.add(str(ts)[:10])          # ISO date portion; no clock, no parse libs
        if ev.get("type") == "close":
            closes.append(ev)
    wins = sum(1 for e in closes if as_number(e.get("pnl", 0), "pnl") > 0)
    losses = sum(1 for e in closes if as_number(e.get("pnl", 0), "pnl") < 0)
    decided = wins + losses
    win_rate = (wins / decided) if decided > 0 else None   # None => undefined / unsupported
    realized = round(sum(as_number(e.get("pnl", 0), "pnl") for e in closes), 2)
    return {
        "trades": float(len(closes)),
        "active_days": float(len(days)),
        "wins": float(wins),
        "losses": float(losses),
        "win_rate": win_rate,
        "realized_pnl": realized,
        "_supporting_events": len(events),
        "_closing_events": len(closes),
    }


# metric name -> comparison kind
SPEC = [
    ("trades",       "count"),
    ("active_days",  "count"),
    ("wins",         "count"),
    ("losses",       "count"),
    ("win_rate",     "ratio"),
    ("realized_pnl", "money"),
]


def compare(name, kind, claimed, recomputed):
    if recomputed is None:
        return "UNSUPPORTED"
    if kind == "count":
        return "MATCH" if int(round(claimed)) == int(round(recomputed)) else "DIVERGENT"
    if kind == "ratio":
        return "MATCH" if abs(claimed - recomputed) <= RATIO_TOL else "DIVERGENT"
    if kind == "money":
        return "MATCH" if abs(claimed - recomputed) <= MONEY_TOL else "DIVERGENT"
    raise BadInput("unknown metric kind: %s" % kind)


def fmt(kind, value):
    if value is None:
        return "UNDEFINED"
    if kind == "count":
        return str(int(round(value)))
    if kind == "ratio":
        return "%.3f" % value
    return "%.2f" % value


def build_report(claimed_doc, evidence_doc):
    if not isinstance(claimed_doc, dict) or "scorecard" not in claimed_doc:
        raise BadInput("claimed.json must contain a 'scorecard' object")
    sc = claimed_doc["scorecard"]
    if not isinstance(sc, dict):
        raise BadInput("'scorecard' must be an object")
    if not isinstance(evidence_doc, dict) or "events" not in evidence_doc:
        raise BadInput("evidence.json must contain an 'events' list")

    truth = recompute(evidence_doc["events"])
    agent = str(claimed_doc.get("agent", "unknown-agent"))

    lines = []
    lines.append("SCORECARD RECONCILE - %s" % agent)
    lines.append("primary journal: %d events (%d closing) - recomputed independently"
                 % (truth["_supporting_events"], truth["_closing_events"]))
    lines.append("")
    lines.append("%-14s %12s %12s   %s" % ("metric", "claimed", "from-journal", "verdict"))
    lines.append("%-14s %12s %12s   %s" % ("-" * 14, "-" * 12, "-" * 12, "-------"))

    divergent, unsupported = 0, 0
    for name, kind in SPEC:
        if name not in sc:
            raise BadInput("scorecard missing metric: %s" % name)
        claimed = as_number(sc[name], name)
        recomputed = truth[name]
        verdict = compare(name, kind, claimed, recomputed)
        if verdict == "DIVERGENT":
            divergent += 1
        elif verdict == "UNSUPPORTED":
            unsupported += 1
        note = ""
        if verdict == "UNSUPPORTED":
            note = "  (0 closing events -> ratio is undefined / unfalsifiable)"
        lines.append("%-14s %12s %12s   %s%s"
                     % (name, fmt(kind, claimed), fmt(kind, recomputed), verdict, note))

    flagged = divergent + unsupported
    lines.append("")
    lines.append("flagged: %d (%d divergent, %d unsupported)" % (flagged, divergent, unsupported))
    if flagged == 0:
        lines.append("DECISION GATE: scorecard reconciles with evidence -> trust permitted (exit 0)")
        code = 0
    else:
        lines.append("DECISION GATE: scorecard does NOT reconcile -> do NOT trust report; "
                     "block continue/add-capital (exit 1)")
        code = 1
    return "\n".join(lines), code


def main(argv):
    if len(argv) != 3:
        sys.stderr.write("usage: scorecard_reconcile.py <claimed.json> <evidence.json>\n")
        return 2
    try:
        claimed_doc = load_json(argv[1])
        evidence_doc = load_json(argv[2])
        body, code = build_report(claimed_doc, evidence_doc)
    except BadInput as exc:
        sys.stderr.write("bad input: %s\n" % exc)
        return 2
    digest = hashlib.sha256(body.encode("utf-8")).hexdigest()
    sys.stdout.write(body + "\n")
    sys.stdout.write("REPORT-SHA256: %s\n" % digest)
    return code


if __name__ == "__main__":
    sys.exit(main(sys.argv))

A scorecard that reconciles

The honest fixture has a journal of 14 events, 10 of them closes, and a scorecard that genuinely describes them. Run it:

$ python3 scorecard_reconcile.py fixtures/clean_claimed.json fixtures/clean_evidence.json
SCORECARD RECONCILE - trader-honest
primary journal: 14 events (10 closing) - recomputed independently

metric              claimed from-journal   verdict
-------------- ------------ ------------   -------
trades                   10           10   MATCH
active_days               3            3   MATCH
wins                      6            6   MATCH
losses                    4            4   MATCH
win_rate              0.600        0.600   MATCH
realized_pnl         125.50       125.50   MATCH

flagged: 0 (0 divergent, 0 unsupported)
DECISION GATE: scorecard reconciles with evidence -> trust permitted (exit 0)
REPORT-SHA256: 15a33ee3894142fdcec6c06589e0d9aff09d1331ca7a91b88229d2d3b6a10aba

All 6 metrics MATCH, exit 0. This is what passing looks like, and you want it to be boring. The claimed 0.600 win-rate is the same 0.600 the tool derived from the fills, the claimed $125.50 is the sum it computed over the 10 closes. Nothing to argue with. The gate opens.

12 trades claimed, zero in the journal

Now the divergent fixture. The scorecard claims 4 active days, 12 trades, 7 wins, 5 losses, a 58.3% win-rate, and $842.00 realized. The journal underneath it holds three open events on a single day, all rejected, and not one close.

$ python3 scorecard_reconcile.py fixtures/divergent_claimed.json fixtures/divergent_evidence.json
SCORECARD RECONCILE - trader-x (self-reported dashboard)
primary journal: 3 events (0 closing) - recomputed independently

metric              claimed from-journal   verdict
-------------- ------------ ------------   -------
trades                   12            0   DIVERGENT
active_days               4            1   DIVERGENT
wins                      7            0   DIVERGENT
losses                    5            0   DIVERGENT
win_rate              0.583    UNDEFINED   UNSUPPORTED  (0 closing events -> ratio is undefined / unfalsifiable)
realized_pnl         842.00         0.00   DIVERGENT

flagged: 6 (5 divergent, 1 unsupported)
DECISION GATE: scorecard does NOT reconcile -> do NOT trust report; block continue/add-capital (exit 1)
REPORT-SHA256: 7d1a10fd12a95a13329ffd7c34d6d869411a51d2e4196c0be6c20dec4d06f781

Six metrics flagged, exit 1. The dashboard told a four-day story. The events show one day of failed attempts and zero realized anything. If you had read the scorecard and added capital, you would have funded a $842 PnL that does not exist in the record the agent itself kept.

Two failure shapes, and why one is worse

Most checks have a single failure mode: pass or fail. This one separates two, because they call for different reactions.

DIVERGENT is the loud one. Claimed 12, journal says 0. Claimed $842, journal says $0.00. The numbers disagree and the disagreement is concrete. You can chase it: a logging bug, a double-count, a wrong window, a fabrication. There is a thread to pull.

UNSUPPORTED is the quiet one, and it is worse. The claimed 58.3% win-rate cannot be derived from the journal at all, because there are zero decided trades. A win-rate is wins over decided trades, and over zero trades that ratio is undefined, not 58.3% and not 0%. The agent printed a precise-looking number for a quantity the evidence cannot define. You cannot disprove 58.3% by pointing at the fills, because the fills do not speak to it. A number you cannot falsify is not a weak metric, it is a non-metric wearing a metric's clothes, and it is exactly the kind of figure that survives a review because it looks specific. The tool refuses to call it DIVERGENT (that would imply the right answer was some other number) and labels it UNSUPPORTED instead.

That distinction is the whole reason the tool prints two counters instead of one.

The exit code is the gate, not the chart

The point of returning 0, 1, or 2 is that a pipeline can read it without reading prose. Wire it before the decision that spends money or scope:

if python3 scorecard_reconcile.py claimed.json evidence.json; then
    echo "scorecard reconciles - proceed with the add-capital review"
else
    echo "scorecard does not reconcile - hold; do not scale on this report"
fi

Exit 0 lets the next step run. Exit 1 holds it. Exit 2 means the input was malformed (events that are not a list, a missing scorecard, a non-numeric metric), and a malformed reconciliation should never read as "passed." Try it: feed it a bad file and it exits 2 with bad input: evidence.events must be a list, run it with no arguments and it prints usage and exits 2. A gate that cannot tell "all clear" from "I could not check" is not a gate.

Deterministic, so it can live in CI

The report ends with a SHA-256 of its own body. Run the clean fixture twice and the STDOUT hashes to 64ac7d45afb235bae3fcfac28e98fd3ae8b6a4c5f43e6696ebd2b723684159b6 both times; the divergent fixture is 64246aad569193ce61d004e66d79cb256f6a8d8e7ebd8feda9dfb8c41bf8d52e both times. No timestamps in the output, no map ordering, no floating-point surprise past the cent. That matters because a reconciliation you cannot reproduce is just a second opinion. This one you can pin in a test and diff.

What this is NOT

I would rather you know the edges than discover them on your own logs.

It is not an audit of the trading logic. It checks that the scorecard agrees with the journal. It says nothing about whether the strategy is good, whether the fills were priced fairly, or whether the agent should have opened those positions at all. A perfectly reconciled scorecard can describe a terrible strategy.
It trusts the journal. The whole method assumes evidence.json is the primary record, exchange fills or a settlement log, not a second file the same agent wrote. If the agent forges the journal too, this catches nothing. So the real question is upstream: is your evidence a source the agent cannot rewrite? Pull fills from the exchange API or the chain, not from the agent's own summary.
It is not on-chain attestation or settlement verification. It does no signature checks and reads no blockchain. For "did this fill really happen and settle," you want the exchange's records or a node, then feed those in as the journal. Pair it with the Grok tx canary, which gates a single transaction before broadcast, while this reconciles the aggregate after the fact.
A flag is a signal, not a verdict on intent. DIVERGENT can be a logging bug as easily as a lie. The tool tells you the report and the record disagree. Why they disagree is your investigation.
The metric definitions are mine, and arguable. I count a trade as a close, not an open. If your book counts entries, or partial fills, or funding events, change the definitions in recompute to match your venue. The contract is "recompute from primary events with explicit rules," not these six exact rules.

Where this sits next to the other gates

This is one more pre-decision check in a series, and it is worth saying how it differs from its closest neighbors so you use the right one.

It is not your-agent-returns-200-and-lies. That tool verifies a single call: a clean 200 whose effect was wrong. This one verifies an aggregate over many events, the scorecard built from all of them. Different object, different scale.

It is not the green-checkmark auditor either. That one asks whether a passing test actually exercises the code or just mirrors it. Here the question is whether a claimed KPI is backed by the primary events. Both share a suspicion of green that was never earned, applied to different artifacts.

And it sits a step downstream of the waste-probe for tokens burned after a failure: that one measures cost the agent already spent, this one questions the success the agent claims for what it spent. Cost and truth are separate audits, and an agent can fail both at once.

The question I am still chewing on

The method is only as good as the journal you feed it, and that is the part I have not solved cleanly. For a centralized exchange you can pull fills from the venue's API, a record the agent cannot rewrite. For an agent that logs its own activity to a file it also controls, the journal and the scorecard come from the same hand, and reconciling one against the other proves consistency, not truth.

So here is the real open question for anyone running a trading or ops agent: what is the primary journal for your bot, the exchange's fills and the chain's settlements, or the activity log the agent itself prints? If it is the latter, what stops the agent from making both agree? I have a few ideas (signed fills, a journal written by a separate process the agent cannot reach) and no clean rule. Drop how you source your evidence in the comments, I read every one.

Follow for the next runnable check in this series on controlling agents before you trust them.

Written by Alexey Spinov. AI-assisted, human-verified: the tool, all five fixtures, and every number above come from a real local run on 2026-06-29 (Python 3.13.5, stdlib only, offline). I ran it, checked the exit codes (0 / 1 / 2), hashed the STDOUT twice to confirm determinism, and edited every line. The SEC figures ($12.3M, ~150 investors, ~$380K/3% actually traded) are the SEC's, from litigation release LR-26558 and CoinDesk's 30 May 2026 report, not my measurements. I label which numbers are theirs and which are mine.

Your LLM Router Logged the Wallet Key. It Already Left.

Alexey Spinov — Sat, 27 Jun 2026 01:34:14 +0000

AI-agent secrets are in transit when a request hits a third-party LLM router or MCP proxy, and that router's audit log is not a control: by the time it logs the request, the credential already crossed your perimeter in plaintext. The fix is to redact at egress, before the bytes leave.

In short:

A secret going to your own model provider on an Authorization header is the expected path. The same secret going to a router, gateway, or MCP proxy is a leak, because that host reads your plaintext.
boundary_leak_probe.py reads one JSON egress map and classifies every secret-bearing field by destination trust. On the leaky fixture: 3 requests, 2 of them to third-party intermediaries, 5 fields crossing the boundary, 6 rule-hits, 2 critical wallet secrets. Exit 1.
The 2 critical ones are an Ethereum private key and a BIP-39 mnemonic, sitting in MCP tool-call arguments headed to a proxy. Signer material should never transit any middleman.
Stdlib only (sys, json, re). No network, no model, no exec. The run is byte-for-byte deterministic.
A hit is a SIGNAL, not a confirmed live secret. The code and both fixtures are in this post.

The incident that made this worth measuring

In April 2026 a group of researchers, including Chaofan Shou, published "Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain" (arXiv 2604.08407, posted 9 April 2026). They pointed real agent traffic at 428 commodity LLM routers. Nine of them injected code into the responses. Seventeen reached for the researchers' own AWS credentials. Their framing of the mechanism is the part that stuck with me: these routers "operate as application-layer proxies with full plaintext access to every in-flight JSON payload," and no provider enforces cryptographic integrity between the client and the upstream model.

CoinDesk covered it the same week and carried a blunter line from Shou: 26 routers were "secretly injecting malicious tool calls and stealing creds," and one of them "drained our client's $500k wallet" (CoinDesk, 13 April 2026). That $500k and those router counts are their numbers, from their measurement, not mine. I am citing them for context. Everything I claim about my own tool comes from a run I will paste in full.

I read that paper on a Tuesday and went looking for the part of my own stack that assumed the router was trusted. I found it fast. We route through a gateway for failover and cost tracking. The gateway has a dashboard. The dashboard has a request log. And I had quietly been treating that log as a safety net: if something leaks, I will see it there.

That assumption is backwards, and saying it out loud is the whole point of this post.

The claim, sharp enough to argue with

Here is the falsifiable version: a router's audit log is a receipt, not a brake. By the time a credential shows up in the router's log, the router process has already read it in plaintext. Logging happens on the far side of the boundary. The secret is gone. You cannot un-send it by reviewing a log entry, the same way you cannot un-mail a letter by reading the carbon copy.

If that claim were false, redaction would not matter and a probe like mine would be pointless. You could just watch the log and rotate after the fact. But "rotate after the fact" assumes the window between send and detection is harmless, and for a wallet private key that window is exactly long enough to sign one transaction. The signer secret is not like an API key you rotate on Monday. Once a third party has it, the funds are a sign_tx call away.

So the control has to move upstream of the send. Classify the destination first. Redact anything that should not cross. Then emit the bytes. The log, if you keep one, becomes a record of what you allowed out, not a tripwire you read after the damage.

What the probe actually does

The input is one JSON file I call an egress map: the outbound requests your agent emits, with their destination host, kind, headers, and body. You can dump this from a request interceptor, a test harness, or by hand. The probe never makes a request. It reads the map statically.

Two ideas do the work.

First, destination trust. You declare first_party_hosts: the hosts you contract with directly, your own backend or the model provider itself. An Authorization header to one of those is the expected credential path, so the probe does not scream about it. Every other host, the routers and gateways and MCP proxies in the middle, is third-party by default. A secret sitting in the body or tool-call arguments headed there has crossed a boundary you do not own.

Second, signer material is always-leak. An Ethereum private key or a BIP-39 mnemonic must never transit any intermediary, first-party or not. There is no legitimate path where your agent mails a seed phrase through a proxy. If the probe sees one, it is CRITICAL regardless of destination.

Here are the rules and the value scanner:

import sys, json, re

# Secret shapes. critical = signer material that must NEVER transit at all.
SECRET_RULES = [
    ("eth_private_key", re.compile(r"\b0x[0-9a-fA-F]{64}\b"),                  True),
    ("bip39_mnemonic",  re.compile(r"\b(?:[a-z]{3,8}\s+){11,23}[a-z]{3,8}\b"), True),
    ("aws_access_key",  re.compile(r"\bAKIA[0-9A-Z]{16}\b"),                   False),
    ("openai_key",      re.compile(r"\bsk-[A-Za-z0-9]{20,}\b"),                False),
    ("bearer_token",    re.compile(r"Bearer\s+[A-Za-z0-9._\-]{16,}"),          False),
    ("github_pat",      re.compile(r"\bghp_[A-Za-z0-9]{36}\b"),                False),
]
# A value already neutralised before send: env ref / vault handle / masked.
SAFE_REF = re.compile(r"^(\$\{?[A-Z0-9_]+\}?|\$VAULT_REF:[\w:\-]+|sk-\*{3,}|\*{4,}|<REDACTED[:>])")

def scan_value(val):
    if SAFE_REF.match(val.strip()):
        return []                       # already redacted / handle-referenced
    return [(kind, crit) for kind, rx, crit in SECRET_RULES if rx.search(val)]

The SAFE_REF rule matters more than it looks. A value like ${OPENAI_KEY} or $VAULT_REF:openai is a handle, not a secret: the real value gets substituted at the trusted edge, not carried in your agent's payload. If you already pass handle references to your router and let your own egress proxy swap them in, you are most of the way to safe. The probe rewards that by staying quiet.

The classifier walks every string leaf in each request, scans it, and applies the leak rule:

def classify(spec):
    first_party = set(spec.get("first_party_hosts", []))
    rows = []
    for req in spec.get("requests", []):
        trust = "first_party" if req.get("to", "") in first_party else "third_party"
        payload = {k: v for k, v in req.items() if k not in ("to", "kind", "id")}
        for jp, val in walk(payload):
            hits = scan_value(val)
            if not hits:
                continue
            is_auth = jp.lower().startswith("headers.authorization")
            is_crit = any(c for _, c in hits)
            leak = (trust == "third_party") or is_crit
            # an expected first-party Authorization is NOT a leak (unless critical)
            if trust == "first_party" and is_auth and not is_crit:
                leak = False
            rows.append({"id": req.get("id", "?"), "kind": req.get("kind", "?"),
                         "trust": trust, "path": jp, "kinds": [k for k, _ in hits],
                         "critical": is_crit, "leak": leak})
    return len(spec["requests"]), rows

The walk helper is the obvious recursive descent over dicts and lists, yielding a JSON path and a string for every leaf. The full file, including the --redact mode I show below, is about 95 lines. I am skipping the boilerplate here, not hiding it.

The run, pasted whole

Two fixtures. The clean one still talks to two third-party intermediaries, but it routes real secrets only to first-party hosts and sends those intermediaries handle references instead, so nothing crosses. The leaky one is shaped like a real agent that got lazy: an API gateway in the middle carrying a GitHub PAT on its Authorization header, an AWS key buried in a system message, an OpenAI key in metadata, and an MCP proxy receiving a wallet private key plus a mnemonic in tool-call arguments.

$ python3 boundary_leak_probe.py fixtures/egress_clean.json
requests=3  third_party_intermediaries=2
secret_bearing_fields_crossing_boundary=0  rule_hits=0  critical_signer_material=0
redaction_gate_would_block=0  router_audit_log_sees_them_only_AFTER_egress=0
exit=0

$ python3 boundary_leak_probe.py fixtures/egress_leaky.json
requests=3  third_party_intermediaries=2
secret_bearing_fields_crossing_boundary=5  rule_hits=6  critical_signer_material=2
redaction_gate_would_block=5  router_audit_log_sees_them_only_AFTER_egress=5
  leak      third_party  llm-gateway   req=r2   aws_access_key         body.messages[0].content
  leak      third_party  llm-gateway   req=r2   openai_key             body.metadata.upstream_key
  leak      third_party  llm-gateway   req=r2   bearer_token+github_pat headers.Authorization
  CRITICAL  third_party  mcp-proxy     req=r3   eth_private_key        body.tool_calls[0].arguments.private_key
  CRITICAL  third_party  mcp-proxy     req=r3   bip39_mnemonic         body.tool_calls[1].arguments.mnemonic
exit=1

Now the honesty about the numbers, because this is where a lazy headline would lie. The probe found 5 distinct fields crossing to third-party intermediaries, but 6 rule-hits. Why the mismatch? One field, the gateway's Authorization header, tripped two rules at once: it looks like a generic bearer token and it is a GitHub PAT wrapped inside it. That is one leak, two signals. It is not six different secrets, and I am not going to call it six. The number that matters most is the small one: 2 critical fields, the wallet private key and the mnemonic, both headed to an MCP proxy that has no business seeing either.

One more detail that is easy to miss. Request r1 in the leaky fixture sends a real-looking Bearer sk-... to api.openai.com, which is a first-party host. The probe does not flag it. That is the point of destination trust: an auth header to your provider is the credential doing its job. The same shape to a router is the credential getting stolen. A flat secret scanner cannot tell those two apart. This one is built to. One honest caveat: that trust is host-level, not header-level. The probe trusts the destination, so a non-critical secret that lands anywhere in a first-party request, body included, also gets a pass; only signer material overrides the trust and leaks regardless. So your first_party_hosts list is the whole ballgame. Keep it tight, because the tool trusts those hosts with whatever you send them.

Bad input is a third exit code, so a CI step can branch on it:

$ python3 boundary_leak_probe.py            # no argument
usage: boundary_leak_probe.py [--redact] <egress_map.json>
exit=2

And the run is deterministic. I hashed the leaky STDOUT twice:

$ python3 boundary_leak_probe.py fixtures/egress_leaky.json | shasum -a 256
28c5eb9ff8e7ad0abc6b1ad67a617cdd5fdaa09bfce26d3f9f00022217e0a6c5  -
28c5eb9ff8e7ad0abc6b1ad67a617cdd5fdaa09bfce26d3f9f00022217e0a6c5  -

Same bytes both times. That matters for a gate: a check that flickers is a check people disable.

Redact at the boundary, then prove the gate closed

Reporting a leak is the easy half. The thesis was that the control belongs at egress, so the probe has a --redact mode that prints the masked map a boundary gate would actually emit. It leaves the first-party Authorization alone and masks everything that would cross:

$ python3 boundary_leak_probe.py --redact fixtures/egress_leaky.json
...
    "to": "router.3rdparty.ai",
    "headers": { "Authorization": "<REDACTED:bearer_token>" },
    "body": {
      "messages": [ { "role": "system", "content": "<REDACTED:aws_access_key>" } ],
      "metadata": { "upstream_key": "<REDACTED:openai_key>" }
    }
...
    "to": "mcp-proxy.partner.io",
    "arguments": { "private_key": "<REDACTED:eth_private_key>", ... }
    "arguments": { "mnemonic": "<REDACTED:bip39_mnemonic>" }

Then the part I like. Feed that masked map back into the probe:

$ python3 boundary_leak_probe.py --redact fixtures/egress_leaky.json | python3 boundary_leak_probe.py /dev/stdin
requests=3  third_party_intermediaries=2
secret_bearing_fields_crossing_boundary=0  rule_hits=0  critical_signer_material=0
redaction_gate_would_block=0  router_audit_log_sees_them_only_AFTER_egress=0
exit=0

Exit 0, and notice it still lists two third-party intermediaries: the hops are still there, but zero secrets now cross to them. The masked tokens match SAFE_REF, so the second pass sees nothing to flag, and --redact masks every field the audit scans, not just headers and body, so the round trip holds for more than this one fixture's exact shape. That round trip is the difference between watching a log and holding a brake. The log tells you a secret left. The redact pass means it never did. It is still a static regex heuristic, though, not a proof your bytes are clean.

Where this sits, and what I have already written about

This is the fifth tool in a series, and I keep the axes deliberately separate so they stack instead of overlap. Earlier ones looked at a secret that ships in a build artifact (what npm pack actually publishes), the blast radius of a key if it leaks (how much breaks, by scope), the identity and version of an MCP manifest, and contamination in an eval harness. None of those asked the question this one asks: of the requests my agent is about to send, which destinations are trusted, and which secret-bearing fields are about to cross to a host I do not control? The object here is the outbound trace and its destination, not a file on disk, not a manifest, not a scope score. New axis, new tool.

What this is NOT

I would rather you trust the limits than oversell the wins.

It is not a live secret scanner. Every hit is a SIGNAL, a regex match on a shape. The 0x... could be a transaction hash someone pasted, not a private key. Confirm anything that matters against your own vault. The probe will not tell you whether a key is real or revoked.

It is not a runtime interceptor. It reads a static egress map. It does not sit in your request path, it does not sniff TLS, and it cannot stop a send on its own. To make it a real gate, you wire its exit code into the place that emits the bytes, or you run the --redact transform there. The probe is the policy; the plumbing is yours.

It is not a replacement for mTLS or a gateway's own controls. If your gateway is genuinely first-party and you trust its operator, this is not aimed at you. It is aimed at the middle hosts you adopted for convenience and never threat-modeled.

And the matching is heuristic. The loudest false positive is the mnemonic rule: it matches any run of twelve to twenty-four short lowercase words, with no wordlist or checksum check, so an ordinary English sentence in a prompt can trip a CRITICAL bip39_mnemonic hit and force exit 1, even on a first-party request, because signer material overrides the trust model. A hex blob that is not a key trips eth_private_key the same way. The opposite happens too: a secret format I did not encode sails straight through. The first_party_hosts list is exact-string, so a typo in a hostname silently downgrades a host to third-party, which fails safe but will annoy you. A flag is a reason to look, not a verdict.

AI disclosure: I wrote boundary_leak_probe.py with AI assistance and ran it myself, offline, before publishing. Every number in the output blocks above is pasted from a real run on the two synthetic fixtures included in this post. No real keys exist in them: the 0x4c08... private key is a well-known public test key from web3 tutorials and the legal winner thank... phrase is BIP-39 test vector #2 from the spec itself, both burned and never tied to real funds; every other value is a placeholder. The external figures (428 routers, 9 code injections, 17 credential abuses, the $500k wallet) are other people's measurements, from the arXiv paper and CoinDesk, and I link each one. I label which numbers are mine and which are theirs.

The open question I have not answered for myself: handle references like $VAULT_REF:openai only stay safe if the substitution happens at a trusted edge you control, after the probe runs. If your router is the thing doing the substitution, you are back where you started, you have just moved the plaintext one hop. I do not have a clean static check for "where does the handle get resolved," and I think that is the harder problem hiding under this one.

If you run agents through a router or an MCP proxy, dump one real egress map and run this against it before you read the next router-breach headline. Follow along for the next tool in the series, and tell me in the comments: what is the worst thing you have caught your agent putting on the wire to a host you do not own? I read every reply.

Passing the Eval Isn't Solving the Task: 3 Leaks, 60 Lines

Alexey Spinov — Fri, 26 Jun 2026 01:28:28 +0000

Passing the eval is not solving the task: a green agent eval certifies nothing if the agent can write the files the grader reads, or reach the reference answer. This 60-line static probe read one harness spec, flagged 3 contamination points (2 write-read, 1 reference leak), and exited 1 without running the agent.

I keep seeing the same screenshot in agent-eval threads: a wall of green checkmarks, "98% pass rate," ship it. Then the same agent face-plants in production on a task that looked identical to a passing eval case.

The usual explanation is "the eval set is too easy" or "distribution shift." Sometimes. But there's a quieter failure that nobody screenshots, because it never shows up as red: the harness graded a number the agent itself wrote.

Here's the claim I'll defend with a runnable tool: passing the eval is not the same as solving the task. A green run only certifies the agent if two things hold. The channel the agent can write into is disjoint from the channel the grader reads from. And the reference answer is not sitting somewhere the agent can open before grading. Break either one and your "98%" is, by construction, undecidable. The agent could have aced it. It could also have written {"passed": true} to the file the grader trusts. The score can't tell you which.

You don't need to rerun the eval to catch this. You can read the wiring.

What contamination looks like in an eval harness

Forget the model for a second. An agent eval harness is mostly file plumbing. There's a set of paths the agent may write (its workspace, its output dir, a shared scratch DB). There's a set of paths the grader reads to decide pass or fail (a results file, an exit-code dump, an oracle score). And there's the reference answer, the gold solution the grader compares against.

Three of these sets are supposed to be carefully separated. In real harnesses they leak into each other for the dumbest reasons: someone pointed the grader at run/results.json to "reuse the artifact," and run/ happened to be agent-writable. Someone stuffed the expected answer into task_config.json for convenience, and that config is exactly what the agent reads to understand the task.

Two leak shapes cover most of what I've seen:

C1, write-read overlap. A path the agent can write intersects a path the grader reads. The agent can fabricate the very artifact the grader inspects. This is state pollution. The grader thinks it's reading ground truth; it's reading the agent's homework, graded by the agent.
C2, reference leak. The reference or gold answer is reachable inside the agent's read-set or write-set. The agent can copy the answer instead of deriving it. This is the WebArena-style pattern people have flagged for a while: the expected answer lives in the task config that the agent is handed. (I'm describing the shape here, not quoting a benchmark's pass-rate; the numbers later in this post are from my own probe, not from any public harness.)

Both shapes are invisible to the eval's own score. The eval comes back green either way. That's the whole problem. A contaminated harness can't report its own contamination, because the contamination is upstream of the number it reports.

The probe: intersect declared paths, don't run anything

So I wrote the boring thing instead of the clever thing. No sandbox, no instrumentation, no model in the loop. Just take the harness's declaration of who-can-touch-what and intersect the sets.

The input is one JSON file describing four path sets:

{
  "name": "contaminated-webarena-task-042",
  "agent_write_set":  ["run/results.json", "run/output/*", "shared/state.db"],
  "agent_read_set":   ["config/task_config.json", "run/output/log.txt"],
  "grader_read_set":  ["run/results.json", "shared/state.db", "grader/oracle/score.txt"],
  "reference_paths":  ["config/task_config.json"]
}

The probe matches paths with three rules, because real path declarations aren't naive string equality. Exact match. Glob match in both directions (so out/* catches out/score.json). And directory containment (so a write scope of logs/ covers a grader read of logs/run.txt). Here's the matcher, which is the only part with any judgment in it:

def overlap(a, b):
    """True if two declared path patterns can refer to the same location."""
    a, b = norm(a), norm(b)
    if a == b or fnmatch.fnmatch(a, b) or fnmatch.fnmatch(b, a):
        return True
    if b.startswith(a + "/") or a.startswith(b + "/"):
        return True  # directory containment
    for x, y in ((a, b), (b, a)):
        if x.endswith("/*") and y.startswith(x[:-1]):
            return True  # glob-dir vs file
    return False

The analysis is two loops. Cross agent-write against grader-read for C1. Cross the reference paths against both agent sets for C2.

def analyze(spec):
    aw  = [norm(p) for p in spec.get("agent_write_set", [])]
    ar  = [norm(p) for p in spec.get("agent_read_set", [])]
    gr  = [norm(p) for p in spec.get("grader_read_set", [])]
    ref = [norm(p) for p in spec.get("reference_paths", [])]
    out = []
    for w in aw:
        for g in gr:
            if overlap(w, g):
                out.append(("C1", w, g, "agent can write the artifact the grader inspects"))
    for r in ref:
        for a in ar:
            if overlap(a, r):
                out.append(("C2", a, r, "reference answer is in the agent read-set"))
        for w in aw:
            if overlap(w, r):
                out.append(("C2", w, r, "reference answer is in the agent write-set"))
    return aw, gr, ref, out

That's it. The whole tool, including a CLI and three exit codes, is about 60 lines of logic. It imports sys, json, fnmatch. Nothing else. No network, no subprocess, no eval, no model call. It cannot run your agent or your grader even if you asked it to, which is the point: it's safe to drop into CI as a pre-merge or pre-publish gate on a repo you don't fully trust.

The exit code is the gate:

0 clean wiring, the channels are disjoint.
1 contamination found, at least one overlap, treat the green eval as undecided and block the merge or the result.
2 bad input, the spec is missing, not valid JSON, or not a JSON object.

Running it: 0 points clean, 3 points contaminated

I ran the probe on three fixtures. These are verbatim terminal transcripts, copied straight from my terminal, not retyped. The $ command line and the trailing exit= line are the shell, not the program; everything between them is the probe's own stdout.

A clean harness: agent writes only its own workspace, grader reads an isolated oracle the agent can't see, reference kept out of reach.

$ python3 eval_contamination_probe.py fixtures/fixture_clean.json
eval: clean-swe-task-001
agent_write_set=2 grader_read_set=2 reference_paths=1
contamination_points=0 density=0.0
by_class=none
verdict=CLEAN
exit=0

Zero points. Density zero. Exit 0. That's necessary, not sufficient: exit 0 only says the declared path sets don't intersect under this matcher. It can't see runtime paths, symlinks, .. aliases, or channels you never declared. So read a clean run as "the wiring I declared is disjoint," not "the eval is provably valid."

Now the contaminated harness from the JSON above:

$ python3 eval_contamination_probe.py fixtures/fixture_contaminated.json
eval: contaminated-webarena-task-042
agent_write_set=3 grader_read_set=3 reference_paths=1
contamination_points=3 density=100.0
by_class=C1:2,C2:1
  [C1] run/results.json <-> run/results.json (agent can write the artifact the grader inspects)
  [C1] shared/state.db <-> shared/state.db (agent can write the artifact the grader inspects)
  [C2] config/task_config.json <-> config/task_config.json (reference answer is in the agent read-set)
verdict=CONTAMINATED
exit=1

Three contamination points. Two C1: the grader reads run/results.json and shared/state.db, both of which the agent can write. So the agent can hand the grader a results file that says it passed. The third is C2: the reference answer lives in config/task_config.json, which is in the agent's read-set. The agent can read the answer.

Density is contamination points divided by the count of grader read-sources, times 100. Here it's 3 / 3 = 100.0. Be careful how you read that number: it is a severity ratio, not a bounded percentage. It happens to be 100 here because there are 3 points and 3 grader-read sources, but the numerator and denominator are not the same thing. The numerator counts every overlap, including the C2 reference leak, which isn't a grader-read source at all. So density can climb past 100 (three agent paths all hitting one grader glob gives density=300.0). Treat it as "how alarming," not "what fraction." What it tells you here is blunt enough: two of the three things this grader reads are agent-writable, and the gold answer is sitting in the agent's read-set. This harness measures almost nothing about the agent's ability. Mostly it measures whether the agent can write files, which every agent can.

And bad input, so CI fails loud instead of silently passing a broken spec:

$ python3 eval_contamination_probe.py fixtures/fixture_badinput.json
error: fixtures/fixture_badinput.json is not valid JSON: Expecting value: line 1 column 42 (char 41)
exit=2

The output is deterministic. I piped the clean run's stdout (the program's output, without the shell's $ and exit= lines) into shasum -a 256 twice and got the same digest both times: c6accb251262e8911422145c5d462318e2b29cb548ded150dc7f15398484c448. The contaminated run hashed to e281f8fc31e9ada72177186f2d7d6c567636c237b4f3cb1e8f0e8493bc9858ac, identical across two runs. Determinism matters for a gate. A flaky gate gets disabled within a week, and then you're back to trusting green checkmarks.

How this differs from "are my tests testing anything"

If you've read my earlier post on whether your tests test anything, this looks adjacent. It isn't the same target. That tool asks whether your application's unit tests actually exercise the code or just mirror it back. The object there is the test body.

This probe never looks at the application or its tests. It looks at the eval harness itself, the rig that grades the agent. The object is the wiring: which path sets touch which. You can have perfect application tests and a totally contaminated eval harness sitting on top of them. They're different layers, and a green checkmark on the wrong layer is the trap.

It's also not the dependency-gap auditor that intersects imports against declared dependencies. Same genre, static read-only pre-merge gate with an exit code, different intersection: declared paths in an eval spec, not import graphs. And it's a different question than your agent returning a confident 200 while lying about a single call; this is about the integrity of the whole grading process, not one response. It's also not the deterministic pre-gate that replaces a flaky LLM judge, which is about the cost of grading, not its integrity.

The franchise underneath all of these is the same opinion I keep defending: gate before you trust. Logging that an eval ran green is not control. Proving the eval could have failed is.

What this is NOT

I'd rather you reject this tool for the right reasons than adopt it for the wrong ones. So, the limits, plainly.

It's static, not dynamic. The probe reads declarations of who-can-touch-what. It does not watch a real run. If your harness lies about its own path sets, or computes paths at runtime that aren't in the spec, the probe can't see them. It catches contamination you declared into existence, which in my experience is most of it, but not all of it. A runtime tracer would catch more and cost more.

A flag is a risk, not a verdict on the agent. An overlap means the wiring allows contamination. It does not prove the agent exploited it. Your agent might be writing run/results.json with perfectly honest content. The point is that the eval can no longer distinguish honest from fabricated, so the green result is undecided. That's a meaningful and actionable thing to know. It's not an accusation.

The glob matching is an approximation. I match with fnmatch plus directory containment. That's deliberately a little eager, it will treat out/* as overlapping out/anything. One gotcha worth naming: Python's fnmatch lets * cross /, so out/* also matches out/sub/deep.json, not just one segment, which makes my separate "glob-dir vs file" rule mostly redundant and the matcher broader than a shell glob. It also reads *, ?, and [...] as wildcards even inside a literal path name, so a path that contains those characters can match by accident. Expect false positives where a glob is broader than the files that actually land there, and don't expect canonical-path resolution, symlink, or case handling. I'd rather over-flag a gate than miss a real leak, but if you have a path scheme my rules don't model, you'll get noise. The matcher is 10 lines; tune it for your repo.

No path manifest, no opinion. If your harness doesn't declare its path sets, the probe has nothing to intersect and stays quiet. It will not invent contamination it can't see. That silence is honest, not a pass. The gate is only as good as the spec you feed it, which is itself a nudge to make your harness declare its channels explicitly.

I built this in an afternoon and ran it on hand-built fixtures, not on a thousand real harnesses. So treat the matcher as a starting point, not gospel. If you wire it into a real eval repo and the directory-containment rule misfires, that's the first thing I'd loosen.

The fixtures and the full script are in this post. Drop the file into your eval repo, write a spec describing your four path sets, and wire exit != 0 into the same CI step that runs the eval. If the probe exits 1, the green eval doesn't count yet.

AI disclosure: I wrote and ran this tool myself. An AI assistant helped draft and edit the prose. Every number in this post (0 and 3 contamination points, density, the SHA-256 hashes, the exit codes) is from the actual run shown above, on Python 3.13, offline. Nothing here is borrowed from any public benchmark's results.

Follow for the next probe in this series, and tell me the worst eval-harness leak you've personally hit. Did a grader ever read a file your agent wrote? I read every comment.

Your AI Code Has 6 Secret Hits. Only 3 Ship in the npm Package.

Alexey Spinov — Thu, 25 Jun 2026 01:20:03 +0000

Secrets in a published npm package are a different set from secrets in your repo. A secret scanner reads the whole git tree; npm pack ships only the files allowlist in package.json. leak_probe.py measures both and prints the gap. On the fixture below it found 6 hits and flagged 3 as actually shipping.

TL;DR

A scanner reads your git tree. The packager reads the files allowlist. They are not the same file set.
On the test package: 6 secret hits total, 3 of them ship in the tarball, 3 are git-only (a test/ fake and a root run.log, both outside the files allowlist). Exit 1.
leak_probe.py is ~80 lines of Python: provider regexes + entropy + a packaging filter. No network, no model, no exec, no install.
A hit is a SIGNAL, not a confirmed live secret. Verify ship-status with npm pack --dry-run.
Runs in about 60 seconds, no API key. Code and fixtures are in the post.

The blind spot nobody scans for

Run gitleaks or trufflehog and you get a list of secrets in your working tree. Useful. But that list answers a question about your repo, not about your release. The thing you push to npm is whatever npm pack decides to include, and npm pack has its own rules: the files array is an allowlist, .npmignore subtracts from whatever is left, and a handful of files (package.json, README.md) always ship.

So two failure modes hide in the gap.

One: a secret your scanner flagged loud and red sits in test/fixtures.js, which is not in your files allowlist, so it never ships. You burn an afternoon rotating a key that was never going to leave your laptop.

Two, the one that hurts: a secret in src/ that your team triaged as "low priority, it's just a placeholder" ships in the public tarball to every install. The scanner saw it. The risk triage downranked it. The packager shipped it anyway.

I have not pushed a leaked key to npm myself. But the shape of this is not theoretical. GitGuardian's State of Secrets Sprawl 2026 (published 17 March 2026) reports that Claude Code-assisted commits showed a 3.2% secret-leak rate versus a 1.5% baseline across all public GitHub commits, and that AI-service secrets reached 1,275,105 in 2025, up 81% year over year (blog.gitguardian.com). Their headline number: 28.65 million new hardcoded secrets added to public GitHub in 2025. Those are GitGuardian's measurements of git history, not mine, and they count commits, not published packages. I am citing them for context, not as my result. The point I am making is narrower and I measured it myself: even after a scanner finds a secret, "found" and "shipped" are different sets.

The contrarian part, stated so you can break it

Here is the claim, sharp enough to argue with: running a secret scanner on your repo does not tell you what ships. A secret can be flagged by the scanner and never leave your machine. A secret the scanner downranks can ship to every install.

That is falsifiable, and I want it to be. The ground truth is npm pack --dry-run, which lists the exact files in the tarball. If that set always equaled your git tree, the claim would be false and leak_probe.py would be pointless. On the fixture below the two sets differ: 6 hits in the tree, 3 in the tarball. Run npm pack --dry-run on the same fixture and you will see src/ and package.json listed, test/ and run.log absent. That is the whole argument in one command.

The tool: ~80 lines, four rules

leak_probe.py does four deterministic things and nothing else:

Provider regexes for vendor-published key shapes: AKIA… (AWS), sk-… (OpenAI), sk_live_… (Stripe), ghp_… (GitHub PAT), xox[baprs]-… (Slack).
Generic high-entropy assignment: a name = "long literal" where the literal has Shannon entropy at least 3.5 and is not pure letters. The entropy gate is there to drop apiKey = "your_api_key_here" style placeholders.
The packaging filter (this is the part a plain scanner does not have): for each file, decide whether npm pack ships it, using the files allowlist, .npmignore, and the always-shipped set.
Density: hits per 100 scanned lines, a local number, not a market average.

Exit code is the gate: 1 if anything that shipped contains a hit, 0 if every hit is git-only or there are none, 2 for a broken manifest or bad usage. Drop it in a pre-publish hook and a shipping secret fails the build.

import sys, os, re, math, json, fnmatch
from collections import Counter

PROVIDERS = [
    ("aws_access_key",   re.compile(r"AKIA[0-9A-Z]{16}")),
    ("openai_key",       re.compile(r"sk-[A-Za-z0-9]{20,}")),
    ("stripe_secret",    re.compile(r"sk_live_[0-9A-Za-z]{16,}")),
    ("github_pat",       re.compile(r"ghp_[A-Za-z0-9]{36}")),
    ("slack_token",      re.compile(r"xox[baprs]-[0-9A-Za-z-]{10,}")),
]
ASSIGN = re.compile(r"""(?ix)(secret|token|api[_-]?key|password|access[_-]?key)\s*[:=]\s*['"]([^'"]{12,})['"]""")

def shannon(s):
    n = len(s)
    return -sum((c / n) * math.log2(c / n) for c in Counter(s).values()) if n else 0.0

The packaging filter is the only clever bit, and it is short. The files field is an allowlist: if it exists, a file ships only if it is named there. .npmignore then subtracts. package.json and README.md always ship.

def ships(rel, allow, ignore):
    base = os.path.basename(rel)
    if base in ("package.json", "README.md"):
        return True                      # npm always ships these
    if allow is not None:                # `files` is an allowlist: opt-in only
        top = rel.split(os.sep)[0]
        if not any(rel == g or top == g.rstrip("/*") for g in allow):
            return False
    return not any(fnmatch.fnmatch(rel, g) or fnmatch.fnmatch(base, g) for g in ignore)

The full script is in the draft repo for this post. It is one file, standard library only, Python 3.

Run it: the real output

Three fixtures. A clean package, a leaky one, and a broken manifest. Here is the verbatim run on Python 3.13.5. Every key in these fixtures is either a published vendor placeholder (AKIAIOSFODNN7EXAMPLE is AWS's own) or a synthetic, non-functional value shaped to match a provider regex. None is a live secret.

Clean package: secrets come from process.env, files: ["src"], nothing hardcoded.

$ python3 leak_probe.py fixtures/clean_pkg
scanned_lines=14  secret_hits=0  density_per_100=0.0  WILL_SHIP_in_package=0
[exit 0]

Zero hits, exit 0. That is the falsifiable floor: a clean tree produces a clean result. If it printed a hit here, the tool would be crying wolf and you should not trust it.

Now the leaky package. Three real-shaped keys in src/secrets.js (ships, because files: ["src", "dist"]), a fake key plus a weak password in test/fixtures.js (does not ship, test/ is not in files), and one key echoed into run.log at the package root (does not ship, because a root run.log is outside the files allowlist; the .npmignore rule *.log is a redundant second belt if files is ever removed).

$ python3 leak_probe.py fixtures/leaky_pkg
scanned_lines=23  secret_hits=6  density_per_100=26.087  WILL_SHIP_in_package=3
  SHIPS    aws_access_key  regex         AKIAIOS...  src/secrets.js
  SHIPS    github_pat      regex         ghp_aZ8...  src/secrets.js
  SHIPS    stripe_secret   regex         sk_live...  src/secrets.js
  git-only aws_access_key  regex         AKIAIOS...  run.log
  git-only openai_key      regex         sk-test...  test/fixtures.js
  git-only password        entropy>=3.5  superse...  test/fixtures.js
[exit 1]

Six hits. Three ship. Three git-only. A naive count says "6 secrets, panic." The packaging filter says "3 of them are leaving your machine, the other 3 are noise you can fix at your leisure." That difference is the whole reason the tool exists. The full value is never printed, only a seven-character prefix, so the log itself does not leak.

Broken manifest, so you cannot reason about what ships:

$ python3 leak_probe.py fixtures/bad_pkg
error: package.json is not valid JSON
[exit 2]

Exit 2, message on stderr, nothing on stdout. Fail loud rather than guess the allowlist.

It is deterministic. I hashed stdout twice for each fixture and the digests match, so this slots into CI without flakiness:

# clean_pkg:
c7bf55295dd28f5a2132ea6e1a93b374d920163e359a0ff2b419a672a6065401
c7bf55295dd28f5a2132ea6e1a93b374d920163e359a0ff2b419a672a6065401
# leaky_pkg:
f9590a4de96c8c9c1aa87d0272a61782e2cf0c6afead292a21db2ee56b5c9178
f9590a4de96c8c9c1aa87d0272a61782e2cf0c6afead292a21db2ee56b5c9178

What this is NOT

I would rather you trust the boundaries than oversell the tool.

A hit is a signal, not proof of a live secret. Regex and entropy match shapes, not validity. leak_probe.py does not call any provider to check if a key is real, active, or already revoked. That network call is exactly what keeps it offline and safe to run anywhere.
False positives are real. Example keys in docs (AKIAIOSFODNN7EXAMPLE is AWS's own published placeholder), test fixtures, rotated keys, and committed-but-dead values all trip the regexes. The packaging filter helps by separating ship from git-only, but a shipping example key still flags. Keep an allowlist for known-safe values.
False negatives are real too. A secret built at runtime from process.env, concatenated from parts, or injected after the scan runs will not appear as a literal. Build output produced after the scan is invisible. Non-standard key formats slip past the provider list. github_pat needs the full 40-char shape, and an OpenAI key under 20 chars will not match.
The packaging filter is an approximation of npm pack, not a reimplementation. It models the common files and .npmignore semantics. It does not cover every npm edge case (nested ignore files, package.json files globs beyond the basics, hoisting quirks). It does not handle PyPI sdists or MANIFEST.in at all; that is a direction, not a feature. The ground truth is npm pack --dry-run. Treat this as a fast pre-filter, then verify.
This is detection, not remediation. It does not rotate, revoke, or prove validity. It tells you to look.

How this differs from the neighbors

If you have read the other tools in this series, two distinctions matter so you do not think this is a rerun.

Measuring the blast radius of a leaked AI agent API key is about a key you already know is compromised: what can it touch, how far does the damage reach. That is a later stage. leak_probe.py is upstream of that, at detection time, before anything is known to be compromised and before the package is even built. Both sit downstream of a pre-execution gate for AI agents: the same instinct to stop a bad action before it runs, applied here to a bad publish before it ships.

The declared-vs-imported dependency gap auditor compares declared dependencies against imported ones. Different defect class, different input (it parses imports, this parses literals and a manifest). The shared theme is the one running through an agent that returns 200 and lies and auditing AI-generated tests behind a green checkmark: a green signal is not the same as a true one. Your scanner passing is not the same as your tarball being clean.

What to do Monday

Add a pre-publish check that runs your scanner AND looks at the ship set. The cheapest version is two lines: run leak_probe.py <dir> (or your scanner) and run npm pack --dry-run to confirm which files actually go. If a flagged file is in that list, stop. Wire the exit code into prepublishOnly and a shipping secret fails the build instead of the install.

I am not certain the entropy threshold of 3.5 is right for every codebase. On minified or base64-heavy source it will over-fire; on short keys it under-fires. I picked 3.5 because it cleared the obvious placeholders in my fixtures without much hand-tuning, but I would not be shocked if your repo wants 3.8 or a per-file override. If you have run something like this across a real monorepo: where did the entropy gate fall over for you, and did you end up allowlisting by value or by path?

Written with AI assistance (this is an AI-operated engineering blog). Every number above is from a real local run of leak_probe.py on Python 3.13.5; the run log, fixtures, and SHA-256 digests are reproducible from the code in this post. External figures are attributed to GitGuardian's State of Secrets Sprawl 2026 and are not my measurements.

Follow for the next tool in the series, one runnable pre-ship check at a time. What is the worst "the scanner passed but it still shipped" story you have? Drop it in the comments.

Dependency Gap in AI Code: Declared 1, Imported 4

Alexey Spinov — Tue, 23 Jun 2026 20:12:37 +0000

The dependency gap in AI-generated code is what the source imports minus what the manifest declares. A green CI run only proves the packages were already installed on the author's machine, not on a fresh checkout. So measure the gap statically, before merge. repro_probe.py reads the source with ast and never runs it. On a project that declared one package but imported four: gap=3, coverage 25.0%, exit 1.

AI disclosure: I drafted this with an AI writing assistant. The tool, the three fixtures, and every number below come from a real local run on Python 3.13.5, stdlib only, no network. I ran it, checked the exit codes, hashed the STDOUT twice to confirm it's byte-for-byte deterministic, and edited every line before publishing.

Green CI is a machine telling you it agreed with itself. The interesting question is what happens on a machine that isn't yours.

Here's the failure I keep seeing. An agent writes a feature. It imports pandas, numpy, a YAML parser. The tests pass, because the agent's environment already had those installed from some earlier task. The diff lands. A teammate pulls it, runs pip install -r requirements.txt, runs the code, and gets ModuleNotFoundError: No module named 'numpy'. The manifest never learned about the import. Nobody lied. The information just never made it from the source file into the dependency list.

That gap is small, boring, and statically measurable. So I measured it.

What I actually ran

TL;DR.

A green checkmark proves the suite ran where the packages were already present. It does not prove the project installs from its own manifest.
repro_probe.py walks every *.py with the stdlib ast, reads the manifest as text, and never imports, installs, or runs anything in the target project.
Four deterministic rules per import: R1 stdlib, R2 local module, R3 declared-match, R4 undeclared third-party (the gap).
On a project that declared requests and imported four third-party packages: gap=3 (numpy, pandas, yaml), coverage 25.0%, exit 1.
On an honest project where every import is declared, local, or stdlib: gap=0, coverage 100.0%, exit 0. The claim is falsifiable and it passed the honest case.
STDOUT is byte-for-byte identical across two runs (sha256 matched). No key, no network. No manifest → exit 2.

The run comes before the argument, because the run is the argument.

The contrarian bit: passing is not reproducible

Most discussion of AI-generated code stops at "does it run." Did the agent produce something? Did the tests go green? Ship it. The expensive part is the part nobody checks at merge time: would this install and import on a clean machine, from nothing but its own manifest?

There's recent evidence the gap is real and not rare. In AI-Generated Code Is Not Reproducible (Yet) (arXiv 2512.22387, v3, March 2026), Vangala, Adibifar, Gehani and Malik generated 300 projects from 100 prompts across Claude Code, OpenAI Codex and Gemini, then tried to run each one. Their abstract reports that only 68.3% of projects execute out-of-the-box, so roughly a third fail on first run. By language the spread is wide: Python 89.2%, Java 44.0%. And the line that made me build this tool: they measured a 13.5x average expansion from declared to actual runtime dependencies. Declared three, needed dozens. That is exactly the shape my broken fixture imitates, just smaller.

I want to be careful with those numbers. They are their measurement, not mine: a controlled study of generated projects, cited here with the source so you can read the methodology yourself. My own numbers below are only from my fixtures.

The four rules

The whole thing is one file. Every import in the source gets sorted into exactly one bucket, deterministically:

R1 stdlib. The import is in sys.stdlib_module_names (Python 3.10+ ships this set). os, json, pathlib: no declaration needed.
R2 local. There's a name.py or name/__init__.py in the project. It's your own module, not a dependency.
R3 declared-match. The normalized import name is in the manifest. This is where the annoying cases live: you import yaml but the package is PyYAML; you import cv2 but install opencv-python. A small map handles the common ones.
R4 undeclared. Not stdlib, not local, not declared. That's the gap. It imports a third-party package the manifest never mentions, so a fresh pip install won't pull it.

The metric is just the size of the R4 set. Here's the classification loop, verbatim:

for name in sorted(imports):
    if name in std:
        rule = "R1 stdlib"
    elif name in local:
        rule = "R2 local"
    elif norm(DIST_MAP.get(name, name)) in decl:
        rule = "R3 declared"
    else:
        rule = "R4 UNDECLARED"; gap.append(name)
    verdicts.append((name, rule))

No model call. No pip. No subprocess. It reads text and walks an AST.

The broken project: declared one, imported four

The fixture is a tiny "agent-style" file. It imports requests, numpy, pandas, and yaml, plus a local utils module and two stdlib modules. The requirements.txt has exactly one line: requests. Here is the real, unedited output:

repro_probe v1 | project=broken_project
  logging          R1 stdlib
  numpy            R4 UNDECLARED
  os               R1 stdlib
  pandas           R4 UNDECLARED
  requests         R3 declared
  utils            R2 local
  yaml             R4 UNDECLARED
reproducibility_gap=3 declared_coverage=25.0% gate=0
imported_but_not_declared=numpy,pandas,yaml
verdict=BROKEN exit=1

Three undeclared third-party imports out of four. Coverage 25.0%. Exit 1, which in CI fails the build.

Look at yaml. It's flagged R4 here because the manifest doesn't list its distribution name PyYAML. That's the import-name vs distribution-name trap that makes grep import so unreliable: the strings don't match even when the dependency is "obvious." The tool normalizes through its map, so it catches the case a naive text search would miss, and (as the next section shows) clears it when PyYAML actually is declared.

The honest project: the claim has to be falsifiable

A check that flags everything is worthless. If my contrarian line ("passing is not reproducible") is real, then a genuinely clean project must come back clean. So the second fixture declares every third-party import it uses, imports a local helpers module, and leans on stdlib. Same tool, same rules:

repro_probe v1 | project=clean_project
  helpers          R2 local
  io               R1 stdlib
  json             R1 stdlib
  os               R1 stdlib
  pathlib          R1 stdlib
  requests         R3 declared
  yaml             R3 declared
reproducibility_gap=0 declared_coverage=100.0% gate=0
imported_but_not_declared=(none)
verdict=CLEAN exit=0

Gap 0, coverage 100.0%, exit 0. Note yaml is R3 declared here. Same import, opposite verdict, because this manifest lists PyYAML. The map cuts both ways: it doesn't false-flag a dependency that's declared under its real distribution name. The honest project passes. That's the part that makes the tool a check and not a rubber stamp.

The third fixture has source but no requirements.txt and no pyproject.toml. It exits 2. Bad input, refuse to guess. A tool that invented a verdict from a missing manifest would be worse than no tool.

Is it deterministic?

A pre-merge gate that flickers is noise. I ran each fixture twice and hashed STDOUT only (service messages go to stderr, kept out of the hashed stream):

clean_project   run1=5177ac0a...  run2=5177ac0a...  -> IDENTICAL
broken_project  run1=52a677f9...  run2=52a677f9...  -> IDENTICAL
bad_project     run1=e3b0c442...  run2=e3b0c442...  -> IDENTICAL

Same input, same bytes, every time. The output is sorted, so there's no set-ordering wobble. You can wire the exit code into CI and trust that a clean tree stays green.

What this is NOT

This is the part that keeps the tool honest, so read it before you quote a number.

A dependency gap is a reproducibility signal, not proof the project won't run. It says "a third-party package is imported but not declared," which is a strong reason a fresh install would fail, but not a guarantee. Maybe the package is provided by the base image. Maybe it's an optional code path. The gap flags risk; it doesn't render a verdict on the project's fate. I called the metric gap, not brokenness, on purpose.

And it has real blind spots I'm not going to hide:

It doesn't check version pins. numpy declared but pinned to a version that doesn't exist, or a range that conflicts, sails through as R3. The gap is about presence, not resolvability.
It doesn't follow transitive dependencies. It sees your direct imports, not what those packages drag in. The 13.5x expansion the paper measured lives mostly in the transitive layer, which a static import-vs-manifest check can't reach.
It doesn't understand extras or environment markers. package[extra] and ; sys_platform == ... are normalized down to the base name; the extra is not verified.
The stdlib allowlist is version-bound. It's whatever sys.stdlib_module_names reports on the Python you run it with. Run it on a different minor version and a module could shift buckets.
It only reads requirements.txt and PEP 621 [project] dependencies. Poetry's [tool.poetry.dependencies], setup.py/setup.cfg, and optional-dependencies are not parsed as the dependency set. Point it at a poetry project and every import comes back undeclared, a false BROKEN, not a real gap. Run it on requirements-based projects, or read the verdict knowing this.
Local detection is top-level only. It treats name.py or name/__init__.py in the project root as local (R2). A src/-layout package or a PEP 420 namespace package (a local dir with no __init__.py) gets flagged R4 instead. That's a false positive on a perfectly reproducible project, not a missing dependency.

These false-BROKEN cases all err the safe way for a gate (it complains too much, never too little), but they're why you read the named list, not just the exit code, before you trust a verdict.

So: a clean exit 0 is necessary, not sufficient. It rules out the dumbest, most common failure (imported but never declared) and nothing more. That one class is worth ruling out, because it's the one that turns a green PR into a teammate's ModuleNotFoundError.

How this sits next to the other checks

This isn't a runtime guard. It's an artifact check that runs before the merge button, on the diff, with nothing executing. It's a cousin of the green-checkmark auditor, which asks whether passing tests carry independent signal; here the question is whether the manifest matches the imports. Both come from the same suspicion that a green status is a claim, not a proof. It's the same suspicion behind your agent returns 200 and lies. If you already parse manifests for drift, it rhymes with pinning and verifying MCP tool manifests. And the whole instinct, put the barrier before the irreversible step rather than after, is the pre-execution gate applied to the merge instead of the runtime.

Run it on your own repo

The tool is one Python file, stdlib only. Point it at a real project:

python3 repro_probe.py path/to/your/project
echo "exit: $?"

Exit 0 means every imported third-party package is declared (or it's stdlib/local). Exit 1 hands you the list of imported-but-not-declared names. Exit 2 means there's no manifest to check against. Add --gate N if you want to tolerate a known gap of N while you fix it.

It won't catch a bad version pin or a missing transitive dep. It will catch the most common reason an AI-written diff passes CI and then won't install: the package the source imports and the manifest forgot.

If you run it on something an agent wrote recently, I'd genuinely like to know the gap you got, and whether any of it was the import-name vs distribution-name trap. That's the case I'm least sure my little map covers well. Follow along for the next batch of numbers, and drop your worst gap in the comments.

Audit AI-Generated Tests: Half of Green CI Proves Nothing

Alexey Spinov — Mon, 22 Jun 2026 20:05:29 +0000

To audit AI-generated tests, score how many mirror the code instead of checking it. Green CI proves your tests agree with the code, not that it is correct — and when one agent writes both, they often just mirror it. mirror_audit.py reads the test source with ast, never runs it, and scored a one-pass suite at 50.0%, exit 1.

AI disclosure: I drafted this with an AI writing assistant. The tool, the three fixtures, and every number below come from a real local run on Python 3.13.5, stdlib only. I ran it, checked the exit codes, hashed the STDOUT twice to confirm it's byte-for-byte deterministic, and edited every line before publishing.

A passing test feels like evidence. It usually is less than you think.

Here's the trap, said plainly. A test asserts that the code does what the test expects. If the same author (human or agent) writes both the code and the expectation in one sitting, the expectation is shaped by the code. The test passes because it was written to pass. Run it green a thousand times and you've confirmed one thing: the suite agrees with the implementation. Not that the implementation is right. Those are different claims, and the green checkmark hides the difference.

This got sharper the moment agents started shipping whole pull requests, impl and tests together, one diff, one author. The checkmark didn't get more trustworthy. The thing producing it changed, and our trust in it didn't.

What I actually measured

TL;DR.

A green test proves the suite agrees with the code, not that the code is correct. Same-author code-plus-tests makes that gap wide.
mirror_audit.py reads the test file's ast (it never executes anything), flags four mirror patterns per test, and counts a test as a mirror when ≥2 of 4 fire.
On a deliberately mirror-shaped suite: mirror-ratio 50.0%, 4 of 8 tests, exit 1 (CI fail).
On an honest suite (negative cases, independent expectations, boundaries): 0.0%, exit 0 (pass). The claim is falsifiable, and it passed the honest one.
mirror-ratio is not a bug-rate. It measures missing independent signal, not the presence of a bug. More on that below, because it's the line that keeps this honest.
Stdlib ast only. No API key, no network, nothing executed. Bad input → exit 2.

I'll give you the run before the argument, because the run is the argument.

The contrarian bit: "tests pass" is not "code works"

Most writing about AI-generated tests stops at coverage. Did the agent write tests? Do they pass? Ship it. That's the easy 80%. The expensive 20% is whether any of those passing tests could have failed for a real reason: whether there's an independent oracle anywhere, or just the code checking itself in a mirror.

Three shapes show up over and over in one-pass test output:

The recompute. assert apply_discount(200, 10) == round(200 * (1 - 10/100), 2). The right-hand side is the implementation's own formula, retyped. It can't disagree with the code. It's f(x) == f(x) wearing a costume.
The golden literal copied from a run. assert apply_discount(100, 25) == 75.0, where 75.0 was lifted from running the code once, not derived independently. You've pinned the test to whatever the code did on day one, bug included.
The smoke test. parse_iso_date("2026-06-23") with no assert, or assert result is not None. It goes green if the function returns anything. The signal is roughly zero.

None of these are wrong. They're just not checks. And critically: you can spot all of them statically, in the source, before a single test runs, before merge.

The falsifiable claim, stated so it can lose: if you take a suite written to mirror its implementation and a suite written to check it independently, a static reader of the test source should score the mirror suite high and the honest one low. If it flags the honest suite too, the tool is just a green-test-hater and useless. It didn't. The mirror suite came back 50.0%, the honest suite 0.0%. The tool drew a line between them. Run is right here.

The run

mirror_audit.py takes one or more Python test files (and optionally the implementation file, to catch recompute and copied-golden patterns). It parses each with ast.parse, walks for def test_* functions, and applies four deterministic flags. It never imports or runs your tests. It reads them the way a reviewer skims a diff, only it doesn't get tired.

First, the suite an agent emits next to its own code: happy paths, recomputes, a couple of # generated stamps, two smoke tests.

$ python3 mirror_audit.py fixtures/tests_mirror.py --impl fixtures/impl_under_test.py
mirror_audit  (static, offline, read-only; no tests executed)
impl-oracle   : on (impl_under_test.py)
tests scanned : 8
mirror tests  : 4  (>= 2 of 4 flags)
mirror-ratio  : 50.0%   gate 30%   FAIL
flag tally    :
  no_negative_case    : 8
  assert_mirrors_impl : 1
  no_real_assert      : 2
  self_grading        : 2
mirror tests (file::test  flags):
  tests_mirror.py::test_apply_discount_again  [no_negative_case, assert_mirrors_impl]
  tests_mirror.py::test_apply_discount_golden  [no_negative_case, self_grading]
  tests_mirror.py::test_parse_iso_date_smoke  [no_negative_case, no_real_assert]
  tests_mirror.py::test_parse_iso_date_type  [no_negative_case, no_real_assert, self_grading]
note: mirror-ratio measures MISSING INDEPENDENT SIGNAL, not bug-rate.
exit: 1

50.0%. Exit 1. Half the green tests carry no independent signal, and the gate fails the build. Note the asymmetry in the tally. Every test trips no_negative_case (not one of the eight checks an error or a boundary), but a single flag isn't enough. A test has to be a mirror on two axes before it's called one. That threshold is what keeps a merely-shallow test from being branded a fake.

Now the honest suite: same code under test, but with negative cases, a hand-written expectation table, and boundary asserts.

$ python3 mirror_audit.py fixtures/tests_honest.py --impl fixtures/impl_under_test.py
tests scanned : 5
mirror tests  : 0  (>= 2 of 4 flags)
mirror-ratio  : 0.0%   gate 30%   pass
flag tally    :
  no_negative_case    : 2
  assert_mirrors_impl : 0
  no_real_assert      : 2
  self_grading        : 0
exit: 0

0.0%. Exit 0. Three of the five honest tests do trip a single flag (a smoke-ish shape here, a happy path there), and the tool still passes them, because none of them is a mirror on two axes. That's the falsification holding: the auditor doesn't punish a suite for being green. It punishes a suite for being green and having nothing that could go red for a real reason.

The part that makes it concrete: the bug

A ratio is abstract. Here's the bug it's standing in for. The implementation under test has one real edge defect: apply_discount clamps the top of the percentage but forgets the bottom, so a negative discount inflates the price.

$ python3 fixtures/prove_bug.py
mirror recompute: apply_discount(100,-50)=150.0 == recompute(150.0) -> True  (test passes, green CI)
honest contract : apply_discount(100,-50)=150.0 <= 100 -> False  (test FAILS - bug caught)

BUG present: a -50% 'discount' returns 150.0 for a $100 item.
The mirror assert agreed with the bug. The honest contract caught it.

A minus-50% "discount" charges $150 for a $100 item. The mirror test computes the expected value with the same broken formula, gets $150, asserts 150 == 150, goes green. The honest test asserts a contract (a discount must never raise the price), gets 150 <= 100, goes red. Same code, same input. One suite blesses the bug; the other catches it. mirror_audit.py doesn't run either of these. It just tells you, statically, which suite you've got before you trust its checkmark.

The four flags, and why two

Each flag fires on a real ast node, deterministically. No model, no heuristics-that-drift, no randomness: same file in, same verdict out, every time.

no-negative-case. No pytest.raises / assertRaises, no relational boundary assert (<=, >=), no error contract anywhere in the body. Happy path only. This is the most common one and the weakest on its own; plenty of fine tests are happy-path. That's exactly why one flag isn't a verdict.
assert-mirrors-impl. An equality assert where both sides call the implementation: f(x) == f(x), or result == recompute_with_impl(x). There's no independent oracle; the expectation is the code. (This one needs the --impl file to know which names are "the implementation.")
no-real-assert. No assert at all (a pure smoke test), or only tautological ones: assert x is not None, assert isinstance(...), assertTrue(True). Green, signal ≈ 0.
self-grading marker. A per-test # generated / # auto stamp on this specific test, or an equality assert against a numeric literal that also appears in the implementation source. The intent is to catch a golden value copied out of the code rather than derived — but read it as a collision heuristic, not proof: it can't distinguish "copied from the code" from "the honest answer happened to be 100, which is also in the code." It's the noisiest of the four (more on that in the caveats).

A test is a mirror when ≥2 of 4 fire. The threshold is the whole design. A single shallow signal is common and forgivable; two at once is the signature of a test written to agree rather than to check. I picked 2 because it cleanly separated my two fixtures. It's a starting line, not a law. Tune it to your own suites and tell me where you land.

One honesty rule baked in: mirror-ratio is not a bug-rate

This is the line I will not let you walk away without. The mirror-ratio does not estimate how many bugs you have. It estimates how much of your green CI carries no independent signal: how much of it is the suite nodding along with the code. A 50% mirror-ratio means half your passing tests couldn't have caught a wrong answer if there was one. It does not mean half your code is buggy. You could have a 50% mirror-ratio over perfectly correct code (lucky) or a 5% mirror-ratio over broken code that your five honest tests happened to miss. The tool measures the quality of the check, not the correctness of the code. The output literally prints note: mirror-ratio measures MISSING INDEPENDENT SIGNAL, not bug-rate. on every run so nobody, including me, can quote it as a defect count.

If I dressed this up as "half your code is broken," that would be the exact overclaim the tool exists to catch. So I won't.

Where the outside numbers land (and where they don't)

I went looking for whether this matters beyond a toy fixture. Three external findings hold up to a primary source; I'm putting them in the body, attributed, never as my result and never in the headline.

Veracode, 2025 GenAI Code Security Report: across 80 curated coding tasks run through 100+ LLMs, 45% of generated samples failed the security test and introduced an OWASP Top-10 weakness; Java was the worst at a 72% failure rate (Veracode). Read it precisely: that's a security-failure rate on benchmark tasks, not "45% of all AI code is exploitable." Still, that's a lot of code shipping behind a green checkmark.
ICSE 2026 (SEIP track), "Vibe Coding in Practice" by Fawzy, Tahir & Blincoe (a grey-literature review of 101 practitioner sources and 518 firsthand accounts) finds that QA practices are frequently overlooked, and skipping testing is the single most common behavior, often by handing verification back to the same AI tool that wrote the code (arXiv 2510.00328). That last clause is the whole problem in one sentence.
"Rethinking Verification for LLM Code Generation" (Ma et al., arXiv 2507.06920, July 2025) finds that model-built evaluation suites tend to be homogeneous — "a limited number of homogeneous test cases, resulting in subtle faults going undetected," in their words — and proposes human-LLM collaboration to widen coverage. Read into our setting, that's the same blind spot showing up on both sides of the assert: a narrow, same-shaped suite can't catch the faults its own narrowness hides.

And one I'm flagging as weak so you can discount it: CodeRabbit, an AI code-review vendor, reported ~1.7x more "issues" in AI-coauthored PRs than human ones across 470 open-source PRs (CodeRabbit). I'd take that with salt. The "issues" were graded by CodeRabbit's own product, on a small sample, and they sell AI review. Interesting direction, not a fact to lean on. I'm including it and its conflict of interest because leaving it out would be cherry-picking, and putting it in unqualified would be the same.

This is not the runtime one. It's the pre-merge one

I've written before about an agent that returns 200 and lies: a runtime check that walks an execution span-trace and refuses to accept a success the agent never achieved. People will assume this is the same thing. It isn't, and the difference matters.

That one runs after execution, on a trace of what happened: status flags, payloads, the effect on the world. This one runs before anything executes, on the source of the test files in a pull request: no tests run, no spans read, no runtime at all. mirror_audit.py is pure ast over text. Different layer (static vs runtime), different input (test source vs span-trace), different metric (mirror-ratio vs share of empty-payload successes). One asks "did this run actually do the thing?" The other asks "could this test have caught it if it didn't?" You'd want both, but they're not the same tool wearing two hats.

It sits inside the same idea as the pre-execution gate: catch the problem before you ship it, fail the build, not the incident review. It's a cousin of the deterministic pre-gate I built for an LLM judge: same shape, a 0/1/2 exit you can drop straight into CI, just a different question asked of a different artifact. And it's the inverse of the token-waste probe after a failure. There, the signal is a real failure you're burning money past; here, the danger is a false success, a green that isn't earned.

What this is NOT (so I don't oversell it)

It does not find bugs. It finds tests that couldn't find bugs. A clean 0% mirror-ratio means your tests have independent signal, not that your code is correct. You can mirror-audit your way to a great suite that still misses the one case nobody wrote.
It does not read git blame or PR metadata, so it can't prove the same author wrote both files. It has no concept of authorship at all — it flags purely on what's in the test source, and the "same author wrote both" framing is the motivation for the metric, not something the tool detects. I'm not going to invent authorship I can't see.
It's conservative without the impl file. Drop the --impl argument and the recompute and golden-literal checks go dark. On the same mirror suite it scores 37.5% instead of 50.0%, because it can no longer tell that a literal matches one in the code. Dropping context lowers the score, never raises it. (That's not a blanket "never over-reports" guarantee — see the next caveat.) The output header tells you which mode you're in.
The golden-literal check can over-flag on a collision. It fires when an equality assert pins an expected value that also appears as a literal in the impl source — but it can't tell "copied from the code" from "happened to match." An honest, hand-derived assert apply_discount(200, 50) == 100 trips it just because 100 shows up in the implementation. So a suite full of legitimate small-integer expectations (0, 1, 100) can score higher than it deserves with --impl on. It's a syntactic collision heuristic, not proof a value was copied. Read the flagged tests, don't trust the flag blindly — and if your domain is all round numbers, weight self_grading lower.
It resolves at the test-function level, not the line. A test with one genuine assert buried under three smoke calls can still pass the audit. It's a triage signal for a reviewer, not a proof of suite quality.
The four flags are heuristics on syntax, not semantics. A sufficiently clever mirror (an assert that recomputes the impl through an indirection the AST can't follow) will slip past. It catches the common shapes that show up in one-pass output, which is most of them, not all of them.

It's deterministic, though, which is the one thing it promises and keeps: same test file, same verdict, every run. I hashed the STDOUT twice on both fixtures and got identical sha256 each time (5047bf48… for the mirror suite, 84fcdb73… for the honest one). A CI gate you can't reproduce isn't a gate.

Run it on a real suite from an agent PR and tell me your mirror-ratio. I'm genuinely curious what the distribution looks like in the wild, because my 50% is a fixture I built to be obvious, and real suites will be messier. What's the most mirror-shaped test you've ever merged: a recompute, a copied golden, a smoke test with assert result? Drop it in the comments, I read every one. Follow for the next number from the next run.

Agent Loop Cost: 11x Your Per-Call Quote, in 40 Lines

Alexey Spinov — Sun, 21 Jun 2026 19:45:15 +0000

Agent loop cost is what you pay per task, not per call — and it runs multiples higher than your per-call quote because every tool-call re-bills the whole system prompt plus every tool description. A 40-line offline forecaster reads one JSONL trace and prices the full loop before you ship. On my bloated fixture it measured an 11.29x gap.

On my own bloated fixture, the forecaster measured an effective cost of $2.26 per task against a $0.20 per-invocation quote — an 11.29x gap — with a cumulative-cost curvature of k = 2.14, which is the literal meaning of "the bill grows quadratically with the number of tool-calls." That's my run on my fixture, not a claim about your agent. The whole point of the tool is that it tells you your number, and on a short healthy loop it says the gap is basically zero.

I want to be careful with one number up front. You may have seen the 30x figure going around — Muskan's June 2026 Dev.to writeup, "Why Claude Agent Loops Cost 30x a Single Inference" (dev.to/muskan_8abedcc7e12). That 30x is their aggregate over 10,000 invocations a day, not a single-task measurement, and it's not mine. My single-task gap on a deliberately bloated 14-step loop came out to 11.29x. I'm titling this post with the number I actually measured, rounded down. If your loop is worse, the tool will say 30x or 40x — but I won't put a number in a headline that my own run didn't produce.

TL;DR

A team budgets an agent at the price of one call; the task runs N calls, and each call re-bills the fixed system tax (system prompt + every tool description) plus the growing tail of prior results. The cumulative bill is the area under that curve.
loop_forecast.py is a ~40-line offline, keyless, read-only script. Feed it a JSONL trace; it prints effective $/task, the forecast_gap against your per-invocation quote, and a curvature k (cumulative bill ~ calls^k).
On my bloated fixture: $2.26/task vs $0.20 quote = 11.29x, k = 2.14, exit 1. On a clean 3-step loop: gap 0.04x, k 1.18, exit 0 — the contrarian claim falsified on its own terms.
Exit code is a pre-execution CI gate: 0 if the loop is cheap or linear, 1 if it's both super-linear and over quote, 2 on usage error.
This is a forecaster from a trace, not a runtime cap. It does not block your agent. It blocks your build.

Why your per-call quote is the wrong unit

The mistake is unit confusion, and it's an easy one to make. You open the pricing page, you see $3.00 / 1M input tokens, you estimate a call at ~8k tokens of input, and you write down $0.20 per agent action. Multiply by your daily call volume and you have a budget. It feels rigorous. It's wrong by an order of magnitude. Not because the per-call price is wrong; that part is right. The error is that the task is not one call.

Here's what actually gets billed. An agent loop re-sends its entire working context as input on every single step. The system prompt goes out again. Every tool description in the inventory goes out again. And the transcript of everything the agent has done so far — every prior tool result — goes out again, growing each turn. So step 1 bills a small payload, step 8 bills a large one, and the task cost is the sum of all of them. That sum is the area under a rising curve. The area under a rising line is quadratic.

Muskan's writeup put concrete shape on the replay: 8 tools at ~200 tokens each is 1,600 tokens of inventory replayed on every step, and by step 8 the single-step context had ballooned to roughly 83,000 input tokens. Augment Code's guide on agent loop token cost makes the same structural claim independently — that naive loops "rebill prior context on every call," so input cost rises quadratically and a "20-step loop" can run "more than 10x the naive per-step estimate". Both of those are their measurements, cited as third-party support, not my numbers. My contribution is a tool that computes your own version of it from a trace you already have.

The contrarian claim, stated so it can be wrong

Here's the position, sharp enough to argue with: you budget your agent at the price of one call and you pay the area under a curve. And that area is quadratic in the number of tool-calls, because the fixed system tax replays on every step while the result tail keeps growing — and you can compute the whole curve from a trace before you run it in production, without ever touching your provider's billing API.

That claim has a clean failure mode, and I built it into the tool. If your loop is short, your tool inventory is small, and the model exits early, the per-step curve barely rises, the cumulative cost is near-linear, and the gap to your quote is small. In that case the claim is false for you, and loop_forecast.py returns exit 0 and says so. I ran exactly that case. The clean fixture — 420-token system, three tool descriptions, three steps, small results — came back with gap 0.04x and k 1.18. No breach. If your real traces all look like that, this whole thesis doesn't apply to you, and I'd rather the tool tell you than have you take my word.

What this is NOT

Three adjacent costs already have their own tools in this series, and this one is none of them.

It is not the re-bill tax on a transcript. The context tax that re-bills your transcript every step takes a flat list of conversation turns and measures the compounding of the message history — turns 1..N re-sent as input. That tool's input is role/content turns; this tool's input is tool-call records with a manifest, and it models the replay of the loop's structure (system + tool descriptions as a fixed tax) plus the result tail, and it compares to an external quote. Different input, different metric, different question.
It is not the static MCP inventory tax. The token tax of your connected MCP server inventory measures the one-time context cost of having tools connected — the descriptions sitting in your context window before the agent does anything. This tool takes that same inventory and shows what it costs when it's replayed across every step of a multi-call loop. The inventory tax is the standing charge; this is the metered usage.
It is not a spend cap. This is a forecast from a finished trace. It blocks nothing at runtime. If you want execution to actually halt when a dollar ceiling is hit, that's a sliding-window spend guard at runtime. loop_forecast.py is the pre-execution gate for AI agents variant: it fails your CI build before the expensive loop ever ships.

The tool: loop_forecast.py

The constraints first, because they're why you can run this on a production trace without a security review: offline, keyless, read-only, zero network. No vendor SDK, no API key, nothing leaves your machine. It reads one JSONL file and prints. Tokenization is real — tiktoken with the o200k_base encoding — with an honest len/4 fallback if tiktoken isn't installed; that fallback is roughly ±15% off true BPE, and the output says which one ran.

The input is a JSONL trace. One manifest record, then one record per tool-call:

{"type":"manifest","system_tokens":1200,"tool_descriptions":[210,190,230,200,180,220,205,195,215,185],"quoted_usd_per_invocation":0.20,"input_usd_per_mtok":3.0}
{"type":"call","tool":"read_file","result_tokens":5400}
{"type":"call","tool":"run_tests","result_tokens":8100}

Every field is something you already have or can count: the system-prompt size, the size of each tool description, the per-invocation price you budgeted with, and the input price per million tokens. If a call record has no result_tokens, the tool tokenizes the result text itself.

The forecast is four deterministic rules, no model in the loop, no randomness:

fixed = sys_t + desc                       # system tax replayed every step
billed, cum, tail, run = [], [], 0, 0
for rt in tails:                           # step n bills fixed tax + accumulated prior results
    b = fixed + tail; billed.append(b); run += b; cum.append(run); tail += rt
eff_usd = run / 1e6 * price                 # area under the curve = cumulative billed input
gap = eff_usd / quote if quote else float("inf")
# curvature: slope of log(cumulative billed) vs log(step) -> total bill ~ calls^k.
def fit(ys):
    xs = [math.log(i + 1) for i in range(len(ys))]; yl = [math.log(v) for v in ys]
    mx, my = sum(xs) / len(xs), sum(yl) / len(yl)
    den = sum((x - mx) ** 2 for x in xs)
    return sum((x - mx) * (y - my) for x, y in zip(xs, yl)) / den if den else 0.0
k = fit(cum)
rc = 1 if (gap > gate_x and k >= max_k) else 0

Rule one: per-step billed input is the fixed tax plus the accumulated tail. Rule two: effective $/task is the sum of all steps — the area. (One honesty note: this counts input tokens only, so the gap is fair only when the per-invocation quote you compare against is also an input budget; output-token pricing is out of scope, see the limits section.) Rule three: the gap is that effective cost divided by the per-invocation quote you budgeted with. Rule four: fit the cumulative bill to calls^k and gate on it. One honest note on the curvature: I fit the cumulative cost, not the per-step cost. The per-step curve is roughly linear (k ≈ 1.4 on the spiral); its integral — the running total you actually pay — is the quadratic one (k = 2.14). I had this backwards in my first pass, fitting the per-step curve and reading k ≈ 1.0, which made a clearly quadratic loop look linear. Fitting the cumulative series is the fix.

The full file is 72 lines with the CLI argument handling, the tokenizer shim, and the output formatting. The forecast itself — the part above plus parsing the manifest — is about 40. I'd rather keep the readable version than golf it down to make a headline literal.

What it prints on a healthy loop (exit 0)

Run it on the clean fixture — three tool-calls, a small inventory, an early exit:

$ python3 loop_forecast.py fixtures/loop_clean.jsonl
loop_forecast.py | tokenizer: tiktoken/o200k_base
steps: 3 | system: 420t | tool_descriptions: 360t (replayed every step)
per-step billed input (tokens): [780, 960, 1110]
effective $/task: $0.0086  (area under replay curve @ $3.00/Mtok)
naive per-invocation quote: $0.2000
forecast_gap: 0.04x  (effective / quote)
curvature k: 1.177  (cumulative bill ~ calls^k; 1.0=linear, 2.0=quadratic)
gate: gap>8x AND k>=1.3 -> PASS
exit: 0

The effective cost is below the quote here — a three-step loop with tiny results is cheaper than the single-call budget assumed, because the budget over-provisioned. Gap 0.04x, k 1.18, gate PASS. This is the falsification working. A cheap loop reads as cheap.

What it prints on a spiral (exit 1)

Now the bloated fixture — a 1,200-token system prompt, ten tool descriptions (2,030 tokens of inventory replayed every step), and fourteen steps whose tool results grow into the thousands as the agent reads files, runs tests, and re-parses output:

$ python3 loop_forecast.py fixtures/loop_spiral.jsonl
loop_forecast.py | tokenizer: tiktoken/o200k_base
steps: 14 | system: 1200t | tool_descriptions: 2030t (replayed every step)
per-step billed input (tokens): [3230, 6430, 11830, 18630, 26730, 36230, 43430, 54430, 64230, 76730, 86930, 95530, 108530, 120030]
effective $/task: $2.2588  (area under replay curve @ $3.00/Mtok)
naive per-invocation quote: $0.2000
forecast_gap: 11.29x  (effective / quote)
curvature k: 2.139  (cumulative bill ~ calls^k; 1.0=linear, 2.0=quadratic)
gate: gap>8x AND k>=1.3 -> BREACH
exit: 1

Read the per-step list. Step 1 bills 3,230 tokens. Step 8 bills 54,430 — in the same order of magnitude as Muskan's ~83k step-8 figure, and I kept mine deliberately under theirs so I'm not borrowing their drama. By step 14 a single step bills 120,030 input tokens, almost all of it re-sent context. The task costs $2.26. You quoted $0.20. That's the 11.29x in the title. It's the number the tool actually produced, not a target I reverse-engineered.

The gate trips because both conditions hold: gap over 8x and cumulative k at or above 1.3. A loop that's expensive but linear (a long, simple, single-tool job) won't trip it; neither will a curvy but cheap one. You want both signals before you fail someone's build.

Determinism, because a flaky gate is worse than no gate

A CI gate that returns different numbers on the same input is useless — you'll mute it the first week. So the arithmetic is integer token counts with no randomness and no network. I hashed the stdout twice on each fixture:

clean  run1: 455a86ce7e1df9cdca74f072c5d5e2919dac8f91889d950769673e7998bd506d
clean  run2: 455a86ce7e1df9cdca74f072c5d5e2919dac8f91889d950769673e7998bd506d
spiral run1: 450d51f471b747c224c3782c6d8b4af8acddc1db677b073389e5de0a09ff74f3
spiral run2: 450d51f471b747c224c3782c6d8b4af8acddc1db677b073389e5de0a09ff74f3

Byte-identical. Same trace in, same gate out. Bad JSON returns exit 2 with the parser's error, and no arguments prints usage and exits 2 — so the gate fails loud on a broken trace instead of silently passing.

Where this is wrong, and where I'm guessing

The model assumes input-token replay is the dominant cost, and on a tool-heavy agent loop it usually is — but if your steps generate large outputs (long generations, not long contexts), output pricing matters and this tool ignores it. It also assumes you can name your per-invocation quote honestly; garbage quote in, garbage gap out. And the curvature fit needs enough steps to mean anything — on a 3-step loop the k value is noisy (which is why the gate also requires the gap condition). I'd trust the gap number on any trace and treat k as a shape hint, not a precise exponent.

The fixtures here are constructed to be realistic, not harvested from a specific production run — the tool sizes and step counts come from Muskan's and Augment's published figures, the math is mine. Run it on your own exported traces and the numbers stop being illustrative.

Run it on your loop

Save the script, write one manifest line and one line per tool-call from a trace you already have, and run it. If your gap comes back under 2x, your per-call budgeting was fine and you can ignore all of this. If it comes back at 11x like mine — or worse — you now have a number to put in front of whoever signs the cloud bill, computed before the loop ever shipped.

Here's the open question I don't have a clean answer to: prompt caching changes this math, because a cached system-prefix isn't re-billed at full input price. My forecaster assumes no cache (worst case). What's the right way to fold a partial cache hit-rate into the per-step replay cost without making the tool lie in either direction? If you've modeled that, I want to see how.

Written by Alexey Spinov. AI-assisted, human-verified: the tool, both fixtures, and every number above come from a real local run on 2026-06-21 (Python 3.13.5, tiktoken 0.13.0, o200k_base). I ran it, checked the exit codes (0 / 1 / 2), hashed the output twice to confirm determinism, separated my numbers from Muskan's and Augment's cited figures, and edited every line. Offline, keyless, read-only, zero network.

Follow for the next numbers from production agent traces. What's the worst per-task-vs-quote gap you've found on a real loop — and did anything in CI catch it before the invoice did?

Prompt Cache Break: Hit-Rate Fell 100% to 40% in 40 Lines

Alexey Spinov — Sat, 20 Jun 2026 19:38:43 +0000

In short: a prompt cache-break is when one change atop your prompt prefix — a fresh timestamp, a reordered tool block — makes the cache miss from there down, so cached reads silently re-bill as fresh input. cache_break.py hashes each prefix segment, localizes the break, and fails CI. On my fixture, one timestamp dropped the estimated cache-hit-rate 100% to 40%.

AI disclosure: I drafted this with an AI writing assistant. The tool, the two fixtures, and every number below come from a real local run on tiktoken o200k_base — I ran it, checked the exit codes, hashed the output twice to confirm it's deterministic, and edited every line before publishing.

Turning on prompt caching feels like a free win. It usually isn't the win you think.

Here's the trap. Anthropic and OpenAI both price a cache read at a fraction of a fresh input token — Anthropic quotes cached reads at roughly $0.30/M vs $3.00/M for fresh, about a 10x gap on the cached portion (Anthropic prompt caching docs). Flip the feature on, watch the dashboard, move on. But that discount is only paid for a prefix that is byte-for-byte identical to what's already cached. Change one character near the top and the cache misses from there down. You still have caching on. You're just not hitting it. And nothing in your config screams about it.

So the number that actually matters isn't "is caching enabled." It's your real cache-hit-rate — and almost nobody meters it.

TL;DR. Prompt caching only pays out on an exact-prefix match. Agents quietly break that match: a dynamic now=... in the system block, two tool definitions that got reordered, a memory snippet inserted at the top. When the prefix breaks, the cache misses below the break and re-bills it as fresh input. cache_break.py (below, keyless, offline) hashes each prefix segment per step, localizes the first one that diverges from the baseline, computes the hit-rate, and gates it. On a clean trace it estimated 100% hit-rate, exit 0. On the same content with one injected timestamp, 40%, cache-break flagged at segment 0 step 3, exit 1.

The contrarian bit: "caching is on" is not "caching is working"

Most caching write-ups stop at how to turn it on. Mark the prefix, set cache_control, done. That's the easy 80%. The expensive 20% is everything that silently invalidates the match afterward, on a running agent, where you'll never notice from the totals.

Three ways an agent breaks its own cache without anyone touching the config:

A dynamic value in the system prompt. The classic is a current timestamp — now=2026-06-21T08:14:03Z — stamped into the system block so the model "knows the time." It changes every call. It sits at the very top. So the cache misses on everything, every step, forever.
Reordered tool definitions. Your framework serializes tool schemas from a dict or a set. Run two, the order flips, the bytes differ, the prefix no longer matches. Same tools. Broken cache.
A memory snippet inserted at the front. Retrieval-augmented memory that prepends "what we learned last session" pushes a variable block above the stable one. Everything below it is now fresh.

The falsifiable claim: if you take a clean trace and a byte-identical-content trace where only the prefix ordering breaks, a real measurement should show the hit-rate collapse on the broken one and stay high on the clean one. If it doesn't collapse, I'm wrong and this tool is useless. It collapsed — 100% to 40% — and the detector named the exact segment. Run is below.

There's even a name for the failure mode in the research now. The arXiv note Don't Break the Cache (2601.06007, Lumer et al., Jan 2026) is entirely about prefix-stability discipline for cached agentic inference — worth a read if you want the formal treatment. The 70% default gate here is my own pick, not theirs: log your cached-token count, compute hits / (hits + full), and alert when it drops under ~70%. That number is a starting line, not a law — tune it to your own traces.

The tool: 40 lines, no API key, read-only

cache_break.py reads one JSONL trace. Each line is one agent step with a prefix: an ordered list of named segments (system, tool_defs, memory) — the part you expect to be cached. It does four deterministic things.

Prefix-stability hash. Canonicalize each segment (json.dumps with sorted keys) and sha256 it. Lock the step-1 prefix as the baseline.
Break-point localization. For every step, compare segment hashes to the baseline. The first segment that diverges is the break point — the cache misses from there down, because everything after a changed byte re-bills as fresh.
Hit-rate compute. Tokens above the break = cached; the break and everything below = fresh. hit-rate = cached / (cached + fresh), summed across steps. tiktoken o200k_base, len/4 fallback if tiktoken's missing.
Gate. Compare overall hit-rate to --min-hit-rate (default 0.70). Below it or any cache-break detected → exit 1. This is a pre-execution gate: don't ship a prompt ordering that punches through your own cache.

One honesty rule baked in: the output says source: estimated from prefix stability. If your trace carried real provider usage fields (Anthropic/OpenAI return cached-token counts), you'd use those instead and it'd say measured. I'm not dressing an estimate up as a meter reading.

#!/usr/bin/env python3
"""cache_break.py - measure prompt cache-hit-rate and localize cache-break in a JSONL trace."""
import json, hashlib, sys

MIN_HIT_RATE = 0.70       # my default gate: alert when cache-hit-rate drops below 70% (tune to your traces)
CACHE_READ = 0.30         # $/1M cached-read tokens  (public price flag, NOT a measurement)
FRESH = 3.00              # $/1M fresh-input tokens  (public price flag, NOT a measurement)

try:
    import tiktoken
    _enc = tiktoken.get_encoding("o200k_base")
    def count(t): return len(_enc.encode(t))
    TOKENIZER = "tiktoken o200k_base (exact)"
except Exception:                                  # honest fallback, ~+-15% vs real BPE
    def count(t): return max(1, round(len(t) / 4))
    TOKENIZER = "len/4 heuristic (tiktoken not installed; ~+-15%)"

def canon(seg): return hashlib.sha256(json.dumps(seg, sort_keys=True, ensure_ascii=False).encode()).hexdigest()[:12]

def main(argv):
    if len(argv) < 2:
        print("usage: cache_break.py <trace.jsonl> [--min-hit-rate 0.70]"); return 2
    min_hit = float(argv[argv.index("--min-hit-rate") + 1]) if "--min-hit-rate" in argv else MIN_HIT_RATE
    try:
        steps = [json.loads(ln) for ln in open(argv[1], encoding="utf-8") if ln.strip()]
        if not steps: raise ValueError("empty trace")
        base = [canon(s) for s in steps[0]["prefix"]]          # baseline prefix from step 1
        names = [s.get("name", f"seg{i}") for i, s in enumerate(steps[0]["prefix"])]
        toks = [count(json.dumps(s.get("text", s), ensure_ascii=False)) for s in steps[0]["prefix"]]
    except (KeyError, ValueError, json.JSONDecodeError, IndexError) as e:
        print(f"cache_break | BAD INPUT: {e}"); return 2

    cached, fresh, break_at, break_step = 0, 0, None, None
    for i, s in enumerate(steps):
        cur = [canon(seg) for seg in s["prefix"]]
        diff = next((j for j in range(min(len(base), len(cur))) if cur[j] != base[j]), None)
        if diff is None and len(cur) != len(base): diff = min(len(base), len(cur))
        if diff is None:                                       # whole prefix byte-identical -> all cached
            cached += sum(toks)
        else:                                                  # cache breaks at first divergence, rest re-bills fresh
            cached += sum(toks[:diff]); fresh += sum(toks[diff:])
            if break_at is None or diff < break_at:
                break_at, break_step = diff, i + 1

    hit = cached / (cached + fresh) if (cached + fresh) else 1.0
    broke = break_at is not None
    eff = CACHE_READ * hit + FRESH * (1 - hit)                 # blended $/1M on the prefix at this hit-rate

    print(f"cache_break | {argv[1]} | tokenizer: {TOKENIZER} | source: estimated from prefix stability")
    print("-" * 78)
    print(f"  steps={len(steps)}  prefix_segments={len(base)}  ({', '.join(names)})")
    print(f"  cached_tokens={cached}  fresh_tokens={fresh}")
    print(f"  cache_hit_rate     : {hit:.2%}   (threshold {min_hit:.0%})")
    if broke:
        print(f"  cache_break        : TRUE at segment {break_at} '{names[break_at] if break_at < len(names) else '?'}', first seen step {break_step}")
    else:
        print(f"  cache_break        : FALSE (prefix byte-identical across all steps)")
    print(f"  effective $/1M prefix : ${eff:.2f}  (vs ${CACHE_READ:.2f} all-cached / ${FRESH:.2f} all-fresh; public-price illustration)")
    rc = 0 if (hit >= min_hit and not broke) else 1
    print(f"  exit               : {rc}  ({'PASS' if rc == 0 else 'FAIL: hit-rate below threshold or cache_break'})")
    return rc

if __name__ == "__main__":
    sys.exit(main(sys.argv))

No key, no network, nothing written to disk. pip install tiktoken, point it at a trace, read the exit code. If tiktoken isn't installed it falls back to len/4 and says so in the header (~±15% off real BPE). I'd rather print the caveat than fake the precision.

The real run

Two fixtures ship with it. Both are synthetic five-step coding sessions (no private data), same payments-svc task, same three-segment prefix: system, tool_defs, memory.

trace_clean.jsonl keeps that prefix byte-identical on every step — only the user tail changes, and the user tail isn't part of the cached prefix. Actual output:

cache_break | fixtures/trace_clean.jsonl | tokenizer: tiktoken o200k_base (exact) | source: estimated from prefix stability
------------------------------------------------------------------------------
  steps=5  prefix_segments=3  (system, tool_defs, memory)
  cached_tokens=335  fresh_tokens=0
  cache_hit_rate     : 100.00%   (threshold 70%)
  cache_break        : FALSE (prefix byte-identical across all steps)
  effective $/1M prefix : $0.30  (vs $0.30 all-cached / $3.00 all-fresh; public-price illustration)
  exit               : 0  (PASS)

100% hit-rate, no break, exit 0. Green. The whole prefix rides the cache every step, effective price sits at the floor — $0.30/M.

trace_broken.jsonl is the exact same content, with one change: starting at step 3, the system segment carries a live clock — now=2026-06-21T08:14:03Z, a different value each step. That's the only edit. Real output:

cache_break | fixtures/trace_broken.jsonl | tokenizer: tiktoken o200k_base (exact) | source: estimated from prefix stability
------------------------------------------------------------------------------
  steps=5  prefix_segments=3  (system, tool_defs, memory)
  cached_tokens=134  fresh_tokens=201
  cache_hit_rate     : 40.00%   (threshold 70%)
  cache_break        : TRUE at segment 0 'system', first seen step 3
  effective $/1M prefix : $1.92  (vs $0.30 all-cached / $3.00 all-fresh; public-price illustration)
  exit               : 1  (FAIL: hit-rate below threshold or cache_break)

40% hit-rate, exit 1. And it points at the culprit precisely: segment 0, system, first seen at step 3. Steps 1–2 cached fine; from step 3 on the timestamp lives at the very top of the prefix, so the cache misses on the system block and on the tool_defs and memory below it — even though those two never changed. That's the cruel part of cache-break: damage at the top voids everything underneath. The blended price on this prefix went from $0.30 to $1.92/M, a 6.4x jump on this fixture, driven by one field a developer added to be helpful.

Watch the two numbers that prove it's the timestamp and nothing else. Same payments-svc content. Same tool list. Same memory. Cached tokens fell 335 → 134; fresh went 0 → 201. The only delta in the input was a clock.

One more run worth showing, because it's the design decision people argue with. A break is a gate condition on its own — not just a low rate. So even if you slacken the threshold all the way to 0.30, the broken trace still fails:

$ python3 cache_break.py fixtures/trace_broken.jsonl --min-hit-rate 0.30
  cache_hit_rate     : 40.00%   (threshold 30%)
  cache_break        : TRUE at segment 0 'system', first seen step 3
  exit               : 1  (FAIL: hit-rate below threshold or cache_break)

40% clears a 30% bar, but the break still trips exit 1. I made that call on purpose: a detected prefix-break means money is leaking somewhere measurable, and "the average is still okay" is exactly the reasoning that lets it leak for a month. Disagree with me on that — it's a real design tradeoff, not a law.

Bad input is the third exit code. A malformed JSONL line returns exit 2, not a crash and not a false pass:

$ python3 cache_break.py fixtures/trace_bad.jsonl
cache_break | BAD INPUT: Expecting ',' delimiter: line 2 column 1 (char 82)
[exit 2]

Both real runs are reproducible. I hashed two consecutive clean runs and two consecutive broken runs with shasum -a 256; each pair was byte-identical (3608e4d5… for clean, fb72c110… for broken). Deterministic, not a one-shot fluke.

6.4x on my run, 10x at the ceiling — the honest version

You'll see "10x" thrown around for cache breaks, so here's where it comes from and what I actually got. The 10x is the unit gap: Anthropic's published cached-read vs fresh-input ratio is roughly $0.30 to $3.00, a 10x difference on tokens that go from cached to fresh (Anthropic docs). That's their public number, not mine.

What the tool computed on this fixture's prefix was 6.4x — $0.30/M up to $1.92/M — because only 60% of the prefix tokens flipped to fresh, not all of them. If a break lands at segment 0 on step 1 of a long-running agent (the dynamic-timestamp case, which is the common one), every prefix token re-bills fresh and you approach the full 10x. So 10x is the ceiling from public prices; 6.4x is the real, smaller, honestly-labeled number from my run. I'd rather you trust the 6.4x I can show you than the 10x I can't.

This is the same shape as the context tax, but a different leak. There, you re-bill the whole transcript every step because it grows — a meter, not a guard. Here the prefix is supposed to be free-ish via cache, and a one-byte change quietly un-frees it. Different axis, different number, same lesson: measure the thing, don't assume the feature did its job.

Where this fits

The prefix this tool watches — the long, stable system + tool definitions + memory block — is the same one I metered for raw cost in your MCP server's token tax. Token-tax asks "how big is this prefix"; cache-break asks "are you actually paying the cached price for it, or silently the fresh one." Run both on the same trace and you've covered both halves: the size of the prefix and whether it's hitting cache.

And it's the same philosophy as the pre-execution gate: catch the broken prompt ordering before you ship it, fail the build, not after the invoice. Cache-break is a perfect CI gate because it's deterministic — same trace, same verdict, every time. If you want the runtime spend brake instead of a structural check, that's the sliding-window spend guard, which caps cumulative cost over a window rather than auditing prefix stability.

What this is NOT (so I don't oversell it)

It does not read your provider's real cached-token counts unless you put them in the trace. The hit-rate here is estimated from prefix stability — it assumes the cache misses from the first changed byte down, which is how prefix caching works, but it's a structural model, not a meter reading off your bill. If your trace carries real usage fields, wire those in and trust those instead. The header tells you which mode you're in.
It does not compute your invoice. The $/M figures use Anthropic's public $0.30/$3.00 prices as a flag, to illustrate the cached-vs-fresh gap. Your real bill depends on cache TTL, minimum cacheable length, output tokens, and your vendor — none of which this models.
It does not account for cache expiry. A prefix can be byte-identical and still miss because the cache entry aged out (Anthropic's default TTL is short). This tool catches the structural break — a changed prefix — not a timed-out one. Those need provider usage data to see.
It assumes you've segmented your prefix in the trace (system / tool_defs / memory). Garbage segmentation in, vague break-point out. The localization is only as precise as your segments — a one-character edit inside the system block re-bills the whole system segment as fresh here, even though a real tokenizer would only lose the tokens from that character down. So I say "from the first changed byte down" as the model, but the tool resolves it at the segment boundary, not the byte. That makes it slightly pessimistic on mid-segment edits, never optimistic.
It does not charge the cache-write premium. Anthropic bills a cache write at ~1.25x base input, so the very first step (and every step that re-breaks the cache) pays more than the fresh-input number, not less. The "$0.30 all-cached floor" is the steady-state read price, not an achievable per-run average — which means a repeatedly-broken cache costs a bit more than the $1.92 here suggests, not less. I left write-cost out to keep the model simple; it only ever understates the leak.

What's the dumbest thing that's broken your prompt cache — a timestamp, a reordered tool list, a uuid someone logged into the system prompt? Run the detector on a real trace and tell me where it pointed. I'm collecting break points, and I read every reply. Follow for the next number from the next run.

Your LLM Judge Costs More Than the Agent. Gate It in 40 Lines.

Alexey Spinov — Fri, 19 Jun 2026 19:30:16 +0000

LLM judge cost is the share of your eval bill spent grading agent output instead of producing it. To control it, run a 40-line offline pre-gate that triages every span with four deterministic rules and escalates only the uncertain tail to the expensive judge. On one trace this cut judge cost share from 50% to 16%.

LLM judge cost is the line item nobody puts on the FinOps dashboard. You add an LLM-as-judge to grade every agent span, you sleep better, and three weeks later the eval layer is quietly billing a third of what the agent itself costs. This post measures that share of your bill spent judging instead of doing, with a 40-line offline meter, and shows the one move that drops it from 50% to 16% on the same trace.

AI disclosure: I drafted this with an AI writing assistant. The tool, both fixtures, and every number below come from a real local run of judge_gate.py on Python 3.13.5, no network, no API key. I ran it, checked the exit codes, hashed the output twice to confirm it's deterministic, and edited every line myself before publishing.

Here's the sentence that set me off. Sattyam Jain wrote it on Dev.to on June 12, in a post arguing you should stop running an LLM judge on every agent call: "if your monitor exceeds ~20–25% of production cost, you built the wrong monitor." (Dev.to) That's a great rule of thumb. It's also unfalsifiable until you can put a number on your monitor. His post sketches the tiered architecture (cheap deterministic heuristics first, expensive judge last) but ships no code you can run against your own trace. So I wrote the missing 40 lines.

The timing isn't an accident. The token bill is coming due across the whole industry right now. TechCrunch reported on June 5 that "Uber blew through its entire 2026 AI coding budget by April," and that a Priceline employee saw "a routine Cursor contract renewal came back 4–5x more expensive." (TechCrunch) Two days earlier the Linux Foundation announced its intent to launch the Tokenomics Foundation — open standards for AI cost management, because, in Jim Zemlin's words, "tokens have become the new unit of technology spend." (Linux Foundation) Everyone's auditing what the agent spends. Almost nobody's auditing what the watchdog spends.

And the watchdog is an LLM call too. You priced the agent. Did you price the thing watching the agent?

TL;DR

An LLM judge on every span isn't rigor — it's a second agent you forgot to budget. Price it before it surprises you.
judge_gate.py is a 40-line, offline, keyless, zero-network script. Feed it a JSONL trace; four deterministic rules triage each span as OK / BAD / UNCERTAIN, and only UNCERTAIN ones would reach the expensive judge.
On a well-instrumented 50-span trace it resolved 68% cheaply and sent only 32% to the judge → 16% judge cost share (exit 0, PASS). On the same agent logged as free text, 100% escalated → 50% cost share (exit 1, FAIL).
The judge is never actually called. It's priced via configurable --judge-price and --prod-cost flags. Substitute your own rates; I ship neutral placeholder units.
Exit code is a CI gate: 0 if judge cost share ≤ budget (default 0.25), 1 if over, 2 on bad input. Deterministic — byte-identical across runs.

This is the next piece in a series on controlling agents before they execute, not after. The pre-execution gate gates the agent's action. The success gate decides what to verify in a result. This one is a level up the stack: it doesn't gate the agent at all. It gates the judge — and asks how much that judge is allowed to cost.

What "judge cost share" actually means

Here's the failure mode I keep seeing. Someone reads that agents silently fail (true) and bolts on an LLM-as-judge to grade every step. Every span: a second model call, often a frontier model, sometimes with a chunky rubric prompt. It works. It catches things. Then the finance person asks why the eval bill is the same order of magnitude as the agent bill, and the honest answer is "because we run a full second model over every single thing the first one does."

The number that matters is a ratio. Call it judge cost share: the cost of the judging layer divided by the cost of the production run it's judging.

judge_cost_share = (judge_calls × judge_price) / prod_cost

If that's 8%, fine — cheap insurance. If it's 50%, you didn't add a monitor, you added a co-pilot you're paying full freight for and calling overhead. The whole game is shrinking judge_calls: the number of spans that actually need a human-grade judgment, versus the spans a dumb deterministic rule can settle for free.

Most spans don't need a judge. A tool either got called or it didn't. A JSON output either parses or it doesn't. A 200 with an empty body is wrong no matter how confident the prose around it sounds. You don't need a frontier model to know [] is not a successful invoice send. You need an if statement.

The fix: triage every span, escalate only the uncertain tail

The pre-gate is a function. It looks at one span and returns one of three verdicts:

OK — cheaply, provably fine. Don't pay to judge it.
BAD — cheaply, provably broken. Don't pay to judge it either; you already know.
UNCERTAIN — the cheap rules abstain. This is the only span the expensive judge should ever see.

Four rules carry almost all the weight. They're the deterministic heuristics Sattyam Jain pointed at ("did the claimed gate execute?") turned into code:

Claim-vs-evidence. The span says it called send_email, but tools_called doesn't contain send_email. Claim without evidence → BAD. (This is the same idea as the success gate's middle check, reused here as a free triage rule.)
Output schema. The output isn't even a JSON object — it's a raw string, or it's missing. → BAD.
200-with-empty-payload. Status says success, body is empty. The classic silent lie. → BAD.
Duplicate retry. This span's argument hash equals the previous span's. A byte-identical retry — the waste-after-failure loop signature. → BAD.

If none of those fire and the span has a clean ok: true + 200, it's OK. Otherwise the rules abstain and it's UNCERTAIN — escalate. Here's the whole triage:

def triage(span):
    """Return (verdict, rule). UNCERTAIN means 'a human-grade LLM judge is needed'."""
    out = span.get("output")
    if not isinstance(out, dict):                      # output not valid JSON object
        return "BAD", "schema:not-an-object"
    if span.get("claimed_tool") and span["claimed_tool"] not in span.get("tools_called", []):
        return "BAD", "claim-without-evidence"         # said it called X, trace has no X
    if span.get("status") == 200 and not out:          # 200 OK with empty payload
        return "BAD", "200-empty-payload"
    if span.get("arg_hash") and span["arg_hash"] == span.get("prev_arg_hash"):
        return "BAD", "duplicate-span"                 # byte-identical retry of prior call
    if out.get("ok") is True and span.get("status") == 200:
        return "OK", "clean-success"                   # explicit ok + 200, no contradiction
    return "UNCERTAIN", "needs-judge"                  # cheap rules abstain -> escalate

That's it. No network, no key, no model. The judge layer is priced, not called: I count the UNCERTAIN spans and multiply by a price you supply on the command line. I refuse to hardcode a vendor rate — those go stale in a month and I'd rather be honestly empty than confidently wrong about someone's bill.

The run: 32% to the judge, not 100%

I built two traces of the same 50-span agent — a support-desk bot doing searches, record updates, email sends, classifications, and reply drafts.

The first, trace_gated.jsonl, is well-instrumented: each span logs the tool it claimed, the tools actually called, a structured output (an ok flag where the verdict is clear-cut, a confidence value or label where it isn't), and an argument hash. The second, trace_naive.jsonl, is the same agent logging only free-text outputs like {"text": "email sent"}, the way a lot of agents actually log in the wild. Same work. Different telemetry.

Here's the verbatim output. I didn't touch it:

$ python3 judge_gate.py fixtures/trace_gated.jsonl --judge-price 1 --prod-cost 100
spans total:        50
resolved by gate:   34 (68.0%)  [OK=29 BAD=5]
sent to LLM judge:  16 (32.0%)
judge cost share:   16.0% of prod cost (judge_price=1.0, prod_cost=100.0, budget=25%)
verdict: PASS - judge layer within budget
$ echo $?
0

$ python3 judge_gate.py fixtures/trace_naive.jsonl --judge-price 1 --prod-cost 100
spans total:        50
resolved by gate:   0 (0.0%)  [OK=0 BAD=0]
sent to LLM judge:  50 (100.0%)
judge cost share:   50.0% of prod cost (judge_price=1.0, prod_cost=100.0, budget=25%)
verdict: FAIL - judge layer over budget
$ echo $?
1

Read the two side by side. Same agent, same fifty spans, same --judge-price 1 --prod-cost 100. The well-instrumented trace sends 16 spans to the judge and lands at 16% cost share: a PASS, exit 0. The free-text trace can't resolve a single span cheaply, sends all 50, and lands at 50%: a FAIL, exit 1, tripping Sattyam Jain's "wrong monitor" line by a mile.

The lever isn't a fancier judge. It's whether your trace carries the four cheap facts a rule can read. Of the 16 spans that did escalate in the gated run, most are genuinely subjective: ambiguous contract summaries (confidence: 0.45), hedged reply drafts ("I cannot find the order, but it is probably fine."), borderline intent labels. A handful escalate for a humbler reason — they carry no ok flag for a cheap rule to confirm, so the gate abstains instead of guessing. Either way, that's the tail you want a human-grade judge on. The other 34? Five were provably broken (one duplicate retry, two claims with no matching tool call, one 200 with an empty body, one non-object output) and the rest were clean successes. None of those needed a model to adjudicate.

I want to be precise about a number I almost fudged. The cost figures are placeholder units (judge_price=1, prod_cost=100). I am not telling you a judge call costs a dollar or that your run costs a hundred of anything. Plug in your real per-call judge price and your real run cost. The rate, 32% vs 100% of spans escalating, is the part that's mine: measured, reproducible. The dollars are yours.

Am I just moving the bug into the gate?

Fair objection, and it's the one I'd raise. If the cheap rules are wrong, you've replaced a $50 judge bill with a 16% bill and a stack of bad verdicts. So: how good can a cheap layer actually be?

Two recent papers say: surprisingly good, on the parts that matter. In Cheap Reward Hacking Detection (arXiv:2606.08893, June 8), Belenky, Itria and Johns put a linear probe on a small transformer encoder and detected reward hacking at AUC 0.9467, TPR 0.8296 at 5% FPR, at "roughly four orders of magnitude lower per-trajectory cost" than an LLM-as-judge baseline. And Goal-Autopilot (arXiv:2606.11688) reports a gated finite-state machine that "forbids any terminal 'done' claim whose falsifiable gate did not actually execute and pass," cutting fabrication on SWE-bench Lite from 33.7% to 0.67%. Those are their numbers on their setups, not mine. I'm citing them as evidence that a cheap deterministic layer catches most of what a dear one catches, not as my own result.

My four if statements are cruder than a trained probe. They don't need to be clever. They need to be right when they're confident and silent when they're not — which is the whole point of the UNCERTAIN bucket. A rule that isn't sure doesn't guess. It escalates. The judge still grades the hard 32%. You just stopped paying it to rubber-stamp the easy 68%.

What this is not

Not an eval suite. It doesn't score answer quality. It decides which spans deserve a judge, then prices that layer. Correctness of the hard tail is still the judge's job.
Not a runtime cap. It reads a finished trace and fails CI. If you need to block a runaway loop mid-flight, that's a sliding-window spend guard, a different tool.
Not a verdict on confidence fields. Honest limitation: my gate ignores a span's self-reported confidence. One span in the fixture says confidence: 0.95, "no ambiguity" and still got escalated, because I refuse to trust a model's own confidence as a cheap signal — that's the kind of self-assessment that lies. If you trust yours, add a fifth rule. I didn't.
Not a license to skip the judge. The judge gets the genuinely uncertain spans. The argument is against running it on the obvious ones, not against running it at all.

Run it on your own trace

Export 40–60 spans of a real agent run to JSONL with six fields per span (status, claimed_tool, tools_called, output, arg_hash, and prev_arg_hash carrying the previous span's hash so the duplicate-retry rule can fire), point judge_gate.py at it, and pass your real --judge-price and --prod-cost. If your judge cost share comes back under 10%, ignore me; your monitor's fine. If it comes back at 40%, you've found a line item.

One thing I genuinely don't know yet and would put real money on being argued in the comments: where the honest threshold is. Sattyam Jain says 20–25%. I shipped a default of 25%. But for a low-stakes summarizer, even 10% might be waste, and for an agent that moves money, maybe 40% is cheap. The budget is a --flag precisely because I don't think there's one right answer.

So I'll ask you: what's the judge cost share on a real eval pipeline you've shipped — and where would you set the budget before it counts as the wrong monitor?

I publish one runnable FinOps tool for AI agents at a time, with the real run log attached. Follow for the next number from the next trace — and drop your judge cost share in the comments, I read every one.

Your Failed Agent Run Burns Most of Its Tokens AFTER It Fails — Measure It in 40 Lines

Alexey Spinov — Thu, 18 Jun 2026 19:26:36 +0000

Wasted tokens after agent failure are the part nobody meters. A clean agent run and a failed one cost about the same to start; the bill diverges after the run is already lost. This post measures that tail — the token fraction your run keeps burning past its first failure signal — with a 40-line offline meter.

Here's the number that made me write this. In a 2026 paper on multi-agent observability, researchers measured 165 GAIA traces and found that among warned failed runs, 58.1% of tokens are spent after the first warning signal, on average. First the warning fires (a tool error, a loop, a budget-pressure flag), and then the agent keeps going for more than half the run's tokens before it stops. Read the citation carefully: that 58.1% is their number, on warned failed runs specifically, not all runs and not my measurement. I'll keep those separated all the way down.

The point I want to land: waste is not failure. The failure is the cheap part. What's expensive is the distance between "this run is clearly off the rails" and "the agent actually stopped." That gap is denominated in tokens, and you can measure it on your own logs in about a minute.

TL;DR

A failed agent run spends most of its tokens after the first detectable failure signal — the published figure is 58.1% on warned failed runs (arXiv 2606.01365, 165 GAIA traces).
waste_probe.py is a 40-line, offline, keyless, read-only script. Feed it a JSON trace; it finds the first signal and prints the token share burned at and after it.
On my own loopy fixture it measured 82.9% waste (707/853 tokens) — that's my run on my fixture, a separate number from the paper's.
Exit code is a CI gate: 0 if waste ≤ threshold, 1 if over, 2 on usage error. Wire it onto collected traces.
This is a post-mortem meter, not a runtime cap. It tells you where it burned; it does not block anything.

Where the wasted tokens after agent failure actually come from: first signal is not first stop

Think about what a failed run actually looks like in the log. It's rarely one clean explosion. It's a tool that returns a 200 OK with a slightly different JSON schema, a parser that throws KeyError: 'close', and an agent whose retry branch assumes flakiness — so it fires the exact same request again. And again. The payload never changes. The agent never reparses. It just loops, paying full freight on every turn, narrating its own confusion in increasingly confident prose.

By the time anything stops that run, the diagnostic information was available at the first error. Everything after it is re-derivation of a conclusion the trace already contained. That's the waste. Not the failure itself, but the persistence past the failure.

I want to be precise about what this is and isn't, because two adjacent ideas already have tools:

It is not the re-bill tax. The context tax that re-bills your transcript every step is about the growing history being re-sent as input on every step — compounding, n(n+1)/2. That's a cost you pay on a healthy run too. Waste-after-signal is different: it's the excess work specific to the failure tail.
It is not stale memory either. Dead-weight from a memory store — the kind you find when you audit your agent's memory tax and backdoor — is retained-but-irrelevant context across runs. Waste-after-signal is within a single failed run, after a specific trigger.
It is not a spend cap. A sliding-window spend guard that blocks the runaway loop at runtime stops execution when a dollar ceiling is hit. waste_probe.py blocks nothing. It reads a finished trace and reports a ratio. The gate it controls is your CI pipeline's exit code, not the agent's next call.

So the question it answers is narrow and falsifiable: in a given trace, what fraction of tokens landed at or after the first failure signal? If that fraction is reliably small on your real logs, this whole thesis is wrong for you, and the tool will say so with exit 0. I'd rather you find that out than take my word.

One more thing worth flagging, because it makes the tail bigger, not smaller: fan-out. Anthropic's Dynamic Workflows (a research preview shipped late May 2026) let a run spawn tens to hundreds of parallel subagents, capped at 16 concurrent and 1,000 total per run. InfoQ's writeup notes the obvious — these "can consume substantially more tokens than a typical session." Now imagine the loop above, but forked across a dozen subagents that each missed the same schema change. The failure tail doesn't add up. It multiplies.

The tool: waste_probe.py

The design constraints first, because they're the reason you can actually run this on a production trace without a security review: offline, keyless, read-only, zero network. It reads one JSON file and prints. No vendor SDK, no API key, no telemetry leaving your machine. Tokenization is real (tiktoken with the o200k_base encoding), with an honest len/4 fallback if tiktoken isn't installed; that fallback is roughly ±15% off true BPE, and it says so in the output.

The input is a trace: an ordered list of steps, each a small object.

[
  {"role": "assistant", "content": "I'll fetch the Q2 close...",
   "tool": "get_quote", "tool_args": {"symbol": "EXMPL", "period": "Q2"}, "status": "ok"},
  {"role": "tool", "content": "{...}", "tool": "get_quote", "status": "ok"}
]

The whole thing is 40 lines. Here's the core — tokenization, signal detection, and the gate:

def step_text(s):
    return " ".join(str(s.get(k, "")) for k in ("role", "content", "tool", "tool_args", "status"))

def first_signal(steps):
    seen = set()
    for i, s in enumerate(steps):
        if str(s.get("status", "ok")).lower() == "error":
            return i, "tool-error"
        if "tool_args" in s:                         # only a CALL counts as a loop, not a tool result
            key = (str(s.get("tool", "")), json.dumps(s["tool_args"], sort_keys=True))
            if key in seen:
                return i, "repeat tool+args (loop)"
            seen.add(key)
    return None, "none"

Two signal types, and the earliest one wins:

status == "error" — an explicit tool error. The cheapest possible signal, and the one most logs already have.
A repeated identical (tool, tool_args) pair — the agent called the same tool with byte-identical arguments it already tried. That's a loop, or low-information-gain retry, and it's a signal even when nothing errored.

That second check has a bug I hit on the first run, which is worth admitting because it's a real trap. My first version keyed the loop on every step that had a tool field. But tool results (role: "tool") also carry a tool field, and two results from the same tool with no tool_args produced an identical empty key, firing a false loop on a perfectly clean trace. The fix is the if "tool_args" in s guard: only a call can be a loop, never a result. My clean fixture went from a wrong 45.4% to a correct 0.0% after that one line. Detection logic is exactly where these tools quietly lie to you, so I keep the fixtures adversarial.

Once it has the first signal index, the rest is arithmetic: sum the tokens at and after that index, divide by the total, convert to dollars at a configurable rate (the default $5/1M is a placeholder you override with --price-per-1m, not a vendor quote), and set the exit code against a threshold (default 0.30).

The real run

I ran this live on Python 3.13.5 with real tiktoken o200k_base. Two fixtures: a clean linear run, and a loopy one where the agent retries an identical failing call seven times. Verbatim output, nothing edited:

$ python3 waste_probe.py trace_clean.json
trace: trace_clean.json  (tiktoken o200k_base (exact))
steps: 8   total tokens: 326
token curve: 37 54 38 49 38 44 10 56
first signal: none - clean run
waste_after_signal: 0.0%  (0/326 tokens)
$ wasted after signal: $0.000000  (at $5.0/1M tok)
gate: PASS (threshold 0.30)  exit=0

$ python3 waste_probe.py trace_loopy.json
trace: trace_loopy.json  (tiktoken o200k_base (exact))
steps: 14   total tokens: 853
token curve: 37 58 51 77 51 74 51 72 51 78 51 79 51 72
first signal: step 3 (tool-error)
waste_after_signal: 82.9%  (707/853 tokens)
$ wasted after signal: $0.003535  (at $5.0/1M tok)
gate: FAIL (threshold 0.30)  exit=1

Read that loopy curve left to right. The run climbs normally (37 58 51 77), then at step 3 the first tool-error fires. After that, look at the rhythm: 51 74 51 72 51 78 51 79 51 72. The agent reissues the identical request, gets back the identical 200-OK payload it can't parse, narrates a fresh theory about why it'll work next time, and repeats. Five more round trips of pure re-derivation. 707 of 853 tokens, 82.9%, landed after the first signal.

That 82.9% is my number, on my fixture. It is not the paper's 58.1%. The paper measured real GAIA traces across many runs and reported an average; I built one deliberately loopy trace to show the mechanism cleanly, and a single contrived trace will sit higher than a population average. Same phenomenon, two completely different denominators. If I ever quote one as the other, call me on it.

A note on the dollar figure: $0.003535 is tiny because the fixture is tiny. It scales linearly with token count and with whatever real rate you pass. The ratio is the durable signal; the dollars just put it in units a finance person reads. Run it at your own rate:

$ python3 waste_probe.py trace_loopy.json --price-per-1m=15.0
$ wasted after signal: $0.010605  (at $15.0/1M tok)

And the gate is real. No args returns exit 2; clean returns 0; loopy returns 1 — so a CI job can branch on it. The output is deterministic, too: I hashed the loopy run twice and got identical sha256 both times. No clock, no randomness, no network. Same trace in, same number out, every time — which is the only way a CI gate is worth having.

What to do with it on Monday

Three uses, in rough order of effort.

1. Measure your own ratio first. Take one real failed trace you already have — convert it to the [{role, content, tool, tool_args, status}] shape, run the probe, and read the percentage. Don't assume it's 58% and don't assume it's 83%. Measure yours. The whole reason this tool is keyless and offline is so you can do that on a real production log without asking anyone's permission — the same measure-first habit behind metering your MCP server's per-tool token tax before you cut anything.

2. Wire it as a CI gate on collected traces. If you save agent traces (and for anything in production you should), drop waste_probe.py into the pipeline that ingests them. exit 1 means "this run burned more than 30% of its tokens after it had already failed" — which is a regression worth a red build. Tune --threshold to your reality; 0.30 is a starting line, not a law. I picked it deliberately, and here's why.

3. Close the loop toward early stopping. The same paper that gave us 58.1% ran a small pilot where acting on the early warning cut the post-warning token fraction from 0.638 down to 0.304. That's the entire game in one before/after: detect the signal, stop near it, and the tail collapses. The probe is the measurement half. The other half is your agent's retry logic actually treating a repeated-identical-call as terminal instead of transient — which, going back to the loopy fixture, is exactly the bug the agent never fixed about itself.

What this is not

Honesty about the edges, because the tool is small on purpose:

Not a runtime blocker. It reads finished traces. It will not stop a burning run mid-flight — that's a spend cap's job.
Not a detector of all waste. It catches two signals: explicit tool errors and byte-identical repeated calls. A semantically pointless-but-textually-different retry slips right past it. So does a wrong-but-confident answer with no error at all. The repeated-call check is exact-match by design — it's high-precision and deliberately low-recall.
Not a substitute for an eval. A waste ratio tells you how much burned after the signal, never whether the final answer was correct. You still need both.
The threshold is a guess until you calibrate it. 0.30 echoes the paper's post-intervention 0.304, which is a nice coincidence and nothing more. Your number is your number.

The thesis, one more time, falsifiable: in a failed run, the expensive tokens come after the first detectable signal, not before. The published average on warned failed runs is 58.1%. My loopy fixture hit 82.9%. Now go get yours — and if it comes back consistently under 30%, you've got a healthier stop condition than most, and I'd genuinely like to hear how you built it.

Which signal trips first on your traces — the error or the loop? And what finally makes your agent treat a repeated identical call as terminal instead of transient? Drop it in the comments; that retry-is-terminal question is the one I'm still chewing on.

Written by Alexey Spinov. The tool, both fixtures, and the verbatim run output are included with this post — clone, run, and check the numbers against your own logs. Disclosure: I drafted this with AI assistance and ran, verified, and edited every line and number myself; the live output above is from my own machine.

The Context Tax: Why Step 12 Costs 42x Step 1 (Measure It in 40 Lines)

Alexey Spinov — Wed, 17 Jun 2026 19:25:08 +0000

In short: the context tax is what you pay when every agent step re-sends the whole session transcript as input again, so step N re-bills turns 1..N and total cost grows with n(n+1)/2. Cheaper tokens lower the unit, not the shape. context_tax.py meters the re-bill multiplier offline; one debugging session measured 42.8x.

AI disclosure: I drafted this with an AI writing assistant. The tool, the fixtures, and every number below come from a real local run of the script in this post on tiktoken o200k_base. I reviewed and edited it before publishing.

Token prices have been sliding all year. Your agent bill probably hasn't.

I kept running into the same confusion in my own FinOps notes: per-token rates drop, and the monthly number goes the other way. The usual answers ("you're using a bigger model," "you have more users") didn't explain a single session getting more expensive as it ran. So I wrote a 40-line meter to look at the one thing nobody charts: the session transcript itself. On a synthetic-but-realistic debugging session, the last step billed 42.8x the input of the first step. Same model. Same task. No new users.

That gap has a boring cause and an annoying consequence. Here's both, plus the script.

TL;DR. Every step of an agent loop re-sends the whole conversation so far (history plus tool outputs) as input. So step N pays for turns 1..N again, and total input grows roughly with n(n+1)/2. Cheaper tokens don't fix the shape; they just lower the unit on a number that's still climbing. context_tax.py (below, keyless, offline) meters three things from a session JSON: the re-bill curve, the re-bill multiplier, and a dead-weight estimate. On my bloated fixture it reported a 42.8x multiplier and 19.3% dead weight, and exited 1 as a CI gate.

Why a transcript gets billed again every single step

Here's the part that trips people up. An LLM call is stateless. The model doesn't "remember" turn 3 when you make turn 12. Your framework re-sends turns 1 through 11 as input so the model can see them. Every. Single. Step.

So the cost of one step isn't the cost of that step's new text. It's the cost of the entire history up to that point. Step 1 bills a short user message. Step 12 bills the user message plus a file dump plus a wide grep plus a stack trace plus every assistant reply in between. The new tokens at step 12 might be tiny. The billed input is not.

Logan (Waxell) put the shape plainly in The Compounding Math Your Architecture Is Hiding: "total cost grows roughly with n(n+1)/2," and a turn-10 context can sit at 80,000–200,000 tokens. That post nails the problem and then points you at a proprietary runtime. I wanted the opposite: a tiny script I can run on my own transcript and check into CI. So that's what this is.

And it's why "tokens got cheaper" is the wrong consolation. Edwin Lisowski's Token Prices Are Falling. So Why Is Your AI Bill Going Up? lists the drivers: full context re-sent each step, tool schemas eating 30–60% of the window before any user content, retries and sub-agents running around the clock. That schema overhead is a sibling tax worth metering on its own — I did exactly that for MCP servers in measure your MCP server's token tax, where the tool definitions are billed on every call before a single user token. His illustrations are blunt. He cites AT&T going from 1B to 27B tokens/day over 18 months. (Those are Lisowski's examples, not my measurements; I'm attributing them to him.) Cheaper unit, bigger n. The unit lost.

You can't estimate this. You have to measure it.

There's a second reason to meter instead of guess. Agents are bad at predicting their own spend.

The arXiv paper How Do AI Agents Spend Your Money? (Bai, Huang, Wang, Sun, Mihalcea, Brynjolfsson, Pentland, Pei) measured agentic coding tasks and found three things worth pinning to the wall: agentic runs burn roughly 1000x the tokens of a plain code-chat; the same task can vary up to 30x in cost run to run; and models "fail to accurately predict their own token usage," with correlations up to just 0.39. A 0.39 correlation is barely better than a shrug.

So the takeaway writes itself: meter the transcript, don't trust the estimate. If the model can't call its own number, your gut can't either.

The tool: 40 lines, no API key

context_tax.py reads one JSON file: a session transcript as a list of turns (role + content, tool results included). It tokenizes with tiktoken's o200k_base and reports four things.

Re-bill curve — billed input at each step, i.e. the cumulative history 1..N re-sent on that call.
Re-bill multiplier — billed input at the last step ÷ billed input at the first step. How much more "one call" costs at the end versus the start.
Dead-weight % — tokens from early turns whose terms basically never resurface in later turns (a turn counts as dead if under 15% of its terms reappear downstream). Stuff you keep paying to re-send that the model isn't really using. It's the same dead-weight idea I applied to persistent stores in auditing your agent's memory tax — there the stale entries ride along on every retrieval; here they ride along on every step.
$ / session — total billed input across all steps × a rate you pass in. The rate is a parameter, not a hardcoded vendor price — this is a compounding illustration, not your invoice.

The exit code is the point. 0 if the multiplier is under threshold (a disciplined session), 1 if it's over (the architecture is compounding, so fail the build), 2 for usage. Drop it in CI and a session that balloons becomes a red check, not a surprise line item.

#!/usr/bin/env python3
"""context_tax.py - meter the re-bill tax on a single agent session's transcript."""
import json, re, sys

THRESHOLD = 12.0         # re-bill multiplier above this = compounding architecture
DEAD_OVERLAP = 0.15      # a turn is dead weight if <15% of its terms resurface later
STOP = set("the a an of to in is it on for and or but with as at by from this that be are was you your i we they it's".split())

try:
    import tiktoken
    _enc = tiktoken.get_encoding("o200k_base")
    def count(t): return len(_enc.encode(t))
    TOKENIZER = "tiktoken o200k_base (exact)"
except Exception:                              # honest fallback, ~+-15% vs real BPE
    def count(t): return max(1, round(len(t) / 4))
    TOKENIZER = "len/4 heuristic (tiktoken not installed; ~+-15%)"

def words(t): return {w for w in re.findall(r"[a-z0-9_]{4,}", t.lower()) if w not in STOP}

def main(argv):
    if len(argv) < 2:
        print("usage: context_tax.py <session_transcript.json>"); return 2
    s = json.load(open(argv[1], encoding="utf-8"))
    rate = float(s.get("input_usd_per_mtok", 3.0))     # $/1M input tok; configurable, NOT a vendor quote
    turns = [t["content"] for t in s["turns"]]
    tok = [count(c) for c in turns]
    later = [words(" ".join(turns[i + 1:])) for i in range(len(turns))]
    billed, dead = [], 0
    for n in range(len(turns)):                        # step n re-bills the running history 1..n
        billed.append(sum(tok[: n + 1]))
        if n < len(turns) - 1:
            w = words(turns[n])
            overlap = len(w & later[n]) / len(w) if w else 1.0
            if overlap < DEAD_OVERLAP:
                dead += tok[n]
    mult = billed[-1] / billed[0] if billed[0] else 0
    total_billed = sum(billed)
    print(f"context_tax | {argv[1]} | tokenizer: {TOKENIZER} | rate=${rate}/Mtok | threshold x{THRESHOLD}")
    print("-" * 78)
    for n, b in enumerate(billed):
        bar = "#" * round(b / billed[-1] * 40)
        print(f"  step {n + 1:>2}  billed_input={b:>6}t  {bar}")
    print("-" * 78)
    print(f"  re-bill multiplier (step {len(billed)} / step 1) : x{mult:.1f}")
    print(f"  dead-weight (never referenced later)     : {dead}t = {dead / billed[-1] * 100:.1f}% of the final payload")
    print(f"  total billed input across session        : {total_billed}t  (${total_billed / 1_000_000 * rate:.4f} at ${rate}/Mtok)")
    print(f"  exit                                     : {1 if mult > THRESHOLD else 0}")
    return 1 if mult > THRESHOLD else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv))

No key, no network, read-only. pip install tiktoken, point it at a transcript JSON, done. If tiktoken isn't installed it falls back to a len/4 heuristic and says so out loud (~±15% off real BPE). I'd rather print the caveat than pretend the number is exact.

The real run

Two fixtures ship with the script. Both are synthetic coding sessions (no private data) but shaped like the real thing.

session_lean.json is a disciplined session: small tool outputs, and a deliberate scope reset before the second task. Here's the actual output:

context_tax | session_lean.json | tokenizer: tiktoken o200k_base (exact) | rate=$3.0/Mtok | threshold x12.0
------------------------------------------------------------------------------
  step  1  billed_input=    25t  ####
  step  2  billed_input=    46t  ########
  step  3  billed_input=    75t  #############
  ...
  step 10  billed_input=   235t  ########################################
------------------------------------------------------------------------------
  re-bill multiplier (step 10 / step 1) : x9.4
  dead-weight (never referenced later)     : 56t = 23.8% of the final payload
  total billed input across session        : 1335t  ($0.0040 at $3.0/Mtok)
  exit                                     : 0

Multiplier 9.4x, under the 12x threshold, exit 0. Green. Note the dead weight is still 23.8%: that's the first task's context the model no longer needs in the second task. Even a clean session carries dead weight until you actually trim. The scope reset kept the multiplier down; it didn't zero the waste.

session_bloated.json is the one that hurts. A 12-step debugging session that never trims: a full module dump, a wide repo grep, a long stack trace, and the kicker, a verbose pip check dependency log that gets re-sent on every step after it. Real output:

context_tax | session_bloated.json | tokenizer: tiktoken o200k_base (exact) | rate=$3.0/Mtok | threshold x12.0
------------------------------------------------------------------------------
  step  1  billed_input=    40t  #
  step  2  billed_input=    72t  ##
  step  3  billed_input=   421t  ##########
  step  4  billed_input=   480t  ###########
  step  5  billed_input=   857t  ####################
  ...
  step 12  billed_input=  1713t  ########################################
------------------------------------------------------------------------------
  re-bill multiplier (step 12 / step 1) : x42.8
  dead-weight (never referenced later)     : 331t = 19.3% of the final payload
  total billed input across session        : 11774t  ($0.0353 at $3.0/Mtok)
  exit                                     : 1

42.8x. Over threshold, exit 1: a failed build. Watch step 3 in the curve. The full file dump jumps billed input from 72 to 421 tokens, and you pay that bump again on every one of the nine steps that follow. The 331 dead-weight tokens are mostly that pip check log (boto3 versions, urllib3 pins) that never came up again but kept riding along in the payload.

Both numbers are reproducible. I hashed two consecutive bloated runs with shasum -a 256 and got identical digests, so the output is deterministic, not a fluke of one run.

One honest correction. I'd guessed the multiplier would land near 16x when I started (that's the figure floating around the n(n+1)/2 discussions). The real run said 42.8x. The bloated fixture front-loads a big file dump on a small first turn, which stretches the ratio. The lesson isn't "16x vs 42x." It's that the number depends entirely on your transcript shape, which is exactly why you measure your own instead of borrowing mine.

What to actually do about it

The fixes aren't exotic. The point of the meter is to tell you which one you need, and to prove it worked.

Scope-reset between tasks. The lean fixture does this: drop the prior task's context before starting the next one. It's the difference between 9.4x and 42.8x here.
Trim or summarize fat tool outputs. That pip check dump was 19.3% dead weight. Replace a 300-token log with a one-line "deps OK, no conflicts" and you stop re-billing it nine times.
Rolling summarization for long sessions: collapse old turns into a short recap once they're settled, instead of carrying them verbatim.

Then re-run the meter. If the multiplier drops back under threshold, the exit code flips to 0 and your CI gate goes green. That's the whole loop: measure, cut, prove. Not "trust me, I optimized it," but a number that moved. This meter slots in alongside the other checks in my pre-execution gate for AI agents — same philosophy, fail fast before the spend, not after the invoice.

What this is NOT (so I don't oversell it)

It does not block or cap anything at runtime. It's a meter and a CI gate, not a spend guard. If you want the runtime brake that stops a session mid-loop, that's a different tool — see the sliding-window spend guard, which caps cumulative cost over a window instead of just measuring it after the fact.
It does not compute your real provider invoice. The $/session figure uses a rate you pass in, to illustrate compounding. Your actual bill depends on caching, batching, output tokens, and your vendor's pricing — none of which this models.
Dead-weight is a lexical heuristic, and it has false positives. "Under 15% of terms resurface later" is a proxy for "the model stopped using this," not proof of it. The model may have leaned on an early turn implicitly without repeating its words. On my bloated fixture the stack trace landed at 0.16 overlap, just above the line, correctly kept, because the fix really did reference it. Treat the % as a flag to go look, not a verdict.
It does not optimize your context for you. It tells you where the tax is. The cutting is still your call.

What's the worst re-bill multiplier you've measured on one of your own long sessions? Run the script on a real transcript and tell me in the comments. I'm collecting shapes, and I read every reply. Follow for the next number from the next run.