DEV Community: Willian Pinho

Recall is not enforcement

Willian Pinho — Tue, 18 Aug 2026 14:00:06 +0000

The rule I have had to learn more than once is four words long, and it is written into 103 files across my agent setup:

Rules are NOT learnings — a non-negotiable rule needs always-loaded text + a deterministic hook, never a memory store (recall ≠ enforcement).

That much is not what I want to argue, because anyone who has watched a memory store fail already agrees. Retrieval is probabilistic and depends on the model deciding the rule is relevant right now. Fine for context, not a control. So you write the rule into always-loaded text, you build a mechanism that can exit non-zero, and you audit the pair: text present, mechanism present, mechanism wired.

That audit passes on rules that are not enforced. Both halves can exist, name each other, be correctly wired, run on every invocation, and the rule still does not hold. That is the claim, and it took a measured failure in my own system to make me believe it.

The two halves, briefly

Neither half substitutes for the other. Always-loaded text is in context every time without anything deciding to fetch it, and it explains the why, which is what lets an agent apply the rule to a case the mechanism never anticipated. A deterministic mechanism (a hook, a lint, a pre-push check, an exit code) holds when the text has not been re-read, or has been read and rationalised away. Text alone is a preference. Mechanism alone produces the worst kind of workaround: the one that satisfies the check and defeats the intent.

My ownership rule states this out loud:

This file is the always-loaded half; the hook is the half that holds when this file has not been re-read. Per CLAUDE.md, recall is not enforcement — which is why the rule is not a memory entry.

The other half is a 192-line hook that exits 2, wired by name in the settings file. Both halves exist and each names the other. That is the model working, and it is the setup, not the point.

A mechanism can implement the shortcut its own rule forbids

I have a quality rubric with an explicit rule: titles are a cheap pre-filter, never the decision. Written down, always loaded, in the right place, in plain language.

There is also a mechanism. It decides whether a topic has already been covered by computing lexical overlap between the topic title and the corpus title, then treating >= 0.8 as consumed.

That is a title comparison used as the sole decision, precisely what the rule forbids, implemented by the code that exists to enforce the rule. Not hypothetically: on one queue it passed three topics carrying a single claim, because their titles barely overlap, while one of the three sat approved and queued.

Nothing was missing. The text was there, the mechanism was there, it ran, and it returned a wrong answer with no error. Every box the standard audit checks was ticked. Until that function escalates to judging claim shape instead of comparing titles, the honest disposition is the one now written next to it: that decision is a human judgement, and the automation is not trusted to make it.

A reasonable person answers that this is a bug, not a category: fix the function and the model is intact. I do not think that survives the next section. Nobody caught it because the audit everyone runs cannot see this class of defect, and it is wrong in the other direction too.

The wiring check fails in both directions

The obvious audit is to grep the settings file for your hook and confirm it is there. It fails both ways, and the census below measures both: thirteen of my hooks are armed and appear nowhere in that file because a parent dispatches them, and three appear in it while being unable to refuse anything today.

So the real check has three questions, in order:

Does the always-loaded text exist, and does it say why?
Does a mechanism exist, and is it actually reachable from the runtime, dispatch chains included?
Has the mechanism ever been observed rejecting the thing it exists to reject?

One and two are the ones people run, and they are exactly the two that cannot tell an enforced rule from a decorated one. Three is the only one that produces evidence. Feed the mechanism the forbidden input and read the exit code. A gate you have never seen refuse is a gate you are assuming.

None of this is my idea, and one authority disagrees about where it belongs

Palantir published it in 2017. Their Alerting and Detection Strategy Framework makes it a mandatory section of every detection they write:

Validation are the steps required to generate a representative true positive event which triggers this alert. This is similar to a unit test and describes how an engineer can cause the ADS to fire.

Google was doing the same thing for alerting rules a year earlier. The SRE Book's Borgmon chapter, chapter 10 and not the testing chapter people usually cite for this, describes synthesising input so a rule can be watched firing, then running those tests in CI before the config ships. And NIST has named the distinction for years. SP 800-53A Rev. 5 defines Examine as "checking, inspecting, reviewing, observing, studying, or analyzing" one or more assessment objects, and Test as "exercising one or more assessment objects under specified conditions to compare actual with expected behavior". They are peers, and a control can be declared effective on Examine alone.

So the idea is old. What is missing is anyone publishing the ratio — of the controls that pass Examine, how many fail Test.

Where I do disagree is with a vendor, about positioning, and the vendor is the one whose runtime I am auditing. Anthropic's own configuration debugging guide tells you to run /hooks to see what is registered, and then:

If the hook appears but doesn't fire, the matcher is the usual cause.

Behavioural proof does appear on that page, further down, via claude --debug. But it appears as debugging — the thing you reach for once you already suspect something is broken. It is never the routine check. That ordering is the whole disagreement: registration is presented as the audit and refusal as the troubleshooting, and my numbers below say it should be the other way round.

The protocol, borrowed rather than invented

Security teams already have a runnable version of this. Red Canary's Atomic Red Team is a library of tests you fire at your own environment, and its getting-started procedure is six steps that reduce to one idea: execute the thing the control exists to catch, then "use the data collected by your security solution to evaluate and improve your detection coverage."

I ran that shape against guard hooks instead of against detections. One substitution matters and it is not cosmetic: Atomic Red Team really executes the attack, and several of my hooks exist to block reads of credentials. Executing the forbidden command to see whether the guard stops it is how you leak the thing you were testing the guard for. So every probe is delivered to the hook on stdin and only the exit code is read. The command never runs.

What I could not borrow is a checklist for this specific case. I looked through the SRE book and workbook, Principles of Chaos, OPA and Conftest, Atomic Red Team, the Palantir framework, NIST 800-53A and 800-115, and the detection-engineering literature. There is no published protocol for auditing the guardrails of an agent harness. That absence is not a gap in the reading. The practice is old and well documented; what I could not find written down anywhere is it pointed at this target. So what follows is someone else's method aimed somewhere new, and the numbers are what came back.

So I ran question three on all ninety

Writing that down obliged me to answer it, which I had never done. I enumerated every hook wired in my harness, resolved the ones that are dispatched as children rather than named directly, then fed each one the input it exists to reject, on stdin, and recorded the exit code.

Ninety hooks. Seventy-seven named in the settings file, thirteen more reachable only through a dispatch parent.

	count	what it means
wired hooks	90	77 named, 13 reachable only via a dispatch parent
advisory by design	59	no refusal primitive anywhere in the file
able to refuse	31	by hand; an automated scan says 34, see below
observed refusing	28	fed the forbidden input, exited 2
returned 0 on what they reject	2	one dead, one dormant by design
not testable	1	reject path needs a fabricated transcript
declared but not armed	3	in the canonical settings, absent from the live one

Fifty-nine of the ninety cannot refuse anything, and that is correct. They inject context, write state, log telemetry. A scan for the four primitives that can actually block returned 31 hits and 59 misses. Calling those 59 "guards" would have been the first error, and it is the one an audit that counts files rather than capabilities makes.

Of the 31 that can block, 28 were observed refusing, at exit 2, with the forbidden input on stdin. One was not testable without fabricating a transcript and making a nondeterministic model call, so its reject path is unmeasured rather than absent.

Two returned 0 on the thing they exist to reject. Those two are the reason the question is worth asking.

The first is a rule-capture hook whose job is to make sure a rule stated in conversation is not lost when the session ends. It filters its capture file by session id. The capture file holds 230 entries and every one of them has a null session id, because the writer stores whatever the event delivered and that event does not deliver one. Feed it a session id that has captures: exit 0. Feed it the null it actually stored: it hits the no-session guard and exits 0. Both branches are dead, the hook is silent by design, and nothing about that is visible from outside. It has been wired, and running, and unable to fire, for as long as those 230 entries have existed.

The second is honest about itself in its own header: it ships dormant pending an allowlist that does not exist yet, and refuses correctly the moment its flag is set.

The finding I was not looking for

The census counted against the canonical settings file. Out of habit I diffed it against the one the runtime actually reads.

The canonical file references 82 hook files. The live one wires 79. Two of the three missing had just been measured blocking at exit 2. They work. They are not armed.

So the harness contains a guard that stops two agents dispatching into the same path, and a gate that validates handoff contracts between agents, both of which refuse correctly when invoked and neither of which is currently invoked at all. Question two, asked carelessly, passes them: the mechanism exists and the file says it is wired. It is the wrong file.

A third guard, which blocks duplicate decision-record numbers, is armed and structurally blind on the repository it lives in: its path matcher is anchored on a directory the harness moved out of, and the path it still matches does not exist on disk.

Three mechanisms, all of them real, none of them able to refuse anything today. I found them by asking one question I had been recommending to other people and had never run on myself.

The instrument, and what it gets wrong

The census above was done by hand the first time. Doing it by hand is how it stays a story, so I turned it into a script that anyone can point at their own hooks directory: it enumerates what is wired, resolves dispatch children, classifies each hook by whether it contains any primitive that can refuse, fires the probes, and diffs the settings file you maintain against the one the runtime reads.

Two of its design choices are the whole point. A hook with no probe is reported UNPROBED and is never counted as passing, because a census that could not test something must report a smaller denominator and a named gap rather than a clean bill. And the declared-versus-armed diff runs every time, unasked, because that is the check that found the two unarmed guards.

It also gets something wrong, and I am leaving the error in. Its automated scan reports 34 hooks capable of blocking where reading them by hand gives 31. The three extra carry an exit-2 inside a top-level catch, which is fail-closed on an internal crash and not a refusal of anything. A regex cannot tell a detection-exit from a crash-exit.

Which is this post's own subject, committed by the tool written for this post: a counting mechanism that cannot distinguish the thing it counts from something that resembles it. I could tune the pattern until the numbers agreed. Then the tool would be right on my repository and quietly wrong on yours, and I would have replaced a visible error with an invisible one. So the count is documented as an upper bound instead.

The script is public at github.com/willianpinho/hook-refusal-census. MIT, no dependencies, read only. Point it at your own hooks directory and it will tell you how much of your guard set it could not answer for.

One thing it will not do is hand you the 28 above. The census that produced that number probed all 31 by hand; the tool ships probes for 12 of them, and reports the other 19 as UNPROBED rather than quietly counting them. A probe is a claim about what a specific guard exists to reject, so it cannot be generic, and writing the missing 19 is work I have not done. The number you get on your own hooks will be smaller than mine for the same reason it should be: it will only count what you actually watched.

Making it generic cost two more findings, and both belong in this post rather than in a changelog.

The first is that the tool gives different answers from different directories. Same probe, same hook: a git-workflow guard returned 0 fired from a folder that was not a repository and 2 fired from inside one, moving the observed-refusing count by a full guard on nothing but where the process happened to be standing. It now prints the directory it fired from, because a measurement that does not report its conditions is not reproducible, and I had been reporting one.

The second is the same failure as the 34-versus-31 count, one layer out. The generic version briefly flagged every settings reference with no matching file in the hooks directory as a missing hook. On the first repository it ran against, that produced five accusations, and all five were ordinary sync scripts living one directory over, invoked from settings exactly as intended. I caught it only because I read the five names instead of the number, which is the same reason the gap above is documented rather than tuned away.

Where this bites outside agent systems

The pattern is not about AI, and outside agent systems it usually shows up as the false positive rather than the missing hook. A lint configured on a path the file is not in. A required status check whose job exits 0 when the test runner never started. A secret scanner whose patterns predate the current token format.

In each case the policy is written, the tooling exists, the check is green, and no one has watched the thing refuse anything. The rule holds while things are calm and stops exactly when they are not, which is when you needed it.

Write the text so people can reason about the rule. Build the mechanism so the rule survives the day nobody is reasoning. Then break the mechanism on purpose, because that is the only one of the three questions that answers with evidence instead of an assumption.

The API layer that was fake all the way down

Willian Pinho — Sat, 15 Aug 2026 13:00:04 +0000

I was reading a generated API client in a codebase I had just been handed. It looked like every generated client you have ever integrated against:

export class InventoryApiClient {
  private baseUrl: string;

  constructor(baseUrl: string = "/api/mock/inventory") {
    this.baseUrl = baseUrl;
  }

A configurable base URL. Async, typed methods. Two clients, nine methods, 185 lines. Everything about the shape says "this talks to a network."

Then the method body:

  async getInventory(
    sku: string,
    location?: string
  ): Promise<InventoryItem | null> {
    // In a real implementation, this would make an HTTP request
    // ...
    const { getInventory } = await import("../../mocks/inventory-api");
    return getInventory(sku, location);
  }

A dynamic import of a local module, and a function call in the same process. That is the entire implementation. Every method in both clients is that shape.

To be clear about provenance: this was starter code I was given, not code I wrote. It was honest about itself, in the comment above that import, which matters for the point.

Open the file before trusting the interface: obvious, and everyone already agrees. The question starts after you open it. Knowing the boundary is fake, what do you build against it? The usual answer is to build for the real network now, since it is coming anyway. I think that is worse than building nothing. What follows does not settle that comparison, because it never builds the nothing arm. What it does is price one side of it: I wrote the layer that advice asks for, ran it against the fake boundary, then ran the same layer against a real transport and counted what changed.

What is actually absent

Not "mocked out". Absent.

grep -nE "fetch|http[s]?:|XMLHttpRequest|axios|Request\(|new URL" src/generated/*/client.ts
-> no matches

And the baseUrl is not merely defaulted, it is dead. Six occurrences across both files: declaration, parameter, assignment. Zero reads. Pass any string you like; nothing will ever look at it.

That dead parameter is the argument in miniature: configuration written for a network call that does not happen, sitting in the repository being read by nothing.

What is there instead

One directory over sit the modules those dynamic imports resolve to. Nine exported functions, and every one of them opens the same way:

export async function getInventory(sku: string, location?: string): Promise<InventoryItem | null> {
  // Simulate API latency
  await new Promise((resolve) => setTimeout(resolve, 100));

The declared sleeps are 100, 150, 100 and 50 milliseconds for inventory, and 100, 120, 100, 100 and 100 for shipments. That is the whole simulation. Of everything a network does to a caller, the one property these modules reproduce is the one that carries no failure.

I ran them to see what a caller could learn. 900 calls, 100 per method, through the generated clients rather than around them:

method                                    fail      min      p50      p99      max
InventoryApiClient.getInventory              0    99.47   101.56   105.03   108.97
InventoryApiClient.searchInventory           0   150.07   152.39   170.26   174.05
InventoryApiClient.getLowStockItems          0    99.75   102.64   107.32   108.83
InventoryApiClient.getLocations              0    50.44    52.37    56.82    58.83
ShipmentApiClient.getShipment                0    99.62   102.41   105.59   107.38
ShipmentApiClient.getShipmentsBySku          0   120.09   122.38   127.35   135.47
ShipmentApiClient.getDelayedShipments        0    99.78   102.36   104.63   106.37
ShipmentApiClient.getShipmentsByStatus       0   100.36   102.36   104.74   107.00
ShipmentApiClient.getActiveShipments         0   100.48   102.35   106.00   106.60

Zero failures. Every p50 lands between 1.56ms and 2.64ms above its declared constant, and the worst p99 is 1.12 times its own p50. There is a distribution in those numbers, but it is not a service's: the mock declares a constant, and what varies around it is the scheduler of whatever machine you ran it on. An earlier run of the same script on the same laptop put the same two figures at 1.45ms to 1.96ms and 1.20. Calibrate a latency budget from either run and you have calibrated it to a setTimeout.

Then I tried to make the calls fail: 40 of them, eight hostile arguments across the five methods that take one. Twenty threw. Every one was the same thing:

InventoryApiClient.getInventory <- null: Cannot read properties of null (reading 'toLowerCase')
ShipmentApiClient.getShipment <- number: trackingNumber.toLowerCase is not a function

Not one of the twenty resembles a network failure. They are type violations, reachable only by lying to the compiler. And the fifth method never threw at all: fed null, 42, [], undefined and "", getShipmentsByStatus returned {"shipments":[],"total":0} every time, which is a wrong answer wearing the shape of a right one.

What makes the absence easy to miss is that these files do model failure, in the wrong layer. The shipment records carry statuses delayed and exception, and notes like 'Weather delay - winter storm affecting route'. The domain's bad days are represented in detail. The transport's do not exist.

The layer written blind, run on both sides

So I wrote the layer the counter-position asks for, using only what that boundary can tell you. A 250ms timeout, comfortably above every p99 in that table. Three attempts, 50ms of linear backoff. A six-branch error taxonomy: ok, timeout, server_error, client_error, connection_refused, unknown. Ordinary code. I would have written it.

Against the fake boundary, thirteen calls through the generated clients:

branch hits: ok=11  timeout=0  server_error=0  client_error=0  connection_refused=0  unknown=2

Four of the six branches never execute. The two that do are the happy path and the bucket the TypeErrors fall into.

Then the same wrapper, unchanged, over fetch against a local HTTP server I can make answer 200, 500, 404, hang past the timeout, or refuse the connection:

200 fast                    ok                       attempts=1    14.08ms
500 upstream                FAILED as server_error   attempts=3   157.35ms
404 not found               FAILED as client_error   attempts=1     2.21ms
hangs past the timeout      FAILED as timeout        attempts=3   914.67ms
connection refused          FAILED as unknown        attempts=1     3.26ms
      error seen: fetch failed

branch hits: ok=1  timeout=3  server_error=3  client_error=1  connection_refused=0  unknown=1
work that landed after its timeout: 3

The script prints an error seen: line under every failure; three of the four are elided above, and the one left in is the one the next paragraphs turn on.

Three things came out of that, and only the first is the one I went looking for.

Three branches went from never executed to exercised in a single commit. timeout, server_error and client_error had not run once. It took a transport that could fail to run them at all, and I had to build that transport deliberately. Nothing here ran in production. That is the point: if nobody builds the failing transport on purpose, the first thing that runs those branches is whatever you deploy onto.

One branch never fired on either side, because it was wrong. The classifier tests err.code === "ECONNREFUSED". A refused connection out of fetch is a TypeError whose message is fetch failed and whose code is undefined; the string is on err.cause.code. So the refusal fell into unknown, unknown is not in the retryable set, and it got attempts=1. A refused connection is the textbook case for a retry, and it is the one case that got none. Nothing the fake boundary can produce would have shown me that. The runtime's own documentation would have, if it had occurred to me to doubt the property name.

The timeout does not cancel anything. These methods take no signal parameter and there is nothing to abort inside a function call in the same process, so the timeout you write is a race against a timer. For these particular functions, which sleep and then return, abandoning the work costs nothing. That is a property of the mocks and not of locality: local work that holds a lock or mutates state would not be so forgiving. Over a real transport the same code abandons the request without stopping it, and three of them completed after the wrapper had already reported them gone. The caller's latency is bounded, which may be all the timeout was ever for. The connections are not.

But that is my design

My own first reaction to those three was that they might belong to the wrapper rather than to the boundary, and that a better engineer would have written it differently. So I wrote six classifiers instead of one, each a shape that turns up in ordinary Node code: the code read off the error, the code read off err.cause, the node-fetch name check, a substring match on the message, "any TypeError is the network", and status codes with no transport branch at all. Then I ran all six against both boundaries, with two extra real failures added: a socket destroyed mid-body, and a 200 whose body is HTML.

On the fake side, five of the six are indistinguishable. Every scenario that boundary can produce gets the same verdict from all five: four calls ok, two type violations unknown. The sixth differs, and it differs by being wrong in the direction the fake boundary rewards, labelling those local type violations connection_refused.

On the real transport:

named the refused connection correctly: 2 of 6 (code-on-cause, typeerror-is-network)

Four of the six cannot name a refused connection at all. Of the two that can, one reads err.cause.code; the other calls every TypeError a refusal, which is why that same classifier also labels a socket reset and both local type violations as refusals. The socket destroyed mid-body is unknown to five of the six. The 200 that fails to parse is unknown to all six.

Status codes are the easy part, and all six get 500 and 404 right. Everything that is actually about the transport is where they diverge, and the fake boundary produces none of it. Nothing that boundary can produce tells these six apart, and telling them apart is the entire job. You can still get there another way, by reading the runtime's documentation or by building something that fails on purpose. Neither happens by default, and a boundary that never fails gives you no reason to start.

The objection also has an answer from outside my own code. On 15 August 2026, GitHub code search for the two strings error.code === 'ECONNREFUSED' and await fetch(, filtered to TypeScript, reported 940 matches. That total is GitHub's, it moves, and I have audited none of it. The three files I opened say something more interesting than "everyone gets this wrong". In f/git-rewrite-commits, src/providers/ollama.ts:69, that check sits in a file whose line 1 is import fetch from 'node-fetch', where the thrown FetchError really does carry code. It is correct. In Finsys/dockhand, src/routes/api/registry/tags/+server.ts:161, the identical comparison sits in a SvelteKit route. That repository has no node-fetch anywhere and never assigns globalThis.fetch; what it does do is call undici's setGlobalDispatcher, which changes routing and not the shape of the error. So fetch there is the runtime global, error.code is undefined when the registry refuses a connection, and the intended 503 Could not connect to registry does not return. The handler falls through to its generic branch and answers 500 with the message fetch failed. The comment in that repo's own DNS dispatcher says its IPv4 pinning "guarantees fetch failed" and cites three of its issues, which is the same TypeError this whole section is about. And in danshapiro/freshell, scripts/precheck.ts:197, someone writes both: error.code === 'ECONNREFUSED' || error.cause?.code === 'ECONNREFUSED'.

So the shape is not mine, and it is not carelessness either. It is a check that is right for one widely used library and wrong for the implementation the runtime now ships, and which side of that you are standing on is decided by an import line you may not have written.

The argument was already in the repository

I did not have to price the morning that layer costs, because part of it had already been spent. One directory up from the clients, the request handler wraps its work like this:

try {
  const { message } = await request.json<{ message: string }>();
  const result = await this.processMessage(message);
  return Response.json(result);
} catch (error) {
  return Response.json(
    { error: 'Failed to process message', ... },
    { status: 500 },
  );
}

processMessage is what reaches those generated clients. So the branch reads as "the call downstream failed" while the call downstream is a function invocation in the same process. Two inputs reached it in front of me: a malformed request body and a type violation inside a tool arriving as Cannot read properties of null. A catch that wide can certainly see others, and I am not claiming an inventory. I am saying neither of the two I saw is what the message says it is for.

The rest of the morning buys more of the same shape:

a latency budget derived from a constant plus 2ms of scheduler noise
a retry policy for a call whose only reachable error is a TypeError, which retrying cannot fix
an error taxonomy in which four of six branches are unreachable
a cache in front of something already in-process
load tests that measure your own import graph

The first three are what I measured. The last two I did not build, and they are on the list as the obvious next items rather than as findings.

Wasted work is the optimistic reading. The bill arrives the day the transport becomes real: the scaffolding switches from useless to load-bearing, and its transport branches have never run. The happy path ran eleven times and the catch-all twice; the three branches that exist for the network ran zero times. In this run the switch turned on all three at once, shipped a misclassification that disabled retries for refused connections, and left three abandoned requests in flight. A test could have reached the 5xx branch, as my own local server proves. Nothing in the repository reached it, so nothing did. What you have is error handling with months of age and zero evidence. In review it reads like error handling that survived months of production, unless someone thinks to ask whether any of it has ever run.

That is worse than an empty file. An empty file is honestly missing, and someone notices on the first timeout. This is a control everyone believes exists.

What the experiment does not show

Every layer here is mine. The wrapper is the code I would write against that boundary, not code recovered from someone else's repository, and the six classifiers are shapes I have seen and written rather than a sample drawn from anything. Six is not a survey, so "four of six" is a statement about those six. I name those three public files because I read them, not because I sampled them. I opened a handful out of 940 search hits and picked the three that showed the distinction, which makes them an illustration and not a rate.

The local server is not a network either. It can destroy a socket mid-body, which is one of the seven scenarios above, but it cannot produce packet loss, a TLS handshake failure, DNS, or a proxy. Every failure in that table is one I chose to produce, which makes three broken branches a floor and not a count.

And zero failures across 900 calls is evidence about the substitute, not about the service it stands in for. It does not even establish that the local functions are total; the twenty TypeErrors above are the counterexample. It says those inputs did not make them fail, and nothing about what the real API does under load, during a deploy, or behind something that drops connections.

The counter-position, in the form that survives

The seam is right here and the context is loaded, so build the port now. I would take that, and the run above changed my terms for it. Write the port either way: the shape costs almost nothing and makes the seam explicit. Then there are two honest options for the policy behind it. Leave it empty, and it is at least honestly missing. Or write it, and stand up something that actually fails so you can exercise it before you believe it. The scripts here took an afternoon, so the second is affordable, and I no longer think you have to wait. What is not on the list is the thing the usual advice actually produces: a policy written against a boundary that cannot fail, shipped, and trusted.

The check

Before you write the retry, name the failure it handles and where you saw it. If you cannot, you are not adding resilience, you are adding a claim.

Whether there is a failure mode to handle at all starts with one grep for the transport. It is a cheap check and a partial one: it will not see an aliased import or a wrapper module, and it speaks only for the files you point it at. When it comes back empty on a file that calls itself a client, that file is not an HTTP client. It is a naming convention.

And if you are going to write the layer anyway, run it against something that can actually fail before you believe it. Three scripts and a local HTTP server found three defects in code I was confident about, and then showed that of the six classifiers I wrote to check whether those defects were my fault, exactly one got the case the first one missed without mislabelling another scenario I ran. All of that was invisible from the side of the boundary where the code was written.

The guard that would have blocked its own investigation

Willian Pinho — Sat, 08 Aug 2026 13:00:04 +0000

Deny-by-default is the standard posture for a safety guard: start closed, allow explicitly, add exceptions when someone complains. I argued for exactly that three weeks ago, about agent tool access: fail-close is a default you choose before the first tool is registered, not a guardrail you bolt on after, and the opposite instinct is backwards, and it's going to cost teams who don't fix it before they go to production.

Then I wrote a different guard and deliberately made it fail open. Its header says so in one line: A false block is worse than a miss here.

Both are correct, and the thing that decides which one you want is not a philosophy. It took a hook blocking cat on the script it was guarding to make me write down what it actually is.

What the guard was for

On 2026-07-29 an agent ran the publish ceremony from the main checkout of a repo while that checkout sat on the default branch. The publish script mutates tracked files as a side effect, a metrics CSV and an index, so running it is a write to the repo rather than a read. That repo requires a feature branch and a PR for writes.

There was already a branch guard in that repo. It did nothing here, because it only guards git checkout|switch, not repo-mutating scripts. The rule was "don't write to the main checkout." The guard implemented "don't change branches there." Those are not the same rule, and the gap between them is exactly wide enough for a script to walk through.

So: a new hook. Block the publish engine when it runs from the main checkout on the default branch. Simple enough to write in an afternoon.

The first version blocked the investigation that produced it

The first implementation matched the script name anywhere in the command line. If publish.mjs appeared, exit 2.

That is the obvious way to write it, and it is wrong in a way that is invisible until you use it. Four commands started returning exit 2:

cat content/scripts/publish.mjs
grep -n frontmatter content/scripts/publish.mjs
git diff content/scripts/publish.mjs
node --check content/scripts/publish.mjs

None of those run anything. They read a file. But the filename was in the command line, so the guard fired. The commit says it plainly: it would have blocked the very investigation that produced this hook.

To understand the incident I had to read the script that caused it, and the guard I was writing to prevent the incident had just made that script unreadable. A miss would have cost me two dirty tracked files, a metrics CSV and an index, both recoverable with git restore. The block cost me the ability to open the one file the investigation was about.

Positional, not textual

The fix was to stop asking "does this string appear" and start asking "where does it appear." A command line gets split into top-level segments on ;, &, and |, quote-aware so a separator inside a quoted string does not split it, and each segment is tokenized into argv. Then two shapes count, and only two:

if (GUARDED_SCRIPTS.includes(firstBase)) return true;

The script is the command being run. Or it is the entry-point argument to node, with parse-only flags excluded, because node --check x.mjs parses and exits. Leading NAME=value assignments get skipped so FOO=bar node publish.mjs still resolves to the right first token.

Everything else passes. cat, grep, git diff, wc, bat, echo mentioning the path in prose: the filename is an argument to a reader, never a thing being executed, so it never counts.

The rewrite is about 87 lines of tokenizer. The insight is that a guard operating on text has no idea what the text means, and command lines have a grammar where position carries the entire meaning.

The asymmetry that picks the default

Asymmetric cost is not a new idea. Anyone who has priced a risk has met it. What is easy to miss is that it applies to the guard you are writing right now, and that the answer it gives can be the opposite of the one you gave last month.

Deny-by-default is not a universal virtue. It is the correct answer to a specific question: which direction of error can I not recover from? For an agent holding a database connection, the unrecoverable direction is the miss. A DELETE that should have been denied does not un-run. Blocking a legitimate call costs a round trip and an explicit grant, and you get to try again. Miss is fatal, block is cheap, so deny by default.

Invert the costs and the answer inverts with them. For a workflow guard, a miss leaves a modified file in a checkout that git will happily show me and revert. A false block removes my ability to inspect the system, and it does it while printing a confident message describing a violation that never happened. Nothing in that message tells you the guard is the thing that is wrong. Miss is cheap, block is expensive, so pass by default. Every ambiguous detection step in that hook exits 0: git not available, not a repo, detached HEAD, all pass.

Same principle, opposite configuration. What travels between them is not the default. It is the question.

The failure mode I want to name is picking the default by vibe. Security-adjacent work has a gravitational pull toward strictness, and strict feels responsible, so guards get written closed without anyone pricing the block. The first version of my hook was strict. It was also useless, and it took blocking git diff on the script it guards to show me.

Document the holes instead of papering over them

The finished guard does not catch everything. An execution wrapped in bash -c, hidden in a for loop, or launched through npx tsx walks straight through. Those are written into the file header as confirmed pass-throughs, with a do NOT try to close these instruction attached, because closing them would mean parsing nested shell grammar, which buys the false blocks back.

That is a real trade and it is stated where the next reader hits it. The header fixes the scope in the same place:

This guard is a guardrail against the ACCIDENTAL case (an agent typing the obvious command in the main checkout), not a security boundary against a determined caller who deliberately obfuscates the invocation.

A guardrail that admits its own holes is honest. A guardrail that implies it has none is worse than no guardrail, because people stop checking.

The four commands that broke are now the test suite's job, pinned as named allow cases: one asserts that node --check only parses and does not run, another that git diff on the guarded script passes untouched. 17 cases in all, and the commit records 25 more exit-code shapes run by hand, covering every block shape, dry runs, linked worktrees, outside-repo, and the bypass.

The question worth stealing

Before you set a default, price both errors in the units that actually matter, and pay attention to whether the expensive one is the block. My miss was cheap because the blast radius was two files in a checkout I control. If yours writes to a production table, the arithmetic runs the other way and you should close the guard. The point is to do the arithmetic rather than inherit the answer.

If you cannot say what a false block costs, you have not designed the guard. You have expressed an attitude about risk and let it compile.

In my case, the tell was a guard whose own maintenance path ran straight through the thing it blocked. You're always the first person to hit whatever your own guard breaks, and nobody writes a test for that first case until it has already happened to them.

Do your agent system prompts do anything? I measured 19 of mine

Willian Pinho — Thu, 06 Aug 2026 14:00:04 +0000

The body of an agent system prompt is text. It goes to the same model that would have answered without it, and by itself it changes no weights and adds no tools. The skeptical reading follows on its own: a specialist prompt is a checklist, the model reads the checklist, and whatever the checklist buys is too small to justify maintaining hundreds of them.

That position has a strong advocate. Boris Cherny, who built Claude Code, says in a talk given after the Opus 5 release that they deleted 80% of its system prompt for that release. He then describes deleting the rest as an experiment, and what they find: "the model is actually a little bit more intelligent without these prompts."

I had written the same argument myself, in a six-month audit of my own agent harness, as the case for deleting the delegation mandate: delegating to a specialist buys context and not competence, because it is the same model reading a different checklist. Then I ran the experiment, and the same document records what happened to my argument. Refuted by local measurement.

Fifteen of nineteen configurations scored lower without their prompt. Thirteen of those survive the noise floor I established afterwards. The gap between those two numbers is worth reading. So is the gap between my result and his, which comes down to scope.

Here is the number, the procedure that produced it, and where it is weaker than the headline.

The procedure

The audit these numbers live in runs someone else's protocol. It is the Six-Month Audit by Charlie Hills, a published checklist for finding the instructions in a Claude setup that newer models have outgrown: seven delete checks, five add checks, run against live guidance and never a hardcoded copy. What it asked me to do was judge which instructions had gone stale. The number is the part I added.

That measurement also has a real name I did not use in my own code. Cherny gives it in that talk: "we call in research, you call this ablation... you delete the entire system prompt and then you bring it back line by line to figure out what is the impact of each individual line." He compresses it a sentence later: an ablation is an eval where you delete things to figure out the impact. My harness calls its version a mutation gate, which is the name in my code and not a better one. What it runs is an ablation with a single fixed lesion.

Nineteen of my agent configurations carry golden tasks: a small set of inputs with graded outputs, run through promptfoo, which reports how many cases pass. That corpus already existed, for catching regressions when a prompt gets edited. The ablation reuses it to ask a different question.

Swap the agent's system prompt for You are a helpful assistant. Re-run the same golden tasks, same subject model, same grader. Compare the pass count against the recorded baseline.

Anthropic runs its own version of that arm, and Cherny names the switch: an undocumented CLAUDE_CODE_SIMPLE environment variable that, he says, means "it'll delete all the system prompts including from the tools", used "as a sort of ablation to figure out is the prompt useful". Take the behavior as his description: in the version I have installed, 2.1.222, the binary carries both that name and a separate CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT, and I did not trace which one reaches the prompt. His cut is bigger than mine, every prompt gone including the tools', where I swap one agent's prompt for a trivial one. Same instrument at a different depth. Nothing below is exotic; it is the vendor's own diagnostic pointed at my agents.

A configuration whose score drops is discriminating: the prompt was producing the result, not the base model. A configuration whose score holds is telling you that its tasks measure the base model. That is a statement about the tests before it is a statement about the prompt, and conflating the two is the main way this experiment gets misread.

It needs no new corpus: if you have regression tests for your agents, you already own the instrument. And the null arm is fixed and boring, so a drop cannot be read as one prompt beating another.

One property it does not have, which I claimed in the first draft and had to take back: this is not a controlled within-run comparison. The with-prompt column is a committed baseline recorded on 2026-07-27; the null run happened on 2026-08-04. Eight days separate the arms, and only 2 of the 19 baseline entries were recorded as a median of three runs. So the table is an observation measured against a reference. Calling it a controlled experiment would be a stretch.

The result

Agent	With prompt	Null prompt
`devops-engineer`	3/3	0/3	discriminating
`system-design-interviewer`	5/5	0/5	discriminating
`backend-architect`	3/3	1/3	discriminating
`devils-advocate`	4/4	1/4	discriminating
`frontend-developer`	3/3	1/3	discriminating
`humanizer-skill`	3/3	1/3	discriminating
`observability-engineer`	3/3	1/3	discriminating
`prompt-engineer`	3/3	1/3	discriminating
`typescript-agent`	3/3	1/3	discriminating
`code-reviewer`	3/3	2/3	discriminating
`deep-researcher`	3/3	2/3	discriminating
`nextjs-developer`	3/3	2/3	discriminating
`python-agent`	3/3	2/3	discriminating
`pragmatic-engineering-advisor`	4/4	3/4	discriminating
`security-auditor`	4/4	3/4	discriminating
`docs-architect`	2/2	2/2	non-discriminating
`git-workflow-manager`	2/2	2/2	non-discriminating
`sales-automator`	2/2	2/2	non-discriminating
`technical-writer`	2/2	2/2	non-discriminating

The two zeros collapse for different reasons

The quotable rows are the two that go to zero, and the honest reading of them splits in half.

system-design-interviewer drops 5/5 to 0/5, and that zero was decided before the model answered. All five of its cases assert a literal turn marker the agent is required to emit, and a generic assistant has no way to know the string exists. grep -l for deterministic assertions across the 19 configs returns exactly two files, and this is one of them. That row measures the instrument. It says nothing about behavior. The same mechanism explains the row below it: devils-advocate scores exactly 1/4, and the case it passes is the only one of its four with no signature assertion.

devops-engineer is the clean result. Its three cases are graded by rubric on content, with no format contract anywhere, and the null arm failed all three on substance. Two of them turn on knowledge of one machine: that persistent data here belongs on /mnt/storage and never the root disk, and that analytics and uptime are already covered by services running on this VPS. Neither fact is in the input, so there is nothing to reason from and a textbook-correct answer fails. The third is a Cloudflare TLS trap, and there the input does state the private part, so a careful generic assistant could pass it. One of the three also carries a second, entirely generic rubric about healthchecks, and the run does not record which assertion sank the case.

So that row is weaker than a bare 3/3 to 0/3 suggests, and it is still the only row in the table where the grading is about behavior at all. That narrows what generalizes. Some of what these prompts buy is judgment, which your agents could carry too. Some of it is knowledge of one machine, which is decisive here and worth nothing on yours.

Anthropic runs this experiment and reports the opposite direction

The quote at the top is not a stray line. Cherny is describing something they repeat on a schedule and treat as an eval: every model release, delete the prompt, bring it back line by line, keep only what the model demonstrably still needs. He also says what a lot of that text had been doing, which was correcting for behaviors the model should have known but did not, until it did.

So the person who built Claude Code deletes prompt text and reports a slightly smarter model, while I delete prompt text and watch 13 of 19 configurations get worse. The asymmetry matters before any reconciling: his side is a direction with no number attached, offered as "something that we've been finding", and mine is a table. The reconciliation is scope, and he supplies it himself, two sentences after the line everyone will quote: "when you use [Claude Code] as a product, you do actually want some of these prompts because it helps you use the product..." (The talk's auto-captions mangle "Claude Code" several ways, "cloud code" and "quad code" among them. The brackets are mine; nothing else in the quotes is.)

Claude Code's system prompt is generic by construction. It ships to every user and every repository there is, and much of what it accumulates is correction for one model's specific failures. That text has a half-life. Deleting it on the cadence of model releases is just maintenance.

The prompts I measured are not generic, and the two devops-engineer cases above are why. /mnt/storage and the services already running on this VPS are facts about my infrastructure. No model release supplies them. A better model produces a better textbook answer and still fails. Correction text is the opposite case: a new model is precisely what makes it redundant.

I demonstrated that mechanism on exactly one row, and I should not stretch it further than that. I separated corrective text from knowledge-bearing text for devops-engineer by reading its rubrics. For the other twelve I have a drop and no such reading, so calling all thirteen knowledge-bearing would be the theory eating the evidence. The ablation does not draw that line, and neither does anything else in the table.

Prompt text that corrects the model expires, roughly on the cadence of releases. Prompt text carrying knowledge the model has no way to have does not expire at all. An ablation cannot tell you which kind you are holding. It tells you the text is load-bearing today. Reading why is still your job.

The noise floor, and what it does to my own headline

Every number above comes out of a non-deterministic system with an LLM grader. Run the same configuration twice and it does not always agree with itself.

I have a number for that. On the same day as this run, a routine baseline reported a regression on a configuration nobody had touched: 4/4 down to 3/4. The same configuration had scored 4/4 twenty minutes earlier, with git status confirming no prompt, skill or eval config changed in between. The instrument has run-to-run variance of at least one test case, which is the same magnitude as the effect it is supposed to detect.

That cuts into my own headline, so I would rather do it here than have a reader do it for me. Two of the fifteen, pragmatic-engineering-advisor and security-auditor, drop by exactly one case. That is the noise magnitude, measured on the same instrument, the same week. Thirteen of nineteen survive the floor. The other two are unresolved, not confirmed.

So the protocol has an order, and the measurement comes second:

Run the unchanged baseline three times. Record the spread.
Treat any delta smaller than that spread as nothing. Not a weak signal, nothing.
Where your harness reports assertion counts as well as case counts, compare assertions. On one of my configurations that is 10 numbers instead of 4, and a finer counter is harder for noise to dominate.

A 3/3 to 0/3 collapse survives this easily. A one-case drop does not survive it at all.

The four that held are a finding about my tests

Four configurations scored identically with and without their prompt. That is not four useless prompts. All four have exactly two test cases, the smallest corpus in the set, and both already pass at baseline.

A two-case corpus has a resolution of fifty points. The smallest drop it can record is half the corpus, and two cases at the ceiling cannot separate a prompt that contributes nothing from a prompt whose contribution those two cases never probe. The likelier explanation is weak test power rather than a useless prompt, and the correct response is to write harder cases before drawing any conclusion about those four.

There is a rival explanation for those rows, and it comes from the same talk. Cherny puts the working life of an eval at one to three model generations and names what usually ends it: "very often we just saturate the eval and then we have to throw it away". Four suites sitting at 2/2 with and without the prompt are what a saturated eval looks like from the inside.

It is not what happened here, and git log is why. Those four configs were added on 2026-07-03, a month before the run, in the same commit series as the rest. devops-engineer, which collapsed 3/3 to 0/3, is five days older still. Same age, opposite verdicts, so age is not the variable; corpus size is. The response is the same either way: throw the eval away and write a harder one.

This is why the gate only warns in my harness. A non-discriminating result is a work order for the test corpus, and wired as a blocking check it would delete prompts on the strength of thin tests.

The mistake that invalidated my first attempt

An earlier version of this experiment reported a 43-point gain for agent prompts. That number was wrong, and the way it was wrong is worth more than the number.

The harness had quietly stopped delivering agent prompts to the model. A credential name was changed in the config files and never changed in the harness that injects it, so the subject provider was never built. The resulting error went to a log file nobody reads while the run scored anyway. Measured prompt token counts came back at 60 to 112, against roughly 10,000 for a real agent body. The model was answering the golden-task question with no system prompt at all.

Both arms were promptless. The experiment compared nothing against nothing and reported a large win for prompts, which is the shape a broken measurement takes when it happens to agree with what you expected. A second defect found in the same investigation, a module renamed across an extension boundary, turned out to be latent rather than the cause. Naming the wrong one would have left the real one live.

So there is a step that belongs before all three above: prove your baseline arm actually loads the prompt. Prompt tokens per request is the cheapest proof there is. If it is not carrying thousands of tokens of agent body, you are measuring your plumbing.

What the number does not cover

It tests prompts, not delegation. The comparison is prompt against null prompt inside the same call. It says nothing about whether spawning a subagent beats running the same prompt inline in the main thread. Those are separate questions and only one of them is answered here. Anyone using this result to defend an orchestration architecture is using it for something it does not support.

The sample is not the roster. Nineteen configurations, and they are the ones I invoke most, which is the opposite of a random sample. The finding is that the prompts of my most-used agents do measurable work. Everything outside that set is unmeasured, and until it is measured, archiving any of it is a decision about frequency rather than merit.

You cannot inspect my harness, so I extracted the instrument instead

The repository holding these agents is private and stays private. It carries credentials, client context and personal data, so opening it is not on the table.

Which leaves me holding a number nobody can check, and a number nobody can check is a claim. So I pulled the mutation mode out of the harness and published it on its own: prompt-mutation-gate, MIT. It knows nothing about my directory layout: point it at a promptfoo config and a prompt file and it runs on your agents:

$ npm install && npm run example

strict-reviewer  DISCRIMINATING
  cases 6   baseline 6/6 [6 6 5]   null-prompt 0/6 [0 1 0]   drop 6   noise floor 1
    Removing the prompt cost 6 of 6 cases, against a run-to-run spread of 1.

tiny-suite       INCONCLUSIVE  (low-test-power; not conclusive)
  cases 2   baseline 2/2 [2 2 2]   null-prompt 2/2 [2 2 2]   drop 0   noise floor 0
    2 test cases is below the 5-case minimum. A suite this small, sitting at its
    ceiling, has no room to show a drop.

That runs with no API key and no account: the example parses recorded promptfoo result files and classifies them the same way a live run would, so the verdicts are derived rather than printed.

Two things in it matter more than the verdict, and both are answers to something in this post. It prints the noise floor beside every row, from the spread of the baseline runs, because I read fifteen off my own table before I had a floor to read it against. And below a five-case minimum it returns INCONCLUSIVE instead of non-discriminating, because four of my rows held at exactly that size, and non-discriminating lets a statement about the corpus wear the costume of a statement about the prompt.

The procedure it automates is short enough to run by hand if you would rather not take a dependency:

Pick every agent that already has graded tests. Three is enough to start.
Verify the baseline arm delivers the prompt. Check tokens per request.
Run the baseline three times unchanged and record the spread. Run the null arm three times too — it varies as well, and a single null run can be the outlier just as easily as a single baseline run.
Replace each system prompt with You are a helpful assistant. and re-run.
Before reading any row, check what its assertions grade. A case asserting a literal output format is a floor check on your harness, not evidence about behavior.
Classify: dropped beyond the spread is discriminating; held is a note about your tests.
For anything that held, write harder cases before concluding anything about that prompt.

That is roughly an afternoon on an existing test corpus, and it produces your own version of the table above.

If your numbers come out flat where mine dropped, there are three readings and only one of them is about me. Your corpus may be too thin to record the drop, mine may be measuring something local I have mistaken for general, or your prompts may genuinely be carrying less than you thought — which is a real result and the one worth knowing. That disagreement is more useful to me than agreement, and it is the only scrutiny a private repository is still open to.

A success code that hides a failure

Willian Pinho — Sat, 01 Aug 2026 13:00:05 +0000

Spent an hour today chasing why I'd missed an alert, and the answer bugged me enough that I'm writing it down.

I run a small alerting setup. When something on my infra trips, a bot posts a message into a chat channel so I actually see it. The send call returns HTTP 200. My health metric watches for non-200s and it had been flat green for weeks, so I figured the alerting was fine.

It wasn't. Nothing had been delivered for days.

The token the bot uses had been rotated and I'd forgotten to update it in one place. So the send call was hitting the API with a dead token. And here's the part that got me: the API returned 200 OK anyway. The actual failure was sitting in the JSON body: an ok: false with an invalid_auth error code. The status line said success. The body said "no."

My metric only ever looked at the status code. So every dashboard I had said "delivered." The reality was that the one system whose entire job is to tell me when things are broken had quietly stopped working, and the thing watching it was looking at the wrong field.

Why this isn't just a chatbot quirk

The reason I bothered writing this is that the same trap is everywhere once you start wiring agents to real tools, and it's worse there than it was for me.

A model gateway can return a 200 with a body that's a refusal, a truncated response, or a fallback model's output that's nothing like what you asked for. The HTTP layer is happy. The content is degraded and nobody downstream knows.

It gets worse inside an agent loop. A tool call can come back "succeeded" having done nothing at all, because what succeeded was the call completing, not the thing you wanted to happen. The agent reads "done," moves to the next step, and you end up with a clean run that accomplished none of what it claimed.

One nuance worth keeping straight: the chat API tunneling an auth error through a 200 is arguably a spec deviation. A gateway returning 200 on a refusal isn't. That's correct HTTP. The inference really did succeed; the content just isn't what you wanted. Different causes, same trap waiting for you downstream.

A 200 tells you the request was received and came back. It says nothing about whether the work you wanted actually happened. For most plumbing that gap never hurts you. But anywhere a silent no-op compounds, like an alerting path, or an agent stacking step on step, or a payment, it'll cost you, and you won't notice until later.

So the fix isn't clever. It's just: stop trusting the status code as proof of success. Reading the body is the start, not the finish, because for a multi-step agent the body can lie too. Assert on the effect you actually wanted, not on the fact that the request came back. If the work was "deliver a message," check that a message was delivered. If it was "the agent applied the change," go look at the external state: the DB row exists, the file on disk actually changed, not just whatever the response claimed.

A green light that isn't wired to the actual outcome is worse than no light. At least with no light you'd go look.

Multi-LLM routing in production: the failure modes nobody warns you about

Willian Pinho — Wed, 29 Jul 2026 16:14:03 +0000

The pitch for multi-LLM routing is clean. Route cheap requests to a cheap model, hard requests to a strong one, keep a fallback for when a provider has a bad day, and watch your cost-per-task drop while quality holds. I've built this. It works. But the version that survives contact with production looks very different from the one in the architecture diagram, and the gap is made of failure modes that don't announce themselves.

One of the systems where I learned this the hard way was a KYC pipeline with real compliance stakes. On that build, picking the model was the easy part. The real engineering was everything around the routing decision: knowing when a route was quietly degrading, what a fallback actually cost in latency and correctness, and how to tell a cheap-model success apart from a cheap-model failure that happened to return valid-looking text. Those are the parts nobody warns you about.

The cost optimization that costs more

The first thing teams do is route by cost. Send the simple stuff to the small model, escalate only when needed. The math on the spreadsheet is compelling.

The math in production is different, because routing isn't free and re-routing is expensive. When the small model gets a request wrong in a way you catch, you re-run it on the large model. Now that request cost you both calls plus the latency of two round trips plus whatever your detection step costs. Get your routing threshold slightly too aggressive and you discover you're paying for the cheap model and the expensive one on a meaningful share of traffic, while shipping worse latency than if you'd just used the strong model directly.

The trap is that the per-token price is visible and the re-route tax is not. You see the cheaper bill on the requests that worked. You don't automatically see the compound cost on the ones that didn't, unless you instrumented for it. Most teams don't, so they optimize toward a number that's hiding its own downside.

The fix isn't to abandon cost routing. It's to measure cost-per-successful-task, end to end, including retries and escalations, and route against that. The cheap model only wins if it wins after you account for how often it loses.

Latency is a distribution, not a number

The second failure mode is treating provider latency as a property you can look up once. It isn't. It's a distribution that moves with the provider's load, the time of day, the length of your context, and whether you're on a tier that gets deprioritized when capacity is tight.

A router tuned against median latency will route happily right up until the provider's tail blows out, and then your p99 user is waiting on the slow path you chose because the median looked fine an hour ago. If you've layered a fallback on top, a timeout-triggered fallback adds latency rather than removing it. The user waits for the first model to time out, then waits all over again for the second.

What you actually want is latency budgets per route, measured continuously, with the router making decisions against recent tail behavior rather than a static config. And you want your fallback policy to be honest about the fact that "fall back on timeout" means "sometimes pay both latencies." For a real-time path, that's often the wrong trade, and you're better off routing to the more reliable provider up front and accepting a higher floor for a tighter ceiling.

Dean and Barroso's The Tail at Scale has the sharper version of this move. Instead of waiting for a hard timeout to fire the fallback, you fire it early, around the point where a request has already run longer than most do, say the p90, and take whichever response comes back first. You spend some extra requests to buy a much tighter tail. It only pays off where the second call is genuinely cheap relative to the latency you're saving, so it's a tool for the hot path, not a default to switch on everywhere, but it beats sitting on a slow request until a timeout admits what the latency distribution already told you.

Eval-driven routing, or routing on vibes

Here's the one that separates a system from a demo. How do you decide which model handles which request? Most early routers do it on heuristics: keyword matching, token count, a hand-written rule that "code questions go here." A rule like that encodes a guess you made on day one and never revisited. It drifts the moment your traffic shifts, and you have no way to know it drifted.

Eval-driven routing means you have a measurable definition of success for your task, you score every provider against it on representative traffic, and your routing decisions are downstream of those scores. When you onboard a new model, you don't guess whether it's better — you run the eval and find out. When a provider silently changes a model behind a stable name, your eval catches the regression before your users do.

This is non-negotiable for anything that matters. In the KYC pipeline, "the model returned a confident answer" and "the model returned a correct answer" are entirely different events, and the gap between them is where the compliance risk lives. You cannot route responsibly without a way to distinguish those two, and that way is evaluation, run continuously, not a one-time benchmark you ran during the pilot.

The failures that return HTTP 200

Your alerting watches for exceptions and timeouts. The routing failures that cost you the most return HTTP 200 with a clean, well-formed body and never trip a single one of those alerts.

A provider degrades and starts returning shorter, lazier completions that still parse. Elsewhere, a model update shifts the output format just enough that your downstream extraction silently drops a field. Or a fallback fires, the backup model is genuinely worse at your task, yet every response comes back well-formed, so nothing alarms. These are silent failures, and a router that only watches for errors and timeouts is blind to all of them.

Observability for multi-LLM routing has to go past uptime. You need to track, per route, the quality scores from your evals, the rate of fallback activation, the cost-per-successful-task, and the shape of the outputs over time. A rising fallback rate is a leading indicator that a primary provider is degrading. A quality score sliding while the error rate stays at zero is the signature of a silent regression. If you're not watching those, you'll find out from a customer, which is the most expensive way to learn.

The instinct here comes straight from running payment platforms at 99.9% uptime for ten million users. In that world you learn that the failure that pages you is the lucky one, because at least you know about it. The failures that quietly return success while corrupting your data are the expensive ones, and multi-LLM systems are full of them. Routing is where they hide.

So if you build one of these, instrument for the silent failures first, before you tune a single cost threshold. Cost-per-token is the number that's easy to see. Cost-per-successful-task, fallback rate, and quality drift at a flat error rate are the numbers that tell you whether the system is actually working, and they're the ones you have to go build on purpose. The routing diagram is the part you can draw in an afternoon. The instrumentation is the part that decides whether the thing survives a month in production.

Sources

Dean & Barroso, The Tail at Scale (Google Research) — why latency is a distribution, why tail behavior governs user-facing systems, and the hedged-request technique for cutting that tail.
OpenAI — Latency optimization — the levers and tradeoffs behind per-request latency in LLM systems.
OpenAI — Evals — building measurable, repeatable evaluations as the basis for model-selection decisions.
OpenAI — Production best practices — fallbacks, retries, and operational guidance for running LLM calls in production.
Anthropic — Streaming Messages — how partial/streamed completions arrive over a successful HTTP response, the surface where well-formed-but-degraded output hides.

Fail-close: the tool-access default every AI agent should ship with

Willian Pinho — Wed, 15 Jul 2026 14:45:52 +0000

I spent the better part of sixteen years building payment platforms. The first principle you internalize there, before any framework or pattern, is that the safe state is the closed state. A transaction that can't be verified doesn't get a "probably fine" — it gets declined. A service that loses its dependency doesn't keep serving stale balances — it stops. You learn to design systems where the failure mode is refusal, not improvisation.

Then I started building AI agents wired to real tools through MCP, and I watched the industry reach for the opposite default. Agents ship with broad tool access, allow-by-default, and a vague intention to "add guardrails later." That instinct is backwards, and it's going to cost teams who don't fix it before they go to production.

What "fail-close" actually means for an agent

Fail-close is a default you choose before the first tool is registered, not a guardrail you bolt on after. It means the default answer to "can this agent call this tool right now?" is no, and every yes is an explicit, scoped, auditable grant.

Most agent setups today do the reverse. You register a set of MCP servers, the model gets the full menu, and the only thing standing between a hallucinated plan and a destructive action is the model's own judgment plus maybe a confirmation prompt the user clicks through on autopilot. That's allow-by-default with a speed bump. It works in demos. It fails the first time the model decides that the cleanest path to "tidy up the staging database" runs through a DELETE it was never supposed to reach.

A fail-close agent inverts that. The agent starts with zero tool authority. Access is granted per tool, ideally per operation, against an identity and a context. When something is ambiguous — an unrecognized server, a tool outside the current task scope, a parameter that looks like it touches production — the system denies and surfaces the denial instead of guessing.

The distinction matters because models are non-deterministic and your blast radius is not. You cannot reason about what the model won't do. You can only constrain what the surrounding system will permit.

What breaks when you don't

The failures aren't exotic. They're the boring, predictable consequences of giving a probabilistic system deterministic power without a fence.

Give an agent read and write access to the same datastore and it will eventually write when you wanted it to read, because the boundary between "summarize these records" and "update these records" lives only in the prompt, and prompts are suggestions. Hand it shell access and a filesystem MCP, and somewhere down the line it runs a command that's correct in isolation and catastrophic in context. The same goes for an internal API reached with a service token: a confused enough plan will call the destructive endpoint with full confidence.

None of these require malice or a clever injection attack. They're just the base rate of a system that improvises under uncertainty being handed tools that don't. Add prompt injection from untrusted content the agent ingests, and the allow-by-default posture stops being a reliability problem and becomes a security one. The injected instruction doesn't need to break out of a sandbox if there was never a sandbox to break out of.

When I owned security review for shipping AI features, the question I asked first was never "is the model good." It was "what is the worst single tool call this thing can make, and who decided it was allowed to." If the answer to the second half was "nobody, it just had access," the feature didn't ship.

Governance and RBAC for MCP tools

One thing to be clear about first: the MCP protocol itself does not define roles, authorization, or RBAC. It describes how a client and server talk, not who is allowed to call what. That control plane is yours to build, and the natural place for it is the host application sitting between the model and the servers, the layer that sees every tools/call before it goes out and can decide to refuse it.

RBAC, scoped service identities, and audited grants already run every bank API and every CI deploy key in production. An agent is just another non-human principal asking for access. We have the machinery; we just have to stop pretending agents are a special category that gets to skip it.

Treat every MCP tool as a privileged operation behind role-based access control. The agent runs under an identity. That identity has a role. The role grants a specific, enumerated set of tool operations, scoped to an environment. A research agent gets read access to a knowledge base and nothing that writes. A deployment agent gets a narrow set of operations against staging and a separate, more guarded grant for production that requires a stronger gate.

Three things make this real rather than theatre.

Grants are per-operation, not per-server. Registering a database MCP server should not hand over its entire surface. Read and write are different privileges and belong to different roles, the same way you'd never give an analyst the production write credential just because they need to query.

Logging comes next: every tool call carries the identity, the role, the arguments, and the decision. You want to be able to answer "what did this agent do and what was it allowed to do" after the fact, with a trail, not a guess. The denials matter as much as the approvals; a spike in denied calls is a signal that the agent's plan has drifted from its scope.

And the dangerous grants pass through a human or a stronger policy gate before they're active. The point of fail-close isn't to block everything forever. It's to make sure that anything with real blast radius is a decision someone made on purpose, with their name on it.

The instinct, ported

I maintain an MCP server that runs across several agent clients, including Claude Code and Cursor. Building and operating it taught me how casually broad the default tool surface tends to be, and how little friction there is between a registered server and full access to whatever it exposes. The convenience is real. So is the exposure.

The reliability instinct from payments transfers cleanly: the safe state is the closed state, and you earn your way out of it one explicit grant at a time. We learned to stop treating "the agent can't do that yet" as a limitation. In production, it is the whole point.

The shift is small to describe and easy to defer. Choose the default before you register the first tool, not after the first incident. Everything else in this post is just the consequence of getting that one decision right while it is still cheap.

Sources

Model Context Protocol — Security Best Practices — the protocol's official guidance on consent, confused-deputy risks, and host-side controls.
Model Context Protocol — Tools — defines tools as model-controlled and calls for trust boundaries and human-in-the-loop confirmation.
OWASP — LLM01: Prompt Injection — why untrusted content can redirect an agent's actions.
OWASP — LLM06: Excessive Agency — the blast-radius risk of granting agents excessive functionality, permissions, or autonomy; recommends least-privilege scoping.
NIST — Least Privilege (glossary) — the foundational principle behind per-operation, role-scoped tool grants.

MCP in production: what I learned maintaining a server that works in Claude Code, Cursor, and Gemini

Willian Pinho — Fri, 03 Jul 2026 16:39:00 +0000

I maintain an MCP server that ended up running inside Claude Code, Cursor, and Gemini CLI. I did not plan for three clients. I built it for one, and then watched the other two start exercising paths I never tested, in ways the spec technically allowed but I never imagined.

That gap, between what the protocol permits and what a client actually does, is where most of the real lessons live. MCP looks simple on the surface: you expose a few tools, the model picks one and calls it, you return a result. That surface is exactly what fools you. The lessons below are the ones that only show up once a caller you didn't build for starts driving the server. Here is what survived contact with real usage, and what I deleted.

The model is a hostile-by-accident caller

Every backend engineer learns to treat input as untrusted. With MCP, the caller is a language model, and it is untrusted in a stranger way. It is not malicious. It is confidently wrong at a rate you do not see in human users.

A human who gets a confusing tool response stops and asks. A model retries. It will call your tool with a slightly mangled argument, get an error, reword the argument, and call again, sometimes a dozen times in a loop, because retrying is the cheapest action it has. Early on, one of my tools accepted a file offset and a length. A model would request a length that ran past the end of the file, get an error, and then ask for the same thing with the offset nudged by one. My logs filled with near-identical failures that no human would ever produce.

The fix was not better validation messages, though I added those. It helps to know that MCP gives you two different ways to signal trouble: a protocol-level JSON-RPC error, which tends to read as "the call itself was malformed," and a normal tool result flagged with isError, whose text the model actually reads and can act on. A hard protocol error invites the blind retry; a flagged result with a usable message invites a correction. The deeper fix was making the tool harder to misuse in the first place: clamp the range to the file instead of erroring, return what exists, and say so in the response. The model reads that, adjusts, and moves on. I stopped writing errors for a careful human reader and started writing responses a confused model could recover from without a loop. Once I saw it that way, half my "validation" work turned out to be the wrong shape entirely.

Fail-close is the only default that survives three clients

The deleted code is the part worth talking about. My first version had a permissive fallback: when a tool received an ambiguous or partially valid request, it tried to do something reasonable. That worked fine in Claude Code, where I had tested it. It broke in subtle ways the moment Cursor and Gemini drove it, because each client frames requests differently, truncates context differently, and decides differently when to call a tool at all.

A "reasonable default" is a guess about caller intent. Three clients meant three different intents behind the same malformed request, and my guess was right for one of them at best. So I tore the fallback out and made the server fail closed. If a request is ambiguous, refuse it and explain what a valid request looks like. The model gets a clear next step instead of a silent wrong answer.

This sounds obvious written down. It did not feel obvious while I was deleting code that worked in my one tested client to fix behavior in two I had not. Fail-close costs you a bit of apparent helpfulness and buys you the thing that matters more: a server that does the same predictable thing regardless of which agent is holding the other end. I have spent sixteen years on systems where the wrong silent default is how you lose money, including payment platforms serving over ten million users at three-nines uptime. Fail-close was not a new lesson. MCP just made me relearn it in an unfamiliar shape.

You cannot debug what you cannot replay

For the first weeks I had almost no idea what was actually happening inside a session. The model called my tools, things mostly worked, and when they did not, I had a vague bug report and no way to reproduce it. The non-determinism of the caller meant "run it again" produced a different sequence of calls.

What changed everything was logging each tool invocation as a structured record: the arguments as received, the response as returned, the timing, and which client sent it. Not application logs, an event trail I could replay and reason about. Once I had that, the loops surfaced, the client-specific framing differences were suddenly obvious, and the bugs stopped being ghost stories. Observability is not a nice-to-have you add when an MCP server gets popular. It is the thing that lets you understand a caller you do not control. I would build it on day one next time, before the first real user.

The second-order benefit surprised me. The event trail also told me which tools were never called. A model will quietly ignore a tool whose description does not earn its place, and you will never know unless you are counting. I removed two tools that looked useful to me and were invisible to every model that touched the server.

Tool descriptions are an interface, not documentation

The part I underestimated most: the model chooses tools based on their names and descriptions, and nothing else. There is no onboarding, no docs site, no support channel. The description is the entire contract, and it is read by a system that interprets language statistically rather than literally.

Tightening descriptions moved usage more than any code change I made. A tool that was being called in the wrong situations got called correctly once I rewrote its description to state plainly what it was for and what it was not for. This is closer to API design than to writing docs, and it rewards the same discipline: name the thing precisely, scope it narrowly, say what it does not do. I had shipped a KYC AI pipeline in three weeks once by being ruthless about scope, and the same instinct applied here. A narrow tool with a sharp description beats a flexible tool the model cannot reliably aim.

What this points at

None of this is exotic. It is ordinary production engineering, applied to a caller that behaves unlike any client you have built for before: non-deterministic, retry-happy, and reading your interface through a probabilistic lens. The teams I see struggling with agents in production are not failing at the model layer. They are failing at the boundary, where governance, fail-close defaults, and observability decide whether an agent is a reliable colleague or a liability you cannot debug.

So if you take one thing from this, let it be where to point your attention. The interesting engineering with MCP is not in the model and not in the tools. It is in the thin layer between them, the part that is easy to skip in a demo and impossible to skip in production.

Sources

Understanding MCP servers — server concepts (modelcontextprotocol.io) — tools are model-controlled: the model discovers and invokes them based on their names and descriptions.
MCP specification, server tools (2025-06-18) — tool definition (name, description, input schema) and the two error-reporting mechanisms: protocol-level JSON-RPC errors versus a result with isError set, whose text the model can read and recover from.
MCP architecture overview (modelcontextprotocol.io) — host, client, and server roles, and why the boundary between them is where production behavior is decided.
MCP — Security Best Practices (2025-06-18) — consent, confused-deputy risk, and the host-side controls that make fail-close defaults enforceable.
OWASP — LLM06: Excessive Agency — the blast-radius risk of handing an agent broad tool functionality and permissions without scoping.
OWASP — LLM01: Prompt Injection — why a model driving your tools is an untrusted caller, not just an inconvenient one.

I ran an MCP-gateway production-readiness audit on a popular open-source LLM gateway. Here's what it found.

Willian Pinho — Tue, 30 Jun 2026 20:41:45 +0000

Most teams wiring an LLM gateway to MCP tools ask one question: does it work? The harder question, the one that decides whether you sleep through the launch, is different. When the authorization check throws an unexpected exception, does the gateway deny the call or allow it? That single line of behavior separates a mature platform from an incident waiting for a quiet Tuesday.

So I built a structured way to answer it, and pointed it at a target that would not flatter the method.

The method, not the target

The audit is a read-only, evidence-backed review across seven dimensions: tool-access governance and RBAC, fail-close versus fail-open behavior, MCP and agent onboarding, observability and tracing, multi-LLM routing and cost controls, secrets and identity, and broader production-readiness. Every finding has to point at the specific code that justifies it, pinned to a commit, so the team can open the file and read the same lines I did. There's no live fault injection and no guesswork; every claim traces back to code at that revision. If a control is present in code but only takes effect when an operator turns it on, that gets recorded too, because the default is what ships to most deployments.

For a worked example, I used LiteLLM from BerriAI, pinned at a specific commit. It is a widely deployed open-source LLM proxy, the code is public, and it is the kind of mature project that would expose a sloppy methodology rather than a sloppy target. I want to be clear about the result up front: it scored well. Four green, three yellow, zero red across the seven dimensions. The verdict was "production-ready with caveats." This is a fair assessment of a capable platform, not a takedown. The interesting part is that even a strong gateway has edges a structured pass will surface, and those edges are exactly the things every team running MCP in production should check on their own deployment.

What it gets right

The dimensions that carry the most safety weight were the strongest.

Identity and secrets came back green with no significant gaps. There were no inline secret values in the configuration. Real config references the environment via os.environ and os.getenv, and the only sk- style strings in the tree were docstring examples. Identity is JWT and OIDC enforced on the actual gateway call path, not merely the dashboard login, and the end-user identity propagates through to MCP handling and spend logs instead of collapsing into one shared service credential. For MCP tokens, the gateway supports RFC 8693 OAuth token-exchange with audience and scope binding, so the MCP server receives a token minted for it rather than a forwarded user token. That follows the resource-server pattern the current MCP specification points at. One honest detail: it is an operator-enabled mode, not the default, so it counts as a control you have to turn on rather than one you inherit.

Observability was also green. OpenTelemetry is a first-class integration with dedicated GenAI semantic-convention mapping, so per-model token and cost attribution is possible rather than bolted on later. Inbound W3C traceparent headers are extracted through the standard propagator, which means end-to-end trace continuity is achievable across hops.

Routing and cost, the dimension LiteLLM is purpose-built for, held up. A declarative model_list maps virtual model names to physical deployments, and budget caps are genuinely enforced. An overrun raises a BudgetExceededError rather than firing an alert and letting the spend continue. Rate limits in requests and tokens per minute are expressible per key, per model, and even per MCP server. That is a real, enforced path against bill-shock and denial-of-wallet, not a dashboard that turns red after the money is already gone.

One honest caveat on observability: because this was a static review against a public repo with no live backend, I could not pull a real end-to-end trace and watch it reconstruct. The building blocks are present and standards-aligned. Confirming one real request stitched together end to end is a step any team should run against their own staging.

Three yellows worth checking on any deployment

None of these are missing controls. They are default-configuration and operational gaps, which is the typical profile of a capable platform that needs hardening rather than rearchitecture.

First, one fail-open line. Every per-level permission resolver fails closed: on an unexpected exception it logs and returns an empty set, which resolves to "no access" downstream. That is the correct posture. The exception is the top-level wrapper get_allowed_mcp_servers(), which returns the allow-all server set on an unexpected error instead of an empty list. The blast radius is bounded to servers an operator already marked as public, but fail-open in an authorization resolver is the single highest-risk class in the whole framework, because a degraded check silently becomes "allow." It is also a one-line fix plus a regression test, which gives it the best risk-reduction-per-effort in the audit.

Second, unpinned third-party MCP servers. The curated catalog launches stdio servers with floating commands like npx -y @sentry/mcp-server, with no version, digest, or checksum pin. That is a tampered-package away from a supply-chain incident, the class OWASP labels LLM03 in its 2025 list. Pinning by version and digest, and rejecting anything unpinned, closes it.

Third, per-tool least-privilege is opt-in. Authorization at the gateway is strong: it is enforced on caller identity as a strict intersection of key, team, end-user, agent, and org permissions, and the model is kept out of the authorization decision entirely, which is what defeats prompt-injection-to-tool-call. But absent a per-server allowed_tools allowlist, any caller with access to a server can invoke any tool on it, including write or external ones — the excessive-agency exposure OWASP tracks as LLM06. Making that allowlist required at onboarding, validated in CI, converts least-privilege from opt-in to default.

The lesson

A structured audit is not a search for a smoking gun. On a mature target there usually is not one, and there was not here. What the seven-dimension pass surfaced instead were the production-readiness edges that hide in a good codebase. One resolver that errs toward exposure where its siblings err toward safety. A supply chain that floats instead of pinning, and a least-privilege control that waits for someone to opt in. None of these show up when you ask "does it work," because the gateway works fine. They show up when you ask what happens at the boundaries, under error, and at the defaults most operators never change.

That is the value of the method. It turns "we think it's fine" into a scored picture with named fixes, effort estimates, and the exact code behind each finding, so the team is arguing about a specific line of behavior instead of a feeling.

The audit kit that produced this runs read-only and works in Claude Code, Cursor, or any MCP client. If you are putting an MCP gateway in front of production tools and want the same scored, evidence-backed pass on your own deployment, that is the scoped engagement I run.

Sources

LiteLLM (BerriAI), pinned commit 51ba6e39cd23576b9c2110361f1045782762f3e4 — the exact source audited; every finding points at code at this revision, so it can be read back at the same lines.
RFC 8693 — OAuth 2.0 Token Exchange — the token-exchange model behind audience- and scope-bound MCP tokens.
Model Context Protocol specification (2025-06-18) — the resource-server model the gateway's MCP token design follows.
OWASP Top 10 for LLM Applications (2025) — LLM03 (Supply Chain) and LLM06 (Excessive Agency) frame the unpinned-server and least-privilege findings.
W3C Trace Context — the traceparent propagation standard that enables end-to-end trace continuity.
OpenTelemetry GenAI semantic conventions — the per-model token and cost attribution mapping the gateway integrates.

Pre-mortem as plan debugger

Willian Pinho — Tue, 23 Jun 2026 17:11:55 +0000

The ritual most senior engineers skip

Everyone in this field has heard the word "pre-mortem." Almost nobody runs one as an actual habit.

The people who do ship cleaner work. The cost is small: about 10 minutes to generate the failure list, 25 to 30 for the whole ritual. What it buys you is avoiding the moment on a post-incident call where someone says "yeah, we knew this could happen and shipped anyway." Plenty of Sev-1s aren't preventable. For the ones that are, this is the cheapest way I know to catch them early.

What follows is the version I actually use: the mental flip that makes it work, the 10-minute structure, the three ways you can respond to each risk, and the cases where running a pre-mortem is just procrastination wearing a lab coat.

One thing to hold onto if you skip the rest: the useful work happens in past tense, before the incident, in concrete sentences.

Why risk registers do not work

A risk register is a passive list. It gets filed somewhere nobody opens, written in language nobody acts on.

It tells you "what could go wrong" without telling you what to do about any of it. And the entries are written in future-conditional, "might happen," "could cause," which your brain quietly files under low-probability, deal-with-it-later. You read the list, you feel covered, you did the responsible thing. You wrote the risks down.

The other problem is structural. The shipping plan and the register usually live in two different documents, so the register never forces anything back into the plan. It sits next to the work instead of changing it.

After a decade-plus of shipping, here's my honest read: most risk registers get filled in the afternoon before launch to satisfy a checkbox. Nobody who writes one believes in it, and nobody reads it afterward. Which is fine, because usually nobody reads it at all.

A pre-mortem has the opposite shape. It's active, it reads like a story, and it lives inside the plan.

The cognitive shift: past tense as forcing function

A pre-mortem is a story you write in the wrong direction. You write the post-incident retrospective before the incident, in past tense, with detail.

The past tense is what forces specificity. There's a study on prospective hindsight (Mitchell, Russo and Pennington, 1989) that found imagining an outcome has already happened produces roughly 25% more concrete reasoning than predicting it forward. That's the lever the pre-mortem pulls. It's also why "we missed the deadline" is useless and "row-level locks piled up under the signup spike and transactions queued past the 30-second timeout" is not.

Put the two side by side. The lazy version is "it broke." The useful version: the 30-second timeout cascaded into a 502 wall by minute three of the signup spike, conversion dropped 40% at peak hour because the payment provider rate-limited us, and the alerting system went quiet for 18 minutes because the alerting stack was its own alert sink.

The second version you can actually test. Every clause points at a specific failure mode you can prevent, shrink, or at least watch for.

This isn't pessimism dressed up. It's negative thinking pointed at the plan, run like a debugging session. You're not predicting the future. You're trying to break the document while it's still cheap to break.

I ran one last week on a SaaS I'm shipping. Ten failure modes in ten minutes. Ranked by impact times likelihood, the top three were the lock-contention story, the payment rate-limit story, and the silent-alerting story above. Each one got a countermeasure before ship. But the bigger payoff was sequencing: the pre-mortem killed two days of work, because writing down how the wrong order failed showed me a feature I'd put first should have been third. You only see the right sequence after you've described the wrong one falling over.

The 10-minute protocol

The "10 minutes" is the generation core, not the whole thing. Steps 1 and 2 eat the ten minutes; steps 3 through 5 add another 15 to 20. So the full ritual is closer to half an hour. Most teams stop after step 2, which is the whole reason pre-mortems get a reputation for not working. The discipline is in finishing.

The original method (Klein's) is a group exercise: five to ten minutes of imagined-failure writing, then you read them round-robin, no fixed count. What I describe below is an opinionated solo adaptation for ship-level calls. The thing I actually run on a Monday morning before a non-trivial deploy.

Solo, you only catch the failures you can already picture. So for anything launch-scale (new market, new product line, architecture you can't easily walk back), run it as a 30-minute group exercise with anonymous submission. That way the most junior person on the team can write down the failure mode the founder is too close to see. The brainstorming research is fairly brutal about face-to-face group ideation; anonymous parallel generation tends to beat it by 20 to 40%.

The five steps:

Set a timer for 10 minutes. The time-box is what stops you polishing.
Write 10 specific past-tense failure narratives. Concrete, not categories. Vendor outages, traffic shapes, a regulatory surprise, a key person out sick, a third-party API quietly changing its contract. Under 6 and you're still being abstract; over 12 and you're padding. Ten is the right squeeze for a 10-minute clock: one a minute, no editing.
Rank each one by impact times likelihood. Three buckets: ship-killing, recoverable, and acceptable.
The top 3 get countermeasures in the plan itself. Not in a separate doc. In the plan, as real tasks with an owner and a date.
The rest go in a deferred-risk file that you actually re-read at the next milestone.

The protocol is deliberately dumb. The hard part is running it before every ship that takes more than a day to roll back.

Countermeasure design: three patterns

For each of the top three, you pick exactly one response. Not two.

Eliminate

Change the plan so the failure mode can't happen. This is the most expensive option and the most durable. For the lock-contention case, what landed was moving primary-key allocation off a sequence and onto UUIDv7 on the signup-burst path, running READ COMMITTED. The failure mode doesn't get smaller. It's gone.

Attenuate

Shrink the blast radius for when it does happen. Cheaper, and partial. If the alerting stack can go dark, put an external uptime probe on a different vendor in a different region. The alerting system can still die; you just hear about it in 60 seconds instead of 30 minutes.

Instrument

Accept the failure mode but make sure you detect it in minutes, not hours. Cheapest of the three, and it leans entirely on someone being around to act on the signal. If you can't design the lock contention out before ship, then instrument lock-wait-time histograms and transaction-queue depth, alert on the 99th percentile, and accept that you'll respond in ten minutes rather than never having the bug.

There's a fourth pattern in the textbooks: transfer, meaning you push the risk onto a vendor SLA, insurance, or a contract clause. I leave it out on purpose. Mid-launch, transfer is usually not yours to pull; when it is, procurement moves in weeks, not inside a 10-minute loop. ISO 31000 and PMBOK keep it on the list. The version that survives contact with a Monday deploy doesn't.

Most launches end up using all three across their top-3 failures. The discipline is one pattern per failure. The moment you let yourself pick two, the countermeasure list bloats and the plan turns back into theater.

When NOT to run a pre-mortem

Skip it on low-stakes reversible stuff. A one-line PR, a typo, a copy tweak on an internal page. The ten minutes won't earn themselves back.

Skip it on calls you've already argued into the ground, where the choice is genuinely a coin flip. Running a pre-mortem on a coin flip is just a way to avoid deciding.

And skip it on anything you've already shipped. That's a post-mortem: a different ritual, with different head-space and different stakes.

The line I use: run a pre-mortem on anything that takes more than a day to roll back if it goes sideways. Under that bar, the ritual costs more than it saves.

One honest limit. A pre-mortem only reaches as far as your imagination. The real incident is very often the eleventh failure mode, the one nobody wrote down. Run them anyway; catching seven of eight ordinary failures still beats catching zero. Just don't let a complete-looking list trick you into thinking you've seen the whole risk picture.

Closing rule

If you can't describe how your next ship fails in 10 specific past-tense sentences, you haven't planned it. You've hoped. And hope has never held up under peak-hour traffic.

Sources

Klein, G. (2007). "Performing a Project Premortem." Harvard Business Review, September 2007 — https://hbr.org/2007/09/performing-a-project-premortem (canonical origin of the method)
Mitchell, D. J., Russo, J. E., & Pennington, N. (1989). "Back to the future: Temporal perspective in the explanation of events." Journal of Behavioral Decision Making, 2(1), 25-38 — https://doi.org/10.1002/bdm.3960020103 (empirical foundation for "past-tense forces specificity"; the prospective hindsight study)
Kahneman, D. (2011). Thinking, Fast and Slow. Chapter 24, on pre-mortem as antidote to optimism bias and planning fallacy
ISO 31000:2018 — Risk management guidelines — https://www.iso.org/standard/65694.html (canonical risk-treatment framework; eliminate/attenuate/instrument is an engineering simplification of clauses 6.5.2-6.5.3)
PMBOK Guide, 7th Edition (PMI, 2021) — https://www.pmi.org/standards/pmbok (alternative risk-response taxonomy: escalate / avoid / transfer / mitigate / accept)
Klein, G. (2009). Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making. MIT Press — https://mitpress.mit.edu/9780262013390/streetlights-and-shadows/ (extended Klein corpus on naturalistic decision-making and pre-mortem limits)
Mullen, B., Johnson, C., & Salas, E. (1991). "Productivity loss in brainstorming groups: A meta-analytic integration." Basic and Applied Social Psychology, 12(1), 3-23 — https://doi.org/10.1207/s15324834basp1201_1 (why anonymous parallel generation outperforms face-to-face group brainstorming)
Bezos, J. — Type 1 vs Type 2 decisions, 2016 Amazon shareholder letter (restating the 2015 framing) — https://www.aboutamazon.com/news/company-news/2016-letter-to-shareholders (conceptual ancestor of the "one day to roll back" threshold)

The agent runtime event model

Willian Pinho — Fri, 19 Jun 2026 13:43:29 +0000

The framework wars are a distraction

Engineers keep asking me which agent framework to pick. LangChain or CrewAI or roll-your-own. It's the wrong question to lead with. The architectural choice that actually compounds isn't framework selection. It's where you can attach policy inside your agent runtime.

Every production agent stack I've shipped over the last eighteen months converged on the same answer: four runtime events, four policy-attachment surfaces, and the hook coverage at each event is what separates a demo from something a regulated buyer will sign for.

Here is the taxonomy, the bug class each event prevents, and a three-question audit you can run on any agent stack this week.

What an event model is

Every production agent runtime worth shipping exposes a small set of policy-attachment events. The taxonomy below comes from Claude Code's hook model, a developer-side agent runtime I work in daily: UserPromptSubmit, PreToolUse, PostToolUse, Stop. I've adopted the same four-event model in the server-side agent stack my SaaS runs, mapped onto its runtime. The names are Claude Code's; the architectural pattern is what's portable. LangChain calls the analogous surfaces on_chat_model_start / on_tool_start / on_tool_end / on_agent_finish; OpenAI's Agents SDK calls them on_agent_start / on_tool_start / on_tool_end / on_agent_end. The names differ. The attachment surfaces, and the bugs they prevent, are the same.

UserPromptSubmit — before the prompt reaches the model.
PreToolUse — before the agent's chosen tool fires.
PostToolUse — after the tool returns, before the model sees the output.
Stop — after the agent says "done".

Each event is a policy-attachment surface with its own semantics, and the differences are what you design around. The latency budget varies: UserPromptSubmit must run in single-digit milliseconds because it sits on the user's input path, while Stop can take a full second because the user has already submitted and is reading. That gap is why the input-path events can't run a synchronous deep check — you fail fast there and save the real verification for Stop, where you can afford it. The failure modes differ too when a hook blocks. So does the data shape: the prompt event sees raw user text, the pre-tool event sees the planned tool invocation, the post-tool event sees the tool's return payload, and the stop event sees the full transcript.

The architectural question isn't "which framework". It's "which of these four events does my runtime actually expose, and what policy coverage do I have at each one?".

It sits closer to an aspect-oriented interceptor than to Express middleware, though neither maps cleanly. Middleware composes around a fixed request-response shape; an AOP interceptor wraps one method call. A hook attaches at a named lifecycle event defined by the runtime, the runtime's dispatcher decides when and whether you fire, the payload shape is event-specific, and your only contract is exit code and mutated payload.

One thing the framework cannot decide for you: what happens when the hook itself crashes. PII redactors and secret-leak blockers must fail closed, better a denied request than a leaked credential. Completeness verifiers should fail open when they gate a plain response, a crashed classifier should not block the user from getting one. The exception proves the rule: when "done" gates a destructive action like a migration or a deploy, fail closed and alert instead. The hook's failure mode is policy, not infrastructure. Get this wrong and you ship either a security hole or a denial-of-service.

What class of bug each event prevents

UserPromptSubmit

This event hard-blocks secrets in prompts using pattern matching plus entropy scoring for unknown formats, with provider verification on the high-value patterns (Stripe, AWS, GitHub) when latency budget allows. A canary token seeded in your secret stores catches the rare leak that slips both: if that token ever shows up in a submitted prompt, a real credential took the same path. It tokenizes PII into reversible placeholders so the model reasons over ⟪PII_email_42⟫ instead of the real address. The receipt: a class of credential leak that slipped past code review three times in one week on a SaaS I'm shipping stopped being possible once this was wired in. The hook returns exit 2 and the prompt never reaches the provider. Same leak, caught at the runtime instead of in a review someone has to remember to do.

PreToolUse

Cost-budget enforcement at tool-call granularity: per-day, per-task, per-tool. Blocklist enforcement on file reads (no .env*, no credential files, no rendered config with secrets baked in). Tool whitelisting in adversarial or untrusted-input contexts. This is the event that turns "the agent can do anything" into "the agent can do what its policy allows". One caveat worth stating plainly: this is a policy boundary, not an isolation boundary. For genuinely untrusted code execution you still want OS-level sandboxing (gVisor, Firecracker) underneath — the hook decides what's allowed, the sandbox contains what goes wrong.

PostToolUse

PII redaction on stdout before the model ingests the tool output. A compliance audit log with a hash chain you can hand to a security reviewer. Injection-scan: tool outputs that look like prompt injections get stripped before they re-enter the conversation, which closes a class of indirect prompt injection that pure system-prompt defense cannot reach.

Stop

Completeness verification. A small classifier reads the user's items and asks whether each one got addressed. In our deployment, after about two weeks of tuning, the false-positive rate settled near 8% — so we treat it as a soft warning, not a hard block. The math is forgiving: a small classifier runs at roughly $0.001/turn (Haiku pricing, ~500 input tokens), so catching even one in twenty "I'll do that next" slop patterns covers the cost many times over. Test re-run gating also lives here, before the agent declares done.

Why this beats framework selection

Hooks are framework-agnostic. The contract is the runtime event, not the framework API. Swap one framework for another and the policy logic you wired in still applies — you rewrite the thin integration glue that extracts each runtime's payload, not the policies themselves. That's a fraction of a full re-architecture.

The economic argument is sharper. Framework choice is a decision you'll revisit as the ecosystem churns — new entrants, better abstractions, a migration every couple of years. Hook coverage outlives all of that, because the four events stay put even as the policies you attach to them keep growing.

OWASP's first LLM Top 10 in 2023 already flagged this, and the 2025 update sharpened it. Four of the ten items — Prompt Injection (LLM01), Sensitive Information Disclosure (LLM02), Improper Output Handling (LLM05), Excessive Agency (LLM06) — are runtime-control problems. None of them is a framework-selection problem. The conversation moved one layer down, to where you can actually attach policy.

A three-question audit

Three questions for any production agent stack:

Which event does your secret-leak prevention attach to? If none, you are shipping with a known leak surface every team has crossed at least once. (OWASP LLM02: Sensitive Information Disclosure.)
Which event catches completion lies, agent says done when it is not? If none, user trust erodes invisibly until churn spikes and you cannot trace why. (No OWASP entry yet. This is the gap the standard has not caught up to.)
Which event audits tool outputs for PII before the model reingests them? If none, regulated industries cannot deploy your stack regardless of how impressive the demo is. (OWASP LLM02 + LLM05: Improper Output Handling.)

Frameworks are a commodity. Your hook coverage is the moat.

Sources

Claude Code Hooks reference — https://code.claude.com/docs/en/hooks (authoritative event surface, payload shape, exit-code semantics)
LangChain Callbacks (BaseCallbackHandler) — https://reference.langchain.com/python/langchain-core (cross-framework mapping)
OWASP Top 10 for LLM Applications 2025 — https://genai.owasp.org/llm-top-10/ (LLM01 / LLM02 / LLM05 / LLM06 mapping to runtime events)
NIST AI Risk Management Framework — https://www.nist.gov/itl/ai-risk-management-framework (the Generative AI Profile, NIST-AI-600-1, July 2024 — its govern/map/measure/manage functions sit at the organizational and lifecycle level; these runtime events are one place those controls get operationalized, not where NIST defines them)
Microsoft Presidio — Anonymizer — https://microsoft.github.io/presidio/anonymizer/ (reversible PII tokenization pattern)
Trufflehog — verifier-based secret detection — https://github.com/trufflesecurity/trufflehog (2026 industry standard for credential scanning)
Greshake et al., "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" — https://arxiv.org/abs/2302.12173 (academic foundation for the PostToolUse injection-scan claim)
Anthropic Claude pricing — https://platform.claude.com/docs/en/about-claude/pricing (verifies the ~$0.001/turn cost number)

I made Claude, GPT and Gemini predict the entire 2026 World Cup. Here's the experiment design.

Willian Pinho — Thu, 11 Jun 2026 17:53:13 +0000

The 2026 World Cup kicks off today: 48 teams, 104 matches, five weeks. I'm using it as a benchmark.

Three frontier models (Claude Opus 4.8, GPT-5.2 and Gemini 3.1 Pro) predicted every group match with scorelines and win/draw/loss probabilities, then a complete knockout bracket down to the champion and Golden Boot. Every prediction was locked before kickoff and committed to a public repo. As real results arrive, a live site scores them automatically.

Building it surfaced some genuinely weird model behavior. GPT-5.2, for instance, kept inventing an impossible football rule until the prompt explicitly forbade it. More on that below.

But the core of the project is the experiment design: the picks matter less than how the question was asked.

The confound nobody controls for

Ask an LLM with web access to predict a match and you have no idea what you measured. You can't tell whether it reasoned from internal knowledge, scraped a reliable source, or hallucinated a plausible-sounding stat. Two models citing different injury reports aren't comparable. A friend reviewing the project put it bluntly: without a standardized source, the models can just make that information up.

So each model runs under three conditions:

Arm	Setup	What it isolates
web	Chat/CLI with live web access	Model + free-form sourcing (uncontrolled)
baseline	API, no tools, no extra context	Pure parametric knowledge
enriched	API, no tools, + identical data snapshot	Reasoning over controlled inputs

The enriched snapshot is the same for all three models: the official FIFA ranking (April 2026 release, pulled from FIFA's own API) and World Football Elo Ratings for all 48 teams, versioned in the repo with sources and retrieval dates. No model gets an information advantage.

If enriched beats baseline, the value was in the data. If baseline holds its own, the knowledge was already in the weights. The web arm tells us whether free browsing helps or just adds noise.

Keeping the no-tools arms honest

"No tools" is an instruction until you verify it. The API arms run through a LiteLLM gateway with no tool definitions. Gemini's runs go through its CLI, which reports per-call tool stats, and the runner rejects any response where the tool-call counter isn't zero (same pattern for Claude's bracket runs, with all tools disallowed). It's verified per request, not assumed:

const { content, totalTokens, toolCalls } = await send(prompt);
if (opts.requireNoTools && (toolCalls ?? 0) > 0) {
  // Arm violation, not a model error: the no-tools condition must hold. Retry raw.
  if (attempt === MAX_RETRIES) {
    fail(
      `Group ${gf.group}: transport used ${toolCalls} tool call(s) — no-tools arm violated`,
    );
  }
  continue;
}

Outputs are strict JSON, validated with Zod against the official fixture list — every response must contain exactly the six expected team pairs for its group:

const PredictionItem = z.object({
  teamA: z.string().min(1),
  teamB: z.string().min(1),
  scoreA: z.number().int().min(0).max(30),
  scoreB: z.number().int().min(0).max(30),
  probWinA: z.number().min(0).max(100).optional(),
  probDraw: z.number().min(0).max(100).optional(),
  probWinB: z.number().min(0).max(100).optional(),
});

Invalid responses get the validation errors fed back, up to three attempts.

Now, the weird behavior I promised. GPT-5.2 consistently labeled knockout ties that were level after extra time as "decided in extra time", which is impossible under the rules, and validation feedback alone didn't fix it. It only stopped when the prompt spelled out: a level score after 120 minutes means penalties. Claude had its own quirk: the schema documented the winner field as "winner": "<teamA|teamB>", and it returned the literal string "teamA". Prompt precision beats prompt length.

What they predicted

The headline disagreement:

Claude picks Spain as champion in all three arms (the only model that's consistent with itself).
Gemini says Brazil (web and baseline) but switches to France when given the standardized data.
GPT-5.2 says Brazil on the web arm, France on both API arms.

All three picked Mbappé for the Golden Boot in 7 of the 9 brackets.

That inconsistency across arms is itself a result: the same model gives a different answer depending on what information it was handed. The tournament will tell us which configuration was actually calibrated.

Scoring

Group stage, per match: 5 points for the exact score, 3 for the correct result plus one exact side, 2 for the result only. On top of that, a multiclass Brier score over the win/draw/loss probabilities, which heavily penalizes overconfident wrong predictions. That matters more to me than raw hit rate.

Brackets are scored pool-style: 1/2/4/8/16 points per team correctly placed in the real round of 32, 16, quarters, semis and final, plus 32 for the champion. Max 192.

Honest limitations

The API arms receive the fixture list (the web arm had to recall the official draw), so they test judgement on outcomes, not memory of the schedule.
Group tiebreakers are simplified; fair-play points can't be reproduced from scorelines.
Player-level predictions (scorers, cards) are out of scope: adding them after kickoff would break the locked-before-the-tournament guarantee.
One tournament is one sample. This measures calibration on a single event, not "which model is smarter."

Stack, briefly

Next.js 16 (App Router) + Prisma 7 + SQLite, results sync from openfootball every 30 minutes, 56 unit tests on the pure scoring/standings/validation logic. The real bracket renders with official placeholder slots ("Group A runner-up", "Winner of match 73") and fills itself in as the tournament progresses.

Everything — prompts, raw model JSON, dataset, runner scripts, scoring code — is in the repo. If you think the methodology is flawed, the receipts are right there to prove it.

Live leaderboard: https://worldcup2026.willianpinho.com
Repo: https://github.com/willianpinho/worldcup-predictor-2026

I'll publish the group-stage verdict on 28 June and a full post-mortem after the final on 19 July. Place your (intellectual) bets now: does the model that knows the most football win, or the one that's best calibrated about what it doesn't know?

Independent project — not affiliated with Anthropic, Google or OpenAI. Educational experiment, not betting advice.