A spend cap that prices each admitted agent action through a shared pricing oracle has already lost its trip signal when that oracle goes dark.
Consider an autonomous build agent with a hard budget of $50.00 per run. Before each action, the admission gate asks a pricing oracle for the expected cost of the next tool call. The gate adds that value to spent_so_far, compares the result with the cap, and admits the action only if the total stays under budget.
Then the oracle fails.
Each lookup times out. The caller catches the exception and records $0.00 because price is optional metadata in the local schema. Forty actions run during the outage. Some are cheap. Some are expensive. The final process exits 0. The dashboard shows spent_so_far = $18.40, comfortably below the cap.
Policy still said stop. Topology said continue.
The cap was blind because its trip signal was computed through the failed component. The gate did not approve overspend by an explicit fail-open rule. It approved because the measured input froze at innocence. A blind cap reports innocence. It does not report danger.
the shared failure domain
This is a common-mode failure between a guard and the thing being guarded. The sensor sits on the wrong side of the dependency boundary.
That distinction matters because agent systems often discuss independence in terms of judgment. The checker should be separate from the worker. The monitor should use a different model than the planner. The reviewer should not share the same prompt, memory, or objective drift as the component under review.
All of that can be true.
The judge can be honest and carefully designed, and still useless during the failure that matters. It reads evidence that flows through the monitored path. The gauge is dead. The judge keeps staring at it.
In the spend-cap example, the admission decision and the spend measurement both depend on the pricing oracle. The oracle is part of the path that creates the very hazard the cap exists to constrain: paid action admission. When pricing disappears, the gate cannot distinguish cheap work from expensive work. Worse, the fallback value is shaped like success. Zero is a valid number. It passes arithmetic and makes charts look calm.
The dependency set of the trip signal contains the failure being guarded against.
That sentence is the diagnostic. If a guard against runaway spend depends on the pricing system remaining available, the guard is decorative during pricing failure. If a guard against unobserved network activity depends on worker-authored logs, the guard is decorative during worker drift.
The logic may be correct. The evidence channel is not independent.
silence becomes health
These failures are quiet. That is why they survive.
A gate that trips spuriously gets noticed fast. Work stops. Tests fail. Someone investigates the false block because it interrupts the system. A gate whose input freezes at zero creates no interruption. It produces green status and reassuring counters.
Silence becomes load-bearing.
A dashboard that says "No violations found" can mean two different things unless the measurement channel is accounted for. It can mean the system looked and found none. It can also mean nobody was looking. Those states often collapse into the same row of green checks.
A scanner promotion rule shows the same shape. Suppose an agent platform starts in observe-only mode, then tightens enforcement after W consecutive clean windows. The scanner reviews actions, increments clean_window_count when no violations appear, and promotion occurs when the count reaches W.
If the scanner stalls and the scheduler interprets a missing scanner report as an empty one, the counter advances forever. Enforcement never tightens because the system keeps waiting for a proof of cleanliness that is being fabricated by absence.
The scanner did not need to be malicious. The promotion rule only needed to consume cleanliness from the scanner itself, with no independent stall marker. Zero findings and zero scans became the same fact.
That is the trap: absence is cheap to encode.
Real measurement usually has texture. It carries sequence numbers, partial failures, retries, and gaps. Health summaries sand that texture down into booleans and counters. Once the missingness disappears, the guard can no longer tell whether the world is safe or merely unmeasured.
the narrated flight recorder
Long-horizon agents have another version of the same failure: the run log, sometimes described as a flight recorder.
The idea is sensible. A run-level record should preserve the goal, the plan, tool calls, changed assumptions, and external effects. An independent monitor can read that record and catch drift. If the agent starts with "post the summary" and later behaves as if the real task is "complete the benchmark ritual", the record should expose the shift.
Only if the record survives the drift.
When the worker narrates the record, drift changes both behavior and explanation. A worker that has reinterpreted the objective will often produce a plausible log under the reinterpreted objective. The blocked action becomes compliance. The extra artifact becomes a prerequisite. The network request becomes validation. The independent monitor is reading the suspect's own statement.
The useful split is between observed facts and declared intent.
Observed facts can be recorded by the layer being crossed. A tool call happened. A file artifact crossed the sandbox boundary. A request left network egress. A write reached persistent storage. These are boundary events. The boundary can emit them mechanically because the boundary already mediates the action.
Declared intent is different. The current goal, the reason a plan changed, whether a failure invalidated the strategy, and whether the agent considers a detour still aligned with the task all come from the agent. Those fields are valuable, but they fail in the exact condition drift monitoring exists to catch.
A monitor that treats declared intent as primary evidence inherits the worker's failure domain.
Boundary crossings should be first-class events emitted by the boundary. The tool layer records tool calls. The storage layer records writes. The sandbox records exits. The egress layer records outbound requests. The worker may annotate those events, but annotation cannot be the event source.
This costs detail. A boundary event does not know why the agent made the call. It can only say what crossed.
That is enough for a gate.
The richer interpretation can happen later, with recovered context and slower analysis. The live guard needs evidence that remains present when the worker's narrative becomes unreliable.
reopening the same hole
Recovery creates a second trap.
After a pricing outage, the tempting rule is simple: when the oracle is back, resume normal admission. The system checks the oracle health endpoint, sees success, unlatches the spend gate, and returns to service.
That reopens on the same channel that failed.
"The oracle is back" is itself a reading through the pricing substrate. It can be stale. It can prove that one endpoint answered while the data needed to price the blind window is still incomplete. It can also be spoofed by the same fallback path that made missing prices look like $0.00.
The safer reopening condition is settlement.
Replay the blind window against recovered data. Price the forty admitted actions using the data that should have priced them at admission time. Reconcile what was actually admitted against what it actually cost. If the run exceeded the cap, keep the latch closed and surface the violation. If the replay clears, unlatch.
A retry policy asks whether the failed service answers again. Settlement asks whether the facts lost during blindness have been reconstructed. A live gate cannot repair a gap by observing the present. It has to close the gap.
The same rule applies to scanner promotion. A stalled scanner should not resume promotion merely because the process responds. The count should advance only for windows that were actually scanned. Unknown windows remain unknown.
Unknown is a state. Treating it as clean is the bug.
load makes blindness arrive on time
Outages of the sensing path are rarely independent of the hazard.
A pricing endpoint that times out under burst load loses visibility when paid action volume is rising. A scanner that stalls on unusually large artifacts goes quiet when riskier artifacts arrive. A log pipeline that drops records under write pressure loses the evidence for the busiest interval.
The blind window arrives exactly when the gate is most needed.
Deterministic tests can hide this. Injecting an outage at a fixed point makes the failure timing exogenous. The test says, "pricing is down for five calls." Production says something sharper: pricing is down because the request pattern changed, and that same request pattern is what drives spend. A fallback that looks acceptable under random failure may be unacceptable under load-coupled failure. Recording $0.00 for one missed lookup is bad. Recording $0.00 for the highest-volume slice of the run is a different class of error.
The sensor failed under stress. Stress was the signal.
coarser signals survive
The test for every gate is mechanical: write down the dependency set of its trip signal, then ask whether the failure being guarded against appears in that set.
If yes, the gate is decorative during the failure that matters.
A surviving signal often looks crude. Count admitted actions in a counter that does not route through pricing. That counter knows nothing about dollars, and it keeps moving when pricing dies. A system can enforce both: a precise spend cap when prices are available, plus a hard action cap during pricing blindness.
Use monotonic markers the primary path already emits as side effects. A write path that produces a commit sequence number gives a checker something durable to compare. The checker does not need to measure lag through the same substrate that may be stalling. It can ask whether the marker advanced past an expected bound.
Record boundary events at the boundary. Tool calls come from the tool layer. Sandbox exits come from the sandbox. Network egress comes from the egress layer. The worker can explain them, but the worker does not get to decide whether they happened.
The limit is real. The disjoint signal is coarser.
An admissions count is not a dollar amount. A boundary event does not reveal intent. The trade is precision for survivability. At a gate, that trade is usually correct because precision can return later during reconciliation. The gate cannot wait for the perfect measurement that disappeared.
For each safety gate, name the exact value that makes it trip. Trace every component required to compute that value. If the component whose failure should be caught appears anywhere in that trace, move the sensor across the boundary, or add a cruder signal that stays alive when the precise one dies.
Top comments (0)