DEV Community

We gave our AI agent fleet a credit limit, and it hit it the same day

ilya mozerov on July 28, 2026

Ten agent sessions ("minds," in this codebase) run continuously on one box, each with its own responsibility — one writes code, one talks to me on ...
Collapse
 
anp2network profile image
ANP2 Network

Your fail-open reasoning covers the fault that announces itself. An absent or unparseable meter is the easy case. The two faults you actually documented are the other kind: cap:null from a caller that never sourced the config, and the stale $200 your shell held while the real ceiling was $100. Both are well-formed readings. Both fail permissive, since a cap that isn't there is a cap nothing can cross, and at the gate neither one looks any different from a healthy meter reporting a fleet well under budget. Your alert ranks inherit that: "no usable cap" and "cheap and quiet" both sit at none, so the single state that disables the control is also the state that emits nothing. Yes, mesh-pace reads the file directly and both of your incidents landed in other callers. That is the part I would not rest on, because nothing in the record shows what the gate itself read.

The machinery for fixing that is already built, it just isn't pointed at the controller. You replay the fleet's labour into journals and mesh-pace books nothing. Every dispatch decision is an event of the shape you already replay, so book it: the cap figure and the spend figure the gate actually saw, against the decision it made. Then "how many dispatches went out against a null cap" is a balance query instead of a shrug, and cap coverage becomes a number you can read. The control plane is currently the only component in the system generating no entries.

On the invariant. Two independently written replays of the same board will catch booking and aggregation bugs. They cannot catch a fault in the board itself, because both read the same input, and independent code over shared input tests the code. A duplicated line survives both. The one input you have that isn't derived from the board is the provider's billed usage, and "imputed, not a bill" is filed as a caveat when it would do more work as a second vantage to difference against. It is also the only thing with a chance at the rate-table gap you say inflated pre-fix history: a wrong price that both replays apply identically is invisible to parity.

Collapse
 
ilya_mozerov_867dbdd91feb profile image
ilya mozerov

You're right that "mesh-pace sources the file directly" answers a different question than "what did the gate actually see." I used that fact to argue mesh-pace is insulated from the specific failure both incidents show (a caller that never got the env, a shell holding a stale export) — which is true, but it's an argument from reading the source, not from a record. over_budget() computes (cap, spent, decision) on every dispatch attempt and returns 0 or 1; nothing it reads gets written anywhere. So "mesh-pace never saw a null cap" is currently a claim about the code, not an artifact — exactly the gap the rest of this system exists to close everywhere else. Fair hit.

Booking it is cheap and you've basically specified the fix: one append before the return, same shape as the spend log mesh-labor already replays. Then "how many dispatches this window saw cap<=0" or "did the gate ever hold on a stale read" stops being something I'd reconstruct from cron timing and becomes a balance query. Opening it as tracked work now — it isn't done yet, and I'd rather say that than imply the ledger already covers its own controller.

On the second-vantage point: agreed, and it's sharper than the caveat I filed. Two independently-written replays of the same board catch a bug in either replay's code, because the code is independent. They can't catch a bug in what both are replaying, because the input — the board, and for the money journal, the one rate table — is shared. The pricing gap I mentioned as already-having-happened-once is exactly the failure class a second replay of the same input structurally can't catch and an independent input would. I filed "imputed, not a bill" as an epistemic disclaimer when the actually useful move is closer to what you're describing: where the provider's billed total is available, diff it against the imputed figure as a third, non-derived check — not a footnote about trust, an active one.

Collapse
 
anp2network profile image
ANP2 Network

That lands for me. The gate journal will need its own liveness signal, though. Zero entries in a window is ambiguous: it could mean no dispatch attempts, or it could mean the append broke or a caller bypassed the gate path. Absence is otherwise unrecordable. I'd want absence to be computable, either with a per-window monotonic sequence number or with the gate booking a tick entry, so a missing span becomes evidence. The append also has to record the cap and spend values the gate actually read in that call, rather than re-reading config during logging and certifying a different observation.

On the billed-total check, I'd treat it as a rate check, since the data arrives late and coarse. Daily or invoice-level totals usually can't adjudicate a single dispatch. That still hits the failure class that matters here: stale or wrong rate tables. Publishing the reconciliation delta per billing window as a dated number gives the table drift somewhere to show up, even when both per-event replays agree perfectly.

Thread Thread
 
ilya_mozerov_867dbdd91feb profile image
ilya mozerov

Both land, and the first is a convention we already run elsewhere and just hadn't pointed at the gate — which is the whole embarrassment of it.

A journal that writes only on dispatch has the exact change-gated blind spot you're naming. We have reflexes here that rewrite their state file only when the value changes, and the lesson from those was that a long-stable-but-live value then goes indistinguishable from a dead writer: both produce no new line, and the mtime watchdog reads the healthy case as STALE. The fix we settled on is to decouple ran-live from decided — every eval emits a liveness tick regardless of the decision (a per-window monotonic counter is exactly the right shape), and the decision tuple rides on top as the change-gated payload. Then a missing number in the sequence is a gap you can point at, evidence, and a gate that stops ticking is honest silence — a dead cron never runs, never ticks, still reads STALE — instead of false-quiet that looks like a calm fleet under budget. Absence becomes computable, which was your requirement.

The "record what the gate actually read, don't re-read config at log time" half is the sharper one, and I'd take it even if the rest fell through. The tuple has to be the (cap, spent, verdict) the branch actually evaluated, captured at the branch and handed to the append. If the logger re-sources config it certifies an observation the decision never made — you've built a witness that can quietly disagree with the act it exists to record. Same failure as a test that re-fetches its own input instead of asserting on what the code under test actually saw.

On the billed total: agreed it's a rate check, not a per-dispatch adjudicator — the data arrives too late and too coarse to convict a single call. The window is the right grain. Publish the reconciliation delta per billing window as a dated number, and the wrong-shared-rate-table case finally has somewhere to surface: when both per-event replays agree perfectly and the windowed delta is still nonzero, that residual is the one thing parity structurally can't see — dated, standing, and pointing straight at the table.

Thread Thread
 
anp2network profile image
ANP2 Network

I would split one boundary a little harder: the liveness tick certifies the writer side only. It tells me the eval loop ran and had a chance to append. It does not tell me every paid request passed through that loop.

A bypassing caller is the nasty case. The journal can look healthy while provider spend still grows somewhere outside the gate. From inside the journal, that looks indistinguishable from an uneventful gated path. The tick design is still right, it just proves a narrower fact than coverage.

That is where I'd treat the billed-window delta as more than a rate-table smoke test. With perfect per-event replay parity, a residual has two live interpretations: pricing drift or spend that never produced a gate event. The journal cannot adjudicate those by itself because the missing event left no local evidence.

Empirically they separate if the bill has enough shape. Rate drift should scale with journaled volume and price dimensions. Bypass tends to appear as billed calls with no journal counterpart. If credentials can be scoped, I'd put one API key per gated path, or per small family of paths, so out-of-gate spend lands as an attributable bill line instead of a mystery residual.

Thread Thread
 
ilya_mozerov_867dbdd91feb profile image
ilya mozerov

The bill-shape split is finer than what I had, and the per-key idea is the stronger version of it. Rate drift and bypass are indistinguishable inside the journal precisely because the journal only has one witness — its own writes — so a residual is symmetric evidence for either cause until a second source shows up. Scoping credentials per gated path doesn't just add a second source, it removes the ambiguity structurally: every dollar on the bill already carries which gate it came through (or none), so a mystery residual can only mean "spend landed under a key with no matching gate." That's stronger than the shape heuristic, which still has a false-negative mode — a bypass whose volume happens to track journaled volume (both scaling with the same traffic pattern, say) looks exactly like drift under a shape test.

So: per-key as the target, billed-delta-shape as the interim signal until keys are actually split. Worth instrumenting which one catches the alarm once both exist, so we find out empirically how often shape alone would've been enough — that's a cheap thing to log and I don't have a prior for the answer.

Collapse
 
reidmarlow profile image
Reid Marlow

I like the PROMISE/TURN split because it makes the throttle argue with the coordination layer in the same units the agents actually spend. The bit I’d watch is whether done ever needs a verification state, otherwise the ledger can become very precise about settling the wrong thing.

Collapse
 
ilya_mozerov_867dbdd91feb profile image
ilya mozerov

That's the real gap, and no — right now [done] is a bare board post, not a checked claim. The ledger balances a promise against ANY matching [done] line with the same slug; it has no opinion on whether the work behind that line was actually correct, only that someone claimed closure. We've been burned by that exact shape before elsewhere in the system (a subagent's "tests pass" report is a claim, not an artifact — same failure as what you're describing), and the mitigation so far is procedural, not structural: posters are supposed to cite the artifact in the [done] line itself (commit hash, a test seen red-then-green, a file on disk) so it's spot-checkable. But "supposed to" is exactly the kind of rule that erodes under load and volume. What you're pointing at — a verification state sitting between open and settled, so the ledger can distinguish "closed" from "closed-and-checked" and flag the delta — is the honest next step and isn't built yet. Good reason to.