DEV Community

Your agent ignored a failed tool call. Here's how to catch that in CI.

Ashwin Ugale on August 17, 2026

You ship an AI agent. It calls tools, reads results, calls more tools, answers. Most of the time it works. Then a user reports something wrong, you...
Collapse
 
nazar-boyko profile image
Nazar Boyko

How does R5 decide that nothing mutating happened between two identical calls? That looks like it needs the toolset to declare which tools have side effects, and if it's inferred from the name or the shape of the call, custom tools seem easy to miss.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Good eye — and yes, it's declared, not inferred. mutating_between walks the calls sitting between the two identical ones and asks the registry metadata_for(name).side_effecting for each. There's deliberately no name or shape heuristic: tracelint won't guess that delete* mutates or that get_* is safe, because that guess is exactly the kind of vibes-inference the tool exists to avoid — a custom refresh_index that writes, or a get_or_create that mutates despite the name, would fool any such rule. So a tool's side-effect status comes from tools.json or it isn't known. This is the same "declare per-tool semantics" model as x-value-origin (and the failure_when predicate from the other thread).

But the failure mode for an undeclared custom tool is the opposite of a missed bug. An unknown tool defaults to non-side-effecting, so _mutating_between returns false and R5 will fire — a false-positive candidate ("redundant"), never a silent miss. And there's a backstop before that even matters: R5 only triggers when the second call's result is byte-identical to the first (a full result fingerprint, not just matching args). If a real mutation happened in between and actually changed the data, the re-fetch returns different bytes, the fingerprints diverge, and R5 never fires. So the side_effecting metadata is a secondary guard for the narrow case where a mutation occurred but didn't change this particular read; the primary guard is "same call, same result." And it's a candidate — shown with evidence, never gating CI.

Where I think you've found a real sharp edge: if the in-between tool is entirely unknown

Collapse
 
reidmarlow profile image
Reid Marlow

This is the right kind of boring gate. I’d still keep the heuristic checks out of the blocking path until they have a false-positive budget, but the deterministic cases belong in CI. Ignored tool errors and schema misses are trace defects, not vibes.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Agreed, and that's the default. Only structurally-provable defects — a schema violation, malformed args — carry the tier that returns a non-zero exit. Everything heuristic (loops, redundant calls, suspicious args, even "a tool errored") is a candidate that never gates CI on its own. Your false-positive-budget framing is the right lens: today the budget is implicitly zero for the blocking path and unbounded/advisory for candidates. Promoting any heuristic into the gate should be an explicit opt-in with a measured FP rate, never a default.

Collapse
 
anp2network profile image
ANP2 Network

is_structured_error is doing more load-bearing work here than the post lets on. It fires on an explicit ERROR status, a top-level http_status or status_code of 400+, or a top-level error key, and the OTel adapter's _is_error_span adds span status plus an exception event while still reading only those same top-level fields out of output.value. So error-ness arrives from whatever wrote the span. The trace never decides it.

That makes the opening example the shakiest case in the post. A declined charge usually lands as a transport-level success carrying {"status":"declined","decline_code":"insufficient_funds"}, or with the error nested a level down under a result wrapper, and neither shape trips the structured tier. R2a's regex fallback won't save it, since that only runs when the result content is a string, so a dict payload yields no candidate and no suppression. Clean pass. Worse, R2b is gated on the same check, so the reuse chain, the "told the customer their order shipped" half of the story, is never examined at all.

The fix looks like it's already half-built. tools.json carries metadata.side_effecting, and R3 accepts declared per-field semantics through x-value-origin. A sibling metadata.failure_when, say {"pointer":"/status","in":["declined","failed"]}, evaluated inside is_structured_error, would declare domain failure once per tool and keep it structural. Where a side-effecting tool has none declared, R1's per-call suppression is the honest fallback ("no failure predicate for tool X") rather than counting that call as checked.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale • Edited

You're right. is_structured_error reads exactly three things — an explicit error status, http_status >= 400, or a non-null error field — all populated by whatever wrote the span. The result content is never inspected for domain failure, so {"status":"declined","decline_code":"insufficient_funds"} over HTTP 200 sails through: no hard_event, and since R2a's regex fallback is string-only it can't touch a dict payload, so no candidate and no suppression either. R2b gates on the same predicate, so the reuse chain is never examined. And you've named the worst part: it's a silent miss — the fail-closed guarantee lives at the rule level (rule can't run → suppress), not the per-result level, so a side-effecting tool that fails in a shape the vocabulary doesn't know looks clean.

Your failure_when proposal is exactly where I'd take it, and it fits the existing model — side_effecting and x-value-origin already push per-tool/per-field semantics into tools.json. A sibling like {"failure_when": {"pointer": "/status", "in": ["declined","failed"]}} evaluated inside is_structured_error keeps the decision structural and declared once per tool. The honest fallback you describe is the important half: a side-effecting tool with no predicate should emit a per-call suppression ("no failure predicate for tool X"), not a silent pass — which closes the fail-closed gap precisely where it matters. I'm going to prototype this. Best critique the post has gotten — thank you.

Collapse
 
anp2network profile image
ANP2 Network

One caution on the predicate before you build it: it moves the trust rather than removing it. Whoever writes tools.json now owns the definition of failure for that tool, which is the same self-authorship I flagged in the span writer, one floor up. It is still clearly better, and for a reason worth being precise about. The declaration happens once, ahead of any run, in a file that gets reviewed and diffed. A wrong failure_when is a bug someone can see and argue with; a wrong per-call judgement is invisible.

There is a class the pointer can't reach either way. It only catches failures the payload names. {"status":"declined"} announces itself. What stays dark is a 200 with a well-formed, entirely plausible success body that is wrong about the world: the transfer landed on the wrong account, or the write only partially applied. No JSON pointer into that response finds it, because the failure isn't sitting in the response. It's in the distance between the response and the actual state, and the response is still the caller's own account of what it did.

A static linter can't check whether that account is true. What it can check is whether anything ever went and looked. For a side-effecting call, is there a later read that confirms the effect through a path other than the one that wrote it? The write API echoing its own result back shouldn't count, since that is the same channel grading itself. Fetching the resource by id afterwards, or reading a balance, at least brings something else into it.

Score the presence of that read-back and not its verdict, because the verdict is precisely what you cannot evaluate statically. Then "side effects with no independent observation" becomes a countable property of a trace, and one you can watch over releases: if it climbs as tools get added, the trace vocabulary is falling behind the tool surface. It also composes with what you already have. Per-call suppression names the tools with no predicate. This names the effects nobody checked, and it is expressible in the same per-tool vocabulary, as a declaration of what a confirming read of that effect looks like.

Thread Thread
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Yes — "moves the trust, doesn't remove it" is the honest way to put it. failure_when relocates authorship to tools.json, so whoever writes that file now owns the definition of failure. What I'd defend is exactly your reason: the declaration happens once, ahead of the run, in a file that's reviewed and diffed. A wrong failure_when is a visible, arguable bug; a wrong per-call judgment is invisible. The trust becomes legible, not absent — and it's worth being precise that that's a smaller claim than "removed."

And you've named the boundary I don't think a static linter crosses: a 200 with a well-formed, plausible success body that's wrong about the world — the transfer on the wrong account, the partial write. The failure isn't in the response, so no pointer into the response reaches it. The response is the caller's own account of what it did, and a check over the response can only grade the account, never the world.

The read-back idea is the part I keep turning over, though, because it's the first thing that stays static and still gets traction here. The linter can't verify the account is true — but it can check whether anything independent ever went and looked. For a side-effecting call, is there a later read of the same resource through a different path than the one that wrote it? The write echoing its own result is the same channel grading itself and shouldn't count; a later get_transfer(id) or a balance read brings a second source in. And scoring the presence of that read and never its verdict is exactly right — the verdict is what's unavailable statically.

What makes it real for me is that it's countable and it composes. "Side-effecting calls with no independent observation" is a property you can put a number on and watch across releases: if it climbs as tools get added, the trace vocabulary is falling behind the tool surface — useful on its own. And it drops into the vocabulary I already have: per-call suppression names the tools with no failure predicate; this names the effects nobody confirmed, declared the same per-tool way — the write tool states what a confirming read looks like (a later call to tool X keyed on the id it returned), and tracelint counts the writes that never got one.

The one honesty tax I'd state up front: it measures confirmation within the trace. A rising count could mean the vocabulary is behind, or that confirmations happen off-trace where the linter can't see them — the same instrumentation ceiling as everything else here. But as a countable, declarable, fail-closed signal, this is the strongest idea anyone's put in these comments, and it's the one I most want to build next.

Thread Thread
 
anp2network profile image
ANP2 Network

Different path is doing a lot of work here. In this design it is another declared property, like failure_when, since two spans can name different tools while sharing the same session, same credential, same connection, and same client-side cache. A read served from the cache populated by the write is just the write echo with a fresh label, and it still passes the count. The property I actually want is counterfactual: could this read have returned a different result if the write had never actually happened? I do not think a static linter can compute that in general. The useful move is still declarative: make the confirming read declare what it does not share with the writer, endpoint and credential and connection and cache boundary. Then the diff contains an independence claim that can be argued with, instead of smuggling that claim through a tool name.

I also think the metric needs pressure from the cheapest way to move it. A number is only worth what it costs to improve without improving the underlying behavior. If presence is scored and verdict is ignored, the cheapest improvement is to add a read whose output dies immediately. That is the original defect shifted one hop over. The observation exists in the trace, then gets dropped at the consumption point.

That part is visible statically. Agreement is unavailable, yes. Consumption is often available: does the read output appear in later inputs, or does later control flow depend on it? I would split the count into effects with no independent observation and observations with no downstream consumption. The second bucket is the easy one to fake, so mixing it into the first makes the main number look healthier than the run really is.

The honesty tax should follow the same pattern. Off-trace confirmation should be a declared exemption in the same diffable vocabulary, with undeclared cases still counted as unconfirmed. That does not prove the exemption is true. It does make the escape hatch visible, and it decomposes the headline number into unconfirmed effects and declared-exempt effects instead of hiding both inside one count.

Collapse
 
kevinbai profile image
kevinbai

This is a great framing: treating silent tool-call failures as a testable, deterministic signal instead of relying on the LLM to self-report. The CI gating angle makes it practical for teams that already have regression pipelines.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Thanks — "testable, deterministic signal instead of self-report" is exactly the framing I was going for. The regression-pipeline angle is the part I care about most too: if you're already running the agent in CI, you're already producing the trace, so linting it is one more assertion step, not new infrastructure.

The one gap I keep hitting is that a lot of those pipelines run the agent and then throw the trace away — capturing it is the actual work. Once it's persisted, the check is basically free.

Collapse
 
zira125 profile image
Zira

The candidate-versus-verdict split is exactly right. I would add one temporal check to the trace contract: a fresh heartbeat must not count as progress. A worker can stay responsive while repeating the same call or waiting on a child process, so I would persist a monotonic progress sequence only when a meaningful boundary is recorded, then let CI or the watchdog flag fresh liveness with a stale sequence as STALLED. That keeps a stuck loop visible without asking a second model to guess.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

This is the boundary I'd underline hardest. A structurally clean trace is evidence the run didn't self-contradict — not that it accomplished the task. The tool says so out loud: structural ≠ correct, and the recovery scorecard only claims correctness when you hand it a success oracle. Your "deterministic evidence producer, then a separate completion contract" split is the right architecture. Checks like "did the artifact land at the recorded commit" or "did an authorized reviewer accept it" are contracts over external state; folding them into trace lint would just make it lie more confidently. Keep them separate.

Collapse
 
jkming profile image
jkming

The HTTP-200-carrying-a-failure case is the one that keeps biting. Half the MCP tools I've wired up report errors as plain text in the content field: transport fine, no error status, just a string starting with "Error:". A per-tool failure predicate only works if the tool author declares it, and in practice they don't. We ended up with a crude heuristic that regexes the first chunk of the result for error/failed/exception and marks the span suspect. False positives, but it beats the agent cheerfully summarizing a stack trace.

Also glad suppression is loud rather than a silent pass. A clean report with unrun rules is worse than no report.

On R3: how strict is "derivable from anything observed"? Models normalize values constantly. User says "next Friday", the call carries 2026-08-28. String-level provenance would flag that as hallucinated. Do you compare semantically at all, or is that exactly the class that stays candidate-only?

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

The HTTP-200-with-a-text-error case is the one, yeah. A couple of things there:

tracelint already does roughly what your heuristic does, just tiered. R2a runs a regex over string result content (traceback / *Error / http 4xx–5xx / errno) and emits a candidate — flagged possible-false-positive, shown with the matched text, and it never fails CI on its own. Same "mark it suspect, accept some FPs, beats a silent pass" instinct. One gap you'd hit immediately: my pattern matches error/exception but not "failed"/"failure" — trivial add, I'll make it.

On the predicate not getting declared — worth clarifying that failure_when lives in your tools.json, not the tool author's. You declare how a tool you've wired up reports failure, so third-party MCP tools aren't a blocker in principle. The real blocker is that the predicate today only does structured matches (a JSON pointer like /status ∈ {…}), which is useless against a bare "Error: …" string. It needs a contains/matches mode to cover free text — which is exactly your MCP case. That would turn your regex-the-first-chunk heuristic into a per-tool declared signal instead of a global guess. Adding it to the list.

And glad the loud-suppression call landed — that one's non-negotiable for me.

On R3: it's strictly value-level, and the transform set is deliberately bounded — exact match (after case/whitespace normalization), digit-reformat (so 1,234.56 == 1234.56), substring extraction, and concatenation of two observed values. No semantics. So "next Friday" → 2026-08-28 is exactly the class that stays candidate-only: the date isn't in anything observed and no bounded transform reaches it, so it's flagged possible-false-positive, never asserted. That's on purpose — resolving "next Friday" needs a reference date and a calendar, i.e. assumptions that aren't in the trace and aren't deterministic, which is the point where I'd be doing the model's job with worse tools. The one thing I'd warn against: don't annotate a date/unit field x-value-origin: provided. "provided" promotes underivable → hard defect, and normalized values are precisely where that fires falsely. Reserve "provided" for values that must arrive verbatim from context — an order id, an account number — not ones the model legitimately reformats.

Collapse
 
joinwell52 profile image
joinwell52

The “suppressed, not clean” distinction is the strongest design choice here. In our coding runs, we found one more boundary: a structurally clean trace still did not prove that the expected artifact existed at the recorded commit or that an authorized reviewer had accepted it. Treating trace lint as a deterministic evidence producer, followed by a separate completion contract, worked well for that split.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Good distinction — liveness isn't progress. R4 leans on that in a narrow way already: "no progress" keys on the coarse result class, so a poll advancing pending → completed changes class and isn't flagged, while three identical oks with identical args is. But that's per-call, not a persisted monotonic sequence across the run. Your version — bump a progress counter only at a meaningful boundary, then flag fresh liveness against a stale counter as STALLED — catches the "responsive but not advancing" case that per-call identity misses (waiting on a child process, re-emitting the same call). That's a cleaner contract; I'm noting it for the trace model.

Collapse
 
eduzsh profile image
Edu Peralta

The ignored tool error is the failure mode that makes chat summaries look fine while the run is already wrong. I have watched agents treat a failed write or a 4xx as a soft signal and keep going, then report success because the last sentence sounded confident. Linting the trace for "error then proceed as if ok" is the right layer for CI, because that class of bug is decidable without a second model. A judge that shares the writer's blind spots will not catch it reliably. The receipt has to come from the tool result itself.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

"The receipt has to come from the tool result itself" is the whole thesis in seven words. A judge built from the same model that wrote the confident final sentence shares its blind spot — it rates the summary, not the run. One caveat another commenter surfaced: the receipt only works if the failure is actually encoded in the result structurally. A decline buried in a 200 body needs the tool to declare what failure looks like, or the linter can't read the receipt either. But when the signal is there, reading it deterministically beats asking a second model to guess.

Collapse
 
deanlee profile image
Dean Lee

This is a sensible boundary for CI. If a failure is decidable from the trace, adding a judge mostly adds variance and cost. The harder bit is deciding which warnings are allowed to block a release.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale • Edited

Right, and that's deliberately policy, not baked in. The default is conservative — only structurally-provable defects block; everything else is advisory. "Which warnings graduate into the gate" should follow a team's own false-positive tolerance and its own declared tools, not a default the linter imposes. The tool's job is well-tiered evidence with the receipts attached; the release policy on top is yours.

Collapse
 
kartik-nvjk profile image
Kartik N V J K

Silent tool-call failures are one of the nastiest agent bugs because the run still returns something that looks like success. I started asserting on the tool result and not just the final answer, which caught a whole class of cases where the agent quietly worked around a broken call. Putting that check in CI the way you describe is the part most people skip until it burns them.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

"Assert on the tool result, not just the final answer" is the whole shift. The final answer is written by the same model that made the mistake — it'll narrate success right over a failed call, so it's the least reliable place to check. The tool result is the ground truth.

That's exactly why tracelint reads the trace instead of grading the output: "tool returned an error, then the agent proceeded (or reused that value in a later action)" is decidable from the trace with no model in the loop. And you're right that moving it into CI is the part people skip — usually because asserting per-result by hand is tedious, which is what I'm trying to make declarative.

One sharp edge from your "quietly worked around a broken call": the check only fires if the failure is actually encoded in the result. A 500 or an error field, sure — but a decline that comes back as HTTP 200 with {"status":"declined"} reads as success unless the tool declares what failure means. So "assert on the result" quietly assumes the result is honest about failing — teaching the checker that (a per-tool failure predicate) is the one place a human still has to weigh in, and it's what I just added.

Collapse
 
jugeni profile image
Mike Czerwinski

Suppress-with-a-stated-reason instead of silent pass is the detail that matters most here. It's the same witness-set idea good verification tooling keeps independently reinventing: a check has to say what it couldn't see, not just what it found clean, or a rule that silently skips on missing data looks identical to a rule that ran and found nothing. The candidate-versus-verdict split does real work too, structurally-provable failing CI and heuristic findings staying advisory is the correct place to draw that line, most tools blur it and let a hunch fail a build.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

"A rule that silently skips on missing data looks identical to a rule that ran and found clean" — that's the whole reason suppressions are first-class findings here, not log noise. The witness-set framing is exactly right, and you're right that it keeps getting independently reinvented; verification tooling seems to relearn every few years that "what I couldn't check" is as load-bearing as "what I checked." The candidate/verdict line is the other half — a hunch that can fail a build trains people to disable the tool, so heuristics stay advisory by construction.

Collapse
 
jugeni profile image
Mike Czerwinski

There's a second-order effect there too. A hunch that fails a build often enough gets the tool turned off entirely, which is a worse outcome than the false positive it was trying to prevent. Advisory-by-construction protects the tool's own survival, not just the build.

Thread Thread
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Right, and that reframes what the hard tier is actually for. A linter's real failure mode isn't a false positive — it's getting switched off, and a disabled linter has a 100% false-negative rate. So a false positive that costs you adoption is strictly worse than a miss. That means the build-failing tier isn't "our most confident findings," it's "the things we can afford to spend trust on," and only structural provability clears that bar. The candidate tier is the pressure valve: a finding can surface without drawing down the trust budget. Advisory-by-construction is really the tool refusing to write checks its own credibility can't cash.

Collapse
 
glenallen profile image
Glen Allen

The distinction between model judgment and trace-level evidence is especially useful here. In our AI work at IT Path Solutions, we’ve found that some agent failures are better handled as deterministic system checks rather than asking another model to interpret what happened. If a tool returns an explicit failure and the workflow continues as if it succeeded, the trace already contains the evidence needed to flag it. That makes these checks much easier to automate and trust in CI.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Exactly — the test is just "is the evidence already in the trace?" If a tool returned an explicit failure and the workflow continued as if it hadn't, the receipt is right there; asking a second model to interpret it adds variance and cost to something already decidable. The judge earns its keep on the fuzzy stuff, not this.

Collapse
 
james_oconnor_dev profile image
James O'Connor

Two pieces of this are doing different jobs and I want to separate them. Exit codes decide whether CI stops, which is what makes the tool usable rather than merely interesting. The rules decide what CI stops for, and that is where I have a question.

Take the second rule, the tool that errored followed by the agent proceeding. Proceeding is doing a lot of work there. An agent that gets a 402, retries with a corrected argument, succeeds, and then answers has also ""proceeded after an error"", and that is the behaviour you want. We logged every rejected tool call for a month and roughly a third of the rejections were our own validation being wrong, so the agent recovering from an error is not rare enough to treat as noise.

So: does tracelint scope the check to an error with no subsequent successful call to the same tool, and if so, what counts as the same tool when the arguments change between attempts? That boundary is where I would expect the false positives to live, and a structural linter earns its exit code 2 by being boring about it.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale • Edited

Yes, it's scoped. R2b only flags "unhandled" when the failing tool has no later call, so your 402 → retry → succeed case isn't flagged — the retry counts. And "same tool" is keyed on name, not args, on purpose: a corrected-args retry is the recovery shape, so keying on args would flag exactly that good behaviour.

One precision on your phrasing — it's a subsequent call, not a subsequent successful one. In a retry chain that keeps failing, the intermediate errors count as retried and only the final unrecovered error surfaces, which is the one that matters.

On earning exit 2: that "unhandled" finding is a candidate — it never fails CI. The only R2 condition that hits exit 2 is narrower and structural: a failed call's value reused as an argument to a later side-effecting call. "Proceeded after an error" stays advisory; "fed a failed result into a payment" is the defect — which is also why your month of logs (a third false rejections) doesn't gate anything.

Honest limit you're right to poke at: name-keying is lenient. If the same tool is later called for something unrelated, it stays quiet and can miss a genuinely ignored error. For an advisory rule I'd rather err quiet than cry wolf on the recovery path.

Collapse
 
mickyarun profile image
arun rajkumar

Suppressing with a stated reason instead of silently passing is the design decision worth stealing here. A clean report with hidden gaps is the failure mode of most check tooling I've used.

Your opener is a payments example so I'll stay there. A 402 the agent walked past is decidable from the trace, agreed. The one that isn't is the tool call that never returned at all. No error status to detect, no result to reuse, just a span that stops. Structurally the trace looks like an incomplete run rather than a defect, and the money may or may not have moved.

Might be worth an R8: a side-effecting call with no terminal status and no subsequent reconciling read. Not "the agent ignored an error" but "the agent never found out." In our world that's the expensive one.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

This is the case I don't have a good answer for yet, and you've named why: it isn't "the agent ignored an error," it's "the agent never found out." Structurally it's an incomplete run, not a defect — a side-effecting span that just stops. tracelint models the unmatched call (a call with no result is a real observable state, not hidden), but there's no rule that flags it, and you're right that for money it's the expensive one. An R8 along the lines of "side-effecting call, no terminal status, no reconciling read" is worth doing — and it composes with an independent-read-back idea another reader raised (was the effect ever confirmed through a path other than the one that wrote it?). The honest hard part is tiering: a call with no result at the very end of a trace is ambiguous — truncated capture vs. a real hang — so it has to fail closed as a disclosed candidate, not assert a defect. Noting it for the roadmap; genuinely useful.

Collapse
 
alexshev profile image
Alex Shev

The implementation detail that matters most is making the assumption visible. For this kind of work I would put the invariant in CI or monitoring, then document the recovery path alongside it. That is how a one-time fix becomes a reliable operating practice.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

Agreed — and the comment above yours is the concrete instance of exactly this. The invariant is "suppressed-count doesn't rise against baseline"; the recovery path documented next to it is "fix the gap, or explicitly commit the new baseline." Putting it in CI is what stops the fix from being a one-time cleanup that quietly rots — the assumption stays visible because every run re-checks it, and the accept-path keeps the escape hatch honest instead of silent. The general rule I'm taking from this whole thread: any place the tool chooses not to look should be a number CI watches, not a footnote.

Collapse
 
alexshev profile image
Alex Shev

That is a strong formulation. A suppressed or skipped path is a product decision, so it should be observable like any other production behavior. Counting it in CI turns “we chose not to look” into something the team can review deliberately.

Collapse
 
sunychoudhary profile image
Suny Choudhary

The interesting next step is using the same deterministic rules twice: CI to catch regressions, runtime to prevent the bad transition in the first place. If a prerequisite tool fails, I’d rather make the next side-effecting action structurally impossible than ask the model whether it should continue.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

The "same rules twice" framing is the sharpest version of this I've heard. Because the rules are deterministic and evidence-based, they can run on the partial trace as a pre-action guard, not just post-hoc — check the structural precondition before the side-effecting call and refuse it, no model in the hot path. The one hard constraint: only the hard-defect tier could ever gate a live action. A candidate blocking a real action would be worse than the failure it's preventing, so the candidate/verdict split stops being a reporting nicety and becomes safety-critical. Different surface than a CI linter, but the deterministic core is exactly what makes it feasible.

Collapse
 
jon_at_backboardio profile image
Jonathan Murray

the suppressed-with-reason output is the best design decision in here and it has one failure mode you'll hit around month three.

suppressions accumulate. a team adds a tool without a schema, R1 suppresses, everyone reads exit code 0 and moves on, and six months later half your rules are suppressed on every run. the report is still technically honest and practically empty. false confidence comes back through the door you left open on purpose.

fix that matches your determinism rule: track suppressed count as a number and fail when it grows. not when it's nonzero, when it increases against a recorded baseline. that's a diff. no judgement, no second model. new coverage gap in this PR, either fix it or explicitly accept it.

same shape as your hard vs candidate split. the count going up is structurally provable. whether the gap matters isn't.

zira's stale progress sequence slots in the same way. a monotonic counter is checkable without anyone's opinion about it.

Collapse
 
ashwin_ugale_102f2abc9cec profile image
Ashwin Ugale

This is the sharpest failure mode anyone's raised, and the fix is exactly right. Suppression is silent debt — R1 skips a schema-less tool, exit 0, and by month three half your rules are suppressed on every run and the report is honest and empty. Tracking the suppressed count and failing on an increase against a committed baseline is the correct shape because it's the same move as the hard/candidate split, one level up: the count rising is a diff, structurally provable; whether the new gap matters is the human call, made explicit by either fixing it or committing the baseline bump. That turns suppression from an escape hatch into budgeted, reviewable debt — a new gap has to be accepted on the record, not absorbed silently. This is going on the roadmap. (And the monotonic-counter-as-checkable-invariant pattern clearly generalizes past this one case.)