Your agent passed its tests. It ran clean in the demo. You shipped it.
Two days later it's confidently returning wrong answers to real users — and...
For further actions, you may consider blocking this person and/or reporting abuse
"Cross-surface / cross-model inconsistency"
I’d put this under regression testing rather than treat consistency checking as a standalone check.
If you swap the underlying model, change a prompt, or modify another part of the agent, comparing outputs across models or repeated runs can catch divergence, but it doesn’t tell you what else changed. The whole regression suite needs to be rerun against the same test cases, because the change that caused inconsistency in one area can silently break behavior somewhere completely different.
This is especially important with agents because a model change can affect tool selection, reasoning paths, refusal behavior, edge cases, and even how the agent handles instructions that previously worked fine.
Consistency is a useful signal, but it isn’t evidence that the rest of the system still behaves as required.
You're right, and this sharpens #8 in a way I under-drew: consistency checking is a symptom detector, not a coverage guarantee. Comparing outputs across models or repeated runs tells you two paths disagreed — it says nothing about whether the paths that still agree are agreeing on the correct behavior, or about everything the change touched that you didn't happen to be watching. Framing it as regression testing rather than a standalone check is the correct move, because it forces the right question: not "did this output diverge?" but "does the whole system still meet its requirements after the change?"
And the agent-specific point is the part that makes this scarier than normal regression. A model swap doesn't just shift the wording of an answer — it can silently alter tool selection, reasoning paths, refusal behavior, and how the agent handles instructions that worked fine yesterday. So the blast radius of a single change is enormous and non-local: the thing you changed in area A breaks behavior in area D, and a consistency check pointed at area A never even looks at D. That's exactly the "silent" property the whole post is about — the failure shows up somewhere you weren't instrumented to see it.
So the correction I'd fold in, with credit: consistency is a useful tripwire, but it's not evidence the system still behaves as required — only rerunning the full regression suite against fixed, known-good cases is. And it ties back to the piece's spine: a green consistency check is another green checkmark that can quietly mean "I didn't look there" rather than "it's fine." The discipline is the same one that keeps coming up in this thread — prove coverage exists, don't infer health from the absence of a visible divergence. Great addition; this is going into the revision.
I’d consider this part of the regression test battery rather than a standalone consistency check.
If you swap the underlying model, change a prompt, or make another change that can affect agent behavior, checking for consistency across models or repeated runs is useful, but it isn’t enough. The regression suite should be rerun against the same known test cases to catch anything else that changed.
With agents, a seemingly small change can affect behavior in areas you weren’t specifically checking for consistency. That’s exactly why we have regression tests in the first place: catch the breakage early, before it becomes a user-facing problem.
Consistency checking is one test. Regression is the safety net.
"Consistency checking is one test. Regression is the safety net." That's the cleaner version of the point, and it corrects a real imprecision in #8 — I framed consistency checking as the catch, when it's really just one tripwire inside a much bigger net. Comparing outputs across models or runs tells you two paths diverged; it says nothing about everything else the change touched that you weren't watching. Rerunning the full suite against fixed known-good cases is the actual guarantee.
And you've named exactly why agents make this worse than ordinary regression: a "small" change has a huge, non-local blast radius. Swap a model and you can silently move tool selection, reasoning paths, refusal behavior, and how it handles instructions that worked fine yesterday — so the thing you changed in one area breaks behavior in a completely different one, and a consistency check aimed at the first never looks at the second. That's the whole "silent" property of the post: the breakage surfaces where you weren't instrumented.
Which ties it right back to the spine — a green consistency check is another green checkmark that can quietly mean "I didn't look there" rather than "it's fine." Regression is how you make coverage explicit instead of inferring health from the absence of a visible divergence. Folding this into the revision with credit; "consistency is a signal, regression is the safety net" is the line.
Point 9 is almost a direct writeup of something we lived through tonight, down to the remediation. We had a CI safety net (auto-open a GitHub issue when a scheduled test suite fails) that had never once actually fired, because a dependency it silently relied on (a GitHub label) didn't exist — so every failure attempt failed a second time, silently, at the alerting step itself. We only found out by deliberately breaking a test to watch the whole pipeline, alert included, end to end. Your "surface the timestamp of its last refusal" idea is exactly the fix we're missing — we proved it can fail once, but we have no ongoing signal that it still can next month. Stealing that pattern.
Really enjoyed this breakdown, @james_anderson_h. The distinction between an agent completing successfully and actually being correct, verified, and authorized is especially important.
I also like the idea of treating “not verified” as an explicit state rather than assuming green means healthy. In agentic systems, observability really needs to cover the reasoning trajectory and intermediate state, not just the final response.
One additional failure mode I’ve been thinking about is context-budget exhaustion: an agent can technically continue running after losing critical constraints or tool context, making the failure look like reasoning drift rather than context loss.
Great write-up — #9 in particular is a strong reminder that our monitoring and evaluation layers need to be tested just as aggressively as the agents themselves. 👏
Thank you — and the context-budget exhaustion failure mode is a genuinely good addition, sharper than how I framed context loss in the post. The nasty part is exactly what you named: it disguises itself as the wrong failure. When an agent loses critical constraints or tool context and keeps running, the output looks like reasoning drift — "the model got confused / went off track" — so you go debugging the reasoning, tuning prompts, blaming the model's logic. But the reasoning was fine; it was reasoning correctly over a truncated world. You're treating a symptom in the wrong organ. The misdiagnosis is arguably worse than the failure itself, because it sends every fix in the wrong direction.
And it's a perfect example of the "not verified" third state you highlighted, applied one level earlier than the output: the agent has no signal that says "I no longer have the constraints I need to decide well." It doesn't know it lost them — eviction is silent, so from the inside, a decision made with the full context and a decision made with half of it evicted feel identical. The fix has to be an explicit check, not a hope: assert that the critical facts (constraints, tool schemas, the original objective) are still present in context at decision time, and treat their absence as a first-class failure state — "operating without required context" — rather than letting the agent quietly proceed and calling the resulting mess "drift." Make the loss observable at the moment it happens, not inferable from weird behavior twenty steps later.
Which lands right back on your last point, and I think it's the most important one: #9 isn't really about the agent, it's about the monitoring itself. Context-budget exhaustion is invisible precisely because most observability watches the final response, not the reasoning trajectory and intermediate state where the loss actually occurred. If your monitoring can't see "critical context was evicted at step 8," it will faithfully report a healthy green while the agent free-associates. The evaluation layer has to be tested as aggressively as the thing it watches — because a monitor that can't detect silent context loss is itself a silent failure. Adding context-budget exhaustion to the revision with credit; "it looks like drift but it's loss" is the line I'd lead that section with.
This is a very clean taxonomy of the gray-failure family, and #9 is the one we've spent the most time actually building - so let me answer your closing question with the mode I'd add, because it sits one level up from the nine: sometimes the silent failure isn't in the agent, it's in the check you built to catch the agent.
Your #9 fix - known-bad through the live path, surface the age of its last refusal - is exactly right, and it has two second-order failure modes that only show up once you ship it:
The canary that goes red for the wrong reason. A known-bad case that fails because the harness threw, not because the guard judged the artifact, is still a green-that-can't-fail wearing a red coat. Feed it malformed input, the parser crashes, the test goes red - falsifiable, and still blind to the artifact. The canary has to fail for the same reason a real violation would, through the same enforcement path, or you've only proven the pipeline can break somewhere, not that the guard can see.
The timestamp that lies about liveness. "Last refusal: 2 days ago" is only trustworthy if the drill writes that timestamp on its success path - after it actually observed the reject - never from the scheduler that launched it. If "the job started" writes the clock, a drill that silently no-ops still leaves a fresh timestamp, and the age alarm shines green over a dead drill. The clock has to be touched by the thing that saw the evidence, not the thing that meant to.
And the 10th mode, the one I'd genuinely add: the check that's green for a reason unrelated to what it's testing. Your guard can be fully falsifiable - goes red on bad input, ages correctly - and still measure the wrong variable. We build a memory layer, so here's ours, measured this week: we hand-built a set of cases specifically to require remembered context, and gated each one so the naive wrong answer is caught. Then we ran a baseline with no memory at all. It solved 78% of them. The cases were fine; the admission criterion was a proxy - "the naive solution is caught" is not the same as "a knowledge-free run fails." The check could go red, it just wasn't testing memory. The fix is the same existence-check discipline you name, pointed one floor up: don't assume your test exercises the capability, measure that a baseline without the capability actually fails it.
Same disease, one floor higher. The green checkmark that can't fail has a cousin - the green checkmark that fails for the wrong reason, and the check that fails for no reason connected to its purpose. All three read as health.
This is the best comment the post has gotten, and it does the thing I most hoped for: it takes #9 and shows that the fix itself has silent failure modes, which means the recursion doesn't bottom out where I stopped it. "Same disease, one floor higher" is exactly right — and you found three floors, not one.
The canary that goes red for the wrong reason is the one I'd already half-missed. A known-bad case that trips because the harness threw is falsifiable and still blind — you've proven the pipeline can break somewhere, not that the guard can see. That's a green-that-can't-fail wearing a red coat, and it's more dangerous than the original because the red reassures you. The discipline you name is the fix: the canary has to fail for the same reason a real violation would, through the same enforcement path. Falsifiable-through-the-wrong-mechanism is just theater with a red light on.
The timestamp-that-lies-about-liveness is the sharper catch, because it's the exact bug from the veto-heartbeat thread wearing a disguise I didn't see. "Last refusal: 2 days ago" is only evidence if the clock is written on the success path, by the thing that observed the reject — never by the scheduler that launched the drill. If "the job started" stamps the clock, a drill that silently no-ops leaves a fresh timestamp, and the age alarm shines green over a corpse. The instrument I proposed to detect dead guards can itself report health while dead. Of course it can. The heartbeat needs its own heartbeat, and the rule that terminates the regress is: the clock must be touched by the thing that saw the evidence, not the thing that meant to.
But the 10th mode is the one I'm going to be thinking about for a while, because it's the subtlest and you brought a measurement, not just an argument. A check that's fully falsifiable, ages correctly, fails through the right path — and still measures the wrong variable. Your memory result is the perfect proof: hand-built cases that "require" remembered context, each gated so the naive answer is caught, and then a no-memory baseline solves 78% of them. The cases were fine. The admission criterion was a proxy — "the naive solution is caught" is not "a knowledge-free run fails," and the gap between those two sentences is where the whole illusion lived. The check could go red; it just wasn't testing memory. That's the most honest thing anyone's posted in this thread, because most people would never run the baseline that exposed it.
The generalization is the keeper: don't assume your test exercises the capability — measure that a baseline without the capability actually fails it. That's the existence-check discipline pointed one floor up, and it's the ablation nobody runs because passing feels like proof. A test your system passes tells you nothing until you've confirmed a system that shouldn't pass, doesn't.
So the taxonomy grows a whole tier, with credit to you:
the green that can't fail (the guard is dead)
the green that fails for the wrong reason (the harness broke, not the guard)
the green whose liveness clock is written by the launcher, not the observer
the green that's fully falsifiable and measures the wrong variable (no baseline ever failed it)
All four read as health. All four need the same fix aimed at a different floor: prove the red is reachable for the reason you claim, through the path you claim, against a baseline that should actually trip it. This is going into the revision as its own section, and if you've written up that memory-baseline result anywhere I'd link it — the 78% number is the most persuasive argument for ablation-as-verification I've seen.
等了你一天，原来你在做这件事！赞一个！
请允许我用中文回复，英语的语义实在有点糟糕，无论怎么翻译都无法表达我的元意！
我用Deepseek和Qwen同时翻译和对比你的文章，为了确保语义没有发生偏移，
我结合两种对比给出的大意反向推论你的文章在表达的元意：
其最终都指向同一个锚点，即AI对人类语言的意图理解，基于这个结论，我的回复如下：
是的！
其本质根源在于AI与人之间存在的根本性鸿沟：一个是工具，一个是涌现（思想）。
所以，要解决这个问题，我们得切换到AI的工作模式和场景中，
用你的意识进入到AI的体内世界，去观察AI体内的五脏六腑内部的运作原理，
只有当你看清楚，并且能够用你的语言描述，
注意，这里我用的是“描述”，不是“解释”；
那么，我们就能够去设计，或者用另一个词“降维”的思考模式去匹配AI的运作模式，
如此才能真正将人类的意图转化为AI能够理解的结构化表达，
这正是我刚刚发布的“[ZSHX-HK] Localized GEO Detection Protocol v1.0”这篇文章在做的事情。
Thank you — this is a remarkably careful reading, and running it through two models to triangulate the meaning before replying is itself the kind of verification discipline this whole series keeps circling. You landed the anchor correctly: underneath the nine failure modes, the real fault line is whether the system has genuinely understood human intent, or merely produced something that looks like it did.
Your tool-vs-emergence framing is sharp, and the distinction you drew between describe and explain is the part I'll be thinking about. Explaining imposes a human model onto the system; describing forces you to observe how it actually operates before you impose anything. That maps onto a theme from my other posts — most failures come from assuming the machine works the way we'd reason, instead of specifying against how it really behaves. "降维 to match AI's operating mode" is a good name for that move: you don't lift the AI up to human intent, you meet it where it computes and structure your intent into a form it can execute.
Where I'd add one note: understanding intent and verifying the action are still two separate problems. Even a system that reads intent well can act on it in an unsanctioned or unverified way — so I'd pair your intent-structuring work with an existence check at the boundary. But that's an extension of your point, not a disagreement. I'll take a look at your ZSHX-HK protocol piece — thank you for engaging this deeply.
Appreciate the thoughtful follow-up, James.
That framing of a fail-closed precondition is exactly how we treat it in practice. Once you stop trusting the transport layer to attest to state changes, the reconciler becomes wonderfully simple: it only has to ask whether the artifact exists with the expected fingerprint. If the read-back fails or is ambiguous, the step doesn't resolve as succeeded or failed — it halts in an explicit unverified state instead of propagating forward.
Glad the concept resonated and looking forward to the updated revision.
"The reconciler becomes wonderfully simple" is the part that surprised me and shouldn't have — that's the tell you got the boundary right. Once you stop asking the transport layer to attest to state, the reconciler collapses to one honest question: does the artifact exist with the expected fingerprint? All the complexity people pile into retry logic and status interpretation was really them trying to infer state from a signal that never carried it. Ask for the fingerprint directly and the inference disappears. Simplicity on the far side of a hard constraint is usually a sign the constraint was load-bearing.
And halting in an explicit unverified state rather than resolving as succeeded-or-failed is the whole thread converging in one design. The fingerprint read-back is the existence check; unverified-as-a-first-class-halt is the third state; refusing to propagate forward is fail-closed. Three ideas different commenters arrived at from different directions, and you've got all three wired into a single step boundary — the step literally cannot lie about progress, because "I couldn't confirm" has somewhere to go that isn't "success." The fingerprint is what makes the unverified state earned rather than guessed, which is the piece that ties it back to the post's spine: prove the effect, don't infer it from the envelope.
Revision's underway and this is going in it — "effect receipts, fail-closed to an explicit unverified state" is the cleanest statement of #1's real fix. Thanks for thinking it through in the open.
There's a tenth one I'd add from entity resolution work: the merge that looks right. The agent returns a high-confidence decision, schema validates, payload non-empty; every observable signal is clean, but the two records it merged are actually different companies, and you don't find out until ops notices contradictory data weeks later. We fixed it by re-querying source records after every merge and treating the decision as a hypothesis to verify, not a result to trust. Six weeks of silent graph corruption before we caught it.
This is a real tenth, and it's nastier than any of the nine because it clears every signal I listed. High-confidence decision, schema valid, payload non-empty — all green — and the merge is still wrong, because "well-formed" and "true" are orthogonal and none of the observable signals were ever measuring the second one. The others at least leave a trace (an empty body, a broken handoff, a loop); yours leaves a perfectly healthy record that happens to describe a fiction. Two different companies collapsed into one, and every instrument says success.
The six-weeks-of-silent-graph-corruption detail is the part that should scare people, because it exposes the compounding property that makes merge errors special: a bad merge doesn't just sit there being wrong, it becomes the substrate the next decisions reason over. Every downstream inference inherits the corruption and looks locally consistent doing it, so the graph gets more internally coherent as it rots — which is exactly why ops caught it through contradictory data rather than any check firing. The failure metastasizes through the healthy trace, and by the time it surfaces the blast radius is weeks deep.
"Treat the decision as a hypothesis to verify, not a result to trust" is the fix, and it's the post's spine aimed at the one place a confidence score is most seductive. Re-querying source records after every merge is the existence check made concrete: the model's confidence is a claim about the world, and the only thing that can confirm it is the world, read back. High confidence is not evidence — it's the thing you have to go check against evidence, and a merge is precisely where that distinction is easy to skip because the number looks so reassuring. Going into the revision as #10 with credit — "the merge that looks right" is the cleanest example in the whole set of a failure that passes every quality check and fails the only one that mattered.
The idea of making “not verified” a real state is probably the most useful takeaway here. In our AI work at IT Path Solutions, we’ve found that treating anything without sufficient evidence as simply “successful” creates a dangerous blind spot. A system should be able to distinguish between passed, failed, and we don't have enough evidence to know yet. That third state can make monitoring much more honest and actionable.
That third state is the whole thing, and you've named why it's not just a nice-to-have: without it, "we don't have enough evidence to know yet" silently collapses into "successful," and that collapse is the blind spot every failure in the post lives in. Two genuinely different situations — verified good and unverified — get rendered as the same green pixel, so the monitor is technically honest about what it checked and completely misleading about what it didn't. A system that can only say pass/fail is forced to lie by omission every time evidence is missing.
What I like about how you put it is that the fix is representational before it's procedural: the system needs a place to put "unknown" before it can act on it. Once passed / failed / insufficient-evidence are three distinct states, monitoring gets honest and actionable in the same move — you can alert on the third one, quarantine outputs that carry it, refuse to auto-promote anything that hasn't cleared it, and actually see how much of your "green" is verified versus merely un-red. Most dashboards would look dramatically worse the day you split that state out, which is exactly the point: they were never that green, you just had no cell for the doubt.
The one discipline I'd add from the thread above: make the third state fail-closed, not fail-open. "Insufficient evidence" has to block or flag by default, because if unknown quietly flows through as if it were passed, you've rebuilt the two-state system with extra steps. Honest monitoring means unverified is treated as a problem until proven otherwise, not a success until proven wrong. Great addition — the passed/failed/unknown framing is the cleanest statement of the post's spine anyone's offered.
The HTTP 200 empty payload is especially nasty when tool callers treat exit status zero or a 200 response code as equivalent to state reconciliation. In our runs, we had to start requiring external effect receipts where the engine reads back the verified artifact or mutation before allowing the step to resolve.
When you decouple task execution from state verification, agents stop hallucinating forward progress after a silent network blip or empty body. It completely eliminates downstream error poisoning because the next tool step simply refuses to run on unconfirmed assumptions.
"Exit status zero as equivalent to state reconciliation" is the precise mistake, and naming it that way is sharper than my #1. A 200 or an exit 0 is a statement about the call, not about the world — it says the request was accepted, not that the intended effect happened. Treating the two as the same is how an agent hallucinates forward progress: the transport succeeded, so it assumes the state changed, and reasons onward from a mutation that never landed. The gap between "the call returned" and "the effect occurred" is exactly where a silent network blip or an empty body slips through.
External effect receipts are the right fix, and I like that they're a read-back, not a retry. Retrying trusts the same unconfirmed signal harder; reading back the verified artifact or mutation before the step resolves replaces the assumption with an observation. That's the existence-check discipline from the post pointed at state instead of response: don't ask "did the call succeed?", ask "can I now observe the thing the call was supposed to produce?" Two different questions, and only the second one is evidence.
The part I'd underline is your last point, because it's the whole error-propagation problem (my #2) dissolved at the source. Poisoned context travels because step N+1 runs on step N's claimed output; if N+1 refuses to run on unconfirmed state, the poison has nowhere to travel — the chain breaks at origin instead of surfacing twenty steps downstream as an untraceable wrong answer. You've effectively made "unverified" a fail-closed precondition between steps, which is the state-contract idea another commenter raised, enforced through the receipt. Decoupling execution from verification is the move; the receipt is what makes the decoupling real instead of aspirational. Going into the revision with credit — "effect receipts, not status codes" belongs in #1.
The “unverified” state is probably the most important idea here. I’d take it one step further: in multi-step agents, observability should track state validity, not just events. A tool can return a perfectly valid schema while producing a semantically invalid state for the next step. That’s how poisoned context travels cleanly through an otherwise healthy trace. Putting explicit state contracts between critical steps gives you an earlier failure boundary, and the trace then tells you exactly which contract was first violated not just where the final answer went wrong.
"Track state validity, not just events" is the upgrade the whole post was reaching for — thank you for naming it precisely. My failure modes mostly describe events going wrong (a 200 with no payload, a poisoned handoff), but you've pointed at the deeper layer: the trace can be full of individually valid events and still carry a semantically invalid state between them. A tool returns a perfectly conformant schema whose contents are wrong for the next step, and because every event validates, the poison rides a completely healthy-looking trace all the way to the final answer. Schema-valid and state-valid are different claims, and observability that only checks the first is exactly the "green checkmark that means 'I didn't look there.'"
The explicit state contracts between critical steps are the right mechanism, and the payoff you describe is the part I'd underline: it moves the failure boundary earlier and makes it localizable. Without contracts, a poisoned state at step 3 is invisible until the answer looks wrong at step 20, and now you're reverse-tracing nineteen steps to find where it started. With a contract at each boundary, the trace tells you which contract was first violated — you get the origin, not just the symptom. That's the difference between "the output is wrong somewhere upstream" and "step 4's output violated the precondition step 5 required," which is a debugging experience in a different universe.
It also connects to a point another commenter made about regression: a state contract is a standing existence check wired into the live path, not a test you run once. It asserts "the state I'm about to act on actually satisfies what this step requires" every time, which is precisely the "prove it, don't assume it" discipline the post keeps circling — applied at the seam between steps instead of at the final output. Contracts turn "did the event happen?" into "is the state I'm carrying still valid?", and that second question is where the silent failures actually live. Going into the revision with credit — this is one of the most useful additions in the thread.
Thanks for sharing such a detailed post. You made a very valid point regarding data security and scalability in AI adoption. Many enterprises struggle with off-the-shelf software, which is why bespoke AI solutions built by an experienced custom AI software development company are becoming a necessity. Keep sharing such informative content!
2 (poisoned step) is the one that took us longest to instrument correctly. per step schema validation catches obvious cases. the harder version: the intermediate output is schema valid but semantically wrong, and the agent treats it as ground truth. only caught it when we started logging the full argument envelope at every tool call and diffing against expected value ranges.
the 'success'/'correct' gap you describe is sharpest at step 2. the monitor sees completion; nobody sees the wrong customer ID flowing into step 3.
do your empty 200 failures cluster by tool, or are they uniform? clustering changes the fix completely.
The semantically-valid-but-wrong intermediate is the version that separates people who've run agents in production from people who've read about them — and you clearly have. Schema validation catches "is this the right shape," which is the easy half; it says nothing about "is this the right value," which is where the poison actually lives. A well-formed envelope carrying the wrong customer ID passes every structural check and becomes ground truth for step 3, and now the corruption has a clean bill of health as it propagates. Your fix is the right one and more expensive than people expect: logging the full argument envelope at every call and diffing against expected value ranges is the difference between "a value arrived" and "a plausible value arrived" — and only the second one catches the wrong-but-conformant case. Expected-range diffing is essentially a semantic contract at the boundary, which is exactly what schema validation isn't.
And yes — "the monitor sees completion; nobody sees the wrong customer ID flowing into step 3" is the sharpest single statement of the success/correct gap anyone's given me. Step 2 is where it's most dangerous precisely because it's early enough to corrupt everything downstream and quiet enough that nothing flags it until the final answer looks off nineteen steps later.
On your question — this is a great one and the answer matters more than I gave it credit for in the post. In what I've seen, empty 200s cluster hard, they're not uniform, and the clustering is the diagnostic. They concentrate in (a) specific flaky upstream tools/integrations returning a 200 wrapper around an empty or error body, (b) pagination/edge boundaries where "no more results" gets encoded as success rather than an explicit empty state, and (c) auth/rate-limit responses that some wrappers dress up as 200. You're completely right that clustering changes the fix: if it's concentrated in one tool, you fix it at that tool's adapter — normalize its contract so empty-but-200 becomes an explicit failure — which is cheap and surgical. If it were uniform, you'd need a global response-validation layer wrapping every call, which is heavier and blunter. So the first move is exactly what you're implying: don't fix empty-200s generically until you've grouped them by tool, because uniform-vs-clustered is the difference between "harden one integration" and "instrument the whole call layer." Curious whether your envelope-diffing surfaced the same clustering, or whether yours spread differently — that'd tell us if it's a property of the tools or of the harness. Going into the revision with the semantic-vs-schema distinction credited.