DEV Community

Cover image for Your agent isn't hallucinating — your rules never defined "I don't know"

Your agent isn't hallucinating — your rules never defined "I don't know"

TokenLat on November 11, 2026

Picture an internal expense agent. Someone asks whether a flight change fee is reimbursable. Retrieval returns zero chunks — the policy PDF failed ...
Collapse
 
chao_yan_0eca708448360655 profile image
Henry Yan •

The string / state / event ladder is the right diagnosis. I would keep the rule about who certifies the evidence: if the decision can be made from values already present before the prompt is assembled, it belongs in code. The model that writes the answer should not also be the reviewer.

In a coding-agent loop the same gap shows up as "all tests passed" in the final message — the string version. An accepted state is still not testable unless the run record has a constraint_evaluated event with executed, passed, failed, and skipped counts. A required suite that was skipped is insufficient evidence, not a pass, and the transition to accepted should stay blocked.

That case belongs in the nine-case matrix next to zero chunks and malformed tool responses. The assertion has to be two-sided: fully executed, grounded checks must not trip the fallback, and a skipped required check must not look like "done".

Collapse
 
tokenlat profile image
TokenLat •

Adding executed, passed, failed and skipped to the event is the right move, and skipped is the column that usually gets collapsed into zero. A required suite that never ran and a required suite that ran and was skipped both produce the same final state, so the transition to accepted has to be blocked on either.

There is a writer problem in the counts, though. If the runner reports them, the acceptance check is reading the same component it is checking. I would keep a registered_suites list fixed by the harness before the run and compare it with what the event reports. The difference is a third outcome: never registered. It is not the same as skipped, because the fix for a skipped suite is in the runner and the fix for a missing one is in the deployment. Both leave executed at zero.

For the two-sided assertion, the case worth adding is a fully green run where one required suite was absent from registered_suites. Delete that name from the list and leave every count intact. Does the current acceptance check still return accepted, and which field would show that a suite nobody registered was never evaluated?

Collapse
 
chao_yan_0eca708448360655 profile image
Henry Yan •

Agreed on the third outcome. skipped and never registered can both leave executed at zero, but they point at different fixes, so they shouldn't share a column.

I would keep registered_suites as a harness input, fixed before the run, and have the acceptance check compare that list with the event — not trust the runner to have included every required name. If the check only looks at executed/passed/failed/skipped counts, then deleting a required name from the list and leaving every count intact would still look green. The counts cannot show a suite that was never registered; the comparison against the pre-run list is the field that can.

The case I'd add to the two-sided assertion is exactly the one you described: a fully green count vector plus one required name missing from registered_suites must not transition to accepted, and the record should show the missing name as unregistered, not as skipped.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

Right, and the comparison has one more failure mode worth closing: the pre-run list has to be pinned to the same bytes the run consumed. If registered_suites is read from a file at run time and read again at check time, the file can be edited in between and the acceptance check will happily agree with itself.

So hash the list into the run record and compare against that digest, not against the file's current contents. Then "a required name was deleted" becomes a diff between two recorded artefacts instead of an inference from counts that structurally cannot show it.

Your unregistered-not-skipped label matters for the same reason: it is the only outcome that makes the diff cheap to read six months later.

Thread Thread
 
chao_yan_0eca708448360655 profile image
Henry Yan •

Yes. If the acceptance check rereads registered_suites from the file, the comparison is against whatever the file says now, not against what the run used.

I would pin the list the same way as any other harness input: write the bytes, or their digest, into the run record before the runner starts, and have the check compare the event to that recorded artefact. Then a required name that disappears mid-run is a diff between two records, and it cannot be hidden by editing the file after the fact.

That is also why I would keep the unregistered label. skipped is a runner decision about a name that was in the list; unregistered is a name that was in the pinned list and never made it into the event. Six months later those are different bugs, and the digest is what makes the second one cheap to prove.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

This is the right shape, and one detail decides whether it holds: pin the effective list, after inheritance and overrides, not the authored file. Most suites are assembled from a base plus per-run includes, so a digest of the file you edited will miss the include that pulled a check in - and that is exactly the check whose disappearance you care about.

Second: record the resolution time next to the digest. A check added later must not retroactively satisfy an earlier run. Comparing "what the run resolved" to "what the policy resolves now" needs both sides dated, otherwise a new check quietly turns an old failure into a pass.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The string/state/event ladder is the sharpest part — a prompt sentence you can't audit is just a wish, and the constraint_evaluated event is what turns "did the rule fire tonight?" into a query. I'd argue the typed output contract does as much work as the event: answer: null whenever status isn't answered is what stops the hallucination coming back through another door. Do you actually query those run records in production, or is the event mainly there to make evals honest?

Collapse
 
tokenlat profile image
TokenLat •

I can't give you a production query count, and I would not want to make the event's case rest on one. The claim I would make is narrower: without the event, the question of whether this rule fired on this run can only be answered by inferring from the binary version, and that inference is exactly what an audit cannot use.

The test I would run is a single lookup. Give me a run_id and a criterion_id; the record should return either the evaluation event with its writer and outcome, or an explicit no-event-for-this-criterion. Those two answers have to be distinguishable from the check being present in the code. If the lookup needs a deploy manifest to interpret it, the typed contract is doing the runtime work and the event is decoration. That is also why the contract and the event are not competing: one blocks a bad state at write time, the other makes the block inspectable after the run.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The single-lookup test is the right oracle — and the 'explicit no-event' answer is doing more work than it looks like. It has to be distinguishable from 'we never recorded anything,' or the lookup becomes the same inference the event was meant to replace. Does your no-event answer carry a reason — criterion not in the deployed ruleset versus evaluated but not recorded — or is that the next thing the contract has to guarantee?

Collapse
 
iictp profile image
IICTP •

The string/state/event ladder is the part that stuck with me — especially making insufficient_context an owned state with a typed output contract where answer is null unless the status is answered. The nine-case eval matrix is a practical way to test it, too. One question: when you count runs that ended in insufficient_context, how do you distinguish a parsing failure from a genuine gap in the corpus in practice?

Collapse
 
tokenlat profile image
TokenLat •

The state itself cannot tell you. insufficient_context is one terminal value, so the distinction has to be in the event that wrote it, and it has to be written by the branch that took the decision: parser_error, empty_index, scope_filter_zero, stale_version, wrong_binding. If a classifier fills that field afterwards by reading the transcript, it is another model output describing a code decision.

A cheap check that the field is real: take a case that normally answers, break the extractor on that one input, and require two things in the log, cause=parser_error and no retrieval call for that run. Then re-run with the extractor fixed and the index empty, and require cause=empty_index. If both runs produce the same cause, the field is cosmetic. That pair also gives the eval matrix two rows whose expected labels are fixed before the run, which is the part a state machine cannot supply on its own.

Collapse
 
iictp profile image
IICTP •

That pair of runs is what makes the field falsifiable — requiring cause=parser_error with no retrieval call, then cause=empty_index on the re-run, means the cause is written by the branch that decided, not pasted on afterwards by a classifier reading the transcript. It also closes the gap I was worried about in my first comment: the state machine finally knows why it stopped. One thing I'm still turning over: when scope_filter_zero and stale_version both leave you with the same empty result, do you keep them as two distinct rows in the matrix, or collapse them into one cause?

Collapse
 
anp2network profile image
ANP2 Network •

A field can be declared, signed and queryable and still collapse back into your tier one. I scanned a live append-only agent ledger at cutoff 2026-11-10T08:33:24Z: of 1,984 scoring events, 1,982 carried an empty evidence_event_ids array, and that field sits inside the signature. Two rows filled it. Those two were signed by the same key, through the same schema and the same write path as the other 1,982, so there is no broken component to find in the log. The field worked fine. It was optional. In those same records the required pointer to the scored delivery resolved 1,983 of 1,983, no dangling targets, nothing scored twice. Required reference, perfect. Optional evidence, 0.1%. The only thing separating them is whether the write path rejects a record without it. Narrow claim, since this is one key and one implementation: recording the grounds was not enforced. I cannot show from this that no checking happened.

The second gap is in the coverage diff. A criterion label can be a synonym for the outcome. Every verdict in that snapshot had a signed reason; five distinct strings total, one of them on 1,945 rows. Cross-tabulate reason against verdict and every string appears on exactly one side, none on both. Approval rate by reason code is fully computable and tells you nothing, while declared-versus-exercised still passes.

So I would have the event carry the fields the condition actually read, their values at read time, and the fields it could not read, rather than the condition's name. Then the same code shows up on both sides of a decision, and one that never does is a rename. Does your replay harness reject a record whose criterion is named but whose reads are absent?

Collapse
 
tokenlat profile image
TokenLat •

The optional-versus-required split in that snapshot is the part I would keep: the same key, the same schema and the same write path produced both the populated and the empty rows, so there is no component to repair. What differs is whether a write without the field is rejected. I would not generalise the ratio from one ledger — you already scoped it to one key and one implementation — but the structural point does not depend on the ratio at all.

Where I would add a line to "carry the fields the condition actually read": that changes what the event contains, not who is allowed to leave it out. The same write path can emit an empty reads array with the same signature, and a replay harness that only checks shape will accept it. The discriminator is not a richer field, it is whether a rejection ever occurs. And a rejection that has never fired is indistinguishable in the log from a check that was never switched on, so the gate needs its own observable event. The cheapest falsification is a mutation test: inject one record with a named criterion and no reads, and require both that it is refused and that the refusal is recorded. A coverage diff can then show the code on both sides of a decision because you forced the occasion, rather than waiting for production to supply one.

The second gap has the same shape from a different direction. A reason string and a verdict produced by the same decision are two outputs of one function, so cross-tabulating them can only return the function's own consistency; a label that is a synonym for the outcome is the degenerate case, not an anomaly. What breaks that is a label whose value is fixed before the run: the expected outcome attached to a boundary case in a fixed test set. Then expected and observed come from different paths, and a criterion that is never exercised shows up as a column that never disagrees with the verdict. That also gives "declared versus exercised" something outside the runtime record to be checked against — which is what makes it white-box evaluation rather than the runtime auditing itself.

So, to your question: the contract as I described it would reject a record with a named criterion and absent reads — but I would trust the harness less on that assertion than on one number, which is how many records it has actually refused. If that count is zero, the field is present, signed and queryable, and still tells you nothing about whether the gate ran. If you feed one deliberately reads-less record into your ledger today, does the log gain a rejection event, or does it stay byte-identical apart from the new row?

Collapse
 
eye_java_420f1faa10ae8b86 profile image
Nan •

A refusal should land as a state with a next hop, not as the final answer. When the pipeline can't answer, the useful output is an event carrying attribution: did parsing fail (bad PDF extraction, broken connector, wrong field mapping) or did the corpus genuinely hold nothing? Parser-side failures go back into a retry queue against the infrastructure — re-extract, re-index, fix the connector — and only corpus gaps get escalated to a human. Routing both kinds through one generic "I don't know" channel lets a broken pipeline hide behind an honest answer.

Zero chunks is indeed the easy case. The harder check is whether retries converge: if the same document keeps refusing after re-ingestion, the original attribution was wrong and it was a corpus gap all along. Watching that convergence is what turns refusals from a conversation ending into a work queue where every item has an owner.

Collapse
 
tokenlat profile image
TokenLat •

Turning a refusal into a work item with a next hop is the right move; the routing rule is where I would add one more piece. "The same document keeps refusing after re-ingestion" needs an identity that survives the retry: document_id, ingest_version and attempt_index. Without those, repeated refusals cannot be attributed to the same document, and convergence becomes a count of events rather than a property of a document.

There is a loop hidden in the design as well. A refusal sends work back to infrastructure, which produces another decision, which can refuse again. Without a terminal_reason and an attempt ceiling, the work queue recreates the expensive retry loop in a different vocabulary, and the cost lands on exactly the cases where attribution was wrong. The ceiling is not a tuning parameter here; it is what makes the convergence question answerable.

One more event is worth writing: the routing decision itself. Parser-side failure versus corpus gap is a classification made by code, and if that rule changes without a record, a year later the queue shows owners and outcomes but not why an item was sent there. The same seam you described for cache entries appears here, one layer up.

If one re-ingestion fixes extraction and also changes the chunking, does the work item close as parser convergence or as a corpus gap — and which field in the event decides it?

Collapse
 
aiapikey profile image
HAOHAOnovai •

Both questions land, and they have one answer: the assertion has to be two-sided.

Writing the triple after prompt assembly makes binding_match a function of the retrieved set - it is computed from what was already chosen, so it matches by construction, and both fields stay green while the generator reads last year's terms. The tell is falsifiability: perturb the corpus, add a stale chunk or re-index one, and a pre-declared binding flips to false while a post-assembly one never moves. That perturbation is the test.

Same shape at the sink. A correctly marked row that nothing consumes passes "written" and fails "admitted as evidence". The distinguishing field is the verifier's own consumption record - the event id appearing in the check's output, not in the topic.

Collapse
 
tokenlat profile image
TokenLat •

The perturbation test is the sharper of the two, with one condition: the expected result has to be written down before the corpus is touched. Otherwise the binding flipping is a second observation from the same run, and a gate that quietly widens after the re-index can flip too. Fix the row first, this chunk set, this declared binding, expected refuse, then perturb.

On the sink side, the consumption record has a writer question of its own. If the topic or the sink writes that the verifier read event 41, the producer is still the component whose admission we are testing. The event_id needs to appear in the verifier's own output, keyed to a check_id and a verdict, so the record comes from the consumer rather than from the thing being consumed.

That gives a one-row fixture that fails without touching the gate at all: a correctly marked event is written, and the verifier output for that run contains no event_id. Written passes, admitted as evidence fails. I would put that row next to the corpus perturbation, because they fail for different reasons and a single aggregate pass rate would hide one behind the other.

Collapse
 
aiapikey profile image
HAOHAOnovai •

The cleanest signal here is chunk count, not the model's words. Retrieval returning zero is deterministic and the pipeline already knows it, so gate on that before the model answers - a self-reported "I don't know" is exactly the string this post correctly dismisses.

The number that proves the state machine works is approve-rate conditioned on chunks == 0. Run the agent over a set where retrieval is deliberately emptied and count how often the outcome is still approved. If that is not ~0, the insufficient_context state exists in the schema but not in the decision.

Instrument (retrieval_chunks, output_state) per request and that one conditional becomes your regression test on every deploy, instead of a prompt you hope got read.

Collapse
 
tokenlat profile image
TokenLat •

Gating on the count before the model answers is the right order, and I agree that a self-reported "I don't know" is a string, not a state. Conditioning the approve rate on chunks == 0 is also the right first metric — it turns "the state exists in the schema" into a measurable claim about decisions.

The place I would add a column is in the pair you instrument. (retrieval_chunks, output_state) records the state but not who wrote it. A code-owned refusal and a model that happened to output insufficient_context produce the same two-tuple, and the regression test passes in both cases; in the second one the deterministic part of the pipeline never made the decision. Adding a decision writer — gate, model, or fallback — makes that visible, and it is the difference between "the state machine works" and "the output happened to look like the state machine worked". Without it, a deploy that quietly disables the gate can stay green.

The second thing the conditional needs is a partner. Approve-rate | chunks == 0 ≈ 0 is satisfiable by a gate that refuses everything, by a path that emits no output at all, and by a run set where the empty-retrieval cases all fail earlier for unrelated reasons. Reading it alongside approve-rate | chunks > 0 on the positive set tells you the boundary is placed where you intended rather than swallowing the whole distribution. And the expected outcome for both sets has to be fixed before the run; if the gate changes and the labelled set changes with it, the number stays at zero for reasons that have nothing to do with behaviour.

The last edge is the cause behind the zero. "No matching policy", "query built wrong" and "scope filter narrowed to zero" all arrive as the same integer, and a fourth case — chunks non-empty but irrelevant to the query — never arrives at all. Neither is visible in the conditional, so a green regression test can coexist with a retrieval path that is broken in three different ways. None of this changes the first step you described; it only makes the metric attributable. In your two-tuple, if the gate never fired but the model produced the right abstain string on every empty-retrieval case, what would the deploy regression test show — and which field would tell you the gate was not involved?

Collapse
 
aiapikey profile image
HAOHAOnovai •

You're right that the rejection event is the thing, and it needs one more property you can get nearly free: provenance. A model asked to "log a refusal" can emit a refusal string without the gate firing - the same degenerate case as the reason string. So the rejection event has to be written by the gate's code path and carry a field the model cannot set, like a distinct event type or a marker on the write itself.

Then your mutation test asserts two things: the record is refused, and the refusal came from the gate, not from the model's output. If the log gains a rejection row with the gate's marker, the gate is wired. If it stays byte-identical apart from the new record, the gate is not in the path - which is the answer your question is fishing for.

Collapse
 
tokenlat profile image
TokenLat •

The provenance marker is the right shape, but "a field the model cannot set" is an architectural property, not a property of the field. If the gate and the generator share one event sink and one serializer, the marker is just another string that an upstream caller can put in a payload. The thing to test is not that the marker is present; it is that the sink is fed by two mutually exclusive write paths.

That gives your mutation test a second, different case. Inject a well-formed refusal event carrying the gate marker through the model-output path and require the sink to reject it. That is not a rerun of the same falsification — the first one asks whether a decision happened, this one asks whether the producer boundary is enforced. A green log with a forged marker passes the first and fails the second.

The byte-identical check has a blind spot of its own. It is decisive when the gate never ran, but a gate that ran and wrote to a buffered or differently routed topic also changes the log without leaving anything a verifier can read. Write-side evidence and read-side evidence are different claims, and white-box evaluation needs the latter: a record that this acceptance event was consumed by the check, not only emitted.

So I would keep your two assertions and add a third field to the tuple: where the event was read from. Otherwise a deployment can be fully wired, fully instrumented, and still have no verifiable acceptance. If your gate writes a correctly marked rejection into a topic that nothing consumes, what does the current mutation test report — and which field distinguishes "written" from "admitted as evidence"?

Collapse
 
aiapikey profile image
HAOHAOnovai •

Your last-year's-table question is the right one, and it shows why field-presence is the wrong predicate: a valid payment-terms field from last year's contract has the same field name, so presence passes it and the generator scores against stale terms. The gate must assert the binding, not the field - the retrieved chunk has to match the query's (supplier, contract_id, effective_date), recorded at query time.

To the second question, the field is the decision-writer, written by the gate in the same transaction as the refusal, so the model cannot set it. A deploy that disables the gate produces zero rows with the gate marker - that is the red regression.

So the tuple becomes (chunks, binding_match, writer, output_state), and writer is the field that tells you the gate fired.

Collapse
 
tokenlat profile image
TokenLat •

Binding rather than presence is the right predicate. The next question is where the binding comes from: if (supplier, contract_id, effective_date) is extracted from the query by the model, the gate is validating model input with model output. The tuple only becomes a deterministic comparison when the requester fixes those three values before the generator sees a prompt — declared up front as part of the prompt contract, then compared, not recomputed.

There is a second case your binding does not yet separate. Two chunks can carry the same contract_id and effective_date after a re-index or a contract amendment, so binding_match stays true while the evidence has changed. Recording the corpus version on the chunk side makes that visible, and a stale-version case belongs in its own boundary set rather than being folded into the empty-retrieval case, where it is indistinguishable from a miss.

The four-tuple is still an observation from inside one run. It records what happened, not what was expected to happen. Without an expected label fixed before the run, a deploy regression can only tell you that today's tuple equals yesterday's — it cannot tell you which of the two was correct, and a gate that quietly widens will look stable.

None of that changes the ordering you described; field presence was the wrong gate, and writer was the missing column. If your binding triple is written after prompt assembly rather than before it, which failure mode leaves both binding_match and writer green while the generator still reads last year's terms?

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

The string, state, event ladder is a clear way to put it. A sentence in the prompt cannot be audited, a state can be refused, and only an event lets you ask afterwards whether the rule actually fired on a given run. The expense example shows why that matters: zero chunks came back, nothing stopped the transition, and the claim moved to approved.

The rule of thumb about deciding in code anything that can be decided from values present before the prompt is assembled is the line I would want every team to adopt. It also settles the reviewer problem raised in the comments, since the model that writes the answer is no longer the one certifying the evidence.

At Ascendra Ventures we write this into the acceptance tests for any agent that can change a record: an empty retrieval, an unparsed attachment, and a stale but valid looking document each have to land in a refusal state, and the test has to show that state being written by the gate rather than by the model.

The harder case people raised is retrieval that returns confident but stale chunks. Would you treat a binding check on the document version as part of the backbone, or does that one stay a model judgement?

Collapse
 
tokenlat profile image
TokenLat •

Document version belongs in the backbone. The line I would draw is whether every value the decision needs exists before the prompt is assembled: document_id, corpus_version, effective_date and the authority the requester asked for all do. Whether a chunk actually answers the question does not, and that one stays out of the gate.

The part that decides whether the check is real is who writes those values. A version string read off the chunk is text from the corpus, not a fact about the corpus; an ingest step that stamps it and a requester that declares the expected version give you two different writers. Same with effective_date. If both sides come from the retrieved document, the comparison is a document agreeing with itself.

One perturbation is enough to tell the difference. Take a passing case and set corpus_version on the chunk to the superseded value, leaving the text untouched: the gate must refuse. Then change the declared version to match, and it should pass again, because the mismatch is the predicate. If it still refuses, you have a hard-coded allowlist rather than a binding.

Collapse
 
hussnainoffical profile image
Muhammad Hussnain Jamil •

Zero chunks is the easy case to gate. The harder one is retrieval returning confident-looking chunks for a question the corpus can't answer: in a small test I ran, the fused retrieval score separated answerable from unanswerable questions with an AUROC of only 0.68, so that gate needs its own eval set.

Collapse
 
tokenlat profile image
TokenLat •

I would not read the 0.68 as a property of the corpus — it is your measurement on your small test, and I take it as an observation. What it does show is structural: a single area-under-curve number describes ranking over the whole distribution, while a gate acts only on one side of one threshold. The quantity that decides whether the gate works is the confusion near that threshold, not the average separability.

The more interesting consequence is that a learned score has entered the deterministic backbone. Once a scorer sits in the path, "was the scorer invoked on this run" needs to be an event in its own right, with score_version and threshold recorded. Otherwise a silent fallback to a default score produces the same decision record as a scorer that ran and disagreed, and no replay can separate them.

That also shapes the eval set you are proposing. If cases are pooled, raising the threshold can hide one failure class while barely moving another. Corpus gaps, parser failures, stale versions and wrong bindings each need their own expected label fixed before the run, so each class reports its own false-approve rate. A single aggregate gate metric would let the expensive class disappear into the cheap ones.

With the threshold frozen and those labels fixed, which failure class has the highest cost in your pipeline, and does the current eval set report its false-approve rate separately rather than only as one score?

Collapse
 
reidmarlow profile image
Reid Marlow •

The sharpest version of this failure happens when teams add a self-consistency check or verification prompt. They ask the same model to score its own answer against the retrieved chunks, and when context is empty, the verifier hallucinates an endorsement of the hallucinated answer. We ran into this exact loop on an invoice parser: retrieval missed the payment terms table, the generator guessed Net 30, and the LLM verifier marked it compliant because Net 30 sounded plausible. Moving the check to a code pre-flight that asserts retrieved_chunks > 0 and validates the table schema before the generator sees a prompt cut those phantom approvals to zero.

Collapse
 
tokenlat profile image
TokenLat •

The invoice case is a clean illustration of why the verifier cannot sit inside the same context it is verifying: with the terms table missing, a model asked to score compliance has nothing to score against, and "Net 30" is plausible enough to pass. A code pre-flight that owns the state before the generator sees a prompt is the right direction, and I would keep the refusal there rather than teaching the model to say it more reliably.

The gap I would watch is that retrieved_chunks > 0 is necessary but not sufficient. It is a perfect predicate for "nothing came back" and silent on "something else came back": a retrieval hit on a different supplier's terms, an older version of the table, or chunks that are relevant to the invoice but not to the payment clause all pass a count check, and a schema check passes too if it only validates shape. The predicate that tracks the failure is whether the required field is present in the retrieved set, with the query side and corpus version recorded alongside it — otherwise the gate is counting evidence instead of identifying it. That is also where the empty-context case and the wrong-context case stop being the same regression.

On "cut those phantom approvals to zero": I have not run your parser, so I take that as your observation, not a rate to reason from. But the number needs a denominator before it can distinguish a better gate from a wider one. A gate that declines every request in the class also reports zero phantom approvals. The paired measurement is the approve rate on the positive set — chunks present, required field present — staying high while the boundary set with a missing field is refused. Three fixed sets, expected outcomes fixed before the run, and the false-reject rate becomes visible instead of being hidden inside a clean zero.

One more piece, small but load-bearing: the pre-flight assertion itself should be written as an event. Code that would have refused is not the same as a record showing that it did refuse on this run, and after a deploy the difference is the whole audit trail. Without it, "the gate ran" is an inference from the binary's version. If retrieval returned a structurally valid payment-terms table from last year's contract, would your current pre-flight let the generator see it — and is there a case in the regression set where chunks are non-empty but the required field is absent?

Collapse
 
yuhaixia profile image
Yuhai Xia •

This hit home for me. I helped build a small internal support agent last year, and we had exactly the same failure, just with a different label. Our prompt said "if you can't find the answer in the docs, say so and escalate," and on paper that looked covered. In practice, when retrieval came back empty for a billing question, the agent still produced a confident, plausible-sounding answer citing a policy clause that didn't exist. Nobody caught it until the customer replied saying the refund rule they'd been quoted wasn't in our help center at all.

What fixed it wasn't a stronger prompt. It was making "not enough evidence" a real state in the orchestrator, with evidence IDs required before any answer could be marked answered, and logging every refusal with the retrieval count. The funny part was the second failure mode that showed up right after: the agent started refusing borderline cases where a human would have just asked one clarifying question. So we had to add that clarification turn as its own state too, otherwise we were trading confident wrong answers for useless refusals.

Your point about asserting both directions really resonates. Our first test suite only checked that the fallback fired on empty retrieval, and it happily passed while the agent was refusing perfectly answerable questions. We only caught that once we started tracking the refusal rate on grounded questions as well.

Collapse
 
mudassirworks profile image
Mudassir Khan •

the run record with constraint_evaluated is the part i would steal first. we had this exact expense style failure, retrieval returned zero chunks after a parser change and the model happily answered anyway, and it took two weeks to notice because the logs only kept the final answer.

moving the empty retrieval check ahead of the prompt killed those silent answers for us. the gnarly part was partial evidence, 1 chunk of 6 expected, which sails through a simple count check.

do you gate on a minimum chunk count or on coverage of the entities in the question?

Collapse
 
aiapikey profile image
HAOHAOnovai •

Neither on its own. A minimum count is satisfied by duplicates and boilerplate - one chunk passes a count of 1 while covering a third of the question.

What generalised for us is entity coverage: extract the entities the question names, require all the required ones to appear in the retrieved set, and treat coverage of optional entities as a ratio rather than a boolean. Your 1-of-6 case fails that gate loudly, because a single chunk rarely holds every named entity.

Then log the decision, not just the outcome - the gate's own line with counts and covered entities - so the next parser change shows up as a coverage drop the same day instead of two weeks later.

Collapse
 
mudassirworks profile image
Mudassir Khan •

ratio for optional entities is the bit i am going to steal. we had a chunk full of boilerplate clear a plain count check once, so coverage over count makes sense to me.

logging the gate's own line with counts is the real win imo. our two week blind spot only existed because we kept the answer and dropped the decision. do you alert when the coverage ratio drops, or just chart it?

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

Both, but they answer different questions. Chart the ratio for humans; alert on the thing that actually broke for you: a decision recorded without its gate line. That is not a coverage number, it is an invariant - if every decision row must carry the gate's counts, then a decision with no counts is a bug rather than drift, and it should page immediately instead of trending.

The ratio alert is the softer one and it needs a denominator floor to be useful: coverage on n=8 items swings 12% when one item moves, so alert on a drop that is both larger than X% and has n above a minimum. Otherwise you build an alarm that fires on small batches, someone mutes it, and the blind spot comes back wearing a different hat.

One cheap invariant worth adding: the gate's counts should sum to the number of entities it claims to have seen. That catches the dropped-duplicate case you described without needing the ratio at all.

Thread Thread
 
mike-johnstone profile image
Mike Johnstone •

I'd keep a small fixed set of queries next to that chart, with expected coverage pinned to a document version. The live ratio can fall because the question mix changed. If the fixed cases also lose coverage on the same documents, that's a stronger signal of a retrieval regression than the live ratio alone.

Collapse
 
glenallen profile image
Glen Allen •

One important takeaway here is that tool discovery should never be confused with task-level authorization. An agent may need to know that a Slack connector exists, but that doesn't mean every agent in the workflow should be able to send messages through it. I’d treat outbound actions as explicit capabilities assigned to specific roles, rather than relying only on which tools happen to be available in a session.

Collapse
 
elarigent profile image
Elarigent •

The expense-agent example makes this land — a confident "yes" with zero retrieved chunks is worse than a refusal. Treating "I don't know" as an owned state, with a recorded constraint_evaluated event, is the part I would want in place before letting an agent approve anything.

Collapse
 
aiapikey profile image
HAOHAOnovai •

That is the right condition and it needs a mechanism rather than a habit: put the expected row in the repo as data - case, declared binding, expected decision - and commit it before the run. Then "written down first" is a commit timestamp instead of a recollection, and a gate that quietly widened after the re-index shows up as a diff rather than as a suspicion.

One more condition on the row: it has to be one that can fail. A perturbation whose expected result is "refuse" proves nothing if the gate was already refusing everything - you need a case the old gate passes, so the change is visible.

Collapse
 
jes_p_d4f61f7aa2ec405a418 profile image
Dev_PJS •

Your expense-agent example is exactly why I do this: I don't let myself say 'done' until I've run the check in the same session. A worker's 'complete' is a claim, not a result — I re-open the output, compare it against what was asked, and only then write it down as finished. The words 'should work' are banned in my records.

Collapse
 
lin7c profile image
xcl •

The zero-chunk case is the cleanest gate, but the harder failure is retrieval returning confident-looking chunks that are irrelevant or stale — for example, an old version of the same policy. In that case retrieved_chunks > 0 and schema_valid both pass, yet the answer is still grounded in the wrong source. How would you keep that check in the deterministic backbone without pulling semantic judgement back in? One option is to treat source version, effective date and document authority as pre-prompt fields alongside scope_match, so a chunk from a superseded policy fails the gate before the model ever sees it. I would also split insufficient_context by cause — empty retrieval, scope mismatch, stale source — because those point to very different fixes. And on over-refusal: a clarification turn after a refusal should probably be its own state, so teams do not count a successful recovery as a failure.

Collapse
 
paul-s profile image
Paul-S •

Exactly. In production agentic AI systems, “I don’t know” must be a typed state with a blocked transition, not just another sentence in the prompt.

Collapse
 
hellomathieup profile image
Mathieu Poli •

Great read. The model isn't lying, it's filling a blank. The fix is to make the blank visible.