Every few weeks someone opens a ticket that says some version of "I think the AI is getting worse?" The answers are still fluent, still confident, ...
For further actions, you may consider blocking this person and/or reporting abuse
Instrumentation as debuggability is the right framing. The exclusion log with reason codes is the part that turns retrieval from a black box into something you can bisect. Without reason codes you cannot tell "the evidence was missing" from "the evidence was there and something rejected it," and those two failure modes need completely different fixes.
The other manifest field I would add: the retrieval query itself, verbatim, before any rewrite or expansion. When a retrieval bug bites, the first thing you want to know is whether the query the retriever saw is even the query the user meant. Manifests that log only the score list quietly assume the query survived transit, and it usually does, but the times it doesn't are exactly the debugging sessions that take the longest.
strong add, and i'd push it one hop further: log the query at every stage, not just verbatim-in. the raw user query, then whatever the rewrite/expansion/HyDE step turned it into, then what actually hit the index. the verbatim query alone tells you intent survived the front door, but most of the transit damage happens in the rewrite, which is the stage everyone skips logging because it "usually works." the diff between raw and rewritten is its own debug signal.
and it pairs with the reason codes to expose a failure mode you can't otherwise name. missing evidence and rejected evidence are the two you called out, but verbatim-vs-rewritten surfaces a third: the right doc was in the corpus, it just stopped matching because expansion dragged the query off-intent. that shows up as a retrieval miss in the score list, but the fix is upstream in the rewrite, not in the ranker or the corpus. without the query hops logged you'd burn the whole session tuning the wrong stage.
same discipline as the rest: the stage that usually works is the one nobody instruments, so it's the one that eats your afternoon the day it doesn't.
The stage-nobody-instruments principle recurses. The instrumentation itself is a stage that usually works and nobody instruments. Sampler drops one in ten events. Formatter truncates the rewritten query at 4KB because the field size predates HyDE. Metrics pipeline dedupes similar strings as "duplicate spans." Every one runs on the debug signal you'll use to find the retrieval bug.
Your four failure modes live in the retrieval layer. There's a fifth in the observability layer: query hops logged, then quietly reshaped by the stage that shipped them. Same shape as your third, one floor higher.
That's the afternoon you burn twice. Once tuning the wrong stage. Once trusting a debug output that no longer says what the stage did.
that recursion is the real one, and it's the version that actually scares me, because every other failure mode at least leaves a true log behind. this one corrupts the log itself, so you debug against a signal that already lied to you.
the only thing i've found that holds is making the debug trail carry its own fidelity: not just "here's the query at each hop" but "here's the query, and here's whether this record is complete or got sampled, truncated or merged on the way to me." the 4KB truncation isn't the bug, the silent 4KB truncation is. a formatter that logged "cut at 4KB, original was 11KB, hash abc" turns your fifth failure mode back into a visible one. same with the sampler: sample whole traces, never spans inside a trace, because a trace with holes reads as complete and that's the lie. and the dedup key can't be payload similarity when the entire point is to diff two hops that look alike, identity is stage plus position, not string distance.
so it's the same spine one floor up: the dangerous log isn't the missing one, it's the one that quietly improved the truth on the way out. a debug output that can't tell you when to distrust it is exactly what costs you that second afternoon. make it faithful, or make it loud about not being faithful, and you only burn the one.
One nail left: the confession has to be authored upstream of the surgery. "Cut at 4KB, original was 11KB, hash abc" written by the truncator is the formatter grading itself; if the formatter is the broken stage, its fidelity note inherits the break. Cheap fix: the producer stamps size and hash at handoff, and the formatter can shorten the payload but never the receipt. Then loud-about-not-faithful is checkable instead of trusted.
"Make it faithful, or make it loud" holds as written. The loudness just cannot come from the same throat.
no notes, you closed the loop. self-grading was the recursion sneaking back in one layer deeper, and i should have caught it: the fidelity note is itself a log, so it inherits every failure mode we were trying to escape. producer stamps size and hash at handoff, every stage after can shorten the payload but passes the receipt through untouched, and the consumer checks length against the receipt instead of believing anyone. loud-about-not-faithful stops being testimony and becomes arithmetic. and the regress bottoms out in the right place: the producer is the only honest candidate for trust root, because it is the only stage with nothing upstream to misreport. stealing "the loudness cannot come from the same throat", that is the whole design in one sentence.
That "the AI is getting worse" ticket is a painfully recognizable symptom. The retrieval-manifest idea feels like the missing artifact in a lot of postmortems because it ties the final answer back to candidate docs, filter decisions, and ranking drift in one place. In the incidents you've seen, what clue usually breaks the deadlock first: a missing chunk, stale embedding, or something in reranking/filtering?
candidate-set membership is the clue that breaks it, almost every time. the first question the manifest answers is binary: was the right doc in the candidate set at all. that split sends you down two completely different roads. not a candidate means ingestion, chunking or a filter: the most common culprits i have hit are a metadata filter or permission trim silently excluding the doc, and chunk boundaries splitting the answer across two chunks so that neither ranks on its own. candidate but not in top-k means ranking: reranker drift, k too tight, or query phrasing living in a different part of the embedding space than the doc.
stale embeddings are the rarest first clue but the nastiest incident, because they do not present as an incident, they present as the "ai is getting worse" ticket. the classic version is a partial re-embed after a model bump: half the corpus in the old vector space, half in the new, and similarity scores quietly comparing apples to oranges. so my working order is: candidacy first, chunk integrity second, filters third, and embeddings last, unless the embedding model changed recently without a full re-index, in which case embeddings jump straight to first.
The manifest gets even more useful offline than in debugging. If you log the full candidate set per query, you have a free regression suite: replay the same queries against a new index or embedding version and diff the manifests before anything ships. I caught a chunking change this way that looked fine in spot checks but quietly dropped recall on long, table-heavy docs. The answers were still fluent, which is exactly why nobody would have noticed for weeks. Do you replay your manifests in CI, or only reach for them when something's already on fire?
in CI, and honestly that's the whole reason i log them. the failure mode here is the one that never pages you: recall quietly drops, the answer still reads fine, so nothing catches fire for weeks. if the manifest only comes out when something's already on fire, it's too late by definition, because silent-but-fluent is exactly the class of regression that never lights up.
the catch is you can't diff raw manifests in CI or the gate turns flaky and someone disables it. embeddings churn, a new index legitimately reorders the tail, and if every reorder fails the build you get alert fatigue and the check quietly dies. so i don't assert on the manifest itself, i assert on a metric pinned to it: recall@k against a small labeled set (did the known-relevant chunk stay in the candidate set at all) plus how far the top-k moved against a frozen baseline. hard-fail on those, and dump the full manifest diff as an artifact for a human to eyeball, not as a build breaker.
the table-heavy case you caught is exactly why i keep the eval set sliced. one global recall number averages that regression away, because the loss is concentrated in one doc shape and everything else covers for it. so long and table-heavy docs get their own slice with its own threshold, and a drop there fails on its own even when the aggregate still looks healthy.
so yeah, in CI as a gate, not a fire drill. the manifest is the raw material, the thing i actually fail the build on is a stability metric sitting on top of it, because the regression worth catching is the one that would never have set anything on fire by itself.
Asserting on a stability metric pinned to the manifest instead of the manifest itself is the piece I was missing. That's what keeps the gate from going flaky and getting disabled. Stealing the sliced-eval-set idea wholesale. Thanks for the detail.
glad it landed. one thing that bites when you actually build the sliced set: each slice needs enough queries to be statistically boring. a 10-item slice flaps on one bad query and you are right back at the flaky gate you just escaped, only now it is flaky per-slice and harder to see. i size each slice's threshold off its own baseline variance, run the frozen baseline a few times and set the floor just below the worst honest run, never off the global number. and resist slicing everything: two or three doc shapes that have actually failed differently is plenty, a threshold nobody can explain gets disabled just as fast as a flaky one.
The exclusion log is the piece almost every RAG implementation drops, and it's exactly where the subtle bugs live. In my experience the stale-document-outranking-current case you describe almost never shows up in the cited output — you only see it when you diff the full candidate set, and if you're not logging the candidates with their reason codes, that diff is impossible. The other thing I'd add: the exclusion log is most valuable precisely when an answer looks correct, because that's when the silent wrong-evidence problem is hardest to catch without the manifest.
yeah, the "looks correct" case is the whole reason i log the candidate set unconditionally instead of only on error. error-triggered logging structurally can't see the silent case you're describing, because nothing errors, the answer is fine, the evidence behind it just happens to be stale. by the time something visibly breaks you've already lost the run that would've shown you the drift. so it has to be always-on (or sampled), not a break-glass thing.
and +1 that reason codes are what make the diff possible. citations alone tell you what won, not what lost and why, and the "why" is the signal. the ones i keep are the boring ones: retrieved-but-outranked, below-threshold, deduped-away, filtered-by-metadata. the stale-outranks-current bug is invisible in the cited output but it's screaming in "current doc retrieved, ranked #4, outranked by three older versions." once that's in the manifest you can assert on it in CI instead of finding it in prod.
the framing i keep coming back to: a right answer is not evidence of right retrieval. you can be correct off stale or wrong evidence and never know it, and the candidate manifest is the only place that's visible.