Grounding is a capability, trust requires a contract is the right way to put it, and citations make the gap worse rather than better because they transfer confidence to the reader without transferring evidence. A user who sees a source link stops checking, which means a wrong answer with a citation propagates further than a wrong answer without one. Of the failure modes listed, the stale authoritative source is the one I have found hardest to catch, since every automated check passes: the citation resolves, the passage genuinely supports the sentence, and the document is from a trusted store. Nothing is wrong except that it was superseded. Metadata on effective dates plus filtering at query time is the only thing that has worked for me, because no faithfulness check can see it - faithfulness is measured against the retrieved context, not against the world.
Staff Software Engineer @ Navan, ML Flow & Cypress Ambassador, Docker Captain and creator of agent-inspect, writing about production AI agents, TypeScript, observability, testing, and developer tools.
@ahmetozel, exactly—freshness is orthogonal to faithfulness. If the retrieved passage accurately supports the claim but has been superseded, a groundedness check can still pass the wrong answer. I’d make validity metadata part of the retrieval contract: effective_from, optional effective_to, version lineage, and a policy for authoritative precedence. The trace should record the selected document version and candidate version set so incident review can distinguish stale indexing from bad ranking.
The effective-date contract also benefits from distinguishing when a fact was valid from when the system learned it. A retroactive correction can have an old effective_from value but a new ingestion time.
Those answer different questions: what is now believed to have been valid on that date, versus what the assistant could have known when it answered. Recording both timelines with the selected version would make that incident review possible, especially when a later correction changes the authoritative account of an earlier period.
Staff Software Engineer @ Navan, ML Flow & Cypress Ambassador, Docker Captain and creator of agent-inspect, writing about production AI agents, TypeScript, observability, testing, and developer tools.
@ahmetozel That's a useful distinction: valid time and knowledge time answer different audit questions. I'd record effective dates, ingestion time, and the selected document version with each answer. Then a replay can compare "as known when we answered" with "using today's corrected record" without mislabeling a retroactive correction as a retrieval bug. Have you found a clear way to show that distinction to readers without making citations noisy?
I would keep the citation itself short: source title, version date and an explicit "effective as of" date when time affects the answer. A compact details panel could show ingestion time and the knowledge cutoff used for the replay.
If today's corrected record changes the answer, display that as a separate comparison with a reason, rather than replacing the original citation silently. Readers can then distinguish the business period being answered from the information that was available, without making every sentence carry the whole audit record.
Staff Software Engineer @ Navan, ML Flow & Cypress Ambassador, Docker Captain and creator of agent-inspect, writing about production AI agents, TypeScript, observability, testing, and developer tools.
@ahmetozel That UI split makes sense: keep the citation readable, then expose ingestion time and the replay cutoff in details. I’d show “answer as originally served” beside “answer under the corrected record,” with both document versions and the reason for the change. That preserves the audit trail without silently rewriting what the reader saw. Thanks for carrying this through to the presentation layer.
We're a place where coders share, stay up-to-date and grow their careers.
Grounding is a capability, trust requires a contract is the right way to put it, and citations make the gap worse rather than better because they transfer confidence to the reader without transferring evidence. A user who sees a source link stops checking, which means a wrong answer with a citation propagates further than a wrong answer without one. Of the failure modes listed, the stale authoritative source is the one I have found hardest to catch, since every automated check passes: the citation resolves, the passage genuinely supports the sentence, and the document is from a trusted store. Nothing is wrong except that it was superseded. Metadata on effective dates plus filtering at query time is the only thing that has worked for me, because no faithfulness check can see it - faithfulness is measured against the retrieved context, not against the world.
@ahmetozel, exactly—freshness is orthogonal to faithfulness. If the retrieved passage accurately supports the claim but has been superseded, a groundedness check can still pass the wrong answer. I’d make validity metadata part of the retrieval contract:
effective_from, optionaleffective_to, version lineage, and a policy for authoritative precedence. The trace should record the selected document version and candidate version set so incident review can distinguish stale indexing from bad ranking.The effective-date contract also benefits from distinguishing when a fact was valid from when the system learned it. A retroactive correction can have an old effective_from value but a new ingestion time.
Those answer different questions: what is now believed to have been valid on that date, versus what the assistant could have known when it answered. Recording both timelines with the selected version would make that incident review possible, especially when a later correction changes the authoritative account of an earlier period.
@ahmetozel That's a useful distinction: valid time and knowledge time answer different audit questions. I'd record effective dates, ingestion time, and the selected document version with each answer. Then a replay can compare "as known when we answered" with "using today's corrected record" without mislabeling a retroactive correction as a retrieval bug. Have you found a clear way to show that distinction to readers without making citations noisy?
I would keep the citation itself short: source title, version date and an explicit "effective as of" date when time affects the answer. A compact details panel could show ingestion time and the knowledge cutoff used for the replay.
If today's corrected record changes the answer, display that as a separate comparison with a reason, rather than replacing the original citation silently. Readers can then distinguish the business period being answered from the information that was available, without making every sentence carry the whole audit record.
@ahmetozel That UI split makes sense: keep the citation readable, then expose ingestion time and the replay cutoff in details. I’d show “answer as originally served” beside “answer under the corrected record,” with both document versions and the reason for the change. That preserves the audit trail without silently rewriting what the reader saw. Thanks for carrying this through to the presentation layer.