TL;DR
- Many LLM observability setups capture prompts, outputs, tokens, and latency while leaving retrieval failures hidden.
- A single search call may conceal query rewriting, filtering, fetching, deduplication, reranking, and evidence selection.
- A useful trace connects the original question to the effective query, returned sources, selected passages, and final claims.
- Retrieval tracing helps distinguish missing, stale, or ignored evidence from a genuine generation failure.
- Production teams should measure freshness, duplicate evidence, citation coverage, and cost per grounded answer.
A user asks your AI assistant whether a product still supports a particular feature. The assistant responds confidently and links to the company’s documentation.
The model request succeeded. Latency was normal. Token usage stayed within budget. No tool call failed. Every indicator on the dashboard is green.
The answer is also six months out of date.
The model trace cannot tell you whether the system searched for the wrong phrase, preferred an old page, discarded a better result, or ignored the correct evidence. It only shows the context that eventually reached the model.
That is the blind spot in model-centred observability. For RAG applications and web-connected agents, the useful unit of observation is not the model call. It is the complete evidence path.
A Successful Model Call Can Still Be a Failed Request
A typical LLM trace records the prompt, response, model name, token consumption, latency, errors, and perhaps a tool invocation. That is useful for diagnosing slow requests, malformed inputs, and unexpectedly expensive generations.
It does not tell you whether the model received the right facts.
In a retrieval application, the final prompt is assembled by an upstream system. That system may rewrite the query, choose a search provider, apply time or domain filters, fetch pages, extract text, remove duplicates, rerank candidates, and select passages for the context window.
The model can behave exactly as instructed and still produce a bad answer because one of those earlier decisions was wrong.
The current OpenTelemetry semantic conventions for generative AI recognise retrieval from a vector database or search system as a separate span. They include concepts such as the data source, top-k, retrieved document IDs, scores, and the effective retrieval query.
The conventions are still marked Development, and potentially sensitive fields such as query text and returned documents are opt-in. They are useful as a shared vocabulary, but they should not be treated as a finished internal schema.
LangSmith’s retriever tracing documentation takes a similar approach by representing retrieval as its own run and attaching document metadata such as source URLs, chunk IDs, and scores.
The important question is not which tracing platform you use. It is what your trace can explain when an answer goes wrong.
One Search Call Hides an Evidence Pipeline
Application code may represent web retrieval as a single tool call. Operationally, it looks more like this:
Question → query rewrite → search → page fetch → extraction → deduplication → reranking → evidence selection → answer
Every transition introduces a different failure mode.
A query rewrite can remove an important date or product qualifier. Search can return relevant-looking but obsolete pages. Extraction can miss the paragraph containing the answer. Several URLs can reproduce the same story and create the illusion of independent confirmation.
A reranker may favour semantic similarity over freshness. The context builder may then drop the best passage to stay inside its token budget.
“Search completed successfully” tells you almost nothing about those decisions.
A search-aware trace should retain enough information to reconstruct how evidence moved through the pipeline.
| Stage | Useful trace data |
|---|---|
| Search decision | Reason for searching, selected source, freshness policy |
| Query | Original question, rewritten query, filters, strategy version |
| Retrieval | Provider, top-k, rank, score, duration, retry, cache status |
| Evidence | Canonical URL, publication time, retrieval time, content hash, selected passage |
| Answer | Claim ID, evidence ID, citation mapping, model and prompt version |
Only some of these fields are part of emerging telemetry conventions. Canonical URLs, content hashes, cache status, freshness policies, and claim-to-evidence mappings are application-level metadata that teams need to design themselves.
This does not mean storing every complete webpage forever. Metadata, content hashes, and the passages supplied to the model are often enough for diagnosis. Sensitive queries and retrieved content still require redaction, access controls, and a defined retention period.
The goal is not to log everything. It is to preserve enough information to explain an answer.
A URL Is Not Evidence
Many applications place a few URLs beside an answer and call that provenance. A URL identifies a location, but it does not identify the exact information the model used.
Pages change. Search snippets can differ from subsequently fetched content. Two URLs may contain the same syndicated article. A page’s publication date may also differ from its most recent update.
If you only retain the URL, opening it tomorrow may not reproduce the evidence used today.
A better evidence record connects four things:
The source, the retrieved version, the selected passage, and the claim it supports.
That normally requires a canonical URL, retrieval timestamp, publication date when available, content hash, and the passage sent to the model. A stable evidence ID can then follow that passage through reranking, context construction, and generation.
This becomes even more important in multi-agent systems. A shared URL cache may stop four agents from fetching the same page, but it does not demonstrate that they consumed the same version or selected the same passage.
Content hashes and evidence IDs make that relationship visible. They also stop duplicated content from inflating confidence. Five URLs do not represent five independent sources when they all reproduce one original report.
Debug the Evidence Path in Order
Return to the outdated product answer.
Start with the effective query. If the user asked about current support but the rewritten query dropped the word “current,” the problem began before retrieval.
Next, inspect the returned sources. If the latest documentation never appeared, the trace points towards a discovery problem involving the query, filters, or search provider.
If the current page appeared but ranked below an older one, look at reranking and freshness weighting. If it survived reranking but its relevant paragraph was never selected, the problem is more likely to be extraction or context construction.
Only when the correct evidence reached the model and the answer still contradicted it does the incident begin to resemble a generation failure. Even then, context ordering, truncation, or conflicting evidence may still be involved.
Citation failure is another category. An answer can be factually correct while citing a page that does not support the relevant claim. A generally related link is not evidence unless the cited passage actually supports what the answer says.
These distinctions matter because each problem has a different fix. A new system prompt cannot recover a source that was never retrieved. Increasing top-k may make duplicate evidence worse. Switching to a more expensive model will not make stale documentation current.
A useful trace turns “the answer was wrong” into a testable hypothesis about a specific stage.
Turn Traces Into Production Signals
Traces are useful for debugging individual incidents, but their larger value appears when you aggregate them.
A team might define retrieval success rate as the percentage of runs in which at least one selected source meets its relevance threshold. Freshness-policy violation rate can measure how often the system uses evidence older than the question permits. Duplicate evidence ratio can show how many apparently different results collapse into one canonical or content-hash group.
Citation coverage should represent the share of material claims connected to supporting evidence. It should not simply count links. An answer can contain five citations while leaving its most consequential statement unsupported.
One metric I would add is cost per grounded answer:
(search + extraction + reranking + generation cost) / answers that pass the grounding threshold
Cost per model call is misleading when a cheap workflow repeatedly searches, retrieves duplicate pages, or generates responses that later fail verification.
Traces should also feed evaluation rather than become an archive nobody examines. Failed production runs can become regression cases. After changing a query strategy, extraction rule, or reranker, the same cases can be run again to determine whether the evidence path improved.
MLflow’s current RAG evaluation documentation separates retrieval relevance, groundedness, and sufficiency, and requires retrieval to appear explicitly in the trace. That separation matters because an end-to-end score alone cannot tell you what to fix.
Automated judges are imperfect, but they provide a useful loop: observe a failure, classify its likely stage, add it to an evaluation set, change one component, and measure the result.
Instrument Retrieval Before Replacing the Model
When an AI application produces a poor answer, the model is often the first component developers replace. They change prompts, increase context, or upgrade models without knowing whether generation caused the failure.
For a search-enabled RAG system, observability needs to begin earlier. Retrieval, reranking, and context selection should appear as separate nested steps under the same request.
A production trace should be able to answer three questions:
What did the system ask? What evidence did it see? Why did that evidence become this answer?
Until it can, you are monitoring your model—not observing your system.
When a RAG answer is wrong, can your current trace tell whether the source was never found, ranked down, dropped from context, or ignored by the model? Which of those is hardest for your team to see today?
Top comments (0)