When an LLM answer is wrong, the trace is where you look. Some tools make that easy.

A user reports a hallucinated answer in prod. To fix it you need the full trace of that one request, and how fast you can pull it depends entirely on the tracing you set up months earlier.

The ticket

A support user pasted a screenshot: our agent told them a refund window was 90 days. The real policy is 30. Wrong answer, confidently stated, already sent. The ticket had a request id in the response headers and nothing else.

The only useful question at that point is: what actually happened inside that one request. Which chunks did retrieval pull? What was the exact prompt the model saw after templating? What did each tool call return? A wrong answer is almost never the model being creative. It is usually a bad chunk, a stale document, a tool that returned the wrong row, or a prompt that got assembled wrong. You cannot see any of that from the output. You have to open the trace for that specific request id and read the spans.

The axis that matters

For debugging a single bad output, I care about two things. First, given a request id, how fast can I pull that one request's complete trace: retrieved chunks, templated prompt, tool args and returns, token counts, per-span latency. Second, is the tracing OpenTelemetry-native, so the spans drop into the collector and backend I already run, instead of locking me into a proprietary SDK and a second dashboard.

That second point is not aesthetic. When tracing is OTel-native, an LLM span sits in the same trace as the HTTP handler, the vector DB call, and the downstream service. One trace id, request to response. When it is proprietary, the LLM half lives in a separate tool and you are stitching timelines by hand at 2am.

Six tools, by how they capture

Ordered by how they get spans out of your app, not by any ranking.

Helicone (github.com/Helicone/helicone) is the fastest to turn on. You point your model's base URL at their proxy and every call gets logged, no per-span instrumentation. That one-line integration is the appeal. The tradeoff is granularity: a gateway sees the request and response it proxies, so your retrieval step and internal tool calls (which never hit the proxy) do not show up as spans unless you instrument them separately. Great for "what did the model get sent", thinner for the chunk that poisoned it.

LangSmith (smith.langchain.com) gives you the richest single-request view if you live in LangChain or LangGraph. Chains, tool nodes, and retriever steps show up already structured, and the waterfall for one run is genuinely good for reading a bad output. The catch is that the tracing is fairly proprietary. You are sending to their backend through their SDK, and pulling those spans into your own OTel collector is not the native path.

Langfuse (github.com/langfuse/langfuse) is open source and OTel-aware: it exposes an OpenTelemetry endpoint, so OTel spans can land there, and it also has its own SDK and decorators. Self-hostable if you want the data in your own infra. Reading one request means opening the trace by id and walking the observation tree, which shows the prompt, the retrieved context you logged, and per-step latency and tokens.

Future AGI (github.com/future-agi/future-agi) approaches tracing as one surface of a broader platform that also covers evaluation, prompt work, and guardrails, and its tracing library is OpenTelemetry-native, so spans flow through the standard OTel path into a backend you can point at your own stack. For the debugging job the useful part is that a wrong answer's trace carries the retrieved context and tool IO as spans on the same trace id, which is what you open when a request id is all you have from the ticket.

Braintrust (braintrust.dev) centers its logging around its Eval object. Traces are first-class, and the strong version of the workflow is: you catch a bad output, and turn that exact request into a test case in the same tool. If your loop is debug-then-lock-with-an-eval, that tight coupling helps. If you just want raw request-level tracing decoupled from their eval abstraction, it is more opinionated than a plain OTel backend.

Arize Phoenix (github.com/Arize-ai/phoenix) is open source and OpenTelemetry-native through OpenInference, so LLM, retriever, and tool spans use standard OTel semantics and flow into your collector. You can run it locally for a single debugging session or against a persistent backend. Opening one request means filtering to its trace id and reading the span tree, with the retrieved documents and tool calls attached as span attributes.

Three of these (Langfuse, Phoenix, Future AGI) are open source and multi-surface. Two of the six (Phoenix, Future AGI) are OTel-native by design; Langfuse supports OTel alongside its own SDK. All of the above is as of mid-2026, and this space ships fast, so check the current docs before you commit.

Reading one request

Whatever the backend, the move is the same: get the trace id (from the request id you logged), pull the trace, walk the spans, look at retrieval and tool IO first. Here is the instrumentation side, plain OpenTelemetry, so the LLM span carries the attributes you will actually want at 2am.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("rag-agent")


def answer(request_id: str, question: str) -> str:
    # request_id ties this trace back to the support ticket
    with tracer.start_as_current_span("answer") as span:
        span.set_attribute("request.id", request_id)
        span.set_attribute("input.question", question)

        chunks = retrieve(question)                    # own child span inside
        span.set_attribute("retrieval.chunk_ids", [c.id for c in chunks])
        span.set_attribute("retrieval.chunk_count", len(chunks))

        prompt = build_prompt(question, chunks)
        span.set_attribute("llm.prompt", prompt)       # the EXACT text the model saw

        out = call_model(prompt)                       # sets llm.tokens on its span
        span.set_attribute("output.answer", out)
        return out

The one attribute that saves you every time is llm.prompt holding the fully templated string, not the template. On the refund bug, that is where it fell out: the retrieved chunk was from an old policy doc, retrieval.chunk_ids pointed straight at it, and the prompt span showed the model was handed "90 days" as context. The model was not hallucinating. It was faithfully repeating a stale chunk. Total time from request id to root cause was about seven minutes, and six of those were me finding the request id in our own logs.

If your spans do not carry the retrieved chunk ids and the templated prompt, no backend will save you. You will be staring at an input and an output with the interesting part missing.

What I'd check first

• Pull the trace by request id and open the retrieval span first. Wrong chunk in equals wrong answer out, and this is the most common cause.
• Read llm.prompt as the fully templated string, not the template. A prompt assembled wrong looks fine in code and obvious in the span.
• Confirm the LLM span shares one trace id with the HTTP and vector-DB spans. If the LLM half lives in a separate proprietary tool, you are stitching timelines by hand, and that is the setup problem to fix before the next incident.