Every observability tool I use has a button that turns a production trace into a dataset row. I have clicked it a lot, on the assumption that this was the loop closing: agent fails, capture the failure, the failure becomes a regression test.
I have argued the trajectory case elsewhere and will not re-run it here. Assume you already want to test the path rather than the final answer. This post is about the narrower thing that surprised me when I went and read how these tools actually store a captured case, which is that the storage decides what you are allowed to assert, and I had never looked.
Capturing a failure is cheap. Turning it into a test case takes two decisions, and a button can only make one of them for you.
Decision one: what counts as the input
For a single LLM call this is not a question. For an agent it is the whole question. Is the input the user's first message, or the user's message plus the state the agent had accumulated by turn four, which is where it went wrong?
Whichever the tool picks by default is the answer you get, and the defaults differ.
Langfuse lets you capture from a trace or from a single observation. The dialog reached from a trace prefills the trace-level input and output; the same dialog is reachable from ObservationDetailViewHeader.tsx, which is how you pin a case to the step that broke. Phoenix is span-first by construction: the mutation takes span_ids, so there is no trace-level shortcut to get wrong. Future AGI accepts either, and the obvious path is the wrong one: pass trace_ids to add_to_new_dataset and _root_span_ids_for_trace_ids keeps only each trace's root span, discarding the steps. Pass span_ids for the spans you meant. Both endpoints sit in the Apache-2.0 tree as of July 2026.
The rule I follow now regardless of tool: if the failure happened at turn four, the case is turn four. Trace-level capture reproduces a conversation. It does not test a step.
Decision two: what counts as the assertion
This is the one I was wrong about, and wrong in the direction that flatters my own argument, so it is worth being explicit.
I had assumed these tools hand you input and output and leave ground truth as your problem. They mostly do not. Langfuse's DatasetItem has a dedicated expectedOutput column beside input and metadata. LangSmith's docs describe an engine that can generate ground-truth examples from production traces automatically, which is doing the expensive half rather than the storage half. Phoenix goes further than I credited: get_dataset_example_input pulls LLM_INPUT_MESSAGES and the tool definitions from LLM_TOOLS, and the output path merges tool_calls from both the legacy function_call attribute and the modern one, so a captured span carries the tool calls nested inside its messages.
So the assertion slot exists. Here is the narrower thing that is still true: in every trace-native tool above, the assertion is one value. An expected output, a string or a blob. There is no named field in which to write "and it should have called lookup_order before issue_refund, with that order id". Phoenix has the tool calls, as a record of what happened, not as a place to state what should have.
Exactly one tool in this set names that field, and it is the one that never touches your traces. DeepEval's Golden carries tools_called and expected_tools, both lists of ToolCall, and its ConversationalGolden carries scenario, expected_outcome and a list of turns. That is a schema built to hold a trajectory assertion. Nobody else here models it that directly.
Future AGI takes a different route. Its mapping_config takes a {col_name, span_field, data_type} shape per column (required on a new dataset, optional when appending), so a tool argument buried in a span attribute can be projected into a column you name. General mechanism, not a named trajectory field: the schema still has no idea what a tool call is.
Six schemas, side by side
Read on 29 July 2026, from the trees rather than the docs pages, except LangSmith which is proprietary and is therefore their claim rather than my verification.
| Tool | Capture from | Assertion slot | Named trajectory field | Provenance back to the span | Versioned on write |
|---|---|---|---|---|---|
| Langfuse (MIT core, ~32.1k stars) | trace or observation | expectedOutput | no | sourceTraceId + sourceObservationId | yes, temporal (validFrom/validTo) |
| Phoenix (Elastic 2.0, ~10.8k) | spans | example output | no, but tool calls survive in messages | span_rowid | yes, with version description |
| LangSmith (proprietary) | traces, experiment results | examples, can be auto-generated | not documented at this level | n/a | yes, taggable versions |
| Future AGI (Apache 2.0, ~1.5k) | trace root span, or chosen spans | any column you define | no, but any span field can become one | not by default, mappable via span_field | no |
| Promptfoo (MIT, ~23.7k) | generated or imported sets | config-defined expectations | trajectory assertions at eval time | n/a | n/a |
| DeepEval (Apache 2.0, ~17.3k) | goldens you construct | expected_output | yes: tools_called, expected_tools, turns | you carry it | on demand (create_version) |
Two things to say about that table honestly. Future AGI is roughly a twentieth of Langfuse's adoption and the row shows it: no provenance column unless you map one yourself, no versioning. And the pattern I did not expect is in the last row. The schema that models a trajectory best belongs to the tool furthest from your traces. DeepEval is offline-first, so the format that could hold your production failure is the one you have to hand-assemble it into. That is a real gap and no vendor in this set is currently on the right side of it.
Promptfoo I want to be careful about. Its dataset commands are generation-shaped (src/commands/generate/dataset.ts, an MCP generateDataset tool) plus Hugging Face import, and it ships OTel tracing under src/tracing/. I looked for a harvest-a-span-into-a-dataset path and did not find one. That is where I looked and what I saw, not a claim that no such path exists.
The habit that survives switching tools
Write the assertion at triage time, while you still remember what should have happened. This is the step everyone skips, mine included, because the capture button makes it feel finished.
def case_from_span(span, expected_tools, expected_substring):
"""Built at triage. `expected_tools` comes from a human, not from the trace:
the agent is the thing under test, so its own behaviour cannot be the oracle."""
return {
"input": span.input, # the failing step
"state": span.attributes.get("session_state"),
"expected_tools": expected_tools, # e.g. ["lookup_order", "issue_refund"]
"expected_output_contains": expected_substring,
"provenance": {"trace_id": span.trace_id, "span_id": span.span_id},
}
The provenance field earns its keep a month later, when the test fails and nobody remembers why the case exists. Langfuse and Phoenix both keep this for you, on different keys. In the offline formats I carry it myself.
Where I'd push back on this
Hand-writing expected_tools does not scale, and that is the objection I have the least answer for. A busy week produces more failures than anyone will annotate, and a half-annotated dataset is worse than an honest pile of archived traces because it looks like coverage. I think that is right, and it is the real argument for the automatic ground-truth generation LangSmith describes. My hesitation is that a generated assertion inherits whatever the agent did, and I cannot see how to break that circularity without a human somewhere.
Second objection, which I will partly concede: for plenty of agents the final answer really is the thing you care about, and one expected-output column is enough. If your agent is one or two tool calls deep, ignore me. It starts to bite around three or four steps, when the failure and the output stop being the same event.
What I am least sure about is versioning. Langfuse and Phoenix version dataset items implicitly on every write, and DeepEval versions when you ask it to (create_version, against Confident AI). I had assumed none of them did, which was simply me not reading. Having been wrong about that once, I now suspect I am wrong about how much it matters: last month I could not tell whether a suite had improved or the cases had changed under me, and a temporal table would have answered it in one query. If you have a convention for versioning a dataset that grows out of triage every week, I would rather copy yours than invent one.
Top comments (0)