The thesis: an agent that fails at step 2 of 7 and an agent that fails at step 7 of 7 get the same score from an outcome eval, and they should not. The first one picked the wrong tool while holding the right context; the second one did everything right and hit a flaky API. Those are different bugs with different fixes, and the outcome eval hands back one number that cannot tell them apart. Through 2024 and most of 2025 the tooling mostly could not see the difference. As of July 2026 it can, across at least six frameworks, and the interesting divide is no longer "can you evaluate agents" but which step-level questions each tool answers deterministically versus by asking another model.
In June I argued that the trajectory, not the final answer, is the unit of agent evaluation. This is the follow-up: the question has moved from whether the tooling can see the path to which step-level questions it answers deterministically. I spent this week reading the current source and docs of six frameworks to map that divide. Everything below is as of July 2026; all six move fast, so treat version-dependent claims as dated the day you read them. Tools in alphabetical order throughout, because the ranking depends on your stack anyway.
The four questions a step-level eval can answer
When people say "evaluate the trajectory," they mean at least four separable checks, and conflating them is how teams buy the wrong tool.
- Tool choice. Given the state at step i, was this the right tool to call at all.
- Argument correctness. Right tool, but were the parameters right. (In my experience this is where the production failures actually live: the tool choice is right and one parameter is subtly wrong, a date filter scoped a day too wide, an ID passed where a name belongs.)
- Path quality. Right calls, wrong shape: loops, backtracking, redundant steps, order violations.
- Task completion. Did the whole trajectory achieve the goal.
Questions 1 and 4 are judgment calls, and most frameworks route them to an LLM judge. Questions 2 and 3 are checkable by code against a reference or a rule, and the frameworks that treat them that way give you something you can gate CI on without inheriting a judge's variance.
What each framework actually checks
Arize Phoenix (Elastic License 2.0). Phoenix splits single-step tool use into three prebuilt LLM-judge evaluators: ToolSelectionEvaluator ("was the correct tool selected"), ToolInvocationEvaluator (arguments and formatting), and ToolResponseHandlingEvaluator (did the agent use the result properly). All three are judge-based classifications. Whole-path evaluation is a cookbook recipe rather than a shipped metric: their docs walk you through writing a code evaluator for path convergence yourself. The tracing underneath is OpenTelemetry-based with a wide set of agent-framework auto-instrumentors, and since June 2026 evals can run as pytest tests in CI. Worth knowing: the license is Elastic 2.0, source-available rather than OSI open source, which matters if your legal team reads licenses closely.
DeepEval (Apache-2.0). The broadest agentic metric menu of the six. Five metrics consume a full execution trace (TaskCompletion, StepEfficiency, PlanAdherence, PlanQuality, plus AgentLoopDetection), and the split between judge and code is explicit in the source: TaskCompletion, StepEfficiency and the plan metrics are LLM judges over the serialized trace, while AgentLoopDetection and ToolPermission are documented as fully deterministic, no API key required. ToolCorrectness is the interesting hybrid: deterministic matching of called tools against expected tools (exact, ordered, or set), with argument matching optional, but hand it the available-tools list and it quietly adds a judged selection score. If you gate CI on it, know which mode you configured. Ships as a pytest plugin, which makes the CI story the most conventional of the six.
Future AGI (Apache-2.0). The eval library takes the opposite bet from Phoenix: agent metrics as deterministic heuristics rather than judges. The trajectory set (task_completion, step_efficiency, tool_selection_accuracy, trajectory_score, plus goal_progress, action_safety and reasoning_quality) scores a structured trajectory input with keyword and sequence heuristics, no model call. The function-calling set parses calls with an AST and checks name match, parameter validation, and either exact match or a weighted accuracy score, including parallel calls. The tradeoff reads both ways: deterministic scoring is fast, free, and identical on every run; the cost is that a keyword-overlap notion of "task completed" is bluntly literal in a way a judge is not. The platform around it spans tracing, simulation, and a gateway, a scope comparable to Phoenix, Langfuse, and LangSmith.
Langfuse (MIT core, ee folder for enterprise plumbing). Langfuse's angle is that the trace is the eval target: evaluators, judge or code, attach to any observation in a multi-step trace, and tool calls arrive in the evaluator's context with names, arguments, and counts. The 23 managed judge templates are quality-flavored (hallucination, correctness, plus Ragas partner metrics like Goal Accuracy and Topic Adherence); there is no off-the-shelf tool-call correctness metric, so step grading means writing a sandboxed code evaluator or a custom judge against the tool-call fields. The agent graph view for traces is in beta. Notably, the eval features live in the MIT tree, and self-hosted deployments get unlimited judge evaluators; the code-evaluator runtime needs an explicit dispatcher config when self-hosting.
LangSmith + agentevals/openevals (platform proprietary; evaluator libraries MIT). The two open libraries are the most complete deterministic trajectory matchers of the group: strict, unordered, subset, and superset match modes over message-plus-tool-call trajectories, with per-tool argument comparators you can override down to a custom equality function. LangGraph users additionally get graph-trajectory matching at the node level. Judge variants exist for the same shapes when you have no reference trajectory. The libraries run standalone under pytest or Vitest with response caching for CI; the LangSmith platform itself, per its own FAQ, is proprietary software, so the split to understand is MIT evaluators, closed dashboard.
Promptfoo (MIT). The change that surprised me most this year: Promptfoo now ships a deterministic trajectory assertion family that runs against traced execution rather than final output. trajectory:tool-used, trajectory:tool-args-match (partial or exact, with ignore-lists for volatile arguments), trajectory:tool-sequence (in-order or exact), and trajectory:step-count, plus a judged trajectory:goal-success and a tool-call F1 scorer. The catch is honest and structural: these require trace data, so your agent must emit OpenTelemetry spans to Promptfoo's receiver before any of it works. Pair that with its declarative configs and CI exit codes and it covers questions 2 and 3 with plain YAML.
The pattern, stated plainly
Lay the six side by side and the divide is clean. Argument correctness and path shape, the checkable questions, are deterministic where they are best developed: agentevals' match modes and comparators, Promptfoo's trajectory family, DeepEval's tool matching and loop detection, Future AGI's AST-based function-calling checks. Tool choice and task completion, the judgment questions, are LLM judges nearly everywhere they are prebuilt, with two exceptions taking the deterministic bet at the cost of bluntness (Future AGI's heuristics) or narrow scope (DeepEval's loop detector).
So the selection question for a team is which of the four questions matches your actual failure mode, and whether you want it answered by code or by a judge. If your agents fail on arguments and paths, the deterministic matchers gate cleanly in CI. If they fail on judgment, on picking the wrong tool while every call is well-formed, you are buying a judge somewhere, and its run-to-run variance comes with it into any blocking check.
One more cost that applies across the board: every step-level anything requires the steps to exist. Traced spans, structured trajectories, instrumented tool calls. The instrumentation tax comes before the first metric fires, whichever framework collects it. Budget for that first; it is most of the adoption work.
Where I'd push back on this
The steelman against step-level evaluation: outcomes are what users experience, and a trajectory metric can flag an "inefficient" path that a model chose for good reasons the metric cannot see. Grading the path risks optimizing agents into brittle choreography, matching the reference trajectory instead of solving the task. That objection lands, and the honest answer is that step-level checks earn their keep as diagnostics and regression tripwires, not as the definition of success. Gate on outcomes plus the deterministic invariants you truly require (no unauthorized tools, no argument corruption, no loops), and use the rest of the trajectory data to explain failures rather than to score them.
Where I hold the line: the claim that outcome evals alone are enough. An outcome eval on a seven-step agent is a test with one assertion at the end of the program. I have never seen a team accept that coverage for ordinary code, and I have not heard a good argument for why agents should be graded that way.
Disclosure: capability descriptions above come from each project's public repositories and documentation, read the week of July 20, 2026. All six ship changes weekly; verify against current docs before deciding.
Top comments (0)