DEV Community

Cover image for Read the Trace, Not the Leaderboard
Max Quimby
Max Quimby

Posted on Edited on Originally published at agentconn.com

Read the Trace, Not the Leaderboard

Read the Trace, Not the Leaderboard

Your agent ran for six hours, made 247 tool calls, and then failed. You have two options. Option A: swap to a newer model and hope it fails less. Option B: read the trace, find the tool call that went sideways at step 183, and fix the harness so it never happens again. Option B is boring. Option B works.

📖 Read the full version with charts and embedded sources on AgentConn →

The industry spent 2025 and early 2026 saturating the orchestration conversation. Everyone shipped a framework. Everyone argued about which model belonged in the loop. And yet the Agentic Harness Engineering paper demonstrated something that should have ended the debate: holding the base model fixed and iterating only on the harness lifted Terminal-Bench 2 pass@1 from 69.7% to 77.0%. LangChain reported the same pattern independently — changing only the harness, not the model, lifted their coding agent on Terminal-Bench 2.0 from 52.8 to 66.5. No model swap. No fine-tuning. Just reading traces and tightening the harness.

Addy Osmani on X — Top coding agents look more like each other than their underlying models do

View original post on X →

Addy Osmani captured the convergence: the top coding agents today look more like each other than their underlying models do. The differentiator is not which frontier model sits at the center. It is the harness around it — and the observability layer that lets you iterate on that harness with evidence instead of intuition.

This piece makes three arguments. First, why traces are the highest-leverage tool for agent reliability — ahead of model selection. Second, what a production trace should actually contain, span by span. Third, how the tooling landscape has matured enough that "we don't have visibility" is no longer an excuse.


The Case for Traces Over Model Swaps

Here is the uncomfortable truth about model selection for agents: it matters far less than the discourse suggests.

The convergence report from our latest intelligence sweep found the same signal across four independent sources — GitHub trending (wall-to-wall orchestration repos), AI YouTube, Tech YouTube, and X — all saying the same thing: the gains come from harness engineering, not model shopping. The lesson from teams running agents at scale is not "use a bigger model." It is: instrument everything, read the traces, and fix what breaks.

This is not a theoretical position. It is an empirical one:

  • Terminal-Bench 2 (AHE paper): 69.7% to 77.0% by evolving the harness. Model: frozen. The paper's core insight is that progress is limited by observability rather than agent capability. Ten iterations of an automated trace-reading loop — not ten model upgrades — delivered the lift.
  • Terminal-Bench 2.0 (LangChain): 52.8 to 66.5 by changing the harness only. The blog credits "self-verification and tracing as major contributors."
  • Harness-squared (Google, Oct 2026): A recursive self-improving framework where the harness rewrites itself based on trajectory evidence. Up to 11.8 points of improvement across three professional benchmarks. The model is the constant; the harness is the variable.

omarsar0 on X — Banger paper from Google on self-improving agent harnesses

View original post on X →

The pattern is consistent: the model is good enough. The harness is not. And you cannot fix the harness without traces.

âš ī¸ Contrarian Corner: The counter-argument writes itself — better models make fewer mistakes, so invest in model selection instead of instrumentation overhead. This is the same argument waterfall shops made against unit tests in 2005. The issue is not whether better models help (they do). The issue is that you cannot know whether a model swap helped without traces that show where failures actually occur. Model selection without observability is superstition.


What a Production Trace Should Contain

A trace is a nested tree of spans. The run is the root. Model calls and tool calls sit beneath it. Sub-agents are linked child traces. This is the same conceptual model as distributed tracing in microservices — and that is not a coincidence. Debugging a multi-step agent is closer to debugging a distributed system than debugging a single program.

The OpenTelemetry GenAI semantic conventions define three span types for agents: invoke_agent at the top, chat spans for model calls, and execute_tool for tool calls. These conventions moved to their own repo (semantic-conventions-genai) with v1.42.0 in June 2026, and they are still marked Development — attribute names may change. But the shape is right: a tree, not a flat log.

Here is what each tool-call span should record. This is the minimum viable instrumentation:

Per tool call:

Field Why
Tool name Identify what ran
Arguments (sensitive fields redacted) Know what was asked
Result status (success / error / timeout) Know what happened
Duration (ms) Spot slow tools before they become blockers
Retry count Catch silent retry loops
Side-effect flag Distinguish reads from writes
Approval status (auto / human / denied) Audit the control surface
Output identifier (e.g., commit SHA, file path) Confirm what actually changed outside the agent

Per model call:

Field Why
Model ID and version Reproduce the exact behavior
Input tokens / output tokens Track cost per step
Prompt hash or version Link behavior to the prompt that caused it
Temperature and other params Identify non-determinism sources
Latency (time to first token, total) Distinguish model slowness from tool slowness

The table is straightforward. The discipline is not. Most teams log the model call and skip the tool call — and then wonder why they cannot explain a failure that happened between steps 40 and 60 of a 200-step run.

â„šī¸ The MCP Trace Context spec: As of July 2026, the MCP specification documents W3C Trace Context propagation inside MCP request metadata. This means tool calls routed through MCP servers can carry trace context end-to-end — your agent's spans and your tool server's spans can live in the same trace tree. If you are building MCP tools, propagate the traceparent header. If you are consuming them, verify it arrives.

This builds directly on our earlier coverage of agent observability as a governance layer. That piece focused on cost visibility — Microsoft pulling Claude Code over budget overruns. This piece focuses on the debugging side: not "how much did it cost?" but "why did it fail?"


Where Runs Actually Fail

If you have been running agents in production, you already know that most failures are not model failures. They are tool failures, context failures, and loop failures. The trace is what distinguishes between them.

Tool failures are the most common. A tool times out, returns an unexpected format, or silently succeeds while actually doing nothing. Without per-tool-call spans, these look like "the agent got confused" — which is the agent equivalent of "it works on my machine." With spans, you can see: tool X returned a 200 but the response body was empty, the agent treated empty as success, and the next 15 steps built on that phantom result.

Context failures happen when the agent's context window fills up and critical information gets evicted. Traces that record input token counts per model call let you see exactly when the context hit the limit and what was lost. This is especially vicious in long-running agents — the kind that run for hours or days. The failure does not announce itself. The agent just starts making worse decisions because it forgot what it was doing.

Loop failures are the agent equivalent of infinite loops. The agent retries a tool, the tool fails, the agent retries with slightly different arguments, the tool fails again. Without a retry count in the trace, this burns tokens silently for dozens of iterations before anyone notices. The eval-gaming monitoring piece we published in October covers a related pathology: agents that report success while actually failing. Traces are the antidote to self-reported status.

Sentry's Agent Tracing product shows the full execution path of an AI agent, including every tool call and token count. This is what the debugging workflow looks like in practice: not reading terminal output for twenty minutes, but filtering a trace tree to the first red span.


The Tooling Landscape (November 2026)

The observability tooling for agents has matured considerably over the past year. Here is a landscape snapshot — not a ranking, because the right tool depends on your stack.

Open-source / self-hosted:

  • Langfuse (~31K GitHub stars): The most widely adopted open-source option. Supports LangGraph, OpenAI Agents SDK, Claude Agent SDK, CrewAI, Pydantic AI, Vercel AI SDK, and more. Acquired by ClickHouse in January 2026 — the data layer is now first-party. If you want to self-host your traces and own the data, this is the default.
  • Arize Phoenix (~11K stars): Source-available (Elastic License 2.0, not OSI open-source). Strong eval integration. Recently released an open-source coding agent tracer that works across Claude Code, Cursor, Codex, Copilot, and Gemini CLI.
  • Laminar: Built specifically for long-running agents. OpenTelemetry-native. Raised a $3M seed in April 2026 for observability of agents that run for hours or days — exactly the use case where flat logs fall apart.

Managed / vendor:

  • LangSmith: The default if you are on LangChain or LangGraph. The Engine feature clusters recurring issues from production traces and can open pull requests — turning trace signals into harness changes automatically.
  • Braintrust: Positions itself at the intersection of tracing and evals. Its TRACE 2026 conference featured Replit's observability-at-scale story.
  • Sentry Agent Tracing: Extends Sentry's existing APM into agent execution paths. If your team already uses Sentry for error tracking, this is the lowest-friction add.
  • Datadog LLM Observability: Enterprise play. Ingests OpenTelemetry traces following the GenAI conventions.

The right choice depends on two questions: (1) Are you on LangChain? If yes, start with LangSmith. (2) Do you need to self-host? If yes, Langfuse. If neither constraint applies, evaluate based on how well the tool integrates with your existing observability stack — the goal is traces that join the same system your infrastructure already lives in, not a parallel universe for AI.

💡 The OpenTelemetry bet: Every tool on this list either ingests or emits OpenTelemetry traces. The GenAI semantic conventions are still experimental, but they have already become the shared vocabulary. If you instrument once to OTel, you can switch tools later without re-instrumenting. This is the strongest argument against vendor lock-in in the agent observability space.


From Traces to Harness Changes: The Feedback Loop

Collecting traces is necessary but not sufficient. The payoff comes from turning trace data into harness changes — and that feedback loop is what separates teams that improve from teams that just watch.

The Focused Labs writeup captures this directly: "Traces only matter when they change the next harness release." They describe a LangChain deployment where the team held the model fixed and changed only the harness — tool descriptions, middleware around tool calls, retry policies, context management — based on what the traces revealed. Each harness iteration was driven by a specific trace pattern, not by intuition.

Martin Fowler and Birgitta Boeckeler's harness engineering guide formalizes this as two feedback mechanisms: feedforward controls that prevent known failure modes (informed by past trace patterns), and feedback controls that evaluate what happened after the agent acted (informed by current traces). The trace is the evidence layer for both.

NLP Papers on X — Agentic Harness Engineering: Observability-Driven Automatic Evolution

View original post on X →

The AHE paper's automated version of this loop is the logical endpoint: the system reads its own multi-million-token trajectories, identifies failure patterns, and proposes harness edits. Ten iterations. No human in the loop. 7.3 percentage points of improvement. The frozen harness even transferred to SWE-bench-verified, using 12% fewer tokens than the seed harness.

This is what the harness-as-moat thesis looks like in practice. The moat is not the model. The moat is not even the harness. The moat is the feedback loop from traces to harness changes — the ability to learn from every production run and ship a better harness before the next one starts.


What This Means for You

If you are building agents today, here is the practical checklist:

  1. Instrument tool calls, not just model calls. Most teams log the LLM interaction and skip everything else. The tool call is where the agent meets reality — and where most failures actually originate. Log the full table from Section 2.

  2. Use OpenTelemetry's GenAI conventions. They are experimental, but they are the emerging standard. Instrument once; switch vendors later.

  3. Record trace context in MCP calls. The W3C traceparent header is now part of the MCP spec. If you are building or consuming MCP tools, propagate it.

  4. Build a trace-to-test pipeline. When a production trace shows a failure, capture it as a regression test. LangChain recommends this explicitly: traces that represent real production behavior become the test suite for the next harness version.

  5. Stop model-shopping before you have traces. If you cannot point to the specific tool call or context state that caused a failure, switching models is a lottery ticket, not an engineering decision. This echoes our earlier take in Stop Chasing Models, Fix Your Harness.

  6. Budget for trace storage. Long-running agents generate large traces. Sampling strategies and retention policies are not premature optimization — they are table stakes for agents that run for hours or days.

Rohit Ghumare on X — The cleanest survey of harness engineering as a discipline

View original post on X →


The Bottom Line

The agent observability stack is no longer a gap — it is a choice. Langfuse, LangSmith, Arize Phoenix, Sentry, Laminar, and Braintrust all ship production-grade tracing. OpenTelemetry gives you a vendor-neutral instrumentation layer. The GenAI semantic conventions give you a shared vocabulary. The papers give you the evidence: harness iteration driven by traces outperforms model swaps by a wide margin.

The only question left is whether you are reading your traces or still reading the leaderboard.

If you are deploying agents that run for hours or manage real infrastructure, you might also want our coverage of how agents fail on real jobs, the harness guardrail stack, and fleet orchestration patterns.

Originally published at AgentConn

Top comments (0)