Your Agent Has Observability. It Doesn't Have Evals.

TLDR: In LangChain's survey of 1,340 practitioners, 89% had implemented observability for their agents and 94% of teams with agents in production had it. Offline evaluations: 52.4%. Online evaluations: 37.3%. Fewer than a third of respondents run both. That gap is not a tooling gap — it's a category error. A trace records what the agent did. Nothing in a trace records whether it was right, because a confidently wrong tool-call chain emits exactly the same telemetry as a correct one: same token counts, same latencies, same finish_reason, same green spans end to end. The OpenTelemetry GenAI conventions make this literal — there is no attribute for correctness until you run an evaluator and write one in.

Two traces

Here is the shape of the problem, in the only form that makes it obvious. A support agent handles refund requests. Two runs, side by side, as your observability platform renders them.

// Run A
{ "trace_id": "a41f…", "gen_ai.operation.name": "agent_run",
  "gen_ai.request.model": "claude-sonnet-4-6", "duration_ms": 3180,
  "gen_ai.usage.input_tokens": 2847, "gen_ai.usage.output_tokens": 193,
  "gen_ai.response.finish_reasons": ["stop"],
  "spans": [
    { "name": "lookup_customer",  "status": "OK", "duration_ms": 112 },
    { "name": "list_orders",      "status": "OK", "duration_ms": 340 },
    { "name": "issue_refund",     "status": "OK", "duration_ms": 908 }
  ], "error_count": 0 }

// Run B
{ "trace_id": "b73c…", "gen_ai.operation.name": "agent_run",
  "gen_ai.request.model": "claude-sonnet-4-6", "duration_ms": 3204,
  "gen_ai.usage.input_tokens": 2851, "gen_ai.usage.output_tokens": 188,
  "gen_ai.response.finish_reasons": ["stop"],
  "spans": [
    { "name": "lookup_customer",  "status": "OK", "duration_ms": 118 },
    { "name": "list_orders",      "status": "OK", "duration_ms": 336 },
    { "name": "issue_refund",     "status": "OK", "duration_ms": 913 }
  ], "error_count": 0 }

Every field that your monitoring dashboard aggregates is, for practical purposes, identical. Both runs are three spans, all OK, no errors, ~3.2 seconds, ~2,850 input tokens, a clean stop finish. On a latency percentile chart they are the same point twice. On a token-spend chart they are the same point twice. On an error-rate chart they contribute equally to a rate of zero.

In Run B the customer had two orders — a $40 one they were complaining about and a $2,300 one they weren't. The agent called issue_refund on the second.

Nothing in that trace is wrong. issue_refund really was called. It really did return OK, because refunding $2,300 is an entirely successful refund. The span is green because the API call succeeded, and the API call succeeding is the only thing a span status has ever meant.

The gap has a number attached to it

This is not a hypothetical asymmetry, and it isn't rare. LangChain's State of Agent Engineering report, published 23 May 2026 and drawn from 1,340 responses collected between 18 November and 2 December 2025, puts numbers on both halves:

	Adoption
Agents in production	57% (67% at 10,000+ employees)
Observability implemented	89% overall, 94% among production teams
Full per-step tracing	62% overall, 71.5% among production teams
Offline evaluations on a test set	52.4%
Online evaluations in production	37.3% (44.8% among production teams)
Both offline and online	fewer than a third

Observability is close to universal; systematic evaluation is roughly a coin flip. And the same survey names quality — accuracy, relevance, consistency, tone — as the top barrier to getting agents into production, cited by about a third of respondents. Those two findings sit awkwardly together. Quality is the thing most likely to block a deployment, and it is the thing least likely to be measured. Roughly nine teams in ten can reconstruct what their agent did last Tuesday. About five in ten can tell you whether it was any good.

The instinct to instrument first is a reasonable one, and the tooling market rewarded it — tracing is a solved, buyable, drop-in problem. Evaluation is not buyable in the same way, because it requires you to state what correct means for your domain, and nobody can do that for you. Observability got adopted at the speed of a vendor integration. Evaluation adoption moves at the speed of the hardest product conversation on your team.

Why the dashboard structurally cannot tell you

It's worth being precise about why the traces above are indistinguishable, because the reason isn't an oversight anyone will patch.

The OpenTelemetry GenAI semantic conventions define the standard attribute registry for instrumenting LLM and agent workloads — the fields your platform is almost certainly collecting. The core ones:

• gen_ai.request.model, gen_ai.provider.name — which model answered
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, plus cache-read, cache-creation and reasoning token counts — how much it cost
• gen_ai.response.finish_reasons — why it stopped
• gen_ai.operation.name — chat, tool_call, agent_run
• gen_ai.input.messages, gen_ai.output.messages — what went in and came out

Read that list looking for the field that goes red when the agent refunds the wrong order. It isn't there, and it can't be, because every one of those attributes is a property of the call rather than of the answer. Cost, latency, model version and stop reason are all knowable from inside the request/response cycle. Correctness is only knowable with reference to something outside it — a ground truth, a business rule, a resulting state you can inspect.

The conventions do include an evaluation namespace — gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label, gen_ai.evaluation.explanation — and score.label accepts values like "correct". But that namespace is a slot, not a sensor. Those attributes are empty until something you wrote computes a judgement and populates them. The standard has a correctness-shaped hole in it and expects you to bring your own filling. Most teams ship the ninety per cent that auto-instruments and never fill the hole.

This is the practical consequence: your alerting is wired to signals that do not move when quality moves. Latency won't regress when the agent starts refunding the wrong order — it might even improve. Token counts won't move. Error rate stays at zero, because from the runtime's point of view nothing failed. You will find out from a customer.

Every field a trace carries is a property of the call. Correctness is a property of the outcome, and the outcome has to be checked against something the request doesn't contain.

What measuring the outcome actually looks like

The fix has a well-established precedent, and it predates the current agent wave: verify the resulting state, not the transcript.

τ-bench, the tool-agent-user benchmark from Sierra, is built on exactly this move. Its authors describe the method plainly: they "employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state." Not the wording of the reply. Not whether the right tool name appeared. The database, afterwards, against what the database should have contained.

Applied to Run B above, that check is unglamorous and about four lines long:

def verify_refund(conversation, db_before, db_after):
    expected = conversation.annotated_goal      # refund order #4471, $40.00
    actual   = diff(db_before, db_after)        # refunded order #4473, $2,300.00
    assert actual.refunded_order_ids == expected.refunded_order_ids
    assert actual.total_refunded_cents == expected.total_refunded_cents

That assertion fails on Run B and passes on Run A. No telemetry field distinguished them; a four-line state diff does it instantly. The same pattern generalises to whatever your agent actually touches — if it writes code, run the test suite; if it files tickets, assert on the ticket's fields; if it produces a document, check the document against the source.

τ-bench also contributes a metric worth stealing outright. Alongside the usual pass@k ("at least one of k attempts succeeded"), the paper proposes pass^k — all k attempts succeeded — "to evaluate the reliability of agent behavior over multiple trials." This is the honest metric for anything customer-facing, because your customers are not running your agent eight times and keeping the best result; they're getting one run each. The arithmetic is brutal and worth internalising: an agent that succeeds 90% of the time, if failures are roughly independent, is fully reliable across eight consecutive runs only 0.9⁸ ≈ 43% of the time.

The benchmark's own headline finding, from the abstract, is the part most teams have not absorbed: "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)." That was measured on a purpose-built benchmark with annotated goal states. Your production agent has neither, and the assumption that it is doing better is currently unfalsifiable — which is precisely the problem.

"We'll just add an LLM judge"

This is the standard next move, and it's the right direction, but it is not free and it is not a shortcut past defining correctness. A judge is another model whose agreement with your actual standard is an empirical question — one that has now been measured at scale, and the results argue for treating judges as instruments that need calibration rather than as oracles.

The largest systematic study of the approach to date — 21 judges from nine providers across MT-Bench, JudgeBench and RewardBench, 118 runs and roughly 541,000 individual judgments — opens by naming the methodological problem directly: "judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability." Two findings in particular should change how you read a judge's score:

• Chance correction matters enormously. The paper reports that "kappa deflation between exact match and Cohen's κ is universal (33–41 pp on MT-Bench)." A judge advertising 80% raw agreement with human labels may be delivering far less genuine discriminative power than that number implies, because a meaningful share of those agreements are what you'd get by guessing.
• Consistency is not accuracy. The authors document a "consistency–bias paradox," finding that "high test–retest reliability (>0.95) coexists with severe position bias (>0.10)" in production judges. A judge that returns the same verdict every time looks trustworthy on a stability check and can still be reliably wrong in a direction that tracks the order you presented the options in.

None of this makes LLM-as-judge unusable — it remains the only practical option for open-ended generation where no reference string exists. It does mean a judge is a component that itself requires a labelled validation set, a chance-corrected agreement statistic, and a position-swap check before you let it gate a release. Where you can verify state instead, verify state: an assertion on a database row has no position bias.

Turning the traces you already have into the evals you don't

The useful thing about being in the 89% is that you are already sitting on the raw material. Traces are unlabelled evaluation data. The work is labelling and replaying them.

1. Pull fifty real traces from last month, weighted toward the unusual: long tool chains, retries, sessions a human took over, anything a customer followed up on. Not fifty happy paths — happy paths inflate every number you're about to compute.
2. For each one, write down what should have happened, in a form a program can check. An order id. A row count. A set of fields. An exit code. If you cannot express it as an assertion, that's the product conversation surfacing, and it is better to have it now than during an incident.
3. Replay them on every change — prompt edits, model version bumps, tool-schema changes, dependency upgrades — and gate the release on the result. This is the step that converts the set from a document into a regression gate. It is also, mechanically, unit testing; the only novel part is that the assertions run against end state rather than return values.
4. Report pass^k, not just the mean. Run each case several times and count only the cases that pass every time. The number will be worse than your average, and it is the one that corresponds to what a single user experiences.
5. Feed failures back. Every production failure becomes case fifty-one. The set gets stronger exactly where the system is weakest, which is the whole point of curating it by hand.

Steps 2 and 3 are where teams stall, and the reason is rarely technical — it's that "what should have happened" turns out to be genuinely contested between engineering, support and finance. Working through that with a versioned golden set and a regression gate on every change is a chunk of what SophiArch's AI Applications with LLMs course spends its validation module on, alongside the observability side that most teams have already built. The two lessons sit deliberately close together, because shipping one without the other is the failure this whole article is about.

The one-line version

Observability answers "what happened?" Evaluation answers "should it have?" The first question is solved, buyable and nearly universally adopted. The second is the one your customers are actually asking, and roughly half of teams running agents in production still have no systematic way to answer it.

If you have 89% of the stack, the remaining work isn't another dashboard. It's fifty labelled traces and an assertion that fails.

References

• LangChain. (2026, May 23). State of Agent Engineering — survey of 1,340 practitioners, fieldwork 18 Nov–2 Dec 2025; source of the observability (89%), offline eval (52.4%), online eval (37.3%) and production (57%) figures. Accessed 2026-09-23.
• Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045 — source of the database-state evaluation method, the pass^k metric, and the pass^8 <25% retail result. Accessed 2026-09-23.
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias. (2026). arXiv:2606.19544 — 21 judges, nine providers, ~541,000 judgments; source of the kappa-deflation and consistency–bias findings. Accessed 2026-09-23.
• OpenTelemetry. Gen AI Attributes registry — the standard gen_ai.* span attributes, including the gen_ai.evaluation.* namespace referenced above. Accessed 2026-09-23.
• SophiArch. AI Applications with LLMs — course covering the probabilistic contract, context window architecture, output validation layers, evaluation frameworks and golden sets, orchestration patterns, and LLM observability, including the regression-gate workflow described here.