Your Evals Aren't Broken. Your Eval Users Are.
Langfuse just published a negative result. At the AI Engineer conference this week, Annabell Schafer walked through an auto-improvement loop they ran on a paper classification task — the kind of narrow, measurable problem where self-improvement should work if it works anywhere. It did not converge. The loop burned tokens without meaningful quality gains until they injected domain-specific expertise into the evaluation criteria. The system was not learning. It was confidently iterating on its own noise.
📖 Read the full version with charts and embedded sources on AgentConn →
This is a rare thing in AI: a company publishing what did not work. And it matters because the failure mode Langfuse exposed is not a Langfuse problem. It is the default state of every agent evaluation pipeline that relies on synthetic test cases generated from the same distribution as the model itself.
The same week, Nick Ung from Lyft gave the talk that names the real failure precisely: Build Evals That Actually Matter. Lyft handles 270,000 AI-managed customer interactions daily. Their first synthetic eval users wrote polite, complete sentences. Real Lyft users send two-word fragments, misspelled demands, and emoji-only messages. The eval was passing. The agent was failing. And the intuitive fix — generate more synthetic test cases — is mathematically wrong, because it amplifies the same distribution gap that caused the failure.
This is the article nobody has written plainly: the practitioner frontier has migrated from "can it do the task" to "can you tell when it stopped doing the task." And the answer to the second question is almost always no, because your eval user is not your user.
The Langfuse Negative Result: Why Auto-Improvement Has a Ceiling
Langfuse's experiment was clean enough to be instructive. They built an auto-improvement loop: run a task, evaluate the output, feed the evaluation back into the system prompt, iterate. The task was paper classification — assigning research papers to categories — against a ground-truth dataset. If auto-improvement works anywhere, it should work here. The labels are unambiguous. The evaluation is mechanical. The feedback loop is tight.
It did not work. The loop oscillated. Quality would improve on one category, degrade on another, and the net score stayed flat. The system was not learning; it was redistributing errors across categories while maintaining total error count.
The fix was injecting domain expertise — specific classification rules that a human expert would apply — into the evaluation criteria. Once the eval knew what good looked like in the domain, the loop converged. Without it, the loop was optimizing against its own reflection.
This is not a novel finding in machine learning. It is the fundamental theorem of evaluation: you cannot improve a system beyond the quality of the signal you are using to measure it. But it is a novel admission in the agent tooling space, where the default pitch is "just add an eval loop and the system improves itself." Langfuse sells evaluation infrastructure. They are telling you their own product has a ceiling. That takes integrity, and the ceiling is real.
The ceiling in one sentence: An auto-improvement loop without domain-calibrated evaluation criteria does not converge — it oscillates, redistributing errors while maintaining total error count. More iterations make it worse, not better, because each iteration compounds the evaluator's blind spots.
The Lyft Distribution Problem: Your Synthetic User Is Not Your User
Nick Ung's Lyft talk is the load-bearing insight of the entire agent-evals conversation, and almost nobody has written it up in terms a statistics class would recognize. So here it is.
When you build a synthetic eval user — an LLM that simulates a customer interacting with your agent — you are sampling from the model's distribution of "what a user looks like." That distribution is wrong. It is wrong in specific, measurable ways:
Lyft's first simulators wrote like this: "Hello, I was charged $23.50 for a ride from downtown to the airport on July 15th, but the quoted price was $18.00. Could you please look into this discrepancy?"
Real Lyft users write like this: "wrong charge fix it," "wtf $23," or simply a screenshot with no text.
The eval was passing at 90%+ because the synthetic user was cooperating with the agent in ways real users never do. The agent looked great in eval. It was failing in production. And the gap between "passing evals" and "failing in production" was not a model problem. It was a distribution problem in the evaluation itself.
Here is why "generate more synthetic test cases" is the wrong fix. More samples from the wrong distribution do not converge on the right distribution. They converge harder on the wrong one. You are not reducing variance — you are reducing variance around the wrong mean. This is the same error that makes oversampling in imbalanced classification dangerous: more data from the majority class does not help you find the minority class. It buries it deeper.
Lyft's production numbers: 79 million trips per month generating 270,000+ AI-handled interactions daily. 55% containment rate, 35% full resolution. Their bespoke evaluator fine-tuned on Lyft-labeled production traces outperformed every off-the-shelf LLM judge they tested. The evaluator that works is the one trained on what your users actually do.
Lyft fixed it by prompt-tuning their simulator with actual production message distributions. Not synthetic patterns — production patterns. The simulator learned to send two-word fragments, misspelled demands, and angry emoji because that is what the data said real users send. Once the eval distribution matched the production distribution, eval scores became predictive of production performance.
This is the part that matters for every team building agent evals: the fix is not better prompts for your synthetic user. The fix is instrumenting your production traces and building your eval FROM them. The synthetic user is a downstream artifact of production data, not a substitute for it.
The Conference Signal: Evals Are the Whole Conversation Now
A batch of AI Engineer conference talks published this week tells a story through its composition. Of 17 videos from the AI Engineer channel in a 24-hour window, 8 were talks on evals, observability, and cost. Almost nothing on model capability. Not one talk on a new architecture, a novel attention mechanism, or a benchmark breakthrough.
A caveat matters here: these 8 talks are one channel's publishing batch from a single conference, not 8 independent signals. The AI Engineer conference curated a program heavy on production operations. That is editorial judgment, not market consensus. But editorial judgment from a conference that sells tickets to practitioners is itself a signal. The conference programmed what its audience wanted to hear, and its audience wanted to hear about evals.
The broader pattern is visible across three independent source platforms on the same day:
GitHub trending was dominated by ops tooling, not capability tooling. herdr — an agent multiplexer for monitoring multiple agents from a single terminal — hit 18,000+ stars. code-review-graph — a local-first code intelligence graph that achieves 8.2x context reduction for agent reviews — crossed 20,000 stars. SigNoz, an OpenTelemetry-native observability platform that now explicitly targets agent workloads, was trending at 31,000+ stars. The tools developers are reaching for are not "make the agent smarter" tools. They are "tell me what the agent did" tools.
X surfaced the Harness Handbook as the most-upvoted paper on HuggingFace Papers for the week — a paper about making agent harnesses readable, navigable, and editable. Not a paper about making agents more capable. A paper about making the infrastructure around agents comprehensible to the humans who have to maintain it.
The convergence across conference talks, GitHub stars, and research attention is genuine. The direction is uniform: the practitioner community has moved past "can my agent do the task" and is now living in "can I tell when my agent stopped doing the task correctly."
The Buyer's Side: Organizational Psychosis Without Observability
While practitioners were discussing eval methodology at conferences, Nik Suresh published an essay that went viral — hitting the Hacker News front page and earning coverage in the Northeast Times. Suresh runs Hermit Tech, a consultancy that sets up data infrastructure and recovers failed projects. His claim: he has observed zero successful AI projects over 18 months across diverse organizations.
The specific failure modes he documents are the eval problem viewed from the buyer's side:
- Staff inflating token usage to meet AI adoption quotas while ignoring output quality
- Engineering teams fictionally attributing standard work to AI to satisfy management mandates
- A Mitsubishi voice bot that promised callback service but never delivered — and the failure never appeared in any metric because no error was logged
That last example is the eval distribution problem wearing a suit. The bot completed the interaction. It resolved the ticket. It just never called anyone back. An eval built around "did the conversation end without an error" would pass it. An eval built around "did the user get what they needed" would fail it. The difference between those two eval criteria is the difference between a product and a demo, and most teams are running the demo eval in production.
Suresh frames this as "organizational psychosis" — a collective inability to speak honestly about AI's results because doing so would undermine peers' credibility. But it is also an observability problem. If you cannot measure whether your agent is actually delivering value — not just completing tasks, but delivering the outcomes users need — then organizational psychosis is the natural consequence. People fill measurement vacuums with narratives, and the narrative that AI is working is easier to maintain than the one that says it is not.
Contrarian Corner: Maybe Synthetic Evals Are Good Enough for Most Teams. Not everyone operates at Lyft's scale. If your agent handles 50 interactions a day, the distribution gap between synthetic and real users may be too narrow to matter. The cost of building production-calibrated evaluation infrastructure — instrumenting traces, labeling production data, fine-tuning custom evaluators — may never pay back for a small team. The counter-argument: if you cannot measure the gap, you cannot know it is narrow. And the failure mode is silent. Your agent passes evals, your users leave, and you never connect the two because the eval said everything was fine.
The Harness Handbook: When Eval Infrastructure Becomes Unmaintainable
The Harness Handbook paper, published on arXiv this week and immediately voted to the top of HuggingFace Papers, addresses a second-order problem that becomes first-order at scale: the eval harness itself becomes too complex to maintain.
The paper introduces Behavior-Guided Progressive Disclosure (BGPD) — a method that synthesizes a behavior-centric representation of a harness codebase via static analysis and LLM-assisted structuring, linking each behavior to its source code. The result: agents navigating a harness to make modifications can localize behaviors and produce better edit plans while using fewer tokens.
This matters because of a pattern we have covered before on AgentConn: it fails on the harness, not the model. The harness — the restraint layer, the eval framework, the orchestration shell — is where production agents succeed or fail. And harnesses grow. They accumulate eval cases, custom judges, per-failure-mode rubrics, integration tests, and observability hooks. Within six months, the harness is more complex than the agent it wraps.
The Harness Handbook's contribution is making that complexity navigable. But its existence is the signal: we have reached the point where the infrastructure around agent evaluation needs its own tooling. The meta-problem — "how do I understand my eval system" — is now as important as the object-level problem of "how do I eval my agent."
The practitioner discussion on Hacker News reinforces the same theme. When asked "how are you monitoring AI agents in production?", the top answers emphasize intent-execution gap detection ("the agent's stated intent was Z, but it executed W instead"), defense-in-depth architecture (treating LLMs as "dangerous tools" that will fail), and standard observability stacks extended into agent territory — OTEL, traces, and cost tracking. The tools exist. The problem is that most teams are not using them because their evals told them everything was fine.
What This Means for You
The shift from capability to observability is not theoretical. It is happening in the tools practitioners reach for, the talks conferences program, the papers researchers publish, and the failures consultants document. Here is what changes in your workflow:
1. Build evals FROM production traces, not FOR production.
Your synthetic user is a hypothesis about what your users do. Validate it against production data or accept that your evals are measuring a fiction. Lyft's approach — prompt-tuning the simulator with actual production message distributions — is the template. If you do not have production traces yet, that is your first investment, not more eval cases.
2. Accept the auto-improvement ceiling.
Langfuse's negative result is generalizable: auto-improvement loops without domain-calibrated evaluation criteria do not converge. If your eval criteria are generic ("was the response helpful?"), your improvement loop will oscillate. Inject domain expertise into your evaluation rubrics before investing in iteration infrastructure. As we explored in our agent judge layer analysis, the judge needs to be a domain expert, not a generic grader. Sebastian Raschka's deep-dive on reasoning effort levels (3,400+ likes) illustrates the adjacent problem: inference-time reasoning budgets are themselves an evaluation parameter, and most teams set them without measuring impact on output quality.
3. Treat your eval harness as production code.
The Harness Handbook exists because eval harnesses grow until they are unmaintainable. Apply the same engineering practices to your eval infrastructure that you apply to your production code: version control, documentation, observability, and regular refactoring. The teams running agents at scale are the ones whose observability investment preceded their capability investment — a pattern we identified in Agent Observability Is the Next Battleground.
4. Monitor for silent failures, not loud ones.
The Mitsubishi voice bot failure that Suresh documents is the canonical example: the agent completed the task, the metrics stayed green, and the user never got what they needed. Your monitoring must distinguish between "the agent finished" and "the user succeeded." Those are different signals, and most teams are only measuring the first one. Agent failures at production scale are the rule, not the exception — and the dangerous ones are the ones your current evals cannot see.
5. Watch the tooling layer for your next investment.
The GitHub trending signal is concrete: herdr (agent multiplexing), code-review-graph (context reduction), SigNoz (OpenTelemetry for agents). The tools gaining stars are not making agents smarter. They are making agent behavior visible. If you are still spending your infrastructure budget on model capability, you are solving last quarter's problem.
The one-sentence version: Stop generating more synthetic eval users. Start instrumenting what your real users actually do. The gap between those two distributions is where your agent is failing, and more evals of the wrong distribution is more noise at higher confidence.
The conference talks, the trending repos, the viral essays, and the published negative results all point the same direction. The agent capability race produced a fleet of agents that can do remarkable things. The agent observability race — the one that started this quarter — will determine which of those agents anyone trusts enough to keep running. The hard part was never building the agent. The hard part is knowing when it broke.
Originally published at AgentConn
Top comments (0)