DEV Community: SyncSoft.AI

The Eval Gap: Your Agent Has Observability but No Idea If It's Any Good

SyncSoft.AI — Tue, 09 Jun 2026 02:03:11 +0000

Here's a number worth sitting with. In LangChain's 2026 State of Agent Engineering report, which surveyed more than 1,300 practitioners, 89% of teams running agents in production have implemented observability — but only 52% have implemented evaluations. That 37-point gap is where most agent quality quietly dies.

If you've shipped an LLM agent, you already feel this gap even if you've never named it. You have traces. You have dashboards. You can replay any session and watch the agent reason, call tools, and respond. And yet, when someone asks "is it actually getting better or worse this week?", the honest answer is a shrug. You can see everything that happened and still have no idea whether any of it was good.

That's the difference between observability and evaluation, and conflating the two is the most expensive mistake in agent engineering right now.

Observability tells you what happened. Evals tell you whether it was right.

Observability is a microscope. It shows you the trajectory: the agent received a query, retrieved three documents, called the search_orders tool with these arguments, got this response, and produced this answer. Invaluable for debugging. Completely silent on the question that matters to your users — was the answer correct, helpful, and safe?

Evaluation is the judgment layer on top of the trace. It takes the same trajectory and asks: did the agent call the right tool? Did it recover when the tool returned an error? Was the final answer factually grounded in what it retrieved, or did it hallucinate a plausible-sounding order number? Did it follow your refund policy or invent one?

The reason so many teams have the first and not the second is simple: observability ships with your framework. Evals you have to build, and building them well means confronting a problem most engineering teams are not set up to solve — you need labeled examples of what good looks like, and you need them to be trustworthy. That is a data problem long before it's a tooling problem.

The three tiers of agent evaluation

The teams closing the gap aren't running one giant eval. They're running evaluation as infrastructure, in three tiers that map cleanly to how you already think about testing.

Tier one: fast checks on every change. These are the unit tests of the agent world. Did the agent call the expected tool with valid arguments? Did it stay under the latency and token budget? Did it avoid an obvious refusal or loop? These are cheap, deterministic, and run on every PR. They catch the dumb regressions — the prompt edit that broke tool-calling for an entire category of inputs.

Tier two: quality regression suites. This is where it gets hard, because "quality" isn't a boolean. Here teams lean on LLM-as-judge — using a strong model to score outputs against a rubric for things like factual accuracy, completeness, and guideline adherence. In the LangChain data, about 53% of teams running evals use LLM-as-judge, because it's the only thing that scales to thousands of test cases overnight.

Tier three: production monitoring. Sampling live traffic and scoring it continuously, so you get an alert when answer quality drifts after a model swap or a sneaky distribution shift in user queries.

Most of the engineering conversation fixates on tier two tooling. But the tooling is the easy part. The hard part is the rubric and the reference data feeding it — and that's where the gap actually lives.

LLM-as-judge has a calibration problem

Here's the uncomfortable truth about LLM-as-judge: an unaligned judge is a confident liar. A model scoring your agent's outputs has its own biases — it favors longer answers, it rewards confident tone over correctness, it misses domain-specific errors a real expert would catch instantly. If your judge says quality is 94% and your users are churning, your judge is wrong, and you won't know until the dashboard and reality have fully decoupled.

The fix is calibration against human judgment. You take a representative sample, have qualified humans score it, and then tune your LLM-judge's prompt and rubric until its scores correlate with the human ones. The same LangChain data shows why this matters: roughly 60% of teams running evals still rely on human review for nuanced and high-stakes cases, more than rely on LLM-as-judge. Human review isn't the legacy approach being automated away. It's the ground truth that makes automation trustworthy.

This is the part nobody likes, because it's labor that doesn't look like engineering. Someone with actual domain expertise — a clinician for a medical agent, a developer for a coding agent, a financial analyst for a finance bot — has to sit down and judge a few hundred trajectories carefully. The quality of that judgment is the ceiling on the quality of your entire eval system. Garbage reference labels produce a garbage judge, which produces a dashboard that lies to you with great confidence.

Where this connects to data quality

If you take one practical idea from this, make it this: your eval system is only as good as the human-labeled data underneath it. Not the framework. Not the dashboard. The labels.

That's why teams who are serious about agent quality treat evaluation data with the same rigor they'd apply to training data — clear rubrics, expert annotators, multiple passes to catch disagreement, and a measured inter-rater reliability so they know the labels themselves are consistent. This is exactly the discipline that high-quality model evaluation and QA work is built on: benchmark dataset construction, response scoring against rubrics, hallucination detection, and red-teaming for the failure modes your happy-path tests will never surface.

It also overlaps heavily with the world of reasoning and human-feedback data — preference ranking, agent trajectory correction, and tool-use validation. The skill of looking at a multi-step agent trajectory and pinpointing exactly where it went wrong is the same skill whether you're generating RLHF data to improve the model or eval labels to measure it. The pipeline that produces good human feedback for training is the same pipeline that produces a trustworthy judge for evaluation. Most teams discover this the hard way, after their first calibration run reveals their judge and their experts disagree on a third of cases.

A concrete starting point

You don't need to boil the ocean. If your team is in the 89% with observability and the 48% without evals, here's a week-one move:

Pull 100 real production trajectories from your traces — ideally a mix of successes, complaints, and weird edge cases.
Write a rubric. Three to five dimensions, each with concrete pass/fail criteria. Force yourself to define what "grounded" and "policy-compliant" actually mean for your product.
Have a domain expert — a real one — score all 100 by hand. Measure how often two experts agree. If they don't, your rubric is too vague; fix it before you automate anything.
Now build your LLM-judge, and validate it against those 100 human labels before you trust a single automated score.

That sequence — human ground truth first, automation second — is the whole game. The teams that skip step three and jump straight to LLM-as-judge are the ones whose dashboards drift away from reality.

Observability told you the agent did something. Evaluation, built on honest human-labeled data, tells you whether it did the right thing. In 2026, with agents making real decisions in production, that's not a nice-to-have. It's the difference between an agent you can trust and one you're merely watching fail in high resolution.

I work at SyncSoft.AI, where our bilingual, SME-led teams build evaluation datasets, human-feedback data, and QA pipelines for AI teams. If you're wrestling with the eval gap and want to talk through what good reference data looks like for your use case, we're happy to compare notes — reach out anytime.

The SLM-First Agent: Why 2026's Best Agentic Systems Run on Small Models

SyncSoft.AI — Tue, 02 Jun 2026 02:03:22 +0000

For most of 2024 and 2025, the default architectural answer to "what model should we use for this agent?" was: the biggest frontier model your budget could carry. In 2026, that default is breaking. A wave of small language models — Phi-4-mini, Qwen3.5-4B, SmolLM3-3B, Gemma-4-E2B, Mistral-7B — are quietly winning production agentic workloads. They are not winning because they beat frontier models on MMLU. They are winning because, for the narrow, schema-constrained, tool-calling-heavy work that real agents actually do, a well-fine-tuned 3B–7B model is faster, cheaper, more predictable, and easier to evaluate.

The interesting consequence — and the part most teams underestimate — is that this shift moves the engineering problem out of the model and into the data. If you are going to deploy a 4B model into a critical workflow, your training and evaluation data has to do work that the frontier models used to do for you by sheer scale. That is the real story of SLMs in 2026.

Why agentic workloads are unusually well-suited to small models

When you watch an agent run in production, three patterns dominate. The model emits a tool call against a fixed JSON schema. The model selects between a small, known set of next steps. The model summarizes or transforms a chunk of structured input into a structured output. Almost nothing the agent does requires the breadth of a 200B-parameter generalist. What it requires is reliability on a narrow distribution.

Narrow distributions are where small models shine. Recent surveys of agentic deployments have found that models in the 1–12B range are sufficient — and often superior — for workloads where the objectives are schema- and API-constrained. The frontier model's extra parameters are mostly paying for capabilities the agent never exercises: open-domain trivia, rare-language translation, creative writing. You are paying frontier prices for capacity you immediately throw away.

Latency is the second forcing function. An agentic loop with five tool calls multiplies model latency by five. A 4B model running locally or on a single H100 can complete a step in 50–200 ms; a frontier model through an API rarely beats 600–1500 ms per step. For a loop with ten steps, that is the difference between a four-second agent and a fifteen-second agent — and product teams notice fifteen seconds.

The third reason is operational. Smaller models are auditable. You can run a deterministic eval suite against every commit, you can fine-tune in hours instead of weeks, and you can deploy in environments — air-gapped, regulated, on-device — where shipping data to a frontier API is not an option. That last point matters more than it used to. Healthcare, finance, and ADAS teams in particular have spent the last year building SLM stacks specifically because their data cannot leave the building.

What changes when you go SLM-first

Here is the catch. The reason a 4B model performs well on your workload is not the model. It is the post-training. Phi-4's results are a useful proof point: Microsoft trained it on roughly 5T tokens, but the headline was that the data was reasoning-dense synthetic content, carefully filtered web material, and structured educational text. The model is small. The data was enormous and curated.

When you ship an SLM-first agent, three data problems become your problems instead of OpenAI's or Anthropic's:

1. Tool-call trace quality. A 4B model fine-tuned on a clean corpus of correct tool calls — with the right arguments, in the right schema, against realistic context — will outperform a frontier model used zero-shot on the same task. A 4B model fine-tuned on a messy corpus will hallucinate arguments, miss required fields, and silently produce JSON that almost validates. The gap between those two outcomes is entirely a function of how the training traces were collected, labeled, and validated.

2. Preference and trajectory correction. Tool calling is the easy part. The harder part is what the agent does when the tool returns something unexpected — an error, a partial result, a missing record. Frontier models recover gracefully because they have absorbed billions of human-corrected interactions. Your SLM has not. To get the same recovery behavior, you need RLHF-style preference data over agent trajectories: pairs of "this is what the model did" versus "this is what it should have done," labeled by people who actually understand the domain. Generic crowd labelers will not do it. Bilingual SME-led teams — which is what providers like SyncSoft.AI's reasoning and human feedback data service specialize in — are the practical way to source this kind of correction at scale.

3. Domain-grounded evaluation. You cannot ship an SLM into a regulated workflow on the strength of MMLU and HumanEval. You need a domain-specific benchmark — built from real failure modes in your real pipeline, with adversarial cases for the situations you care about. Production teams in 2026 are converging on a pattern: a held-out set of a few hundred carefully constructed prompts that exercise tool calling, multi-step reasoning, refusal behavior, and recovery, scored by a combination of programmatic checks and human review. That benchmark becomes the gate for every model update.

A concrete pattern that works

The teams shipping SLM-first agents successfully tend to converge on a similar pipeline. It is worth describing concretely because the steps are unglamorous and easy to underinvest in.

Start with a base model that already has strong tool-calling behavior — Qwen3.5-4B and Phi-4-mini are the current defaults, both Apache-2.0 or MIT licensed. Collect a few thousand traces of your target workflow being completed correctly. These can be human demonstrations, traces from a frontier model used as a teacher, or — most commonly — a mix. Have domain experts review and correct a meaningful fraction of those traces; this is the supervised fine-tuning corpus.

Run SFT on the base model. Evaluate against your domain benchmark. The first round almost never clears the bar. The interesting question is not "did it pass" but "what kinds of mistakes did it make." Almost all of them will fall into one of three buckets: schema violations (fix with more SFT examples covering the schema's edge cases), wrong tool selection (fix with preference pairs that contrast the right and wrong tool for ambiguous prompts), and bad recovery (fix with trajectory data showing how to handle tool errors).

Iterate. The right cadence in practice is weekly: collect last week's production failures, have annotators correct them, mix them into the next training run. After three or four cycles, the model's behavior on your workflow tightens dramatically. After ten, it tends to be more reliable on your specific task than a frontier model used zero-shot — because the frontier model has not seen your schema, your tools, or your error modes, and your model has seen little else.

The bottleneck in this loop is almost never compute. It is the speed and quality of the data work — particularly the trajectory correction, which has to be done by people who understand both the domain and the agentic pattern. Teams that try to crowdsource this with general labelers tend to stall; teams that work with SME-led annotation partners — for example through SyncSoft.AI's multimodal data annotation service — tend to keep the cadence going.

What this means for your stack

If you are building agentic systems in 2026, the question is no longer "which frontier model is best?" It is "do I have the data pipeline to make a small model good at my specific job?" Three practical implications:

Budget for data work, not just compute. The cost ratio is shifting: a typical SLM-first agent project spends two to four times more on labeled trajectory data than on GPU hours. That is the right ratio.

Build the evaluation benchmark before the model. Teams that build the eval first end up shipping faster, because they have an unambiguous signal for "is this better." Teams that build the model first spend months arguing about whether changes are real improvements.

Treat your data partners as part of the model team. Whether you build the annotation function internally or work with a specialist, the people producing your tool-call traces and preference data are functionally part of your ML engineering org. The handoff between "data partner" and "training team" is where most projects lose months. Pick partners — internal or external — who can ship reviewed traces on a weekly cycle, with real QA. Triple-pass QA pipelines are not overhead; they are the only way to keep the SFT corpus clean enough to be useful.

The model arms race will continue, and frontier models will keep their place — for research, for one-shot complex reasoning, for novel tasks where no domain data exists yet. But for the systems that run quietly inside products and ship value every day, the architecture is shifting under us. The next two years of competitive advantage in applied AI will be won by the teams that get their data flywheel right around a small, fine-tuned model — not the teams that pay for the largest one.

The author works at SyncSoft.AI, where we help AI teams build the data pipelines — SFT corpora, RLHF preference sets, agent trajectory corrections, and domain-grounded evaluations — that make small models production-ready. If you are wrestling with any of the patterns above, we would be glad to compare notes.

Coding Agents Don't Fail at the Start — They Fail in the Middle

SyncSoft.AI — Thu, 21 May 2026 09:24:10 +0000

If you've shipped anything built on a coding agent — a SWE-style PR bot, a computer-use agent, an autonomous refactor tool — you've probably noticed a strange pattern in the failures.

The agent reads the task correctly. It makes a clean first move. It looks like it's going to work. And then, twelve steps later, it hands you a confidently wrong result. Not a crash. Not a syntax error. A plausible answer that's quietly built on top of a mistake it made somewhere around step 4.

This is the part of agent behavior that almost no one talks about, and it's the part that decides whether your agent is a demo or a product.

Outcomes are easy to measure. Trajectories are not.

Here's the uncomfortable truth about how most coding agents are trained and evaluated: we optimize for the outcome and ignore the path.

Think about how a benchmark like SWE-bench works. There's an issue, there's a "gold" patch, and there's a test suite. The agent either makes the tests pass or it doesn't. Pass@1 goes up, everyone celebrates.

That signal is real, but it's also incredibly coarse. A binary pass/fail at the end of a 30-step trajectory tells you that the agent failed. It tells you nothing about where or why. Two agents can both score 0% on a task and have failed for completely different reasons — one misread the issue, the other had the right plan but botched a single file edit on step 9 and never recovered.

When your training signal is "did the final state match," you get models that are very good at producing things that look like correct final states. You do not get models that are good at noticing when they've wandered off the path.

The "first wrong step" is where the value is

If you sit down and actually annotate failed agent trajectories — step by step, the way a senior engineer would review a junior's work — one observation shows up over and over:

There is almost always a single, identifiable step where the trajectory first goes wrong.

Everything before that step is fine. Everything after it is conditioned on a broken state, so it's also going to look wrong — but those later steps aren't the real bug. They're downstream symptoms. The agent picked the wrong file to edit, or misread a stack trace, or assumed a function signature, and then it spent the next twenty steps reasoning impeccably about a world that no longer existed.

That first divergence point is the highest-information label you can attach to a trajectory. It isolates the causal error from the noise. And it's exactly the thing outcome-only data throws away.

A trajectory labeled only "failed" teaches a model almost nothing. A trajectory labeled "failed; first wrong step is #7; here is why #7 was wrong; here is the action that should have been taken instead" is a genuine teaching signal.

Agents need to be taught recovery, not just correctness

There's a second pattern that's just as important and gets even less attention.

Real engineers don't execute a perfect plan from start to finish. They make a wrong move, notice, back up, and try something else. That recovery loop — detect, diagnose, correct, continue — is most of what senior engineering actually is.

Coding agents are largely not trained to do this, because the data we feed them rarely contains it. Instruction-tuning datasets are full of clean (problem → correct solution) pairs. They are essentially a highlight reel. They show the model a world in which mistakes never happen, so the model never learns what the inside of a mistake feels like or how to climb out of one.

If you want an agent that recovers, you have to show it recovery. That means training data that deliberately includes:

A trajectory that goes wrong at a known step.
The moment of detection — what signal should have told the agent something was off (a failing test, an unexpected diff, a tool error it shrugged off).
The corrected reasoning at that step.
The next good action, and the continuation toward a real completion.

This is a fundamentally different artifact from a static (prompt, response) pair. It's a record of judgment under uncertainty, and it has to be produced by people who can actually do the underlying engineering work — because labeling the first wrong step in a multi-file refactor is itself a hard engineering task. It's the core of what specialized reasoning-data and trajectory-correction work looks like in practice.

What this means for how you build

You don't need to be training a frontier model to act on any of this. A few things are worth doing on almost any agent project:

Log full trajectories, not just outcomes. Every step, every tool call, every observation. If your telemetry only captures "task succeeded / failed," you've already lost the data you need to debug the agent. You can't fix what you can't see.

Evaluate at the step level. Outcome accuracy is a fine north-star metric, but it's a terrible debugging tool. Build eval sets where you know the correct trajectory, so you can measure where divergence happens and not just whether it happened. A heatmap of "which step do failures originate from" is worth more than another pass@1 number.

Build evals that contain mid-trajectory failure. If every example in your eval starts from a clean state, you are never testing recovery. Seed some evals with a deliberately broken intermediate state and measure whether the agent notices. Most don't. That gap is your roadmap.

If you fine-tune, invest in trajectory and correction data, not just more instruction pairs. The marginal (problem → solution) example is cheap and low-value. The marginal annotated failure-and-recovery trajectory is expensive and high-value. Spend accordingly.

The teams getting real reliability out of coding agents in 2026 aren't the ones with the cleverest prompts. They're the ones who treat the agent's path as a first-class object — something to be logged, labeled, evaluated, and trained on — instead of staring only at the final diff.

The middle of the trajectory is where your agent actually lives. It's worth looking there.

Disclosure: I work at SyncSoft.AI, where a chunk of our work is building exactly this kind of data — agent trajectory annotation, first-wrong-step labeling, and reasoning-alignment / RLHF datasets for teams training coding and computer-use agents. If you're wrestling with mid-trajectory failures and want to compare notes, I'm happy to talk. Opinions here are my own.