If you've shipped anything built on a coding agent — a SWE-style PR bot, a computer-use agent, an autonomous refactor tool — you've probably noticed a strange pattern in the failures.
The agent reads the task correctly. It makes a clean first move. It looks like it's going to work. And then, twelve steps later, it hands you a confidently wrong result. Not a crash. Not a syntax error. A plausible answer that's quietly built on top of a mistake it made somewhere around step 4.
This is the part of agent behavior that almost no one talks about, and it's the part that decides whether your agent is a demo or a product.
Outcomes are easy to measure. Trajectories are not.
Here's the uncomfortable truth about how most coding agents are trained and evaluated: we optimize for the outcome and ignore the path.
Think about how a benchmark like SWE-bench works. There's an issue, there's a "gold" patch, and there's a test suite. The agent either makes the tests pass or it doesn't. Pass@1 goes up, everyone celebrates.
That signal is real, but it's also incredibly coarse. A binary pass/fail at the end of a 30-step trajectory tells you that the agent failed. It tells you nothing about where or why. Two agents can both score 0% on a task and have failed for completely different reasons — one misread the issue, the other had the right plan but botched a single file edit on step 9 and never recovered.
When your training signal is "did the final state match," you get models that are very good at producing things that look like correct final states. You do not get models that are good at noticing when they've wandered off the path.
The "first wrong step" is where the value is
If you sit down and actually annotate failed agent trajectories — step by step, the way a senior engineer would review a junior's work — one observation shows up over and over:
There is almost always a single, identifiable step where the trajectory first goes wrong.
Everything before that step is fine. Everything after it is conditioned on a broken state, so it's also going to look wrong — but those later steps aren't the real bug. They're downstream symptoms. The agent picked the wrong file to edit, or misread a stack trace, or assumed a function signature, and then it spent the next twenty steps reasoning impeccably about a world that no longer existed.
That first divergence point is the highest-information label you can attach to a trajectory. It isolates the causal error from the noise. And it's exactly the thing outcome-only data throws away.
A trajectory labeled only "failed" teaches a model almost nothing. A trajectory labeled "failed; first wrong step is #7; here is why #7 was wrong; here is the action that should have been taken instead" is a genuine teaching signal.
Agents need to be taught recovery, not just correctness
There's a second pattern that's just as important and gets even less attention.
Real engineers don't execute a perfect plan from start to finish. They make a wrong move, notice, back up, and try something else. That recovery loop — detect, diagnose, correct, continue — is most of what senior engineering actually is.
Coding agents are largely not trained to do this, because the data we feed them rarely contains it. Instruction-tuning datasets are full of clean (problem → correct solution) pairs. They are essentially a highlight reel. They show the model a world in which mistakes never happen, so the model never learns what the inside of a mistake feels like or how to climb out of one.
If you want an agent that recovers, you have to show it recovery. That means training data that deliberately includes:
- A trajectory that goes wrong at a known step.
- The moment of detection — what signal should have told the agent something was off (a failing test, an unexpected diff, a tool error it shrugged off).
- The corrected reasoning at that step.
- The next good action, and the continuation toward a real completion.
This is a fundamentally different artifact from a static (prompt, response) pair. It's a record of judgment under uncertainty, and it has to be produced by people who can actually do the underlying engineering work — because labeling the first wrong step in a multi-file refactor is itself a hard engineering task. It's the core of what specialized reasoning-data and trajectory-correction work looks like in practice.
What this means for how you build
You don't need to be training a frontier model to act on any of this. A few things are worth doing on almost any agent project:
Log full trajectories, not just outcomes. Every step, every tool call, every observation. If your telemetry only captures "task succeeded / failed," you've already lost the data you need to debug the agent. You can't fix what you can't see.
Evaluate at the step level. Outcome accuracy is a fine north-star metric, but it's a terrible debugging tool. Build eval sets where you know the correct trajectory, so you can measure where divergence happens and not just whether it happened. A heatmap of "which step do failures originate from" is worth more than another pass@1 number.
Build evals that contain mid-trajectory failure. If every example in your eval starts from a clean state, you are never testing recovery. Seed some evals with a deliberately broken intermediate state and measure whether the agent notices. Most don't. That gap is your roadmap.
If you fine-tune, invest in trajectory and correction data, not just more instruction pairs. The marginal (problem → solution) example is cheap and low-value. The marginal annotated failure-and-recovery trajectory is expensive and high-value. Spend accordingly.
The teams getting real reliability out of coding agents in 2026 aren't the ones with the cleverest prompts. They're the ones who treat the agent's path as a first-class object — something to be logged, labeled, evaluated, and trained on — instead of staring only at the final diff.
The middle of the trajectory is where your agent actually lives. It's worth looking there.
Disclosure: I work at SyncSoft.AI, where a chunk of our work is building exactly this kind of data — agent trajectory annotation, first-wrong-step labeling, and reasoning-alignment / RLHF datasets for teams training coding and computer-use agents. If you're wrestling with mid-trajectory failures and want to compare notes, I'm happy to talk. Opinions here are my own.
Top comments (0)