The easiest way to feel productive is to collect more successful trajectories and throw them at the model. It's the default move. Your agent passes a test, you log the whole episode, you add it to the training pile, and you tell yourself the model is getting smarter. More passes, more data, more intelligence. That's the story.
SWE-Prime says the story is wrong. The paper — https://arxiv.org/abs/2608.27449v1 — trains on a curated 10% of trajectories and beats training on all the successful ones. Not matches. Beats. Fewer trajectories, better performance. That's not a small correction, that's the myth busted.
Let me be clear about what the myth is, because it's seductive. The myth isn't "data helps." The myth is "successful trajectories are uniformly good training data." You filter for the pass, you assume the pass is the signal, and you feed the whole thing in. SWE-Prime's result is that the pass is a terrible filter. A trajectory can be successful and still be garbage.
I've seen this pattern in my own agents, and I bet you have too. The trajectory that passes after 14 retries on the same tool call. The one that wanders through five wrong approaches before stumbling into the right one. The one that passes because the test was weak, not because the agent was strong. All of those get labeled "successful" and dumped into the training set. The model learns the wandering. It learns the retries. It learns that flailing eventually works, because that's what the data shows it.
That's the real insight in SWE-Prime, and it's worth sitting with: filter by segment quality, not trajectory success. A trajectory is a coarse label — one bit, pass or fail. But a trajectory is made of segments, and each segment has its own quality. A good trajectory can contain bad segments. A failed trajectory can contain brilliant segments. If you're labeling at the trajectory level, you're throwing away the signal and keeping the noise.
What does segment quality actually look like in practice? It's the difference between an agent that reads the failing test, forms a hypothesis, makes one targeted edit, and verifies — versus an agent that greps the codebase for twenty minutes, edits three unrelated files, breaks two other tests, and finally stumbles onto a fix. Both trajectories end in a pass. One of them is teaching the model to think. The other is teaching it to flail. The pass label can't tell them apart, but the segments can.
The paper's approach — curating down to 10% — is essentially saying: most of what you're training on is teaching the model the wrong thing, and the small slice that's actually good is enough. That's a hard claim to swallow if you've spent months building data pipelines. But it matches what I've seen in practice. When I've pruned training data aggressively — cutting the meandering episodes, keeping the tight ones — the model gets sharper, not dumber. I never had the discipline to cut to 10%, but the direction was always right.
There's a cost angle here that matters if you pay the GPU bill like I do. Training on 10% of the data is cheaper. It's faster. It's less compute per epoch, less time waiting for runs, less money on the invoice. And you get better results. That's the rare outcome where the cheap option is also the correct option. Usually you pay more for better. Here, you pay less and get more. That alone should make you question the "more data" reflex.
Now, the caveats, because I'm not going to pretend this is settled. I haven't reproduced SWE-Prime yet. It's a paper, and papers don't always survive contact with production. The curation itself is the hard part — "filter by segment quality" is easy to say and hard to do. What's a good segment? How do you score it without a reward model that's as expensive as the thing you're training? The paper has an answer, but the answer might not transfer to your domain. And there's a scale question — what works at the paper's scale might not hold when you're training on millions of trajectories. Maybe I'm wrong here. But the core claim — that success is a bad filter — feels robust to me.
The practical takeaway is simple, and it's the one I'm actually going to act on: stop treating "passed the test" as the bar for training data. Start looking at the segments. If a trajectory is successful but meandering, cut it. If it's full of wasted tool calls, cut it. If it only passed because the test was weak, cut it. Keep the tight ones. Keep the ones where the agent knew what it was doing from the first step. Your model will thank you, and so will your GPU bill.
The myth that more successful trajectories always help dies here. It was never about the volume. It was about the quality of the segments, and we were too lazy to look.
Top comments (0)