The Agent RL Trick Is Making the Model Explain Its Own Mess

A new paper called SEED dropped on arXiv yesterday, and the interesting part is not the usual "agentic RL got better" headline. The paper is trying to fix a very ordinary problem in agent training: the model gets a pass/fail signal at the end of a long task, but most of the actual mistakes happened five, ten, or fifty actions earlier.

That is the annoying part of training agents. A coding agent, browser agent, or game agent can wander through a task for minutes, take a wrong turn, recover accidentally, then fail at the end. A terminal reward says "bad run." It does not say whether the bad move was the search query, the file it opened, the command it trusted, or the moment it forgot the goal.

SEED's bet is simple: after the run is over, make the current policy read its own trajectory and turn that mess into reusable hindsight.

Not a motivational poster. A training signal.

The sparse reward problem is worse for agents

Classic reinforcement learning can often get away with a thin reward because the action space is constrained. Move left, move right, pick an action, get feedback. Language-model agents are messier. The action is text. The environment may be a shell, a browser, a shopping site, a text game, or a visual planning task. The model is not just picking an action; it is writing the interface to its next action.

That makes credit assignment ugly.

If an agent fails a WebShop task, the final score may know that the purchase was wrong. It does not know that the model filtered on the wrong attribute three pages earlier. If an ALFWorld agent fails to put an object in the right place, the failure may come from an early bad assumption about where objects are likely to be, not the final command.

Outcome-only RL can still improve behavior, but it wastes a lot of runs learning what a human would call obvious after reading the transcript.

SEED tries to extract that obvious-after-the-fact part automatically.

What SEED actually does

The paper calls the method Self-Evolving On-Policy Distillation. Stripped down, it has two moving parts.

First, the model is trained to look at completed trajectories and write hindsight skills: short natural-language lessons such as reusable workflows, decisive observations, or failure-avoidance rules. These are not prompts used at inference time. They are privileged training-time explanations derived from the run that already happened.

Second, during RL, the latest policy plays both roles. It collects new trajectories, then analyzes those same trajectories. The system compares how likely the policy was to produce its sampled actions with and without the hindsight skill in context. That probability shift becomes a dense token-level distillation signal, trained alongside the normal outcome-based RL objective.

In plain English: the model watches itself fail or succeed, writes down what mattered, then uses the difference between "ordinary me" and "me with the lesson visible" as extra supervision.

The important detail is that the hindsight comes from on-policy trajectories. It is not a static teacher handing down generic advice. As the policy changes, the mistakes change, and the lessons change with it. That is the "self-evolving" part.

The generated skills also disappear at inference time. The deployed policy is not carrying an extra scratchpad of hints. If the method works, the behavior has been internalized.

The numbers are good, but the shape matters more

The paper reports gains across embodied interaction, web navigation, search-based QA, and visual planning tasks. On ALFWorld, SEED reaches 91.8 average with Qwen2.5-3B. The project page says it beats full-data GRPO using only 60% of the training data. In the paper's detailed sample-efficiency table, SEED at 60% data hits 80.7 on ALFWorld, while GRPO at 100% data reaches 75.0.

On unseen ALFWorld tasks, the average goes from 70.9 for GRPO to 86.2 for SEED, a +15.3 gain. For visual tasks, the project page reports 91.0 average versus 77.0 for GRPO.

Those are paper numbers, so apply the usual caution. Benchmarks are benchmarks. But the pattern is the part worth keeping: denser hindsight supervision seems to make each trajectory count for more.

That tracks with how debugging agents feels in practice. The failure transcript is usually rich. The reward is usually poor. SEED is a way to stop throwing away the transcript after reducing it to a single scalar.

Why this is more useful than "give the agent a memory"

A lot of agent systems respond to failure by adding memory. Store the mistake. Retrieve it later. Hope it matches the next situation.

That can help, but it also creates a new operations problem. Which memories are still true? Which ones were artifacts of a bad environment? Which ones leak task-specific trivia into a new context? How much retrieval is enough before the agent starts dragging a filing cabinet through every run?

SEED takes a cleaner route. It uses the completed trajectory as training data, not as a permanent runtime dependency. The lesson is distilled into the policy instead of stapled to the prompt.

That distinction matters. Runtime memory is a product feature. Training-time hindsight is a learning mechanism. One makes the agent remember a note. The other tries to make the agent less likely to need the note.

I like that boundary.

The obvious risk: the model is grading its own homework

There is also a reason to be cautious. SEED asks the current policy to act as both actor and analyzer. That is elegant, but it raises the usual self-training problem: if the analyzer has a blind spot, it may distill the wrong lesson with confidence.

The paper tries to reduce this by keeping the signal on-policy and comparing ordinary versus skill-augmented probabilities on the sampled actions. Still, the analyzer is not magic. If the model misunderstands the environment, its hindsight can be tidy and wrong.

That is not a dealbreaker. Humans do the same thing. We write postmortems that explain the last outage too neatly, then get surprised by the next one. The fix is not "never write postmortems." The fix is to treat them as training signals, not scripture.

For agents, the same rule applies: hindsight is useful when it changes behavior and survives new tasks. It is dangerous when it becomes a prettier name for overfitting.

The bit I would steal

If I were building an agent harness today, I would not start by copying the full RL method. Most teams do not have that loop running. I would steal the shape:

After every meaningful run, generate a short hindsight note from the transcript:

• what decision actually moved the task forward,
• what assumption wasted time,
• what check would have caught the failure earlier,
• what should change in the next run.

Then do not blindly paste that note into every future prompt. Use it as review material for the harness, tests, evals, and retry policy. If the same hindsight note appears three times, it probably belongs in the system design, not in a growing memory blob.

That is the practical lesson here. Agent improvement is not just "more rollouts" or "bigger model." It is better use of the boring evidence you already paid for: the full trace of what the agent did.

SEED is a research paper, not a drop-in recipe for every developer workflow. But it points at the right abstraction. The transcript is not a log file to archive after the reward lands. It is where the useful supervision is hiding.

The agent already made the mess. You might as well make it explain the mess before you throw the run away.

---

Sources: SEED paper on arXiv, project page, Hugging Face paper page.