DEV Community: Vatche Isahagian

Your Agent Aced the Task. Will It Do It Again?

Vatche Isahagian — Wed, 21 Oct 2026 16:34:48 +0000

Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task.

That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper.

Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap.

Most benchmarks report the first number. We built a way to measure the second — and improve it.

In an earlier post, we introduced ALTK-Evolve — a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines, a new guideline type in altk-evolve built on top of a diagnostic tool we call the Consistency Analyzer, that targets this gap directly.

TL;DR

Accuracy hides an unreliability problem. A ReAct agent (GPT-4.1 on AppWorld test_normal) that succeeds 77.4% of the time on average succeeds on all 5 repeated runs for only 53.0% of tasks — a 24.4-point consistency gap. On hard tasks it reaches 30 points.
We built a diagnostic for exactly this. The Consistency Analyzer resamples an agent's own recorded trajectory to find flip-prone decision points — steps where the model was one token-sample away from doing something different. It needs one trace and no ground truth — it resamples each decision point in that trace with a single call requesting k completions (k=5 by default), rather than re-running the task end-to-end.
Turning that diagnosis into guidelines halves the gap — from 24.4pp to 12.0pp (same-task Pass⁵ +16.0pp, similar-task +13.0pp), without costing anything in average accuracy.
Full methodology and evaluations are in the technical report on arXiv.

The Metric Almost Nobody Reports

Standard agent evaluation reports Mean@k: run a benchmark k times, average the pass rate. Often k=3, sometimes just 1. It's the number on every leaderboard, and it's what "77% accurate" means in practice.

Mean@k answers "how good is this agent, on average?" It does not answer the question a real user cares about: will it still be good if I ask this exact question again? For that you need Pass^k: the fraction of tasks where the agent succeeds on all k runs.

⚠️ Pass^k is not Pass@k. The familiar Pass@k is optimistic — it asks whether at least one of k attempts succeeded, the right question when you can verify and retry. Pass^k is its pessimistic mirror image: every attempt must succeed. Same letters, opposite question. Pass^k ≤ Mean@k ≤ Pass@k, always.

A ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4% — genuinely strong. But Pass^5 is only 53.0%. Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes can't, with nothing about the task changing between runs. We call this gap — Mean@k minus Pass^k — the consistency gap.

This isn't a capability problem you fix with a bigger model. It's an orthogonal axis: an agent can be capable and inconsistent at the same time.

Why Agents Flip: Sharp Decisions vs. Flat Ones

Every time an LLM agent decides something — which API to call, what argument to pass, whether to retry — that decision comes out of a probability distribution over next tokens. What matters is the shape of that distribution. A sharp one puts most of its mass on a single token: the runners-up are far behind, and the same choice comes out run after run. A flat one spreads comparable mass across several near-tied tokens, and which one wins is close to a coin flip.

The shape decides how much noise it takes to change the outcome. Sharp distributions are resilient — GPU floating-point non-associativity, request batching, and other platform-side effects nudge the numbers slightly, but nowhere near enough to reorder a clear winner. Flat distributions are vulnerable to exactly that nudge: near-ties may reorder under small perturbations. And because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. That's where a 24-point gap comes from.

This is also why the problem survives your decoding settings. Greedy decoding and a fixed seed both govern how a distribution gets turned into a token — they say nothing about the distribution itself. On a hosted endpoint the probabilities shift slightly from run to run, so the same prompt to the same model at temperature zero can still resolve a near-tie one way today and the other way tomorrow.

Our setup: the ReAct agent runs at temperature 0.0, so none of the variance above is ordinary sampling.

Diagnose, Then Fix

Which turns the problem into a search: which steps in a given trajectory were the flat ones — and what do you do about them once you know?

Consistency guidelines come out of a two-stage pipeline that plugs into ALTK-Evolve's existing machinery — with a new source signal driving what gets written.

1. Detect — the Consistency Analyzer.
Given one recorded trajectory, the analyzer replays each decision step through controlled resampling, measuring how much the model's output actually varies at that point. Concretely, that's one additional model call per decision step, done once offline — issued with the sampling parameter set to draw k completions at once (k=5 by default) — replayed against the already-recorded context, not new tool calls, not new environment interactions, and not a second end-to-end rollout of the task. This yields a consistency score per decision step that is written into a scorecard to pinpoint exactly which decisions are at risk of flipping on the next run. Detection is fully black-box — no logits, no model internals, no instrumentation beyond the trace you already have.

2. Generate — targeted guidelines. Every flagged step becomes a candidate consistency guideline in the standard ALTK-Evolve format, so it slots into the existing storage and retrieval pipeline. Here's a real example, generated by GPT-4.1 from a trajectory of the AppWorld task "How many activities are done in my bucket list as per my SimpleNote note?":

[Guideline 1] When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count — note titles often repeat the marker symbol in a legend line.

[Guideline 2] Always verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding.

Nothing here is task-specific trivia. String-counting bugs and unverified search results are decision points that show up with high uncertainty across many AppWorld tasks. That's the point: the analyzer targets instability, not failure — so it catches steps the agent happened to get right this time but could easily get wrong next time.

Watch the 2-minute demo — five parallel runs of the agent split 3-2 on this task because of agent uncertainty about the counting strategy, then run again after these guidelines in context: all five agree.

Results: Reducing the Gap Without Losing Accuracy

We evaluated on AppWorld test_normal (168 tasks) with a ReAct agent on GPT-4.1, generating consistency guidelines from a single baseline trajectory per task and testing them on 5 fresh runs.

Mean@5 (%), aggregate — same scale as Pass^5 above.

The consistency gap is cut roughly in half. Aggregate Pass^5 rises 53.0% → 69.0% while Mean@5 rises 77.4% → 81.0%, narrowing the gap between "looks capable" and "can be counted on" from 24.4pp to 12.0pp. Nearly a third of previously-inconsistent tasks become tasks the agent passes on every single run.

The middle and hard tiers gain most. Medium +22.9pp (+44% relative), Hard +14.3pp (+45% relative) — effectively tied in relative terms, with Medium ahead absolutely. Easy gains +12.2pp, having had the least room. This is consistency guidelines doing what they're designed to do: finding and stabilizing the specific decision points where an agent's own uncertainty was leaking into the outcome.

Mean@5 never drops. Preserving average accuracy was a hard requirement, not a nice-to-have: a system that boosts Pass^5 by trading away Mean@5 would just be shifting unreliability around, not fixing it. Mean accuracy holds or improves at every difficulty level.

The guidelines generalize — they aren't patching one trajectory

Applied to a different but related task in the same AppWorld scenario — another variant of the scenario the guidelines were mined from — consistency guidelines still lift Pass^5 by +13.0pp, only 3 points below the same-task number. A guideline derived from one run isn't just patching that run; it's capturing something that transfers.

The sharper evidence comes from a weaker model, gpt-oss-120b. Same-task Pass^5 rose +6.0pp from a much lower baseline (10.1% → 16.1%) — and, interestingly, the similar-task generalization number (+8.7 pp) actually exceeded the same-task gain, suggesting the guidelines were capturing genuinely reusable failure patterns rather than memorizing one trajectory's specifics.

If You're Shipping an Agent

Report Pass^k next to Mean@k. Averages can't distinguish a reliable agent from a lucky one; even k=3 will surface a gap you didn't know you had.
Expect the gap to widen with difficulty. Your hardest tier is where a single averaged number is most misleading.
Don't reach for a bigger model first. Consistency is orthogonal to capability. A stronger model raises Mean@k; it doesn't necessarily reduce the consistency gap.
Diagnosis needs no grader and no live replay. One extra LLM call per decision step (sampling k=5 completions by default) is enough — no ground truth, no re-running the task against the environment. That's what makes it usable on production traffic, where you often can't replay a task end-to-end even once.

Try It

Try the ALTK-Evolve — the open-source repo now includes the Consistency Analyzer and consistency-guideline generation used in these experiments — or read the technical report on arXiv for the complete methodology.

If accuracy numbers you can't reproduce on your own tasks sound familiar, we'd like to hear about it — concrete examples of flip-prone behavior in your own agents are exactly the kind of feedback that shapes what we build next. Open an issue or a discussion.

Appendix: Understanding the Metrics

Mean@k. Run a task k times, report the average pass rate — what most benchmarks call "accuracy."
Pass^k. The fraction of tasks where the agent succeeds on all k independent runs. Always ≤ Mean@k. What a user experiences if they run the same query twice.
Pass@k At least one of k runs succeeds — the optimistic counterpart, common in code-generation papers.
Consistency gap. Mean@k − Pass^k, in percentage points.

Linked artifacts / references

ALTK-Evolve open source repo — github.com/AgentToolkit/altk-evolve
Technical report — arXiv
Consistency guideline demo —2 min video

How Much Memory Does Your Agent Actually Need?

Vatche Isahagian — Mon, 05 Oct 2026 18:21:37 +0000

In our previous post, we compared ALTK-Evolve with ACE and showed that how you deliver an agent's self-distilled guidelines — a few retrieved per task vs. the whole set injected — drives both accuracy and cost. This post steps back to the question that comes before it: how much should you give it?

Equipping an agent with agentic memory sounds simple: distill lessons from its past work, put them back in context, and more experience should mean better performance. It doesn't always work that way. When we scaled the evaluation to eight models — from a 30B dense model to frontier proprietary systems — one finding stood out:

Agentic memory is not a feature you switch on. It's a dose you calibrate to the model.

TL;DR

ALTK-Evolve lets an agent learn from its own past trajectories: distilling reusable guidelines and injecting them back at inference time, with no weight updates and no human annotation.
The right dose differs by model tier: strong models with headroom want the full guideline set, weaker models do best with a compact core plus per-task retrieval, and saturated models show no measurable gain.
Curated retrieval can be both the most accurate and the cheapest option: gpt-oss-120b gained +16.1pp task completion at only +5% tokens — and prompt caching keeps even the full guideline set affordable in production.

The Key Insight: Dosage Depends on Capability

Not every model benefits from the same amount of memory. Across eight models spanning the capability spectrum, we saw three recurring patterns:

Strong models with headroom want the full guideline set — every guideline, including rare edge-case lessons. They have the capacity to absorb and apply all of it. DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in task completion when given its full self-mined guideline set.
Smaller or weaker models get drowned by a large guideline set. For these, a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task works best. gpt-oss-120b (117B MoE) gained +16.1pp with this selective approach — while the full guideline set gained less and cost ~50% more tokens.
Already-saturated models show no measurable gain. We call this the saturated pattern — the label describes what we observed, not a proven cause. The model may already have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. GLM-5 (745B MoE) sat here in our runs.

What puts a model into one pattern rather than another isn't simply parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all appear to shape where a model lands, and separating those factors is ongoing work. The practical takeaway holds either way: the right dose of memory depends on the model, and we can calibrate it.

Learning happens around the model, not inside it

"Memory" here doesn't mean replaying a past transcript. It means a guideline set — strategies that worked, mistakes to avoid, and edge cases — distilled from the agent's own prior trajectories. The loop is straightforward:

The agent attempts tasks and produces trajectories.
ALTK-Evolve extracts behavioral guidelines from both its successful and unsuccessful runs.
It consolidates those guidelines into a reusable set.
At inference time, the agent receives either the full guideline set or a task-relevant selection of it.

No model weights are updated. The learning loop changes the guidance available to the agent, not the underlying model — which is exactly why it's cheap to adopt and portable across the eight models we tested.

Results Across the Spectrum

We evaluated on AppWorld — 585 multi-step tasks (168 test_normal + 417 test_challenge) across 9 simulated apps (calendars, messaging, payments, and so on). Tasks are scored two ways: whether the agent fully completes each task (TGC — Task Goal Completion) and whether every variant of a scenario passes (SGC — Scenario Goal Completion, a stricter, all-or-nothing bar). Full definitions are in the appendix.

The three configurations we compare

Because the confusing part of any memory study is what's actually in the context window, we define the configurations up front.

Both memory configurations draw from the same guideline set, mined once (via the loop above) from AppWorld's training split only. What changes between them is only how that one set is delivered — the full guideline set injects all of it every step, while curated retrieval delivers a selected subset — never how the guidelines were produced, and no test-split data ever goes into building it.

Configuration	What's in the agent's context
Baseline	No memory — the agent as shipped.
Full guideline set	Every mined guideline, injected on every ReAct step.
Curated retrieval	A fixed, high-confidence core of those same guidelines plus a few task-relevant ones retrieved for each task (a fixed portion + a variable portion).

The number of guidelines a model mines depends on its own capability, so we report configurations by strategy — "full guideline set" vs. "curated retrieval" — rather than by raw counts, which aren't comparable across models.

The three patterns, in one view

Representative models from the eight-model sweep, measured by task completion (TGC) on test_normal:

Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld test_normal for baseline vs. the best-memory configuration; the x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below.

The figure plots TGC to keep it readable; the table adds the stricter SGC metric, where the gains are often larger:

Model	Pattern	Baseline TGC / SGC	Best-memory TGC / SGC	Best config	Δ TGC	Δ SGC
gpt-oss-120b (117B MoE)	Weak / selective	39.9 / 21.4	56.0 / 37.5	curated retrieval	+16.1	+16.1
DeepSeek-V3.2 (671B MoE)	Strong w/ headroom	79.8 / 64.3	89.3 / 80.4	full guideline set	+9.5	+16.1
Claude Opus 4.6	Strong w/ headroom	90.5 / 87.5	94.6 / 94.6	full guideline set	+4.1	+7.1
GPT-5.5	Strong (near-ceiling)	92.3 / 82.1	95.2 / 89.3	full guideline set	+2.9	+7.2
GLM-5 (745B MoE)	Saturated	87.5 / 80.4	87.5 / 80.4	full guideline set	0.0	0.0

Reading the SGC column, the stricter metric usually moves more than TGC — DeepSeek's SGC jumps +16.1pp against a +9.5pp TGC gain — because good guidelines especially help an agent clear every variant of a scenario, not just the average case. And the effect doesn't disappear at the top of the range: GPT-5.5 and Opus, both near the ceiling on TGC, still gain +7.2 and +7.1pp SGC respectively. Memory keeps paying off as long as a model has a remaining failure mode to target.

The Cheapest Memory Strategy Can Also Be the Best

A practical concern: injecting a full guideline set inflates every ReAct step's input, because the guidelines are re-sent each turn. Here's what we observed:

Model	Config	Tokens/task (baseline)	Tokens/task (+ memory)	Overhead
DeepSeek-V3.2	full guideline set	148K	263K	+78%
gpt-oss-120b	full guideline set	110K	166K	+51%
gpt-oss-120b	curated retrieval	110K	116K	+5%

Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline.

Two takeaways:

Curated retrieval keeps cost near baseline. For weaker models, where selection wins on accuracy, it also wins on cost — the best of both worlds (+16.1pp TGC at only +5% tokens for gpt-oss-120b). Better performance here does not require more inference cost.
Memory doesn't blow up the reasoning loop. DeepSeek runs about the same number of ReAct steps with memory as without (≈18–19 on average), so the added cost is input-token inflation, not longer trajectories.

The real efficiency lever in production is prompt caching: the static portion of the guideline set is identical across steps and can be cached, cutting effective cost substantially. Cache-aware prompt design — keeping the shared guideline-set prefix stable so it stays cacheable — is worth engineering for. We also hypothesize that context-window size plays a role: models with larger windows may absorb the full guideline set more effectively, while smaller-context models benefit more from retrieval that keeps injected content compact. We have not yet run controlled experiments isolating this factor.

Memory Should Be Calibrated, Not Merely Accumulated

The lesson isn't to give an agent everything it has learned. It's to give it the amount of experience it can actually use.

For weak models, that means a compact core plus a few task-specific lessons — which, conveniently, is also the cheapest option.
For strong models with headroom, it means preserving the full guideline set, kept affordable in production via prompt caching.
For saturated models, it means spending no extra context until their remaining failure modes are better understood.

The gains are real across the board — automatic, leakage-free, and requiring no human annotation — but only when the dose fits the model.

What's Next

This is a starting point, not the finish line:

A learned selector. Our current retrieval ranks guidelines by cosine similarity, which we've shown doesn't perfectly predict which guidelines help a given task. A selector trained on outcome signal is the natural next step.
Memory for very weak models. Below a minimum capability baseline, self-distillation lacks signal. Teacher-distilled memory for very weak models is a separate problem we're exploring.
Beyond AppWorld. These results are validated on AppWorld — a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress.
Isolating context window. As above, we want controlled experiments that separate context-window size from raw capability.

Try the ALTK-Evolve library — which includes the extraction, consolidation, and retrieval pipeline used here — or read the full technical report for the complete method and ablations.

Appendix: Understanding the Metrics

AppWorld tasks are graded by two metrics, both reported as percentages (higher is better):

TGC — Task Goal Completion. The share of individual tasks the agent completes fully and correctly. This is the headline "did it get the job done" number.
SGC — Scenario Goal Completion. A stricter, all-or-nothing metric. Each scenario bundles several variants of the same task (the same request with different data, phrasing, or edge conditions). SGC counts a scenario as passing only if the agent succeeds on every variant. It measures reliability — an agent that solves a task most of the time but fails on one variant scores on TGC but not on SGC.

Thinking of Stanford’s Agentic Context Engineering? We Can Do It with Fewer Tokens

Vatche Isahagian — Thu, 24 Sep 2026 15:08:31 +0000

TL;DR

ACE and ALTK-Evolve both turn agent trajectories into reusable lessons without updating model weights.
The key difference is delivery: ACE supplies the full playbook, while ALTK-Evolve calibrates how many guidelines each model receives.
In our controlled AppWorld runs, ALTK-Evolve matched or exceeded ACE while using about 40% of the tokens on DeepSeek-V3.2 and one-seventh on gpt-oss-120b.

Give an LLM agent a realistic multi-step task — split a bill, find a song, reconcile an order across nine simulated apps — and when it fails, it usually isn't for lack of knowledge. It mis-paginates an API, resolves the wrong person, or returns a value when none was asked for. The model knows the APIs; what it hasn't internalized is how to use them reliably. That's learnable from the agent's own history.

Two recent systems do exactly this, on the same kind of agent: ACE (Agentic Context Engineering) and our ALTK-Evolve (introduced here). Both are a form of agentic memory — turning an agent's past trajectories into reusable lessons and feeding them back at inference time, no weight updates, no human labels. They even agree on the hard part. Where they part ways is delivery.

A note on words, because the two systems name things differently: we'll call the raw thing an agent learns a lesson. ACE organizes its lessons into one comprehensive, evolving playbook; we consolidate ours into individually retrievable guidelines. Same lessons, two containers.

What we agree on

Both systems refuse to compress.

ACE names the failure modes precisely: brevity bias — optimization collapsing toward short, generic instructions — and context collapse — a model asked to rewrite its whole context each step summarizing the detail away. Its answer is to keep a rich, itemized playbook, with a helpful/harmful counter on every bullet, and let the model distill relevance at read time.

We reach the same conclusion from the other direction. Every distinct guideline keeps a support count — how many independent episodes produced it — and we never summarize the store down to a handful of rules. A lesson five different tasks discovered is a different object from one that appeared once, and both are worth keeping.

So on the core question — should you compress an agent's hard-won lessons into a tidy summary? — ACE and ALTK-Evolve give the same answer: no. Count them, don't collapse them. ACE's per-bullet counters and our support counts are two spellings of the same idea.

Where we differ

Two places: how the memory is built, and how it's delivered — and it's the delivery difference that shows up in the token bill.

Consolidation (how the store is built). ACE grows one playbook through a Generator → Reflector → Curator loop, applying incremental delta updates and de-duplicating by embedding. We cluster near-duplicate lessons and merge within a cluster, support-conserving — when several lessons merge, the survivor inherits their combined count, so the store shrinks without losing the record of how much experience backs each guideline. We also extract typed guidelines — strategy, recovery, and optimization — with causal attribution and provenance back to the source trajectory, and at subtask granularity, so a lesson learned on one app can transfer to another.

Delivery (what reaches the model at inference). This is the one that drives the numbers. ACE injects the comprehensive playbook on every step, the same way regardless of model or task. We treat delivery as a dial, not a constant: a small fixed core of high-support guidelines, extended per task with a handful selected for the task at hand (cosine or LLM-guided, priority-weighted) — or, when a model has the headroom to use it, the full consolidated set. The same lessons are available to both agents; the difference is that ACE always sends all of them, and we send however many a given model can actually use.

Why it matters

On AppWorld, with the same base ReAct agent, running both systems in-house:

Model		TGC / SGC	Tokens/task
DeepSeek-V3.2	ACE	80.4 / 73.2	634K
	ALTK-Evolve	89.3 / 80.4	263K
gpt-oss-120b	ACE	54.8 / 35.7	777K
	ALTK-Evolve	56.0 / 37.5	116K

On the strong model we're better on both metrics at ~40% of ACE's inference cost. On the weak model we edge ACE 56.0 to 54.8 — close enough that we call it a tie on accuracy (a repeat run of ours landed at 54.8, matching ACE almost exactly, which is within this benchmark's run-to-run noise) — at about one-seventh the cost.

A fair word on cost: ACE's own efficiency story is about building its context cheaply. Ours is on a different axis — serving it. Retrieving a few guidelines per task instead of injecting the whole playbook on every step is where the tokens go, and it's the direct consequence of the delivery difference above.

Where does the accuracy come from? The by-difficulty breakdown tells two different stories:

Figure 1. Post-memory Task Goal Completion by difficulty, ours vs. ACE. On DeepSeek-V3.2 (right) we win Easy, Hard, and Overall; ACE only edges Medium. On gpt-oss-120b (left) ACE leads easy and medium, but per-task selection wins the hard tasks — and the aggregate. Each system improves from its own no-memory baseline (see the by-difficulty reference tables under Method notes below).

The two models tell different stories. On gpt-oss-120b, ACE's full playbook has the edge on Easy and Medium — there's enough of the task solved by generic instruction-following that a comprehensive prompt helps more than it distracts. But on Hard tasks, where the model has to pick the right lesson rather than wade through all of them, curated retrieval pulls ahead — and that's the tier that decides the aggregate. On DeepSeek-V3.2 the story flips: the stronger model absorbs ACE's full playbook well enough to edge us on Medium, but we lead Easy, Hard, and Overall — with more capacity to spare, more lessons (delivered our way) keep helping instead of crowding each other out.

We give each model its best configuration — the full consolidated set for the strong model, selective retrieval for the weaker one, because a large context overwhelms a weaker model rather than helping it. (Exactly how much to inject, and how it scales across the capability spectrum, is the focus of our follow-up post.)

Same lessons, different delivery

Both systems refuse to compress an agent's hard-won experience into a tidy summary — that part, we agree on. The difference is whether delivery is fixed or calibrated: ACE sends the whole playbook every step no matter what; we send however much of the guideline set a given model can actually use. That calibration is what bought the numbers above — same-or-better accuracy at a fraction of ACE's inference cost — and on the weaker model, it was the difference between guidance that helped and guidance that got in the way.

Try the ALTK-Evolve library — which includes the extraction, consolidation, and retrieval pipeline used here — or read the full technical report for the complete method and ablations.

Linked artifacts / references

Earlier post: ALTK-Evolve introduction — link
ACE (Agentic Context Engineering) — link
AppWorld benchmark — link
ALTK-Evolve — link
Full technical report — link

Method notes

AppWorld test_normal, 168 tasks. A ReAct code agent (each step writes Python; the environment returns the output). TGC = Task Goal Completion; SGC = Scenario Goal Completion, which requires every variant of a scenario to pass. Memory is mined from train/dev only; results are single runs (pass@1), as is standard on this benchmark.

The ACE numbers are our own runs of the ACE agent, evaluated in-house on the same AppWorld splits and the same base models as ALTK-Evolve (DeepSeek-V3.2 and gpt-oss-120b). The ACE paper reports on a different base model (DeepSeek-V3.1), so running it ourselves keeps the comparison controlled for model and harness. Both systems are the same ReAct agent and differ only in the prompt template — which is why the two no-memory baselines differ (72.0 vs 79.8 TGC); we don't rest the comparison on that baseline gap, only on the claims a prompt tweak can't touch: same-or-better accuracy at a fraction of the tokens.

Reference tables

DeepSeek-V3.2 — test_normal (168 tasks):

System	Guidelines	TGC	SGC	Tokens/task
ReAct, no memory	0	79.8	64.3	148K
ReAct + ACE	106	80.4	73.2	634K
ReAct + ALTK-Evolve	191	89.3	80.4	263K

gpt-oss-120b — test_normal:

System	Guidelines	TGC	SGC	Tokens/task
ReAct, no memory	0	39.9	21.4	110K
ReAct + ACE	full	54.8	35.7	777K
ReAct + ALTK-Evolve (selected)	~29	56.0	37.5	116K

gpt-oss-120b — by difficulty (TGC):

Difficulty	Baseline	ACE	ALTK-Evolve
Easy	66.7	84.2	82.5
Medium	35.4	60.4	56.2
Hard	19.1	23.8	31.8
Aggregate	39.9	54.8	56.0

DeepSeek-V3.2 — by difficulty (baseline → +memory): the two systems start from different no-memory baselines (79.8 vs 72.0 TGC overall) because of the prompt-template difference above.

Tier	ALTK TGC	ALTK SGC	ACE TGC	ACE SGC
Overall	79.8 → 89.3	64.3 → 80.4	72.0 → 80.4	57.1 → 73.2
Easy	93.0 → 94.7	84.2 → 84.2	78.9 → 84.2	63.2 → 78.9
Medium	81.2 → 97.9	62.5 → 93.8	85.4 → 100.0	75.0 → 100.0
Hard	66.7 → 77.8	47.6 → 66.7	55.6 → 61.9	38.1 → 47.6

ALTK-Evolve: On-the-Job Learning for AI Agents

Vatche Isahagian — Thu, 17 Sep 2026 14:01:41 +0000

TL;DR

Most AI agents re‑read transcripts instead of learning principles, so they repeat mistakes and don’t transfer lessons to new situations.
ALTK‑Evolve turns raw agent trajectories into reusable guidelines.
In benchmarks, the approach boosted reliability, especially on hard (Δ 14.2% on AppWorld), multi‑step tasks, without bloating context.

The “eternal intern” problem

Imagine a brilliant line cook who has memorized every cookbook but forgets your kitchen every morning. They don’t remember your oven runs hot, or that regulars like extra salt; they’ll follow a recipe card yet freeze when you’re out of lemons. That’s most AI agents: excellent at following prompts, poor at accumulating wisdom about your environment. Feeding yesterday’s logs back into the prompt just makes them re‑read history; it doesn’t help them generalize from it.

A junior needs different recipes for “vinaigrette” and “duck à l’orange.” A chef learns “acid balances fat” and applies it everywhere. Likewise, reliable agents should distill principles from experience and apply them to new tasks, not just near duplicates of old ones. This long‑term memory subsystem does exactly that: it converts interaction traces into candidate guidelines, filters for quality, and injects only relevant guidance at the moment of action. Agents need principles, not transcripts.

A recent MIT study found that 95% of pilots fail because agents don't adapt and learn on the job. ALTK-Evolve addresses this learning gap using long term episodic memory to help agents reason better.

Solution: long term memory with ALTK-Evolve

Evolve is a memory system for AI agents, that can help agents improve over time, learning from and using guidelines generated from previous executions.

Operationally, the system runs as a continuous loop:

1) Downward flow (observation & extraction): Capture full agent trajectories (user utterances, thoughts, tool calls, results) in an Interaction Layer (e.g., Langfuse or another OpenTelemetry‑based observability tool). Pluggable extractors mine traces for structural patterns and persist them as candidate entities.

2) Upward flow (refinement & retrieval): A background consolidate‑and‑score job merges duplicates, prunes weak rules, and boosts proven strategies, evolving a high‑quality library of entities such as guidelines, policies, and SOPs. Retrieval pulls only the relevant items via the Interaction Layer and injects them back into context at the Application Layer.

This approach works for a few key reasons:

Teaches judgment: Converts one‑off events into portable strategies that transfer across tasks.
Controls noise: Scoring keeps memory lean and useful, not a growing junk drawer.
Progressive Disclosure: Retrieval is just‑in‑time, not stuffing everything into the context.

Results: better reliability, especially on hard tasks

We evaluated the framework on AppWorld, where agents complete realistic multi‑step tasks via APIs, averaging 9.5 APIs across 1.8 apps, with hard cases requiring more complex control flow. A ReAct agent received the task instruction plus the top 5 retrieved guidelines generated on a prior run (train/dev) and tested on an unseen partition (test-normal). We report Scenario Goal Completion (SGC), a strict consistency metric requiring success across variants.

Difficulty	Baseline SGC	+ Memory	Δ
Easy	79.0%	84.2%	+5.2
Medium	56.2%	62.5%	+6.3
Hard	19.1%	33.3%	+14.2
Aggregate	50.0%	58.9%	+8.9

Here are some key conclusions from the evaluations:

Generalization: The agent improves on the unseen Test‑Normal tasks, evidence that it’s learning principles, not memorizing recipes.
Complexity scaling: The harder the task, the more the agent benefits from concise learned guidelines, with the largest lift on the more difficult tasks. The Hard tasks saw a 74% relative increase in success, where guidelines are useful to navigate the intricate control flows.
Consistency: SGC gains exceeded raw pass‑rate improvements, reducing “flaky” behavior across scenario variants. The guidelines don’t just help the agent solve tasks, they help them solve them reliably across variants.

Find more details about the experiments in the paper at https://arxiv.org/abs/2603.10600.

Getting started (choose your path)

You have a choice in how to integrate ALTK‑Evolve into your agent.

No‑code with Claude Code, Codex, and IBM Bob (Lite mode)

Install the plugin into Claude Code:

claude plugin marketplace add AgentToolkit/altk-evolve
claude plugin install evolve-lite@evolve-marketplace

That’s it! The plugin extracts entities from trajectories and stores them as files on your filesystem. It uses Claude Code’s hooks for automatic retrieval.

Prefer to watch instead of read? See the short Evolve-Lite Claude Code walkthrough (video): Demo

Check out the walkthroughs here for examples of how to learn with Claude Code in Lite mode.

Lite mode is easy to test‑drive but has limitations. For example, it doesn’t glean insights from across agent sessions or perform consolidation and garbage collection of entities. The low‑code and pro‑code versions below address these limitations.

There are also one-step integrations with Codex and IBM Bob. Try them out!

Low‑code with a ReAct agent

Add a single altk_evolve.auto import and flip a flag to emit traces to an Arize Phoenix UI. Then sync traces to generate improvement guidelines without changing your current stack. It works with popular LLM clients and agent frameworks (e.g., OpenAI, LiteLLM, and Hugging Face agents), so you keep your current stack and simply gain visibility.

To see just how easily this fits into existing projects, explore our hands‑on examples showcasing different framework integrations. For full details on configuration and capabilities, read our low‑code tracing documentation.

Pro‑code with CUGA

We integrated ALTK‑Evolve directly into CUGA via MCP to create a tight, low‑overhead learning loop. Before each run, the get_guidelines MCP tool is called to surface task‑specific steering and reduce trial‑and‑error. After the run, CUGA sends back structured execution traces via save_trajectory, so Evolve can learn from what actually happened and improve future guidance. The result is an integration that gets better over time while staying transparent, composable, and easy to adopt.

Prefer a visual tour? Watch the CUGA integration walkthrough: video

Try it & tell us what your agent learned

Your agent shouldn’t wake up as an intern every morning. This approach helps it learn on the job.
If you're using Claude Code, Codex, and IBM Bob, try it out in minutes and see how it improves your agent.

Tip: Star the repo, it helps others discover the project and directly guides what we build next.

Code: https://github.com/AgentToolkit/altk-evolve
Docs: https://agenttoolkit.github.io/altk-evolve
Quick start tutorials: https://agenttoolkit.github.io/altk-evolve/tutorials/
Feedback & ideas: Open a GitHub issue or join the discussions — concrete use cases, benchmarks, and integration requests are especially helpful.

Watch the demos

Claude Code walkthrough (video): Demo
OpenAI Codex walkthrough (video): Demo
IBM Bob demo walkthrough (video): Demo
CUGA integration walkthrough: video

DEV Community: Vatche Isahagian

Your Agent Aced the Task. Will It Do It Again?

The Metric Almost Nobody Reports

Why Agents Flip: Sharp Decisions vs. Flat Ones

Our setup: the ReAct agent runs at temperature 0.0, so none of the variance above is ordinary sampling.

Diagnose, Then Fix

Results: Reducing the Gap Without Losing Accuracy

The guidelines generalize — they aren't patching one trajectory

If You're Shipping an Agent

Try It

Appendix: Understanding the Metrics

Linked artifacts / references

How Much Memory Does Your Agent Actually Need?

The Key Insight: Dosage Depends on Capability

Learning happens around the model, not inside it

Results Across the Spectrum

The three configurations we compare

The three patterns, in one view

The Cheapest Memory Strategy Can Also Be the Best

Memory Should Be Calibrated, Not Merely Accumulated

What's Next

Appendix: Understanding the Metrics

Thinking of Stanford’s Agentic Context Engineering? We Can Do It with Fewer Tokens

TL;DR

What we agree on

Where we differ

Why it matters

Same lessons, different delivery

Linked artifacts / references

Method notes

Reference tables

ALTK-Evolve: On-the-Job Learning for AI Agents

TL;DR

The “eternal intern” problem

Solution: long term memory with ALTK-Evolve

Results: better reliability, especially on hard tasks

Getting started (choose your path)

No‑code with Claude Code, Codex, and IBM Bob (Lite mode)

Low‑code with a ReAct agent

Pro‑code with CUGA

Try it & tell us what your agent learned

Watch the demos