DEV Community: Vatche Isahagian

How Much Memory Does Your Agent Actually Need?

Vatche Isahagian — Mon, 05 Oct 2026 18:21:37 +0000

In our previous post, we compared ALTK-Evolve with ACE and showed that how you deliver an agent's self-distilled guidelines — a few retrieved per task vs. the whole set injected — drives both accuracy and cost. This post steps back to the question that comes before it: how much should you give it?

Equipping an agent with agentic memory sounds simple: distill lessons from its past work, put them back in context, and more experience should mean better performance. It doesn't always work that way. When we scaled the evaluation to eight models — from a 30B dense model to frontier proprietary systems — one finding stood out:

Agentic memory is not a feature you switch on. It's a dose you calibrate to the model.

TL;DR

ALTK-Evolve lets an agent learn from its own past trajectories: distilling reusable guidelines and injecting them back at inference time, with no weight updates and no human annotation.
The right dose differs by model tier: strong models with headroom want the full guideline set, weaker models do best with a compact core plus per-task retrieval, and saturated models show no measurable gain.
Curated retrieval can be both the most accurate and the cheapest option: gpt-oss-120b gained +16.1pp task completion at only +5% tokens — and prompt caching keeps even the full guideline set affordable in production.

The Key Insight: Dosage Depends on Capability

Not every model benefits from the same amount of memory. Across eight models spanning the capability spectrum, we saw three recurring patterns:

Strong models with headroom want the full guideline set — every guideline, including rare edge-case lessons. They have the capacity to absorb and apply all of it. DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in task completion when given its full self-mined guideline set.
Smaller or weaker models get drowned by a large guideline set. For these, a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task works best. gpt-oss-120b (117B MoE) gained +16.1pp with this selective approach — while the full guideline set gained less and cost ~50% more tokens.
Already-saturated models show no measurable gain. We call this the saturated pattern — the label describes what we observed, not a proven cause. The model may already have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. GLM-5 (745B MoE) sat here in our runs.

What puts a model into one pattern rather than another isn't simply parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all appear to shape where a model lands, and separating those factors is ongoing work. The practical takeaway holds either way: the right dose of memory depends on the model, and we can calibrate it.

Learning happens around the model, not inside it

"Memory" here doesn't mean replaying a past transcript. It means a guideline set — strategies that worked, mistakes to avoid, and edge cases — distilled from the agent's own prior trajectories. The loop is straightforward:

The agent attempts tasks and produces trajectories.
ALTK-Evolve extracts behavioral guidelines from both its successful and unsuccessful runs.
It consolidates those guidelines into a reusable set.
At inference time, the agent receives either the full guideline set or a task-relevant selection of it.

No model weights are updated. The learning loop changes the guidance available to the agent, not the underlying model — which is exactly why it's cheap to adopt and portable across the eight models we tested.

Results Across the Spectrum

We evaluated on AppWorld — 585 multi-step tasks (168 test_normal + 417 test_challenge) across 9 simulated apps (calendars, messaging, payments, and so on). Tasks are scored two ways: whether the agent fully completes each task (TGC — Task Goal Completion) and whether every variant of a scenario passes (SGC — Scenario Goal Completion, a stricter, all-or-nothing bar). Full definitions are in the appendix.

The three configurations we compare

Because the confusing part of any memory study is what's actually in the context window, we define the configurations up front.

Both memory configurations draw from the same guideline set, mined once (via the loop above) from AppWorld's training split only. What changes between them is only how that one set is delivered — the full guideline set injects all of it every step, while curated retrieval delivers a selected subset — never how the guidelines were produced, and no test-split data ever goes into building it.

Configuration	What's in the agent's context
Baseline	No memory — the agent as shipped.
Full guideline set	Every mined guideline, injected on every ReAct step.
Curated retrieval	A fixed, high-confidence core of those same guidelines plus a few task-relevant ones retrieved for each task (a fixed portion + a variable portion).

The number of guidelines a model mines depends on its own capability, so we report configurations by strategy — "full guideline set" vs. "curated retrieval" — rather than by raw counts, which aren't comparable across models.

The three patterns, in one view

Representative models from the eight-model sweep, measured by task completion (TGC) on test_normal:

Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld test_normal for baseline vs. the best-memory configuration; the x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below.

The figure plots TGC to keep it readable; the table adds the stricter SGC metric, where the gains are often larger:

Model	Pattern	Baseline TGC / SGC	Best-memory TGC / SGC	Best config	Δ TGC	Δ SGC
gpt-oss-120b (117B MoE)	Weak / selective	39.9 / 21.4	56.0 / 37.5	curated retrieval	+16.1	+16.1
DeepSeek-V3.2 (671B MoE)	Strong w/ headroom	79.8 / 64.3	89.3 / 80.4	full guideline set	+9.5	+16.1
Claude Opus 4.6	Strong w/ headroom	90.5 / 87.5	94.6 / 94.6	full guideline set	+4.1	+7.1
GPT-5.5	Strong (near-ceiling)	92.3 / 82.1	95.2 / 89.3	full guideline set	+2.9	+7.2
GLM-5 (745B MoE)	Saturated	87.5 / 80.4	87.5 / 80.4	full guideline set	0.0	0.0

Reading the SGC column, the stricter metric usually moves more than TGC — DeepSeek's SGC jumps +16.1pp against a +9.5pp TGC gain — because good guidelines especially help an agent clear every variant of a scenario, not just the average case. And the effect doesn't disappear at the top of the range: GPT-5.5 and Opus, both near the ceiling on TGC, still gain +7.2 and +7.1pp SGC respectively. Memory keeps paying off as long as a model has a remaining failure mode to target.

The Cheapest Memory Strategy Can Also Be the Best

A practical concern: injecting a full guideline set inflates every ReAct step's input, because the guidelines are re-sent each turn. Here's what we observed:

Model	Config	Tokens/task (baseline)	Tokens/task (+ memory)	Overhead
DeepSeek-V3.2	full guideline set	148K	263K	+78%
gpt-oss-120b	full guideline set	110K	166K	+51%
gpt-oss-120b	curated retrieval	110K	116K	+5%

Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline.

Two takeaways:

Curated retrieval keeps cost near baseline. For weaker models, where selection wins on accuracy, it also wins on cost — the best of both worlds (+16.1pp TGC at only +5% tokens for gpt-oss-120b). Better performance here does not require more inference cost.
Memory doesn't blow up the reasoning loop. DeepSeek runs about the same number of ReAct steps with memory as without (≈18–19 on average), so the added cost is input-token inflation, not longer trajectories.

The real efficiency lever in production is prompt caching: the static portion of the guideline set is identical across steps and can be cached, cutting effective cost substantially. Cache-aware prompt design — keeping the shared guideline-set prefix stable so it stays cacheable — is worth engineering for. We also hypothesize that context-window size plays a role: models with larger windows may absorb the full guideline set more effectively, while smaller-context models benefit more from retrieval that keeps injected content compact. We have not yet run controlled experiments isolating this factor.

Memory Should Be Calibrated, Not Merely Accumulated

The lesson isn't to give an agent everything it has learned. It's to give it the amount of experience it can actually use.

For weak models, that means a compact core plus a few task-specific lessons — which, conveniently, is also the cheapest option.
For strong models with headroom, it means preserving the full guideline set, kept affordable in production via prompt caching.
For saturated models, it means spending no extra context until their remaining failure modes are better understood.

The gains are real across the board — automatic, leakage-free, and requiring no human annotation — but only when the dose fits the model.

What's Next

This is a starting point, not the finish line:

A learned selector. Our current retrieval ranks guidelines by cosine similarity, which we've shown doesn't perfectly predict which guidelines help a given task. A selector trained on outcome signal is the natural next step.
Memory for very weak models. Below a minimum capability baseline, self-distillation lacks signal. Teacher-distilled memory for very weak models is a separate problem we're exploring.
Beyond AppWorld. These results are validated on AppWorld — a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress.
Isolating context window. As above, we want controlled experiments that separate context-window size from raw capability.

Try the ALTK-Evolve library — which includes the extraction, consolidation, and retrieval pipeline used here — or read the full technical report for the complete method and ablations.

Appendix: Understanding the Metrics

AppWorld tasks are graded by two metrics, both reported as percentages (higher is better):

TGC — Task Goal Completion. The share of individual tasks the agent completes fully and correctly. This is the headline "did it get the job done" number.
SGC — Scenario Goal Completion. A stricter, all-or-nothing metric. Each scenario bundles several variants of the same task (the same request with different data, phrasing, or edge conditions). SGC counts a scenario as passing only if the agent succeeds on every variant. It measures reliability — an agent that solves a task most of the time but fails on one variant scores on TGC but not on SGC.

Thinking of Stanford’s Agentic Context Engineering? We Can Do It with Fewer Tokens

Vatche Isahagian — Thu, 24 Sep 2026 15:08:31 +0000

TL;DR

ACE and ALTK-Evolve both turn agent trajectories into reusable lessons without updating model weights.
The key difference is delivery: ACE supplies the full playbook, while ALTK-Evolve calibrates how many guidelines each model receives.
In our controlled AppWorld runs, ALTK-Evolve matched or exceeded ACE while using about 40% of the tokens on DeepSeek-V3.2 and one-seventh on gpt-oss-120b.

Give an LLM agent a realistic multi-step task — split a bill, find a song, reconcile an order across nine simulated apps — and when it fails, it usually isn't for lack of knowledge. It mis-paginates an API, resolves the wrong person, or returns a value when none was asked for. The model knows the APIs; what it hasn't internalized is how to use them reliably. That's learnable from the agent's own history.

Two recent systems do exactly this, on the same kind of agent: ACE (Agentic Context Engineering) and our ALTK-Evolve (introduced here). Both are a form of agentic memory — turning an agent's past trajectories into reusable lessons and feeding them back at inference time, no weight updates, no human labels. They even agree on the hard part. Where they part ways is delivery.

A note on words, because the two systems name things differently: we'll call the raw thing an agent learns a lesson. ACE organizes its lessons into one comprehensive, evolving playbook; we consolidate ours into individually retrievable guidelines. Same lessons, two containers.

What we agree on

Both systems refuse to compress.

ACE names the failure modes precisely: brevity bias — optimization collapsing toward short, generic instructions — and context collapse — a model asked to rewrite its whole context each step summarizing the detail away. Its answer is to keep a rich, itemized playbook, with a helpful/harmful counter on every bullet, and let the model distill relevance at read time.

We reach the same conclusion from the other direction. Every distinct guideline keeps a support count — how many independent episodes produced it — and we never summarize the store down to a handful of rules. A lesson five different tasks discovered is a different object from one that appeared once, and both are worth keeping.

So on the core question — should you compress an agent's hard-won lessons into a tidy summary? — ACE and ALTK-Evolve give the same answer: no. Count them, don't collapse them. ACE's per-bullet counters and our support counts are two spellings of the same idea.

Where we differ

Two places: how the memory is built, and how it's delivered — and it's the delivery difference that shows up in the token bill.

Consolidation (how the store is built). ACE grows one playbook through a Generator → Reflector → Curator loop, applying incremental delta updates and de-duplicating by embedding. We cluster near-duplicate lessons and merge within a cluster, support-conserving — when several lessons merge, the survivor inherits their combined count, so the store shrinks without losing the record of how much experience backs each guideline. We also extract typed guidelines — strategy, recovery, and optimization — with causal attribution and provenance back to the source trajectory, and at subtask granularity, so a lesson learned on one app can transfer to another.

Delivery (what reaches the model at inference). This is the one that drives the numbers. ACE injects the comprehensive playbook on every step, the same way regardless of model or task. We treat delivery as a dial, not a constant: a small fixed core of high-support guidelines, extended per task with a handful selected for the task at hand (cosine or LLM-guided, priority-weighted) — or, when a model has the headroom to use it, the full consolidated set. The same lessons are available to both agents; the difference is that ACE always sends all of them, and we send however many a given model can actually use.

Why it matters

On AppWorld, with the same base ReAct agent, running both systems in-house:

Model		TGC / SGC	Tokens/task
DeepSeek-V3.2	ACE	80.4 / 73.2	634K
	ALTK-Evolve	89.3 / 80.4	263K
gpt-oss-120b	ACE	54.8 / 35.7	777K
	ALTK-Evolve	56.0 / 37.5	116K

On the strong model we're better on both metrics at ~40% of ACE's inference cost. On the weak model we edge ACE 56.0 to 54.8 — close enough that we call it a tie on accuracy (a repeat run of ours landed at 54.8, matching ACE almost exactly, which is within this benchmark's run-to-run noise) — at about one-seventh the cost.

A fair word on cost: ACE's own efficiency story is about building its context cheaply. Ours is on a different axis — serving it. Retrieving a few guidelines per task instead of injecting the whole playbook on every step is where the tokens go, and it's the direct consequence of the delivery difference above.

Where does the accuracy come from? The by-difficulty breakdown tells two different stories:

Figure 1. Post-memory Task Goal Completion by difficulty, ours vs. ACE. On DeepSeek-V3.2 (right) we win Easy, Hard, and Overall; ACE only edges Medium. On gpt-oss-120b (left) ACE leads easy and medium, but per-task selection wins the hard tasks — and the aggregate. Each system improves from its own no-memory baseline (see the by-difficulty reference tables under Method notes below).

The two models tell different stories. On gpt-oss-120b, ACE's full playbook has the edge on Easy and Medium — there's enough of the task solved by generic instruction-following that a comprehensive prompt helps more than it distracts. But on Hard tasks, where the model has to pick the right lesson rather than wade through all of them, curated retrieval pulls ahead — and that's the tier that decides the aggregate. On DeepSeek-V3.2 the story flips: the stronger model absorbs ACE's full playbook well enough to edge us on Medium, but we lead Easy, Hard, and Overall — with more capacity to spare, more lessons (delivered our way) keep helping instead of crowding each other out.

We give each model its best configuration — the full consolidated set for the strong model, selective retrieval for the weaker one, because a large context overwhelms a weaker model rather than helping it. (Exactly how much to inject, and how it scales across the capability spectrum, is the focus of our follow-up post.)

Same lessons, different delivery

Both systems refuse to compress an agent's hard-won experience into a tidy summary — that part, we agree on. The difference is whether delivery is fixed or calibrated: ACE sends the whole playbook every step no matter what; we send however much of the guideline set a given model can actually use. That calibration is what bought the numbers above — same-or-better accuracy at a fraction of ACE's inference cost — and on the weaker model, it was the difference between guidance that helped and guidance that got in the way.

Try the ALTK-Evolve library — which includes the extraction, consolidation, and retrieval pipeline used here — or read the full technical report for the complete method and ablations.

Linked artifacts / references

Earlier post: ALTK-Evolve introduction — link
ACE (Agentic Context Engineering) — link
AppWorld benchmark — link
ALTK-Evolve — link
Full technical report — link

Method notes

AppWorld test_normal, 168 tasks. A ReAct code agent (each step writes Python; the environment returns the output). TGC = Task Goal Completion; SGC = Scenario Goal Completion, which requires every variant of a scenario to pass. Memory is mined from train/dev only; results are single runs (pass@1), as is standard on this benchmark.

The ACE numbers are our own runs of the ACE agent, evaluated in-house on the same AppWorld splits and the same base models as ALTK-Evolve (DeepSeek-V3.2 and gpt-oss-120b). The ACE paper reports on a different base model (DeepSeek-V3.1), so running it ourselves keeps the comparison controlled for model and harness. Both systems are the same ReAct agent and differ only in the prompt template — which is why the two no-memory baselines differ (72.0 vs 79.8 TGC); we don't rest the comparison on that baseline gap, only on the claims a prompt tweak can't touch: same-or-better accuracy at a fraction of the tokens.

Reference tables

DeepSeek-V3.2 — test_normal (168 tasks):

System	Guidelines	TGC	SGC	Tokens/task
ReAct, no memory	0	79.8	64.3	148K
ReAct + ACE	106	80.4	73.2	634K
ReAct + ALTK-Evolve	191	89.3	80.4	263K

gpt-oss-120b — test_normal:

System	Guidelines	TGC	SGC	Tokens/task
ReAct, no memory	0	39.9	21.4	110K
ReAct + ACE	full	54.8	35.7	777K
ReAct + ALTK-Evolve (selected)	~29	56.0	37.5	116K

gpt-oss-120b — by difficulty (TGC):

Difficulty	Baseline	ACE	ALTK-Evolve
Easy	66.7	84.2	82.5
Medium	35.4	60.4	56.2
Hard	19.1	23.8	31.8
Aggregate	39.9	54.8	56.0

DeepSeek-V3.2 — by difficulty (baseline → +memory): the two systems start from different no-memory baselines (79.8 vs 72.0 TGC overall) because of the prompt-template difference above.

Tier	ALTK TGC	ALTK SGC	ACE TGC	ACE SGC
Overall	79.8 → 89.3	64.3 → 80.4	72.0 → 80.4	57.1 → 73.2
Easy	93.0 → 94.7	84.2 → 84.2	78.9 → 84.2	63.2 → 78.9
Medium	81.2 → 97.9	62.5 → 93.8	85.4 → 100.0	75.0 → 100.0
Hard	66.7 → 77.8	47.6 → 66.7	55.6 → 61.9	38.1 → 47.6

ALTK-Evolve: On-the-Job Learning for AI Agents

Vatche Isahagian — Thu, 17 Sep 2026 14:01:41 +0000

TL;DR

Most AI agents re‑read transcripts instead of learning principles, so they repeat mistakes and don’t transfer lessons to new situations.
ALTK‑Evolve turns raw agent trajectories into reusable guidelines.
In benchmarks, the approach boosted reliability, especially on hard (Δ 14.2% on AppWorld), multi‑step tasks, without bloating context.

The “eternal intern” problem

Imagine a brilliant line cook who has memorized every cookbook but forgets your kitchen every morning. They don’t remember your oven runs hot, or that regulars like extra salt; they’ll follow a recipe card yet freeze when you’re out of lemons. That’s most AI agents: excellent at following prompts, poor at accumulating wisdom about your environment. Feeding yesterday’s logs back into the prompt just makes them re‑read history; it doesn’t help them generalize from it.

A junior needs different recipes for “vinaigrette” and “duck à l’orange.” A chef learns “acid balances fat” and applies it everywhere. Likewise, reliable agents should distill principles from experience and apply them to new tasks, not just near duplicates of old ones. This long‑term memory subsystem does exactly that: it converts interaction traces into candidate guidelines, filters for quality, and injects only relevant guidance at the moment of action. Agents need principles, not transcripts.

A recent MIT study found that 95% of pilots fail because agents don't adapt and learn on the job. ALTK-Evolve addresses this learning gap using long term episodic memory to help agents reason better.

Solution: long term memory with ALTK-Evolve

Evolve is a memory system for AI agents, that can help agents improve over time, learning from and using guidelines generated from previous executions.

Operationally, the system runs as a continuous loop:

1) Downward flow (observation & extraction): Capture full agent trajectories (user utterances, thoughts, tool calls, results) in an Interaction Layer (e.g., Langfuse or another OpenTelemetry‑based observability tool). Pluggable extractors mine traces for structural patterns and persist them as candidate entities.

2) Upward flow (refinement & retrieval): A background consolidate‑and‑score job merges duplicates, prunes weak rules, and boosts proven strategies, evolving a high‑quality library of entities such as guidelines, policies, and SOPs. Retrieval pulls only the relevant items via the Interaction Layer and injects them back into context at the Application Layer.

This approach works for a few key reasons:

Teaches judgment: Converts one‑off events into portable strategies that transfer across tasks.
Controls noise: Scoring keeps memory lean and useful, not a growing junk drawer.
Progressive Disclosure: Retrieval is just‑in‑time, not stuffing everything into the context.

Results: better reliability, especially on hard tasks

We evaluated the framework on AppWorld, where agents complete realistic multi‑step tasks via APIs, averaging 9.5 APIs across 1.8 apps, with hard cases requiring more complex control flow. A ReAct agent received the task instruction plus the top 5 retrieved guidelines generated on a prior run (train/dev) and tested on an unseen partition (test-normal). We report Scenario Goal Completion (SGC), a strict consistency metric requiring success across variants.

Difficulty	Baseline SGC	+ Memory	Δ
Easy	79.0%	84.2%	+5.2
Medium	56.2%	62.5%	+6.3
Hard	19.1%	33.3%	+14.2
Aggregate	50.0%	58.9%	+8.9

Here are some key conclusions from the evaluations:

Generalization: The agent improves on the unseen Test‑Normal tasks, evidence that it’s learning principles, not memorizing recipes.
Complexity scaling: The harder the task, the more the agent benefits from concise learned guidelines, with the largest lift on the more difficult tasks. The Hard tasks saw a 74% relative increase in success, where guidelines are useful to navigate the intricate control flows.
Consistency: SGC gains exceeded raw pass‑rate improvements, reducing “flaky” behavior across scenario variants. The guidelines don’t just help the agent solve tasks, they help them solve them reliably across variants.

Find more details about the experiments in the paper at https://arxiv.org/abs/2603.10600.

Getting started (choose your path)

You have a choice in how to integrate ALTK‑Evolve into your agent.

No‑code with Claude Code, Codex, and IBM Bob (Lite mode)

Install the plugin into Claude Code:

claude plugin marketplace add AgentToolkit/altk-evolve
claude plugin install evolve-lite@evolve-marketplace

That’s it! The plugin extracts entities from trajectories and stores them as files on your filesystem. It uses Claude Code’s hooks for automatic retrieval.

Prefer to watch instead of read? See the short Evolve-Lite Claude Code walkthrough (video): Demo

Check out the walkthroughs here for examples of how to learn with Claude Code in Lite mode.

Lite mode is easy to test‑drive but has limitations. For example, it doesn’t glean insights from across agent sessions or perform consolidation and garbage collection of entities. The low‑code and pro‑code versions below address these limitations.

There are also one-step integrations with Codex and IBM Bob. Try them out!

Low‑code with a ReAct agent

Add a single altk_evolve.auto import and flip a flag to emit traces to an Arize Phoenix UI. Then sync traces to generate improvement guidelines without changing your current stack. It works with popular LLM clients and agent frameworks (e.g., OpenAI, LiteLLM, and Hugging Face agents), so you keep your current stack and simply gain visibility.

To see just how easily this fits into existing projects, explore our hands‑on examples showcasing different framework integrations. For full details on configuration and capabilities, read our low‑code tracing documentation.

Pro‑code with CUGA

We integrated ALTK‑Evolve directly into CUGA via MCP to create a tight, low‑overhead learning loop. Before each run, the get_guidelines MCP tool is called to surface task‑specific steering and reduce trial‑and‑error. After the run, CUGA sends back structured execution traces via save_trajectory, so Evolve can learn from what actually happened and improve future guidance. The result is an integration that gets better over time while staying transparent, composable, and easy to adopt.

Prefer a visual tour? Watch the CUGA integration walkthrough: video

Try it & tell us what your agent learned

Your agent shouldn’t wake up as an intern every morning. This approach helps it learn on the job.
If you're using Claude Code, Codex, and IBM Bob, try it out in minutes and see how it improves your agent.

Tip: Star the repo, it helps others discover the project and directly guides what we build next.

Code: https://github.com/AgentToolkit/altk-evolve
Docs: https://agenttoolkit.github.io/altk-evolve
Quick start tutorials: https://agenttoolkit.github.io/altk-evolve/tutorials/
Feedback & ideas: Open a GitHub issue or join the discussions — concrete use cases, benchmarks, and integration requests are especially helpful.

Watch the demos

Claude Code walkthrough (video): Demo
OpenAI Codex walkthrough (video): Demo
IBM Bob demo walkthrough (video): Demo
CUGA integration walkthrough: video