Reducing bootstrap memory cost in LLM agents

Sandy Shen — Wed, 01 Apr 2026 17:22:25 +0000

LLM agents are stateless by default. To get continuity, the standard approach is to load everything into the system prompt. Logs, past decisions, project state.

It works, but it is wasteful. We were spending 3,500+ tokens on memory before the agent even started doing anything useful. If you load nothing, you get the opposite problem. The agent forgets preferences and repeats the same mistakes every session.

We stopped trying to tune the context window and changed how memory is handled.

Instead of loading everything at once, we split memory into three parts:
Hot: A small set of curated facts that are always loaded, around 625 tokens.
Warm: Recent logs from the last 7 days, only pulled in when needed.
Cold: Older history stored externally and not loaded by default.
Most of the time, the agent only needs one or two specific pieces of context.

That simple change made a big difference.
In our setup, bootstrap memory cost dropped from around 3,500 tokens to about 125 tokens, roughly a 96 percent reduction.

We are preparing the open-source release of the OpenClaw Auto Memory Manager. Full write up here:
https://zflow.ai/zflow_ai_insights_article_4.html

Stop tuning LLM agents with live API calls: A simulation-based approach

Sandy Shen — Wed, 01 Apr 2026 00:51:15 +0000

LLM agent configuration is a surprisingly large search space, including model choice, thinking depth, timeout, and context window. Most teams pick a setup once and never revisit it. Manual tuning with live API calls is slow and expensive, and usually only happens after something breaks.

We explored a different approach: simulate first, then deploy. Instead of calling the model for every trial, we built a lightweight parametric simulator and replayed hundreds of configuration variants offline. A scoring function selects the lowest-cost configuration that still meets quality requirements.

The full search completes in under 5 seconds.

A few patterns stood out:

Many agents are over-configured by default
Token usage can often be reduced without impacting output quality
Offline search is significantly faster than live experimentation

In practice, this approach reduced token cost by around 20-40% on real workloads.

We’re currently preparing the open-source release of the OpenClaw Auto-Tuner. If you’re interested, you can check the full write-up here:
https://zflow.ai/zflow_ai_insights_article_3.html

DEV Community: Sandy Shen

Reducing bootstrap memory cost in LLM agents

Stop tuning LLM agents with live API calls: A simulation-based approach