DEV Community

Shridhar Shah
Shridhar Shah

Posted on

Your Agent's Context Window Is RAM. Start Paging It.

#ai #go #architecture #agents

Treat the context window as a scarce cache: keep the working set resident, evict cold items to a store, and page them back on a fault. Locality keeps the fault rate low and nothing is ever lost.

TL;DR: A long-running agent keeps piling tool results and notes into its context window, but attention is a fixed budget — dilute it and both accuracy and cost degrade. Hard-capping the window and dropping the oldest items loses things the agent needs later. Operating systems solved this with demand paging: keep only the working set resident, evict cold items to a backing store, and page them back on a fault when referenced again. Agent sessions have strong locality, so the fault rate stays low and nothing is lost. In a runnable Go demo, "keep everything" balloons to 7.6× the window budget, while an LRU-paged window stays within budget at a 15% fault rate and zero information lost.

Mental model: your laptop runs apps that need far more memory than it has RAM, and you never notice, because the OS keeps the pages you're actively using in RAM and quietly parks the rest on disk — fetching them back the instant you touch them. An agent's context window is that RAM.

The problem: the window is small, the session is long

The context window is not memory — it's a cache, and a small one. Everything an agent does adds to it: a tool returns 30k tokens, a document gets pasted in, each step leaves notes. Two bad things happen as it fills. First, the attention tax: transformer attention is a fixed budget, and published measurements show accuracy sliding from ~95% to 60–70% as that budget is spread across more tokens. Second, cost and latency scale with tokens in the window, every single step.

The naive fix — cap the window and drop the oldest content — trades one failure for another. The moment a later step references something you dropped, it's gone: the agent silently recomputes it (paying for the tool call again) or, worse, hallucinates it.

The pattern: demand paging for the context window

Operating systems have run programs whose address space dwarfs physical RAM since the 1960s, using demand paging and Denning's working-set model. The mapping onto agents is direct:

OS Agent
RAM the resident context window (fixed budget)
Disk a backing store (a file, a vector DB)
Page a tool result / document / note
Page fault a reference to something evicted → fetch it back

Keep only the working set resident. When the window is full and something new comes in, evict the least-recently-used items to the store (not the void). When a later step references an evicted item, that's a page fault — page it back in. Nothing is destroyed; it's just moved to a cheaper tier.

The eviction-to-fit and fault-on-miss logic is the whole pattern:

for _, item := range stream {
    if el, resident := pos[item]; resident {
        lru.MoveToFront(el)          // hit: already in the window
        continue
    }
    faults++                          // miss: page it back in from the store
    evictToFit(item)                  // make room by evicting LRU residents (to the store)
    tokens += size[item]
    pos[item] = lru.PushFront(item)
}
Enter fullscreen mode Exit fullscreen mode 
func evictToFit(incoming int) {
    for tokens+size[incoming] > budgetTokens && lru.Len() > 0 {
        victim := lru.Back().Value.(int)
        lru.Remove(lru.Back())
        delete(pos, victim)
        tokens -= size[victim]        // evicted to the backing store, not destroyed
    }
}
Enter fullscreen mode Exit fullscreen mode

The result 

Context Demand Paging — the context window is a cache, not infinite memory
  1500-step session, 60 distinct items, 8000-token window budget.

   keep everything resident   60,728 tokens   (7.6× the window — attention tax + cost blowup)
   demand paging (LRU)        7,997 tokens   (stays within budget)

   page faults                  226   (15% of references — the rest hit the working set)
   information lost               0   (faults page back in from the store — nothing destroyed)
Enter fullscreen mode Exit fullscreen mode

Keeping everything resident hits 7.6× the window budget — the exact attention tax the budget was meant to prevent. Demand paging holds the window at budget, and because the agent keeps touching the same working set, only 15% of references miss — each a cheap re-read from the store, never a loss.

Reality check: the 7.6× and the 15% fault rate come from the locality model above — directional, not a benchmark, and your fault rate depends entirely on your workload's locality. What's independently real is the attention tax this targets (published long-context accuracy slides from ~95% to 60–70% as the window fills), and the direction of travel: 2026's Neural Paging formalizes the problem and learns the eviction policy instead of using the plain LRU shown here.

Why this is where 2026 is heading

As agents move from single answers to long-horizon work — hours of coding, multi-document research, sessions that outlive many context resets — context management stops being an afterthought and becomes the systems problem. The field is converging on the OS analogy fast: 2026's Neural Paging formalizes the "Context Paging Problem" and a learned page controller acting as a neural MMU, while practitioner write-ups describe context offloading, just-in-time context, and demand-paged working sets with measured fault rates. The transferable engineering idea: stop treating the window as memory you fill, and start treating it as a cache you manage — resident working set, cheap backing store, fault-driven recovery, eviction by utility.

How faithful is this demo?

It's the paging mechanism, not a real agent: "items" are integers with token sizes, the access stream is a locality model, and the store is assumed cheap and lossless. Real systems earn the two hard parts — what to evict (pure LRU by recency is weak for agents; pin by reference / working set, since a file read during planning stays relevant all session) and what the resident handle says (a pointer the model can't act on is worse than the payload). Start by offloading big tool results behind a summary+handle and paging them back only when a step needs them.

When not to use this

  • Short sessions. If the whole session fits the window with room to spare, paging is machinery you don't need.
  • No locality. If the agent touches items near-randomly, the fault rate climbs and you're just paying store round-trips — plain truncation or a summary may beat it.
  • A good summary beats the raw item. When a distilled note serves later steps as well as the original, consolidate instead of paging the full payload back and forth.

Try it 

go run .   # standard library only
Enter fullscreen mode Exit fullscreen mode

Sources & further reading

Papers & foundations

Engineering write-ups

Top comments (0)