Context Window Management for Big Jobs

Throughout the AI System Design series we follow Mailmind, a fictional AI email assistant that reads an inbox, triages messages, and drafts replies. It is our running example.

Before you read: The ReAct Loop: How Most AI Agents Actually Work, Context Windows: The Model's Working Memory and Chunking: The Decision Everything Downstream Inherits.

In Hallucination Detection: Trust, but Verify we added layers so Mailmind could check its own answers. But checking is useless if the agent never saw the information in the first place. This article is about what happens when the work is too big for one model call.

Mailmind has to process a year's worth of subscriptions, find a hidden flight change fee, and reschedule meetings across 50-message threads. The raw text does not fit in one context window. Even when it does fit, the model may use the middle of a very long prompt less well than the start and the end. So you need a strategy. The main ones are chunking, hierarchical processing, map-reduce, retrieval, and sliding windows with summaries. Each splits the work, processes the pieces, and merges the results. The right choice depends on whether you need completeness, per-item precision, or links between distant parts. At the end we turn to a second problem: the agent's own history, which fills the window as it works.

Why does model quality degrade before the context limit?

Think of reading a 500-page report in one sitting. You remember the opening and the last chapter, but the middle blurs. Language models show a similar pattern. In one well-known study (Liu et al. 2023, Lost in the Middle: How Language Models Use Long Contexts), models answered best when the relevant passage sat at the beginning or the end of a long input, and noticeably worse when it sat in the middle, even models built for long inputs. Newer models handle long inputs better than the ones in that study, but the problem has not gone away: Anthropic's documentation on context windows says that as the token count grows, accuracy and recall degrade, which it calls context rot.

For Mailmind, that means a flight change rule buried in message 25 of a 50-message thread may be used less reliably than the first and last messages. Everything fit inside the window. The model simply used the middle less well. The agent never raises an error. It produces an answer that quietly leaves out the middle content. This is a silent failure mode.

Compare that with hard truncation, where your own code cuts a thread to make it fit. Truncation is easier to detect, because your code counted the tokens and knows exactly what it dropped. Soft degradation is harder to spot: nothing was dropped, and the answer looks confident. Both are problems you have to manage.

So you cannot just cram everything in. You need to split the work.

How does chunking let an agent process more than it can hold?

Chunking splits the raw input into smaller pieces before the model ever sees them. You met it as a retrieval decision in Chunking: The Decision Everything Downstream Inherits; here the same idea splits work. Each piece is processed on its own, and the results are merged later. The simplest version is fixed-size chunking: you cut the text into blocks of a set number of tokens, with a small overlap between neighbors. The overlap keeps a fact that sits near a cut whole in at least one block. Around 1,000 tokens per block with 10 to 20 percent overlap is a reasonable place to start, and you tune it on your own data.

Mailmind uses chunking on long email threads. Each email becomes its own chunk. That is semantic chunking: you split at natural boundaries instead of fixed token counts. An email about a flight change is a logical unit. A thread with 40 messages becomes 40 chunks. A light overlap still helps: each chunk can carry the subject line and the first sentence of the previous message, so the conversation's context travels with it.

Chunk size is a tradeoff. Too small and each piece loses the context that gives it meaning. Too large and each piece carries more noise, costs more to process, and blurs the detail you wanted. For Mailmind's email processing, one message per chunk works well. For a 100-page PDF attachment, you might start with fixed-size chunks of 1,500 tokens and 15 percent overlap.

Chunking alone works for tasks that treat each piece independently: finding all unread emails, tagging spam, extracting dates. But if a task needs the whole picture, flat chunking falls short. That is where hierarchical processing comes in.

How does hierarchical processing handle large-scale tasks?

Hierarchical processing does a coarse pass first, then a deep dive. The coarse pass builds an outline or a category map. The deep dive only looks at the relevant sections. This keeps the total context use small while still getting full detail where it matters.

Mailmind uses hierarchical processing for its weekly "subscription triage" job. The inbox has 500 emails. The agent needs to find all subscription-related messages and decide which ones to cancel, which to keep, and which to downgrade. A flat chunking approach would process every email with the same depth. That is wasteful. The coarse pass scans every email with a small, cheap model or a fast keyword filter and marks each one "subscription" or "not subscription." Then the deep dive processes only the subscription emails, one at a time.

graph TD
    A["Inbox 500 emails"] --> B[Coarse pass 10 emails per chunk]
    B --> C[Identify subscription emails]
    C --> D[Deep dive 1 email per chunk]
    D --> E[Extract subscription details]
    E --> F[Triage report merged]

The coarse pass can use large chunks of 10 emails at a time. The deep dive uses one email per chunk. The overall cost is a fraction of processing everything at full depth. The tradeoff is that the coarse pass might miss something. If a subscription email uses odd phrasing, it might get filtered out. That is a recall versus precision tradeoff. You can tune the coarse pass to let more borderline emails through (higher recall), but then you spend more on the deep dive.

Hierarchical processing works best when you have a clear two-tier structure: find the needles, then examine them. If the task is a summary of the whole corpus, map-reduce is a better fit.

How does map-reduce summarize an entire inbox?

Map-reduce borrows its name from the distributed computing pattern. You split the work into independent map tasks. Each map task processes its chunk and produces a partial result. Then a reduce step combines those partial results into a final answer. The map step never needs to see the whole corpus. The reduce step only sees the partial results, not the raw text.

Mailmind uses map-reduce when a user asks, "Summarize all my subscriptions this year." The inbox has 200 subscription emails. The agent splits them into 20 chunks of 10 emails each. Each map call extracts the service name, the monthly cost, and the next renewal date from its chunk. The reduce call takes all 20 partial lists and produces a categorized summary: "FlixNest: $15.99/month, renews Aug 3. SongCrate: $11.99/month, renews Sep 1." (Both services are made up for this example.) The map calls never see the full list. The reduce call never sees the raw emails. And no single call has to hold all 200 emails, so none of them runs into the long-input problems above.

flowchart LR
    A[Inbox: 200 subscription emails] --> B[Chunker: 20 chunks]
    B --> C[Map agent per chunk]
    C --> D[Partial lists of subscriptions]
    D --> E[Reduce agent]
    E --> F[Final categorized summary]

The risk with map-reduce is that the partial results lose detail. If a map call extracts only the dollar amount but not the fact that it is a promotional rate that expires in 3 months, the reduce step never sees that detail. The fix is to design the map output schema carefully. Include a "notes" field for edge cases. Another risk is that a model doing the reduce invents connections between the partial results. That is a hallucination problem, made worse because the reduce step has no raw text to check against. When the map output is structured, much of the reduce can be plain code instead: merging lists, removing duplicates, adding up totals. Save the model for the parts that need judgment.

Map-reduce scales to very large inputs, but only if you reduce in levels. A million emails in chunks of 10 gives 100,000 partial lists, far too many for one reduce call. So you combine 20 partial lists into one, then combine those results 20 at a time, until one summary remains. This is a tree reduce: 100,000 lists become 5,000, then 250, then 13, then 1, in four levels. Each level is just another map-reduce over smaller inputs. For tasks that are about one specific fact, retrieval is more efficient.

What role does retrieval play in context management?

Retrieval turns the problem inside out. Instead of processing everything, you embed the chunks and store them in a vector database. When a question comes in, you retrieve only the 3 to 5 most relevant chunks. Those chunks go into the context window, and the rest is ignored. This is the most token-efficient strategy.

Mailmind uses retrieval for targeted questions like "What is the change fee for my flight to Boston?" The inbox has thousands of emails. The retriever embeds the question and finds the 3 chunks closest in meaning to it: the booking email and the fare rules. The agent only sees those 3 chunks. The answer costs a few hundred tokens of context instead of the millions it would take to read the whole inbox. Retrieval is also fast at question time, because each chunk was embedded once, ahead of time; only the question needs embedding before the search.

The catch is that retrieval works only for questions that can be answered from a few chunks. If the user asks a comparative question like "Which subscription has the best value?", the retriever might not pull every chunk the answer needs. You need map-reduce or hierarchical processing for that. Retrieval also depends on the quality of the embeddings. Good embeddings help here: emails that talk about the same topic in different words still land close together, so they get retrieved.

Retrieval and map-reduce are not mutually exclusive. You can use retrieval to find the relevant chunks, then map-reduce over those chunks. That hybrid works for many real tasks. For links between distant parts of one long document, sliding windows and progressive summarization are the tools.

How do sliding windows and progressive summarization preserve long-range dependencies?

Some tasks need to track a discussion that spans many messages. A project thread might have decisions in message 1, objections in message 15, and a resolution in message 30. A flat chunker would split this into chunks that lose the cross-references. A sliding window processes the thread with overlapping windows. Each window covers a few messages and shifts by half a window, so each message (apart from the first and last few) appears in two windows, and the agent can see the connections between neighbors.

Mailmind's meeting scheduler uses a sliding window for long threads about date and time proposals. A window of 4 messages slides by 2 messages, so the windows cover messages 1 to 4, 3 to 6, 5 to 8, and so on. The agent in each window extracts the relevant proposals and objections. A final synthesis step combines the findings. The overlap ensures that a proposal in message 3 and a counterproposal in message 5 are seen together in the window that covers messages 3 to 6.

Progressive summarization takes a different approach. It summarizes sections of the thread, then summarizes the summaries. The final answer is built from a hierarchy of summaries. This is useful when the thread is too long even for sliding windows. Mailmind uses it for threads that span hundreds of messages. The first pass summarizes each message into a sentence. The second pass groups 10 messages and summarizes the group. The third pass groups the groups. The final summary is a few paragraphs. The agent can then answer "What was decided about the Q3 budget?" by looking at the top-level summary.

The tradeoff is that progressive summarization loses detail. A specific number buried in a message might get smoothed over in the summary. That is why Mailmind always keeps the original chunks for retrieval. The summary is for coarse understanding. The raw chunks are for fact-checking.

Both strategies have to handle the same boundary problem that chunking does. When related facts end up on different sides of a window or summary boundary, the agent can miss the connection. That is the boundary bug.

What are boundary bugs and how do you fix them?

A boundary bug happens when a logical unit of information is split across two processing chunks. The agent in each chunk sees only a fragment. Neither fragment is enough to act on. The result is a silent failure. The agent never errors. It just never produces the right answer.

Mailmind hit this with a booking confirmation thread of 20 messages. The flight details were in message 3. The hotel details were in message 7. The change fee rules were in an attachment summary in message 14. The chunker split the thread by message count into two chunks: messages 1 to 8 and 9 to 20. The map call for the first chunk saw the flight and hotel but no change fee. It flagged "incomplete trip information." The map call for the second chunk saw the change fee but no flight details. It also flagged "incomplete." The reduce step got two incomplete flags and produced a generic "I could not find the full trip details" response. The user never got the itinerary.

The obvious fix is more overlap, and here it does not work. Overlap helps only when related facts sit close together, near the cut. The flight (message 3) and the fee rules (message 14) are 11 messages apart, so even an overlap of 4 messages leaves them in different chunks.

The real fixes are layered:

1. Do not split what fits. Twenty emails fit easily in one context window. Chunk at thread boundaries: one thread, one chunk. Split a thread only when it is genuinely too long.
2. When you must split, split by meaning. Treat the attachment summary in message 14 as a unit and keep it with the messages it belongs to, rather than cutting at a message count.
3. Detect cross-references first. Before processing, scan the thread for references between messages. The phrase "see attached fare rules" in message 3 tells the system to keep message 3 and the attachment together.
4. Track coverage. After processing, Mailmind checks that every message in the thread was assigned to at least one chunk and that every logical entity (trip, subscription, meeting) was fully processed. If any entity is flagged as incomplete, the system reruns with a different chunking strategy. Silence is not success. You have to verify that the work was actually done.

{
  "type": "comparison",
  "title": "The Booking Thread, Split Two Ways",
  "caption": "Illustrative, not measured. The 20-message booking thread above, split by message count and then kept whole.",
  "before": {
    "label": "Split by message count (1 to 8, 9 to 20)",
    "points": [
      "Flight (msg 3) and fee rules (msg 14) land in different chunks",
      "More overlap cannot help: they are 11 messages apart",
      "Both map calls report incomplete, silently"
    ]
  },
  "after": {
    "label": "One thread, one chunk",
    "points": [
      "All 20 messages fit easily in one window",
      "Flight, hotel and fee rules are read together",
      "Coverage check confirms the trip is complete"
    ]
  }
}

Together, these layers turn silent failures into explicit retries. The cost is extra computation, but for critical tasks like trip planning, it is worth it.

What about the agent's own growing context?

Everything so far treats context as a big input to split up. But an agent also fills its own window as it works. You saw why in The ReAct Loop: How Most AI Agents Actually Work: every iteration adds the model's turn and a tool result, and the whole conversation is sent again on the next call. An agent that runs for an afternoon and makes a hundred tool calls can fill its window with its own history, most of it stale. Anthropic's guide Effective context engineering for AI agents (September 2025) calls this work context engineering: choosing what goes into the window on each call. It describes four techniques that apply to any agent.

• Compact the history. When the conversation grows long, replace the older turns with a summary and keep the recent turns word for word. A good summary keeps decisions, open questions, constraints and IDs, and drops the rest. The guide says Claude Code works this way: it keeps decisions, unresolved bugs and implementation details, and carries on with the summary plus the five files it accessed most recently. Some APIs now do the summarizing for you; Anthropic's compaction, for example, replaces older turns with a summary written on the server (in beta as of October 2026).
• Clear old tool results. A search result the agent has already acted on rarely needs to be read again in full. Replace it with a one-line note ("read 40 travel emails; booking found in message 3") or drop it. The guide calls tool result clearing one of the safest, lightest forms of compaction, and Anthropic's API offers it as context editing.
• Write notes outside the window. The agent keeps its findings in a file or a store (a progress list, a notes file) and reads back only what the next step needs. This is memory kept outside the model, which Agent Memory: The Four Kinds and How They Fit Together covers.
• Hand sub-tasks to sub-agents. A sub-agent starts with a fresh window, does one focused job, and returns a short summary. The guide describes sub-agents that may use tens of thousands of tokens exploring and hand back a condensed summary, often 1,000 to 2,000 tokens. Multi-Agent Orchestration: Managers, Workers, and Pipelines covers how to split the work.

Here is how that looks for Mailmind. Its travel assistant spends an afternoon planning a four-city trip: it searches the inbox for bookings, reads loyalty emails, checks the calendar and drafts changes. By call 60 the window is full of raw search results it no longer needs. With these techniques, the early turns become a short summary ("flights booked for legs 1 to 3; no hotel yet in Lisbon; the user wants no flights before 9 am"). Each raw search result is cleared once its findings are written to the trip notes. The search for a Lisbon hotel goes to a sub-agent that returns three lines. The main agent's window stays small, and every call stays fast.

One warning: a summary can drop the thing that matters. If compaction loses "no flights before 9 am", the agent will happily book a 6 am flight later. Keep the user's request and constraints word for word, outside the part that gets summarized, and test your summaries on real tasks.

Quick Reference

Strategy	Best for	Typical size	Overlap needed?
Fixed-size chunking	General splitting	A set token count per chunk (around 1,000 is a common start)	Yes, 10 to 20 percent is a common start
Semantic chunking	Natural boundaries (one email, one section)	Variable	Light (subject line plus the previous message's first sentence)
Hierarchical processing	Find the needles, then examine them	Coarse pass: large chunks; deep dive: one item	No
Map-reduce	Summaries over a whole corpus	Very large inputs, if you reduce in levels	No
Retrieval	Targeted questions	Top 3 to 5 chunks	No
Sliding window	Links between nearby messages	About 4 messages, shifting by 2	Yes, half a window
Progressive summarization	Very long threads	Summaries of sections, then summaries of summaries	No
Compaction	A long-running agent's own history	Summary of older turns, recent turns word for word	Not applicable

Frequently Asked Questions

Q: How do I choose between chunking and retrieval for a given task?
Use retrieval when the task is a specific question that can be answered from a few chunks. Use chunking when the task requires processing every piece of input, like extracting all dates or tagging all messages. The two can be combined: retrieve the relevant chunks, then process them.

Q: What is the biggest pitfall with map-reduce?
The reduce step can lose detail if the map outputs are too high-level. If a map call extracts only the dollar amount but not the fact that it is a promotional rate, the reduce step never sees that detail. Always include a notes field in the map output schema for edge cases.

Q: Can I just use a model with a huge context window and skip all this?
Some models now take a million tokens in one request (as of October 2026). Long windows have improved, but recall of material in the middle can still drop, so test on your own data. Filling a huge window on every call is also slow and costly, and prompt caching helps only when the same prefix repeats from call to call. For most tasks, a well-designed chunking or retrieval strategy is faster, cheaper and easier to check.

Q: How do I detect boundary bugs in production?
Implement coverage tracking. For each logical unit, like a thread or a document, verify that every piece has been processed. Log when an agent returns an "incomplete" flag. If the coverage check fails, escalate to a human or rerun with a different chunking strategy.

Q: When should I use hierarchical processing versus progressive summarization?
Use hierarchical processing when you need to decide what to look at closely based on a coarse pass. For example, finding all subscription emails and then analyzing them. Use progressive summarization when you need to reduce a long document into a dense summary without losing its structure. Both are forms of multi-pass processing but serve different goals.

Q: When should an agent compact its history?
Before the window gets crowded, not at the last moment, because quality slips well before the hard limit. Compact at a token threshold you set, keep the original request and constraints word for word, and clear tool results the agent has already used first, since that is the cheapest saving.

Test yourself

A user asks Mailmind: "Which subscriptions have I paid for in the last six years, and which am I still paying for?" The inbox holds 80,000 emails, far too many for one context window. A teammate proposes retrieval: embed the question, fetch the top 5 chunks, and answer from those. What goes wrong, and what pipeline would you build instead?

Answer: Retrieval fails because the question asks for completeness, not one fact. The top 5 chunks might cover five services and silently miss the rest. Build a pipeline instead. First, a coarse pass with a cheap classifier or keyword filter marks billing and subscription emails, tuned for recall so borderline emails get through. Then map over those emails, one per chunk, extracting a structured record: service, amount, date, and a notes field for promotions and cancellations. If the records are too many for one reduce call, reduce in levels, and do the merging in plain code where you can: group by service and keep the latest charge. "Still paying" then means a recent charge and no cancellation since, which code can check. Finally, track coverage: every email the coarse pass flagged must produce a record or an explicit "not a subscription" verdict, so a skipped email shows up as a gap, not as silence.

Next: Self-Improving Loops: Agents That Learn From Outcomes. It covers how an agent's prompts and examples can get better from tracked outcomes, behind guardrails that people control.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

---

Originally published at Internals Decoded. AI internals, explained conversationally.