On a chat screen, you type a message and get a reply. Keep talking, and it feels like one continuous conversation.
Behind that screen, your app sends your message to a backend. The backend calls an AI provider—the service we use to run the model—and sends the model’s reply back to the app.
Each user message and assistant reply becomes part of the conversation history. The backend stores that history, but still has to decide which parts to include in the next model request.
Say the user’s next message is: “What about the second option?”
That sentence needs context. Which options? What were we comparing? Did the user set any constraints?
So the backend may need to send earlier messages, previous answers, and tool results alongside that short follow-up.
The user sends one sentence. We might send a whole conversation to explain it.
And as the chat grows, that history keeps growing too.
Send everything on every request, and token consumption climbs. Cut too much, and the assistant starts forgetting things the user expects it to remember.
That’s the problem I’m working through in a conversational AI system: how do I reduce token consumption while keeping the relevant context available?
The user expects the conversation to flow. The cost of maintaining it needs to make sense. And the system needs clear rules for what stays in context, what gets compressed, and when.
I ended up implementing an approach I call grow and flush. Getting there meant working through a few other options, because each one solved part of the problem and introduced another.
Engineering. There’s always another “but.” 😅
The starting point: send everything
The most straightforward approach is to pass the entire conversation back to the model on each request.
The model gets the history it might need. We don’t have to decide which messages to leave out.
But consider what happens as the chat continues.
An early message gets sent again on the next request. And the next. And the next. Meanwhile, new messages and tool results keep joining it.
A one-line follow-up can end up carrying a large amount of history.
Even before we reach the model’s context limit, there’s a practical question: how much of that input is helping the current reply, and how much are we repeatedly sending just because it’s there?
So, let’s start cutting it down.
1. Sliding window: keep the latest messages
Keep only the most recent messages.
For example, with a window of eight, we initially send messages 1–8. When message 9 arrives, we send messages 2–9. Then 3–10, and so on.
Each new message pushes the oldest one out.
Simple enough. The number of messages we send stays small.
The problem
Recent doesn’t always mean relevant.
Suppose the user sets a budget in message 2. At message 10, they ask us to compare something using that same budget.
Message 2 has already left the window.
The user remembers the conversation. The assistant is working with a smaller slice of it.
From the user’s side, that feels like: “We literally just discussed this.”
And honestly, fair.
The old message might still be in the database, but it won’t help the current reply unless we bring it back into the model’s context.
A sliding window can be enough for short chats or exchanges that don’t depend much on earlier messages. For the conversations I needed to support, it left too much behind.
A smaller prompt helps with cost. Making users repeat their instructions adds friction to the product.
We need to preserve something from those older messages.
2. Summarize once, then keep going
What if we summarize the older messages before removing them?
Once the conversation reaches a certain point, we make a summary of the older part and keep the newer messages unchanged.
For example, at message 16:
- Summarize messages 1–8.
- Keep messages 9–16 in their original form.
- Send the summary alongside those recent messages.
From that point, keep appending new messages after the same summary.
Now the assistant has a compact version of the early conversation. Nice.
The problem
The summary stays small. The messages after it keep growing.
At message 20, we’re sending the summary plus messages 9–20.
At message 30, it’s the same summary plus messages 9–30.
We’ve reduced the history once, but the amount of context we send keeps climbing again.
So yes, we’ve bought some breathing room. The token problem is still coming back.
That initial saving doesn’t give us a lasting way to control the cost of a long conversation.
The summary needs to keep moving with the chat.
3. Rolling summary + sliding window
Keep the recent messages in their original form and maintain a summary of the older ones.
As the window moves, merge whatever is leaving it into the existing summary.
With an eight-message window, message 9 pushes message 1 into the summary. Message 10 pushes message 2 into it. The summary grows in coverage while the recent window stays at eight.
That gives us two useful things: recent detail and compressed history.
The raw conversation no longer grows forever. Earlier context has somewhere to go.
The problem
If we update the summary every turn, we’ve added another model call to every turn that needs compaction.
And the summarizer also consumes tokens. It doesn’t run on vibes.
Those calls have their own input and output tokens. Across many active conversations, that adds work and cost just to maintain the memory.
There’s also the timing. If the reply waits for summarization, the user waits. If summarization happens in the background, we need to handle requests arriving before it finishes.
Then there’s repeated compression. An older detail has to survive every rewrite of the summary.
“Keep the budget under $500, including delivery” might eventually become “user has a limited budget.”
That’s quite a different instruction.
A summary that changes every turn can also reduce how much of the prompt remains reusable through prompt caching. The effect depends on where the summary sits and how the provider handles caching.
Maybe we can keep this structure and summarize less often.
4. Batch the summaries. Meet the amnesia gap.
Keep the eight-message window, but wait until six messages have left it before summarizing them together.
That gives us fewer summary calls.
Sounds reasonable. Until we look at what happens while those six messages are accumulating.
The problem
At message 9, message 1 has left the window. But we haven’t summarized it yet.
At message 13, messages 1–5 are waiting for the batch threshold.
They’re in neither the summary nor the raw messages being sent to the model.
At message 14, six messages have accumulated, so we finally trigger summarization.
That waiting period is the amnesia gap. A temporary black hole for context. 💀
Those messages might be safely stored somewhere. The model still can’t use them for the current reply.
If the user refers to something in that gap, the assistant is missing the very context we were planning to preserve.
This is where the system’s internal scheduling starts showing up as a product problem. The assistant can appear to forget something, then have access to it again after the summary catches up.
The gap exists before the summarization call even starts. A slow or failed call can make it last longer.
Batching helps with summary frequency. Evicting messages while they wait for the batch creates the problem.
That led to the next change.
5. Grow and flush
Keep the waiting messages in the prompt.
Instead of forcing the raw window to stay at eight, let it grow until there’s enough older material to summarize together.
Using eight and sixteen as example thresholds:
- Keep the original messages while the buffer grows.
- When it reaches 16, summarize the older eight.
- Once the summary is successfully saved, remove those eight from the active prompt.
- Retain the recent eight and let the buffer grow again.
That’s grow and flush.
On the first flush, messages 1–8 become the summary. Messages 9–16 stay in their original form.
As new messages arrive, we send that same summary plus messages 9–17, then 9–18, and so on.
At message 24, we combine the previous summary with messages 9–16. The updated summary covers messages 1–16, while messages 17–24 stay raw.
The buffer grows, then shrinks. The summary advances in batches.
The important rule is:
Keep a segment in its original form until a successfully saved summary covers it.
The messages waiting for compaction remain available to the model. They don’t disappear while we wait for the batch threshold.
“Flush” here means removing messages from the active prompt. It doesn’t require deleting the stored transcript.
If compaction runs after a response finishes, that response still uses the expanded raw buffer. Later requests can use the new summary once it has been saved.
A background job can’t go back and help a reply that already happened. Sadly, no time travel included.
The problem
We’ve given the buffer room to grow, so the main prompt is sometimes larger than it would be with a strict sliding window.
That’s the trade: fewer summary updates, more original messages carried between updates.
Fewer summary calls don’t automatically mean a lower total bill. The extra raw context costs tokens too.
The useful comparison includes the main requests, the summarization requests, summary size, and any cache savings.
The goal is to find a balance that keeps the conversation useful at a sensible operating cost.
How the approaches compare
| Approach | Main benefit | Raw context | Summary updates | Main tradeoff |
|---|---|---|---|---|
| Sliding window | Simple; no summarization calls | Fixed recent window | None | Older context drops out |
| One-time summary | Compresses early history while retaining a summary | Grows after the summary point | Once | Token growth returns |
| Rolling summary + window | Keeps recent detail alongside summarized history | Fixed recent window | Every turn that moves the window | Frequent calls and repeated compression |
| Batched summary + fixed window | Reduces summary frequency | Fixed recent window | After enough evictions | Waiting messages fall into a gap |
| Grow and flush | Batches updates while retaining messages awaiting compaction | Grows, then shrinks | After enough retained messages accumulate | Larger prompts between flushes |
Why grow and flush works for us
Grow and flush fits our current needs because it lets us batch summary updates while keeping the waiting messages available. Recent exchanges also stay in their original form, so the model gets their detail directly.
That gives us the balance we’re after: less frequent summarization, recent detail, and no amnesia gap caused by premature eviction.
In the simplified grow-and-flush example, compaction happens once per eight new messages after the initial warm-up.
Between flushes, the summary stays unchanged and new messages append to the raw history. That creates an opportunity to reuse a stable prompt prefix.
Caching can reduce processing cost. The cached content still occupies context, though. We still need to manage its size.
The tradeoffs I still need to account for
A summary can lose an important detail.
Grow and flush prevents the gap caused by dropping messages before they’ve been summarized. It doesn’t make summaries lossless.
It also still rewrites an existing summary, just less often. We need to check whether constraints, exact values, corrections, and unfinished work survive those updates.
Compaction involves choosing what to preserve. Something that looks unimportant now might matter later.
Message counts aren’t token limits.
Eight short messages and eight pasted documents are very different amounts of input.
One user pastes a giant log, and suddenly “it’s only eight messages” isn’t so comforting.
A possible solution is to use two triggers and compact when either threshold is reached:
- The unsummarized buffer reaches the message-count limit, such as 16 messages.
- Its estimated token size reaches the token budget, even if there are fewer messages.
Whichever comes first.
That way, a few large messages can trigger compaction before the buffer reaches the message-count limit.
The amount we retain after compaction also needs to fit the budget. Keeping the latest eight is a useful target, but eight enormous messages could still be too much.
We also need to check the full request size, leaving room for instructions, tools, the summary, and the response. A single oversized message may need separate handling.
And messages aren’t the same as complete turns. One user request can produce several assistant and tool messages.
Background compaction can fall behind.
While an update is pending, keep the unsummarized messages available.
Before sending the next request, check whether the full context still fits. If it doesn’t, the system needs a fallback, such as waiting for compaction or reducing oversized content.
The summary and its boundary need to stay in sync.
We need to save both the summary and the point in the transcript it covers.
If those disagree, messages can be skipped or unnecessarily repeated. Overlapping jobs also need protection so an older update doesn’t overwrite a newer one.
Critical action state needs reliable storage.
A pending confirmation or an exact action parameter should live in application state.
The summary helps the assistant follow the conversation. The action flow still needs a reliable source of truth.
For me, this is where the Product, Business, and System concerns meet: the conversation should feel continuous, the cost should remain sensible, and the implementation should behave predictably when work is delayed or fails.
Bonus: don’t cut a tool exchange in half
This applies whenever we trim or summarize conversation history.
An assistant can issue a tool call and then receive a matching result. A blind cut at “the last eight messages” can separate that result from its call.
Summarization can create the same split. If the tool call gets absorbed into the summary while its result stays in the raw history, the next request contains a result without its matching structured call.
Mentioning the call in a text summary doesn’t replace the original tool-call message.
Parallel calls make the boundary more interesting: one assistant message may need several matching results.
Depending on the API, an incomplete exchange can produce an invalid request.
When trimming or choosing a chunk to summarize, preserve complete tool exchanges, including the matching IDs and required results. Compact a completed exchange together, and keep unfinished exchanges protected in the raw context.
All that thought about memory, only to trip over one bad slice. Lovely. 😅
Where I’m taking this next
Grow and flush is what I’ve implemented for this problem.
It gives me a clear rule for when history can leave the prompt, while allowing summary updates to happen in batches.
I’m still investigating better ways to preserve useful context with less token consumption. Structured memory, selective retrieval, and different compaction schedules are all worth exploring.
If you’ve worked through this, I’d like to hear what worked, what broke, and what you’d change here. Suggestions are welcome.






Top comments (0)