Forty turns into a slow-burn mystery roleplay, the character I had spent two hours building forgot her own name. She started apologizing politely and asking what I wanted to talk about today. If you have run any long conversation with an LLM-backed character, you have hit this wall too.
It feels like the model "lost its memory." That framing is wrong, and the wrong framing leads people to the wrong fixes. Here is what is really happening under the hood, and why throwing a bigger context window at it does not solve it.
There is no memory. There is a context window.
A chat model is stateless between calls. Every turn, the app stuffs the system prompt, the character definition, and as much prior conversation as fits into one input, then asks the model to predict the next message. What you experience as "the character remembering" is just old turns still being inside that input window.
When the conversation grows past the window, the oldest turns fall out. The turn-12 detail that made your character who she is gets evicted to make room for turn-41. The model is not forgetting. The app stopped showing it the thing you wanted remembered.
Why a bigger window is not the fix everyone expects
The obvious answer is "use a model with a 200k or 1M token window." It helps less than you would think, for two reasons.
The first is cost and latency. Attention is quadratic in sequence length, so doubling the window roughly quadruples the compute per token. Apps cap the effective window well below the advertised maximum to keep replies fast and cheap, which is why a model marketed at 128k often behaves like it is working with a fraction of that.
The second is degradation inside the window. The "Lost in the Middle" research (Liu et al., 2023) showed that models retrieve facts well from the start and end of a long input but reliably miss facts buried in the middle. Recall does not stay flat as the window fills. It sags, and the sag gets worse the longer the input runs.
So a bigger window buys you a longer rope, then quietly frays the middle of it. Your turn-12 anchor is exactly the kind of mid-context detail that gets dropped first.
What holds a long scene together
The apps that survive long roleplay do not rely on raw window size. They add an architecture layer on top of the model.
Recursive summarization is the simplest version. The app periodically compresses old turns into a dense summary and keeps that summary in-context while letting the verbatim history age out. You trade exact wording for a small, durable footprint of what mattered.
Retrieval is the stronger version. The app embeds prior turns into a vector store and, each turn, pulls back only the few passages relevant to the current moment. This is what "lorebooks" and "world info" systems are doing: a keyword in the last few messages fires a retrieval that injects the right backstory entry right when it is needed, instead of carrying all of it all the time.
The pattern across both: stop trying to keep everything in the window, and get deliberate about what re-enters it and when.
Where this shows up if you are a user, not a builder
The engineering matters because it explains a thing every heavy user notices. Two apps running similar base models behave completely differently past turn 50, and the difference is almost never the model. It is whether the app built a memory layer or shipped a thin wrapper over a raw context window.
I got curious about which consumer apps had done this work, so I ran the same 80-turn scenario across fifteen of them and watched for the exact failure above: character consistency, plot recall, and whether the story circled back to a planted detail without being reminded. The full breakdown of which apps held the thread and which collapsed is here. The short version: the survivors were the ones with a retrieval or summarization layer, and raw window size barely predicted the outcome.
The takeaway
If your character keeps dissolving into a generic assistant, the fix is not a longer prompt or a bigger model. It is structure: summarize the old, retrieve the relevant, and protect the handful of facts that define the scene.
Treat the context window as a cache, not a hard drive. The moment you design for that, long conversations stop falling apart in the same predictable place.
Top comments (0)