With Context Windows This Large, Why Do We Still Need Memory?

jj_uc — Sun, 06 Sep 2026 11:43:19 +0000

I've been reading through dev communities lately, and this exact topic keeps showing up in different forms: context windows hit a million tokens, so is memory even necessary anymore. Enough people are arguing both sides that I wanted to actually dig into it and put together my own take instead of just picking whichever post I read most recently.

Just How Big Are They Now?

A few years ago, a few thousand tokens felt generous. Now, 1 million is the baseline. Meta released the 10-million-token Llama 4 Scout last year, and a startup called Magic built a 100-million-token model (LTM-2-mini). That means about 10 million lines of code or 750 novels can fit into a single prompt. At this point, you have to wonder what is left for a separate memory system to do.

Camp 1: It's Already Enough

Fabio Akita pointed out that looking at the leaked Claude Code source code, Anthropic's own coding agent doesn't use a vector DB at all; it just uses the file system and grep. By his math, a 200,000-token query costs about $0.63 including caching, which is cheaper long-term than maintaining a vector DB pipeline. Meta made the same bet, marketing the Llama 4 Scout as capable of holding years of chat history "without a vector store."

It's an attractive argument, and honestly, the cost aspect is the strongest part of this camp. But I think the question this camp is answering is narrower than reality. "Do we need a vector DB?" and "Do we need memory?" are different questions. Akita's point is really about search complexity, using grep instead of embeddings, not about whether state needs to persist between sessions. Meta's marketing also conveniently skips over the fact that a session with 10-million-token chat history eventually ends, and the next session starts at zero.

Camp 2: Memory Solves a Completely Different Problem

Mem0 compares the context window to RAM rather than storage: the moment a session ends, everything inside vanishes. Redis puts it more sharply: agents don't fail because a single invocation lacks space, but because they lack continuity, they can't carry what they learned in one session over to the next. No window size fixes this. Even a 100-million-token model completely forgets you the moment you open a fresh chat.

The most reliable evidence here is Chroma's "context rot" research. Testing 18 models (including GPT-4.1, Claude 4, Gemini 2.5, Qwen3), they found that performance degrades as inputs get longer, well before hitting the model's actual limits. A single piece of irrelevant distractor info noticeably drops accuracy. In some tests, a scrambled mess of info actually performed better than an organized one. This is what genuinely convinced me: fitting into a window and a model actually utilizing it well are two different things, and sellers of larger windows have plenty of incentive to blur that boundary.

And Camp 3: The Window Is Simply Too Small

Factory.ai points out that current 1-million to 2-million-token models are already smaller than their enterprise customers' codebases. CloudGeometry goes a step further, arguing that even if you gave them 100 million tokens tomorrow, it wouldn't be enough because codebases are graph structures while context windows are linear, no matter how large they get, that structural topology disappears.

What's fascinating is that Factory and CloudGeometry draw the exact same fact, "therefore, we need better search," while Magic and Meta go with, "therefore, let's build bigger windows." Same observation, opposite prescriptions. It's also worth noting that the "bigger window" crowd usually sells models, while the "better search" crowd usually builds things on top of other people's models.

My Conclusion

Context windows need to keep growing, I'm not arguing against that. What I disagree with is treating "bigger windows will fix memory problems" as a given. These are two separate investments, and relying entirely on one to substitute for the other doesn't work. Why? Because a bigger window solves exactly one problem: the fitting problem.

It does nothing for session amnesia, accuracy loss from irrelevant tokens, or the cost of reprocessing the exact same context on every single invocation. Memory has to evolve on its own to solve these three, and scaling the context window won't do that work for it. So the real paradigm isn't "context versus memory." Rather, while context handles one problem, memory must solve the remaining three through its own evolution, not by waiting for someone else's context window to get bigger.

The Strongest Counter-Argument to My Point

I don't want to give memory a free pass, so let's briefly counter my own argument. Memory poisoning is a real attack where someone sneaks mundane content into an agent's long-term memory to plant false info, which the agent then retrieves and trusts in a completely unrelated situation later on. MINJA, presented at NeurIPS 2025, achieved this with just a few mundane queries.

Add in silent failures (memory retrieves the wrong thing, yet the model speaks with absolute confidence and no errors) and stale information (studies measure that forcing answers yields stale info 15 to 40 percent of the time), and you have genuine reasons to doubt anyone saying, "just throw memory at it."

Yet my answer doesn't change, and here is why. Silent failures and stale info are mistakes, not malicious acts. They are mostly fixed by displaying confidence scores and recency on retrieved memories, and automatically invalidating old info when new data arrives. This isn't a design flaw, it's engineering maturity, and it's precisely what improves as memory technology matures. Security, however, never fully goes away, just like defense and offense evolving together, much like anti-spam filtering never truly ends. But saying "this requires ongoing work" is different from saying "memory is a dead end." It's simply a trade-off you must accept the moment you ask an agent to persist anything, via memory or otherwise.

Wrapping Up

So let's sum it up: if your agent needs to remember anything past the current session, waiting for context windows to get bigger is betting on the wrong lever.

Change my mind. What is actually breaking in your production environment right now, massive contexts, RAG, structured memory, or something entirely different?

How Do ChatGPT and Claude Actually "Remember" Our Conversations?

jj_uc — Thu, 03 Sep 2026 08:13:41 +0000

Introduction

We’ve all experienced the frustration of yelling at an LLM, "I literally just told you that!" But on the other hand, there are moments when it casually recalls a minor detail you mentioned weeks ago, making you wonder, "Wait, how did it remember that?"

Why is LLM memory so inconsistent? Is it actually storing our chat history somewhere, or is something else happening under the hood? Let's break down how modern AI services handle context and memory from a technical perspective.

1. LLMs Don't Actually "Remember": The Reality of Context Windows

The first thing to understand is that LLMs do not possess human-like long-term memory stored in an internal state.

At their core, LLM-based chat applications are stateless. Every single time you send a message, the system passes the entire conversation history up to that point back into the model along with your new prompt.

The bottleneck here is the Context Window.

Think of it like a desk with limited workspace.
As the conversation grows and hits the token limit, older messages get pushed off the edge of the desk (pruning) and disappear from the model's field of view entirely.
If the AI suddenly forgets a core architectural constraint you established earlier and starts hallucinating bad code, it's usually because those earlier tokens have fallen out of the active context window.

2. Surviving Long Conversations: Summarization and Pruning

So, do long conversations just completely wipe their early history?

Not quite. When a conversation approaches the window size limit, services typically rely on background summarization. The system compresses older messages into a brief summary to save tokens while attempting to preserve the main narrative flow.

However, this introduces a severe engineering trade-off:

Loss of Granularity: During summarization, hyper-specific details—such as exact variable names, strict typing constraints, or edge-case handling logic—inevitably get squashed.
If you find yourself asking, "Why is it still recommending that function when I explicitly told it not to?" chances are that specific detail was lost in the summarization pipeline.

3. Persistent Memory Across Sessions: How Services Differ

What about information that persists across weeks or in brand-new chat sessions? This is where the product design and underlying system architectures of different AI providers diverge.

OpenAI ChatGPT: Global User Profiling

ChatGPT’s "Memory" feature continuously analyzes conversations in the background to build and update a global profile summary of your preferences, tech stack, and coding style. When you open a new chat session, this summary is injected into the system prompt. This is why memory updates often take effect asynchronously or in subsequent sessions.

Anthropic Claude: Structured Context & Projects

Claude takes a slightly different approach through features like 'Projects', allowing developers to isolate files, documentation, and specific system instructions into a dedicated context container. When you correct Claude on a specific rule, it tends to apply updates more immediately through targeted prompt injection and RAG (Retrieval-Augmented Generation) pipelines.

The Unified Limitation: Fragmented State

The common denominator across all these systems is state fragmentation. Information or preferences you teach ChatGPT remain completely unknown to Claude. It's the ultimate siloed developer experience.

Conclusion

If you throw an infinite amount of context and code at a model, token costs and latency explode. If you summarize too aggressively, you lose all the engineering details. Current LLM services are constantly iterating to find the optimal trade-off between token efficiency, inference speed, and memory fidelity.

How do you manage your AI interactions and prompt context in your daily workflow? Have you ever hit a bizarre debugging loop because of context window limitations? Let's discuss in the comments below!

DEV Community: jj_uc