AI is moving fast, and it feels like there's a new concept to learn every week. In an effort to actually understand this whole new world instead of...
For further actions, you may consider blocking this person and/or reporting abuse
Thanks for the analogy and explanation. Would you be open to create a new post explaining what loop engineering is and its usage?
This is not a concept is more like a practise, a while loop that let the ai run, test and self correct. I try to write about the concepts and algorithms that are around ai
I really liked the "smart intern" analogy. 😄 I also think a lot of people assume bigger context windows mean the model magically remembers everything. In reality, feeding more text isn't the same as giving it the right information. Good explanation for beginners without oversimplifying it.
I guess people cannot comprehend how "if this box answers me quite accurately doesn't have memories and doesn't remember much". It's very hard to explain easily that everything they see is a statistic probability
The "lost in the middle" point connects nicely to why RAG isn't just about fitting things in. Even when the whole doc fits, burying the answer at 50% depth can hurt you, so good retrieval also decides where in the prompt the relevant chunk lands, not only which chunk. Have you looked at re-ranking yet? It's the natural next ELI5 after this one.
I was caught up with reading about embeddings which are quite fascinating. Re-ranking will come in the future! ^_^
I enjoyed this explanation, especially the emphasis that a context window is closer to working memory than long-term memory.
One thing it made me think about is that we've started treating larger context windows as though they're a substitute for memory. I'm not convinced they are.
To me, the context window is really an execution surface. It's the information available for this reasoning step. Once the inference completes, that context disappears unless something intentionally preserves it.
That's where I think concepts like Memory as Infrastructure become interesting. Instead of assuming memory "just happens," the surrounding system decides what deserves to survive beyond the context window. Decisions, evidence, corrections, provenance, and reasoning history become durable architectural assets instead of transient prompt content.
I also wonder if this is where Prose Tax starts to matter. Bigger context windows certainly allow us to provide more information, but they don't eliminate the cost of recovering intent from verbose or ambiguous language. A million-token context full of loosely organized prose may still be less effective than a much smaller context containing precise, well-structured information.
A larger context window lets an agent think longer.
Memory as Infrastructure lets a system learn longer.
Those feel like related, but fundamentally different, architectural problems.
Exactly. Bigger context windows are a genuine technical improvement, but they're also an easy metric to market because they're simple to compare: 200k, 1M, 10M tokens. It's like megapixels in cameras. More isn't automatically better.
Speaking as the audience this article is actually for: I'm a freelancer in
Indonesia, I subscribe to a couple of chat products, and I have never built a
RAG pipeline in my life.
The amnesia section explained something I'd been misreading for a year. When a
long chat starts giving worse answers, my assumption was that the model was
somehow getting tired or that I'd annoyed it into being lazy. It genuinely felt
like that. Knowing it's a sliding window and the early instructions have simply
scrolled out is much less mysterious, and it changes what I do — start a fresh
chat and re-paste the constraints, instead of repeating myself louder in the
same one.
The intern analogy is the part I'll steal when clients ask me why the AI got
something wrong about their business.
One small thing from the cheap seats: for people at my level, the practical
version of "context window" is mostly "when do I start over." That's not in the
article, and maybe it doesn't belong, but it's the decision the concept actually
maps to for a non-builder.
Thanks, yes a clear window can help, claude also provide the ability to summarise the previous messages so it can free space, but the caveat is that it might lose information
Nice explanation! One distinction I’d add is that people often confuse the context window with memory. A context window is just the information available for a single inference, while memory (when a product supports it) is an application-layer feature that retrieves or stores information across sessions. Keeping those concepts separate helps explain why a model can have a huge context window and still not “remember” previous conversations by itself.
References:
Definitely worth mentioning! Thanks!
Nice explanation. One nuance worth adding is that a context window is closer to the model’s temporary workspace than human memory. The model does not necessarily retain everything inside it equally well, and different products may truncate, summarise or retrieve earlier messages rather than simply letting them slide out.
This is also why huge context windows do not remove the need for RAG or persistent memory. Context determines what the model can see right now whereas RAG determines what information should be brought in and memory systems decide what may be useful again later. They solve related, but different problems.
the "lost in the middle" section is the one that changes how you structure prompts in prod. ran evals where the same fact at position 30% vs 80% in a 50k context had a 15% accuracy gap on retrieval questions — "200k context = just cram everything in" is genuinely bad advice for tasks that need precision on a specific buried fact.
the fix we landed on: anchor critical info right before the instruction. retrieval at the start for richness, the specific thing the model must not miss right at the end.
did you cover the attention distribution research in a followup, or is this series staying more conceptual?
This series is for beginners to understand the basic concepts of how AI systems work.
The RAG caveat at the end is right but understates the failure mode. Good retrieval fixes the "wrong information" problem, it doesn't fix lost-in-the-middle on its own. If retrieval hands the model twenty relevant chunks and you dump all twenty into the prompt, you've just moved the amnesia problem downstream, now it's happening inside the retrieved set instead of the raw conversation.
The part that actually holds up in practice is ranking and trimming what retrieval returns before it goes into the prompt, not just getting the retrieval itself accurate.
That's a fair point, and I agree. This article is intentionally focused on explaining what a context window is, not on RAG or retrieval strategies. I wanted to keep the mental model simple before introducing concepts like ranking, reranking, chunk selection, and other techniques that make RAG effective in practice. Those definitely deserve an article of their own.
Makes sense to scope it that way, the mental model has to exist before the retrieval failure modes are legible anyway. Looking forward to the RAG follow-up, that's the part where "more context" and "better context" stop being the same claim.
The sliding-window framing is where the ELI5 quietly diverges from what actually happens in most chat products. A raw API call doesn't slide anything — if you exceed the limit, it errors out. The "old messages disappear" behavior is something the product implements on top: it truncates or summarizes history before it hits the model. That distinction matters because it means the forgetting isn't a law of physics, it's a policy decision someone made, and different apps make it differently. Some drop the oldest turns, some summarize them, some pin the system prompt and evict the rest. When a user complains the model "forgot," the bug is usually in that eviction logic, not the model.
Worth adding a caveat to the RAG section too: retrieving the wrong page is the loud failure, but the quiet one is retrieving the right page plus four mediocre ones. That's exactly the "lost in the middle" problem you cited biting you again — the correct chunk lands in position 3 of 5 and gets underweighted. So retrieval quality isn't only precision at rank 1; it's how aggressively you prune before stuffing the context. More retrieved chunks is the same trap as a bigger window, just wearing a RAG costume.
Those are good nuances, but I think we're talking about different goals. This article is an ELI5 introduction, not a deep dive into LLM infrastructure.
To explain context windows to someone who's never heard of them, simplifying implementation details is intentional. Going into API behavior, eviction policies, chat product architectures, retrieval ranking, and lost-in-the-middle would make the article harder to understand than the concept it's trying to explain.
All of the points you raised are valid in a more advanced discussion, but they're beyond the scope of an introductory article. Sometimes the goal is to build the right mental model first, then explore the edge cases later.
This is good explanation agree on the context window
but take eg of internal docs even RAG couldn’t be much helpful when retrieving answers for questions but say if you take two documents and give them to model and if you question them answer accuracy will be good but same if you ask embedded with RAG isn’t great it is just ok
My article was mainly about explaining the role of the context window, but RAG definitely has its own trade-offs and limitations that are worth discussing separately.