If you use ChatGPT, Claude, Grok, Copilot, or Gemini daily, it feels like you're talking to a person. It remembers what you said three messages ago. It references the project details you shared yesterday. It feels like the model has a persistent brain that is learning about you.
But it’s a lie.
From an architectural standpoint, an LLM is the most "forgetful" piece of software you will ever use. Every time you hit "Send," the model starts at a blank slate.
So, how does it maintain your chat history? The answer lies in the Context Window and the engineering that happens outside the model’s weights.
The Reality: LLMs Are Stateless
Large Language Models (Transformers) are stateless functions. In computer science terms, a stateless service processes a request based solely on the input provided at that moment.
When you send a prompt:
- The model receives your current message.
- It generates a response.
- It then discards everything. The model’s internal weights—the "brain" that was trained for months—do not change based on your conversation. It does not update its database, and it does not store your name or your preferences in its parameters. If you close the chat and start a new one, the model has absolutely no idea who you are.
The Solution: The Context Window "Buffer"
If the model is stateless, why does it seem to remember? Because of the Context Window.
Your UI (the chat interface) acts as a high-speed messenger. Behind the scenes, the UI maintains an array of your conversation history.
Every time you send a new message, the UI application does the following:
- Retrieves your current input.
- Fetches the previous $N$ messages from your chat history.
- Packages the entire conversation—your prompt plus the last 10-20 turns of history—into one giant, concatenated string.
- Sends that entire bundle to the LLM as the "context.
"When the LLM receives this bundle, it "reads" the entire conversation from the top down. It generates the next token based on the entire history provided in that specific prompt.
The LLM isn't remembering your past; the UI is just resending the past to the LLM every single time you speak.
The Engineering Trade-offs
This "resend everything" approach is why we have the concept of a Context Limit:
- Token Costs: Since you are resending the entire history with every prompt, the number of tokens processed grows significantly as the chat gets longer. This increases latency and API costs.
- The "Lost in the Middle" Phenomenon: As the context window fills up, the model’s performance can degrade. Models sometimes struggle to "attend" to information buried in the middle of a massive context block, focusing instead on the beginning or the very end.
- Context Management: Modern AI applications use advanced techniques like RAG (Retrieval-Augmented Generation) or Summarization/Memory Buffers to decide which parts of your history are relevant enough to be included in the context bundle, ensuring the model stays focused without exceeding token limits.
For the Software Professional: The "Stateless" Mindset
Understanding this distinction is vital for anyone building AI-native applications:
- Don't rely on the model for storage: If you need to store user preferences, conversation logs, or specific facts, do it in a traditional database (e.g., PostgreSQL, Redis, or a Vector DB).
- Manage your own context: When building an API, you are responsible for the "memory." You must manage the conversation array, truncate old messages, or summarize long sessions before sending them to the LLM.
- Scalability: Treat the LLM as the processing engine, not the data store. Your application layer should handle the "state."
The Big Takeaway
The feeling that an LLM has a "memory" is a masterclass in Application Layer Engineering. We have essentially built a sophisticated "stateful wrapper" around a "stateless core."
The next time you chat with an AI, remember: it’s not remembering you—it’s just reading the notes your interface handed it, seconds before it replied.
This makes it very clear to your readers that the "Memory" lives in the Application Layer, not the Model Layer.
Top comments (0)