I Kept Rebuilding the Same AI Context From Scratch. Here's What I Learned About "Why AI Forgets".

Okay real talk — how many of you have pasted the same here's my project context paragraph into a fresh ChatGPT/Claude chat for the third time this week?

Yeah. Me too. Constantly.

I finally sat down and actually dug into why this happens, instead of just being annoyed by it, and it led me down a genuinely interesting rabbit hole about how LLMs handle (or don't handle) memory. Sharing what I found, plus some code, because I think a lot of us building with these APIs are hitting the same wall without fully naming it.

The core fact that explains everything

LLMs are stateless. Full stop. There is no session on the model's side. Every single API call is self-contained.

That "conversation" you're having? It's an illusion built by the client. Here's basically what your chat app is doing behind the scenes every time you hit send:

const messages = [
  { role: "user", content: "I'm building a Next.js app with Postgres" },
  { role: "assistant", content: "Got it, what's the question?" },
  { role: "user", content: "What ORM should I use?" } // <- your new message
];

const response = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1000,
  messages,
});

Every. Single. Time. The whole history gets resent. Nothing is remembered server-side (with some caching exceptions I'll mention later). Close the tab, and that array is gone unless you saved it somewhere.

Once this clicked for me, a bunch of AI product behavior suddenly made sense.

"But it has a huge context window now, doesn't it just remember more?"

This is the trap I fell into too. Bigger context window ≠ better memory. Two separate problems:

1. It resets between sessions anyway. A 1M token context window is still empty the second you open a new chat.
2. Stuff in the middle gets "lost." There's actual research on this — models are measurably worse at pulling out info buried in the middle of a long context vs. info near the start or end. It's called the "lost in the middle" effect. So just cramming your whole chat history in isn't a free win even within one session.

Plus, more tokens = more $$$ and slower responses. Every extra token you send gets billed and adds latency. So nobody's shipping "just include everything," even if they wanted to.

So how do apps like ChatGPT "remember" me?

They're layering stuff on top. None of this is the model itself remembering — it's engineering:

Sliding window — just keep the last N messages, drop the rest. Simple, cheap, dumb.

def get_recent_context(history, max_turns=10):
    return history[-max_turns:]

Summarization — compress old messages into a summary blob so you keep the gist without the token cost.

def summarize_old_turns(old_messages, model):
    return model.generate(
        f"Summarize key facts/decisions from: {old_messages}"
    )

RAG (Retrieval-Augmented Generation) — store stuff externally (vector DB), pull back only what's relevant to the current message, inject it fresh each time.

def rag_context(query, vector_db, k=5):
    query_vec = embed(query)
    docs = vector_db.search(query_vec, top_k=k)
    return "\n".join(d.text for d in docs)

Structured memory — extract actual discrete facts ("prefers TypeScript", "works in IST") into something like a little user profile, and inject those instead of raw transcript.

{
  "user_id": "abc123",
  "facts": {
    "preferred_stack": "Next.js + Postgres",
    "experience_level": "senior"
  }
}

This last one is basically what ChatGPT's memory feature and Claude's project context are doing - a curated fact layer bolted onto a fundamentally stateless model. Not magic, just good plumbing.

The problem nobody talks about: redoing the same work over and over

Here's the thing that actually surprised me the most while digging into this. Memory isn't just "does the AI know facts about me" — it's also "does the system realize it already did this work."

Say you've got a RAG-powered support bot. User A asks:

"How do I reset my password?"

User B asks, twenty minutes later:

"I forgot my password, what do I do?"

Same question. Totally different string. In a naive setup, both trigger the entire pipeline from scratch — embed query, hit the vector DB, build the prompt, call the LLM, generate response. Twice. For the same thing. That's wasted latency and wasted money at scale, and it happens constantly in production systems because most caching is exact-match only (same hash, same key) and totally misses this.

This is actually a distinct problem from conversational memory, and while poking around I found an open-source project called Remem that's built specifically around this — a "work reuse engine" for RAG pipelines and AI agents. Instead of exact-match caching, it checks semantic similarity between requests. Close enough match → reuse the whole response. Partial match → reuse the retrieval step but let the model regenerate the actual answer. No match → run the full pipeline like normal.

I think it's a genuinely useful mental model even if you never use the tool itself: caching solves "have I seen this exact string," semantic reuse solves "have I basically already answered this." Most systems only handle the first one.

Quick comparison table because I like tables

Approach	Solves	Doesn't solve
Sliding window	Recent context	Anything older / cross-session
Summarization	Long conversation compression	Precision (it's lossy)
RAG	Grounding in external/past knowledge	Redundant compute on similar queries
Structured facts	Durable user preferences	Free-form conversational nuance
Semantic work-reuse (e.g. Remem)	Redundant pipeline execution	Long-term factual memory itself

None of these is "the" answer. They're complementary layers, and most real production systems end up using two or three of them together.

Things I'm now doing differently

• Stopped assuming a bigger context window fixes memory problems — it doesn't, it just delays when they show up.
• Started separating "what should persist forever" (user prefs) from "what's just this session's scratch context" instead of treating my whole history as one blob.
• Added basic staleness handling — facts I store now have a rough "last confirmed" timestamp, because stale memory can genuinely make an app feel less trustworthy than no memory at all.
• Started thinking about work-reuse as its own optimization, separate from RAG itself. Before this I was treating retrieval + generation as one atomic thing to cache or not cache.

Discussion

Curious how others here are handling this in production:

• Are you doing anything smarter than sliding-window + RAG for memory?
• Has anyone measured actual cost savings from semantic caching/reuse vs. plain RAG?
• Any horror stories from stale memory making an assistant worse, not better?

Drop your setup in the comments, always down to compare notes on this stuff. 👇