I Tried Building LLM Memory the Stupid Way — It Worked Surprisingly Well

rajesh kumar — Mon, 10 Aug 2026 13:56:22 +0000

This started as a fun experiment.
I've been working on a local RAG system , and at some point I started wondering about something that sounds simple but is actually quite difficult:
Can an LLM remember old conversations without sending its entire history back into the context window?
I didn't want to build a complicated memory architecture just to answer that question. So I tried something almost ridiculous. I exported my old chats and extracted conversations from Json files.
The entire memory extractor was basically one rule
If a line had more than 10 words, keep it.
If it had 10 or fewer, throw it away.
That's pretty much it.
No LLM summarization during ingestion. No expensive memory model. No elaborate memory classification.
The resulting text was then chunked and fed into my existing RAG. For every question, I allowed retrieval of only 8 chunks.
So the experiment looked roughly like this:
Old conversations
↓
Keep lines with >10 words
↓
Create chunks
↓
BM25 + semantic retrieval
↓
Top 8 chunks
↓
Feed back to same LLM
Then I decided to actually try to break it.
I used my own old conversations
Instead of asking generic questions, I went back to conversations from the extracted chats and asked the LLM very specific questions.
Things like old procurement discussions, financial figures, dates, coding decisions, project discussions and relationships between completely separate conversations.
I asked roughly 13–14 questions.
LLM failed.
Then I fed the chunks from my RAG

And honestly, I expected it to fail much more often.
It didn't.
It recovered almost everything I asked about.
It could retrieve exact amounts and dates. It could recover old technical discussions. In some cases, it could connect information from separate conversations and reconstruct what had happened.
And remember:
The model only received eight retrieved chunks.
It wasn't given my entire historical conversation corpus.
Then I found the obvious weakness
The one clear failure was actually quite revealing.
It was terrible at remembering code and commands may be since they are often less than 10 words. So my brilliant memory algorithm simply throws them away. But I tried to make it as a context builder rather than a storage. So I am satisfied with results.
The LLM was restricted to use internet or think only to answer from its memory. I think if that was allowed that single miss would haven’t occurred. Its not a production grade memory management system. But it works gr8.

DEV Community: rajesh kumar

I Tried Building LLM Memory the Stupid Way — It Worked Surprisingly Well