Why I stopped letting LLMs decide what they remember

Kaue Oliveira — Thu, 27 Aug 2026 06:16:36 +0000

I've been working on a project called ProofRay.

It's not another chatbot, and not another wrapper around RAG. I wanted to test a simpler idea: what if memory and text generation are two different problems?

LLMs are great at writing. They'll make an answer sound confident, natural and useful even when they remembered the wrong thing. That's fine for a lot of tasks, but it's a bad contract for personal memory, technical docs, logs. Anything where "I think this was the answer" isn't enough.

So ProofRay sits before the model.

You give it documents and a question. It routes the question to source material, verifies identities, spans and digests, and tries to close a proof for the answer.

Proof closes → it answers.
Material is useful but incomplete → it returns evidence.
Not enough authority → it abstains.

The important part:

relevance alone can't become an asserted answer.

The core runs locally and doesn't need embeddings or an LLM to make the memory decision. You can plug a model in after ProofRay, but only as a reader or rewriter of a result that was already authorized.

I recently ran four end-to-end consumer paths on MemGym-DR with the current default/general memory profile:

ProofRay standalone, no LLM: 0.7975
ProofRay + Gemini Flash-Lite polish: 0.6125
BM25 RAG + Gemini Flash-Lite: 0.5583
ProofRay + Qwen3 1.7B Q8 local: 0.4975

Higher is better.

The standalone number was the one that got me. I expected the LLM to make the final answer better. Instead the "polish" step often made it worse, it would drop an important condition, change a detail, merge evidence into a claim, or turn an abstention into something that sounded more complete than it actually was.

That doesn't mean Gemini or Qwen are bad. It means they shouldn't be the authority on memory.

The local Qwen run was fun too: a 1.7B Q8 model on my own machine landed reasonably close to a traditional BM25 + cloud Gemini pipeline. I'm not claiming a small local model matches Gemini in general. I'm saying better memory architecture seems to reduce how much model scale you need for a recall task.

Some limits, because they matter:

This is consumed-development benchmark data, not a sealed claim of general performance. ProofRay doesn't understand every possible question, language, relation or domain. The default profile targets general source-grounded memory; there's a separate personal profile for conversational/user-memory cases. The repo keeps the failed experiments and the limitations documented instead of hiding them.

I'm releasing it as a public alpha because I want people throwing real documents at it, weird "remember when..." questions, conflicting notes, cases I didn't think of.

Repo: https://github.com/kyuubyN/ProofRay

If you try it, I'd honestly rather get a good failure case than a generic "looks cool".

DEV Community: Kaue Oliveira

Why I stopped letting LLMs decide what they remember