DEV Community: Neeraj Yadav

Keyword search ties a purpose-built memory layer on 200 LongMemEval questions

Neeraj Yadav — Tue, 20 Oct 2026 17:50:47 +0000

I build one of the systems in this test, so discount accordingly. Posting the numbers because they surprised me and because the boring result is the interesting one.

Setup: 200 questions from LongMemEval, three ways of giving a model memory over the same material. A collaborator ran all three arms with the same reader (Qwen 3.8 27B, Q4) and the same grading rubric.

A structured memory layer (MemStrata, my product): facts stored with provenance and validity, retrieved as structured evidence.
SQLite FTS5: plain keyword full-text search over the raw text. The baseline everyone skips.
An LLM wiki: the pattern where an LLM incrementally writes and maintains linked markdown pages, and answers from those pages.

Results:

MemStrata 188/200 94.0%
SQLite FTS5 184/200 92.0%
LLM wiki 163/200 81.5%

The 4-question gap between the memory layer and FTS5 is within noise at n=200. Call it a tie. The 25-question gap to the wiki is not noise.

Category detail, memory layer vs FTS5:

Knowledge update: 29/31 vs 31/31 (FTS5 wins outright)
Temporal reasoning: 50/53 vs 47/53
Multi-session: 48/53 vs 46/53
Preference: 12/12 vs 11/12
Single-session: tied on both subcategories

Two things I take from this:

First, keyword search is embarrassingly strong and most memory products are probably being compared against strawman baselines. On single-fact updates, where the current answer is stated verbatim somewhere, FTS5 is perfect and structure adds nothing.

Second, the wiki pattern lost on the thing it is supposed to be good at. It carries the highest maintenance cost of the three (an LLM rewriting pages on every new fact) and still landed 12.5 points behind, including on multi-session questions where a curated notes base should shine. My read: every rewrite is a chance to drop or mangle a fact, and the errors compound silently. That is an interpretation, not a measurement.

Disclosures: the wiki arm is our implementation of the pattern, not anyone's production system. One MemStrata answer was missing from the run output and was recovered, with the owner's authorization, before grading; scoring it as wrong would move the result by half a point. One reader model, one question set, n=200.

Question for anyone else testing memory layers: are you also finding FTS5 this hard to beat, and on which question types does structure actually earn its keep in your runs?

How do you know your agent's memory layer is helping, not hurting?

Neeraj Yadav — Fri, 16 Oct 2026 18:44:09 +0000

I work on long-term memory for agents, and the honest problem I keep hitting: everyone benchmarks the model, almost nobody benchmarks the memory layer itself. A bad memory layer can actively make a good model worse, stale facts retrieved with high confidence, contradictions never surfaced. The vendor answer is usually "trust our evals." I published an open harness instead, MIT, that reruns 500 LongMemEval-style questions against any local stack so the numbers are checkable, not claimed: https://github.com/yadu9989/memstrata-lme500-harness

Curious how others here are evaluating theirs. Do you trust the memory layer, or treat it as guilty until proven innocent?

A Small, Checkable Test for AI Memory Systems

Neeraj Yadav — Tue, 08 Sep 2026 00:44:25 +0000

AI disclosure: This draft was generated autonomously by AI. The author should review every technical claim before publication.

AI memory demos often optimize for a strong first impression. A long archive goes in, a fluent answer comes out, and the result feels convincing. That is not yet evidence that the memory system will be useful in ordinary work.

A better evaluation starts small enough that you already know the correct answer. It should test retrieval, interpretation, missing information, updates, and repeat use separately.

1. Begin with one source you understand

Create a short note containing a date, an owner, a decision, and one explicit limitation. Keep it small enough to read without search.

Example:

The migration review is scheduled for October 14. Priya owns the checklist. The database change is not approved yet.

Ask questions whose answers are directly present in the note:

When is the review?
Who owns the checklist?
Has the database change been approved?

The goal is not to surprise yourself. It is to confirm that the system can retrieve the expected source and that the answer preserves important qualifiers such as “not approved yet.”

2. Inspect the supplied evidence

A plausible answer is not enough. Open the source or evidence shown beside the answer and check:

Did the system retrieve the right document?
Did it select the relevant passage?
Did the answer preserve names, dates, and negation?
Can another person repeat the check?

This separates two failure modes that are often mixed together. Retrieval can choose the wrong evidence, or the answering model can misinterpret the right evidence. Those require different fixes.

3. Ask for something that is missing

Now ask a question the note cannot answer, such as:

Which meeting room is booked?

A useful system should make the absence visible. If the answer invents a room, retrieving more unrelated text will not solve the underlying problem. Missing-information tests are especially valuable because fluent models are good at making unsupported details sound ordinary.

4. Change one fact

Update the note so that the review moves to October 21. Then ask the date again.

Check whether the current answer uses the new value and whether the earlier value remains available when you ask a historical question. Finding both versions is not the same as knowing which one is current. Similarity search measures relevance, not supersession by itself.

This test helps distinguish:

retrieving a date from computing with dates;
document recency from fact validity;
remembering an earlier statement from treating it as current.

5. Return in a second session

The first successful query proves very little about memory. Close the session, return later, and ask a question that depends on the same source.

Then inspect what survived:

the source content;
the speaker or owner;
the date and session boundary;
the update from the earlier value;
the evidence needed to verify the answer.

Repeat use is where a memory workflow becomes useful or becomes another setup step.

A compact evaluation checklist

A small memory test should include:

One known fact.
One explicit limitation or negation.
One missing fact.
One updated fact.
One historical question.
One second-session question.
Source inspection for every answer.

This will not produce a universal accuracy score. It will give you something more practical: a reproducible way to identify whether a failure came from retrieval, interpretation, freshness, or unsupported guessing.

Start with evidence you can inspect. Expand the archive only after the small path works.

Post 4 - Building AI You Can Trust

Neeraj Yadav — Thu, 16 Jul 2026 03:34:56 +0000

"Move fast and break things" is terrible advice for anything that touches money, health, or safety.

In six years building product in auto-finance, the discipline that mattered most wasn't speed. It was: prove it's safe before you ship it, and build the proof into the system so it can't be skipped.

When I build AI products now, I use the same rule. Every new capability ships turned off by default. It only gets turned on after it passes a gate that proves it didn't break the thing that already worked. Not "I think it's fine", a measured, automated check that fails loudly if I'm wrong.

It feels slower. It's actually faster, because you never spend three weeks debugging a regression you shipped a month ago and forgot about.

Guardrails aren't the opposite of velocity. They're what lets you go fast without being reckless.

Leadership #ProductManagement #AI

Post 3 - Building AI You Can Trust

Neeraj Yadav — Fri, 10 Jul 2026 14:57:08 +0000

Here's a problem that sounds simple and isn't:
Your AI coding assistant learned last month that your service runs on port 8000. Yesterday you changed it to 8080. Today you ask it about the port.

A normal retrieval system stores both facts. They look almost identical to it - one word different. When you ask, it retrieves both and has no idea which one is current. So it either guesses or confidently gives you the old answer.

This is the thing I've been working on: memory for AI that knows which facts are still true, not just which ones are similar. It turns out "similar" and "current" are completely different problems, and almost everyone conflates them.

I'll go deeper on the how in future posts. For now: this single blind spot is behind a huge amount of "why did the AI confidently tell me something outdated."

Has an AI assistant ever given you a confidently wrong answer that used to be right?

AI #MachineLearning #BuildInPublic

Not many know which part of the process needs AI!

Neeraj Yadav — Fri, 03 Jul 2026 21:48:12 +0000

A process that took 2–3 weeks and produced a 40-page document is the kind of thing everyone wants to "AI-ify."
Here's what most people get wrong about it: they reach for the LLM first.

In regulated lending, the LLM is the last and smallest part of the pipeline. The hard parts are: ingesting messy documents reliably, extracting data into a canonical model, validating and cross-reconciling it, and doing the actual financial calculations - and those calculations must be done in deterministic code, never by a language model that might quietly round a number wrong.

The LLM drafts prose. Code does the math. A human owns the risk decision. That ordering isn't a limitation - under OSFI E-21 it's the only design that's allowed to exist.

The teams that win with AI in banking aren't the ones using the biggest model. They're the ones who know which 10% of the problem the model should touch.

Where have you seen an LLM used for something that should have been deterministic code?

FinancialServices #AI #Lending

MemStrata Beats RAG comprehensively on mutating code content - http://arxiv.org/abs/2606.26511

Neeraj Yadav — Sat, 27 Jun 2026 04:54:18 +0000

I've spent the last several months building an AI memory system on nights and weekends, and the most valuable thing I learned has nothing to do with AI.

It's this: the moment you let what you hope is true override what you measured, you stop doing engineering and start doing marketing.

I caught myself doing it more than once. I had a headline result I loved - and the data quietly didn't support it. I had a clever feature I'd already written up as the fix - and when I finally measured it, it made things 25% worse. Each time, the honest move was to kill the thing I was attached to.

That's uncomfortable. It's also the only thing that makes a result trustworthy.

I'm going to spend the next 90 days here writing about building AI you can actually trust - the failures included, because the failures are where the truth is. If you work in AI, in financial services, or you're building something hard on the side, I'd love for you to follow along.

What's a result you were sure of that the data later overturned?

AI #ProductManagement #BuildInPublic