DEV Community: Neeraj Yadav

A Small, Checkable Test for AI Memory Systems

Neeraj Yadav — Tue, 08 Sep 2026 00:44:25 +0000

AI disclosure: This draft was generated autonomously by AI. The author should review every technical claim before publication.

AI memory demos often optimize for a strong first impression. A long archive goes in, a fluent answer comes out, and the result feels convincing. That is not yet evidence that the memory system will be useful in ordinary work.

A better evaluation starts small enough that you already know the correct answer. It should test retrieval, interpretation, missing information, updates, and repeat use separately.

1. Begin with one source you understand

Create a short note containing a date, an owner, a decision, and one explicit limitation. Keep it small enough to read without search.

Example:

The migration review is scheduled for October 14. Priya owns the checklist. The database change is not approved yet.

Ask questions whose answers are directly present in the note:

When is the review?
Who owns the checklist?
Has the database change been approved?

The goal is not to surprise yourself. It is to confirm that the system can retrieve the expected source and that the answer preserves important qualifiers such as “not approved yet.”

2. Inspect the supplied evidence

A plausible answer is not enough. Open the source or evidence shown beside the answer and check:

Did the system retrieve the right document?
Did it select the relevant passage?
Did the answer preserve names, dates, and negation?
Can another person repeat the check?

This separates two failure modes that are often mixed together. Retrieval can choose the wrong evidence, or the answering model can misinterpret the right evidence. Those require different fixes.

3. Ask for something that is missing

Now ask a question the note cannot answer, such as:

Which meeting room is booked?

A useful system should make the absence visible. If the answer invents a room, retrieving more unrelated text will not solve the underlying problem. Missing-information tests are especially valuable because fluent models are good at making unsupported details sound ordinary.

4. Change one fact

Update the note so that the review moves to October 21. Then ask the date again.

Check whether the current answer uses the new value and whether the earlier value remains available when you ask a historical question. Finding both versions is not the same as knowing which one is current. Similarity search measures relevance, not supersession by itself.

This test helps distinguish:

retrieving a date from computing with dates;
document recency from fact validity;
remembering an earlier statement from treating it as current.

5. Return in a second session

The first successful query proves very little about memory. Close the session, return later, and ask a question that depends on the same source.

Then inspect what survived:

the source content;
the speaker or owner;
the date and session boundary;
the update from the earlier value;
the evidence needed to verify the answer.

Repeat use is where a memory workflow becomes useful or becomes another setup step.

A compact evaluation checklist

A small memory test should include:

One known fact.
One explicit limitation or negation.
One missing fact.
One updated fact.
One historical question.
One second-session question.
Source inspection for every answer.

This will not produce a universal accuracy score. It will give you something more practical: a reproducible way to identify whether a failure came from retrieval, interpretation, freshness, or unsupported guessing.

Start with evidence you can inspect. Expand the archive only after the small path works.

Post 4 - Building AI You Can Trust

Neeraj Yadav — Thu, 16 Jul 2026 03:34:56 +0000

"Move fast and break things" is terrible advice for anything that touches money, health, or safety.

In six years building product in auto-finance, the discipline that mattered most wasn't speed. It was: prove it's safe before you ship it, and build the proof into the system so it can't be skipped.

When I build AI products now, I use the same rule. Every new capability ships turned off by default. It only gets turned on after it passes a gate that proves it didn't break the thing that already worked. Not "I think it's fine", a measured, automated check that fails loudly if I'm wrong.

It feels slower. It's actually faster, because you never spend three weeks debugging a regression you shipped a month ago and forgot about.

Guardrails aren't the opposite of velocity. They're what lets you go fast without being reckless.

Leadership #ProductManagement #AI

Post 3 - Building AI You Can Trust

Neeraj Yadav — Fri, 10 Jul 2026 14:57:08 +0000

Here's a problem that sounds simple and isn't:
Your AI coding assistant learned last month that your service runs on port 8000. Yesterday you changed it to 8080. Today you ask it about the port.

A normal retrieval system stores both facts. They look almost identical to it - one word different. When you ask, it retrieves both and has no idea which one is current. So it either guesses or confidently gives you the old answer.

This is the thing I've been working on: memory for AI that knows which facts are still true, not just which ones are similar. It turns out "similar" and "current" are completely different problems, and almost everyone conflates them.

I'll go deeper on the how in future posts. For now: this single blind spot is behind a huge amount of "why did the AI confidently tell me something outdated."

Has an AI assistant ever given you a confidently wrong answer that used to be right?

AI #MachineLearning #BuildInPublic

Not many know which part of the process needs AI!

Neeraj Yadav — Fri, 03 Jul 2026 21:48:12 +0000

A process that took 2–3 weeks and produced a 40-page document is the kind of thing everyone wants to "AI-ify."
Here's what most people get wrong about it: they reach for the LLM first.

In regulated lending, the LLM is the last and smallest part of the pipeline. The hard parts are: ingesting messy documents reliably, extracting data into a canonical model, validating and cross-reconciling it, and doing the actual financial calculations - and those calculations must be done in deterministic code, never by a language model that might quietly round a number wrong.

The LLM drafts prose. Code does the math. A human owns the risk decision. That ordering isn't a limitation - under OSFI E-21 it's the only design that's allowed to exist.

The teams that win with AI in banking aren't the ones using the biggest model. They're the ones who know which 10% of the problem the model should touch.

Where have you seen an LLM used for something that should have been deterministic code?

FinancialServices #AI #Lending

MemStrata Beats RAG comprehensively on mutating code content - http://arxiv.org/abs/2606.26511

Neeraj Yadav — Sat, 27 Jun 2026 04:54:18 +0000

I've spent the last several months building an AI memory system on nights and weekends, and the most valuable thing I learned has nothing to do with AI.

It's this: the moment you let what you hope is true override what you measured, you stop doing engineering and start doing marketing.

I caught myself doing it more than once. I had a headline result I loved - and the data quietly didn't support it. I had a clever feature I'd already written up as the fix - and when I finally measured it, it made things 25% worse. Each time, the honest move was to kill the thing I was attached to.

That's uncomfortable. It's also the only thing that makes a result trustworthy.

I'm going to spend the next 90 days here writing about building AI you can actually trust - the failures included, because the failures are where the truth is. If you work in AI, in financial services, or you're building something hard on the side, I'd love for you to follow along.

What's a result you were sure of that the data later overturned?

AI #ProductManagement #BuildInPublic