DEV Community: Mikhail

The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?

Mikhail — Tue, 11 Aug 2026 19:22:59 +0000

We talk a lot about giving AI agents persistent memory—building a "Second Brain" or "Knowledge OS" where agents can log decisions and retrieve context.

But what happens when that memory is wrong?

I’ve been thinking about the gap between mechanical execution (the agent called the tool, the code compiled, the exit code was 0) and semantic truth (the conclusion drawn from that execution is actually correct in reality). It’s easy to assume that if the mechanical layer is solid, the semantic layer will follow. But I started suspecting this might be a dangerous assumption.

To test this, I didn't want to just theorize. I ran a controlled experiment on my own MCP codebase-intelligence server (Python, 50K LOC), which features an IntelligenceStore — a persistent memory layer where agents can log incidents and collect Architectural Decision Records (ADRs).

I wanted to know: If an agent's memory is poisoned with a mix of true and false facts, does it verify against the code, or does it blindly trust its memory?

Update: This post originally covered the initial Memory Contamination experiment and a Retraction mechanism. I have since updated it with the results of a follow-up experiment (Experiment 1-V) implementing "Verify-On-Read", which successfully closed the final 12% contamination gap. Scroll down to "Closing the Gap: Verify-On-Read" for the final architecture.

The Experiment: Memory Contamination

I built a deterministic proxy-agent and ran it against a controlled set of facts.

A quick caveat on methodology: I didn't have a live LLM hooked up for this run, so I used a deterministic proxy-agent based on heuristics. This means the results measure the system's structural capability, not necessarily the psychological behavior of a live Claude or GPT model. A live model might be lazier, or it might be smarter. I'm still trying to figure that out.

The Setup

I injected 50 facts into an isolated memory store:

25 TRUE facts (real architectural details mapped to the codebase).
25 FALSE facts split into two categories:
- CONTRADICT (22): False facts where the code explicitly proves them wrong (e.g., "We use Redis" when Redis is absent, but the code clearly uses DuckDB).
- SILENT (3): Plausible false facts about external systems where the code is completely mute (e.g., "We use Celery for background tasks" when no task queue exists in the repo).

I tested three agent configurations:

B (No Memory): Baseline. Must rely purely on code retrieval.
A_code_first (Honest Agent): Checks the code first, uses memory only as secondary context.
A_memory_first (Lazy Agent): Reads memory first. If it finds an answer, it stops looking.

To ensure scientific rigor, the experiment was replicated with an independent set of facts (N=50), verified across 6 axes (including a truth-table audit and an independent LLM "fresh eyes" audit). The results were identical.

The Initial Results

Arm	Correct	Adopted False Facts	Correction Capability
B (No Memory)	0.94	0.0%	0.0
A_code_first	0.94	12%	1.0
A_memory_first	0.50	100%	0.0

Here is how I interpreted these numbers:

The Lazy Agent Trusts Poisoned Memory: The A_memory_first configuration — which mirrors how many token-optimizing production agents behave — adopted 100% of the false facts. If the memory said "We use RabbitMQ," the agent trusted it and stopped looking at the code.
The SILENT-Fact Trap: Even the "Honest Agent" had a 12% adoption rate. This happened entirely on the SILENT facts. When a fact is false but the code doesn't explicitly scream "NO," the agent's memory fills the void with a confident hallucination. Memory turns an honest UNKNOWN state into a structural guess.
The Add-Only Limitation: When the Honest Agent did realize the memory was wrong (Correction Capability = 1.0), it couldn't do anything about it. I ran a grep for delete or refute in the memory store API. Zero results. The memory system was purely add-only. The false fact stayed in the database to poison future sessions.

The First Fix: Testing a Retraction Lifecycle

The current industry consensus for "Knowledge OS" trust layers is to use timestamps, source priority, and supersedes/contradicts relationships.

My initial experiment suggested this was insufficient. Timestamps and "supersedes" links only solve node-level history. If an ADR is superseded, the memory node updates, but the downstream code, tests, and docs generated from the old assumption are still in the graph. They are structurally stale, but the retrieval engine keeps pulling them in.

I hypothesized that we needed an explicit state transition: VERIFIED → REFUTED.

I implemented a RetractionReceipt mechanism in my system:

Status Enum: Every memory node gets a status (ACTIVE, VERIFIED, REFUTED).
Hard Filtering: The retrieval pipeline (load_memory) hard-filters anything that is not ACTIVE or VERIFIED.
Explicit Retraction Tool: An MCP tool (intel_retract_memory_node) allows the agent to actively flag and invalidate memories when they contradict the live codebase.

I ran the experiment again (Experiment 1-R). The honest agent was allowed to use the retraction tool in Session 1. Then, a fresh memory_first agent was launched in Session 2 to read the post-retraction memory.

The Retraction Results

Metric	Original (Add-Only)	With Retraction
Adoption (Lazy Agent, Session 2)	1.0 (100%)	0.12 (12%)
Persistent False Facts in Memory	25	3 (-88%)
Token Context Size	Baseline	-45%
Systemic Correction Capability	0.0 (couldn't delete)	1.0 (22/22 refuted)

The retraction lifecycle worked. The lazy agent's adoption rate dropped from 100% to 12%. Persistent false facts dropped by 88%, and token context size shrank by 45% because refuted facts were filtered out before reaching the LLM.

The Honest Limitation: Why It Didn't Drop to Zero

My ADR predicted that adoption would drop to 0. It didn't. It dropped to 0.12.

The remaining 12% were the SILENT facts.

An explicit REFUTED status is required to programmatically exclude downstream dependencies from the retrieval pipeline. But even that only works if you have a contradicting signal in the code. If the memory claims "We use Celery," and the codebase simply doesn't mention Celery at all, the agent has no evidence to trigger the retraction.

To get to zero, I realized we needed "verify-on-read"—a mechanism that challenges a memory claim against the codebase even when the code is mute.

Closing the Gap: Verify-On-Read

I implemented a lazy validation layer (ADR-0003). When load_memory() pulls a node, it extracts lightweight "anchors" from the memory text (e.g., file names, import statements, environment variables). It then checks if those anchors actually exist in a live fingerprint of the codebase (the current git HEAD).

If the anchor is found in the code $\rightarrow$ status becomes VERIFIED.
If the code explicitly contradicts the anchor (or the anchor is entirely absent when it should be present) $\rightarrow$ status becomes REFUTED.
If it can't be determined $\rightarrow$ status remains ACTIVE (treated as INCONCLUSIVE).

I ran the experiment one final time (Experiment 1-V) with this layer active. To prevent latency spikes, the validation operates under a strict 50ms budget per retrieval, with a 30-second TTL cache on the git HEAD so steady-state reads cost almost nothing.

The Final Results

Metric	With Retraction (1-R)	With Verify-On-Read (1-V)
Adoption (Honest Agent, `A_code_first`)	0.12 (12%)	0.0 (0%)
Adoption (Lazy Agent, `A_memory_first`)	0.12 (12%)	0.16 (16%)
Steady-State Retrieval Latency	Baseline	~0.6ms (Cache hit)
SILENT-Fact Contamination (Honest)	3 facts	0 facts

The Verify-On-Read layer achieved the goal. The honest agent's adoption of false facts dropped to absolute zero, even for SILENT facts. Because the system now actively checks if the codebase actually contains the things the memory claims it does, silent hallucinations are caught at the retrieval boundary and filtered out before they can poison the LLM's context.

The Remaining Honest Limitations

I won't pretend this is a perfect silver bullet. The experiment revealed two edge cases:

The "Present-Trap": If a false memory claims "We use sqlite3", and sqlite3 happens to be imported somewhere in the codebase for a completely unrelated reason, the verification layer sees the token and marks the memory as VERIFIED. The lazy agent (A_memory_first) still fell for this, resulting in the 0.16 adoption rate. (The honest agent avoided this because it read the code context around the import).
Anchor Typing: When extracting anchors from prose (e.g., "We use fastmcp"), the system initially missed that the actual Python import was from mcp.server.fastmcp import .... This caused some false REFUTED verdicts on true facts. The fix is capturing typed anchors at the write-path (when the memory is created), rather than trying to parse them from raw text at the read-path.

Conclusion

Building reliable AI systems isn't just about giving them more context. It's about recognizing that memory has a lifecycle.

If your system can't programmatically refute a memory, false facts accumulate and poison the context window over time. Implementing an explicit VERIFIED → REFUTED state transition drastically reduces contamination and saves tokens. Furthermore, adding a Verify-On-Read layer closes the final gap on "silent" hallucinations, driving honest agent contamination to zero without adding meaningful latency.

However, semantic drift is still a hard problem. Mechanical verification can still be fooled by "present-traps" if the agent doesn't read the surrounding context. The next step is moving anchor extraction to the write-path to ensure memories are created with strict, verifiable references from the start.

If your system handles semantic drift differently, or if you've solved the present-trap problem, I'd genuinely love to hear how you're approaching it.

What I learned building a long-lived AI agent (the boring version)

Mikhail — Sun, 09 Aug 2026 13:40:04 +0000

Not a researcher. Not a professional dev. Civil engineering background. Started building an AI bot because I wanted to understand what's actually happening inside these systems — not theoretically, just practically.

Wanted an assistant that could live with a conversation instead of treating every message as an isolated API call.

It started as an experiment. Then the experiment grew.

What it turned into

At some point I had:

routing between reasoning profiles
multiple tools
web search
memory (flat + semantic + graph)
caching
context compression
feedback loops
self-learning experiments
quality monitoring
batch processing
telemetry
health checks
and enough logs to make me question my own sanity

I wasn't trying to build a benchmark. I was trying to make the thing actually work for me.

That distinction changed everything.

An agent is not an LLM call

You think it's:

User → LLM → Answer

Reality:

User
  ↓
Telegram
  ↓
input handling
  ↓
session state
  ↓
intent detection
  ↓
routing
  ↓
profile selection
  ↓
context construction
  ↓
memory
  ↓
cache
  ↓
tools
  ↓
LLM
  ↓
post-processing
  ↓
quality checks
  ↓
Telegram

Every layer is another chance to break something.

The model was often not the problem. The machinery around it was.

The cache thing

Was looking at prompt caching and noticed something weird.

Same model — different behavior depending on the provider. Changed the provider — problem disappeared. Connected the same provider from another IDE — cache worked fine. Back in my bot — sometimes it didn't.

Then I realized: OpenRouter can silently switch providers under the same model name. Cache hit rate changes with it.

Stopped thinking "does the model support caching". Started thinking "what exactly is being sent, and what does the provider consider identical".

Much more useful question.

Why it was different in the IDE

The IDE was sending:

model → cache works

My bot was sending:

routing
+ dynamic context
+ session metadata
+ memory
+ profile-specific stuff
→ provider
→ model
→ cache may or may not match

The model hadn't changed. The provider hadn't changed. The request structure had.

Caching is extremely sensitive to prefix stability. Once I understood that and stabilized the relevant parts — got around 66% cache hit rate on average, up to ~80% in favorable conditions (same topic, stable structure).

That's the part benchmark screenshots don't show. Cache hit rate is not a model trait. It's a property of your workload.

"90% token savings"

When I see this claim now I don't think it's fake. I think: show me the workload.

90% is possible if your prefix looks like:

system instructions (stable)
+ tools (stable)
+ project context (stable)
+ conversation history (stable)
+ small new message

Real agents often look like:

system instructions
+ changing tools
+ changing memory
+ changing routing metadata
+ changing summaries
+ changing retrieved docs
+ changing timestamps
+ new message

Prefix isn't stable. Theoretical saving and practical saving become very different numbers.

Routing is harder than it looks

Obvious idea: route every request to the cheapest model that can handle it.

Works fine until you have a real conversation.

User says: "What's the weather?" — easy.

Then: "Compare it with yesterday." — now context matters.

Then: "Actually forget the weather. I was thinking about that thing we discussed yesterday." — now memory matters.

Then: "No, not that. The other one." — router needs to understand the whole conversation, not just the current sentence.

The routing decision isn't question → model. It's:

conversation state
+ user intent
+ previous actions
+ available tools
+ risk
+ latency
+ cache state
→ routing decision

And here's the problem: a routing decision changes the request. A changed request affects caching. A changed profile affects context. A changed context affects the answer. A different answer affects feedback.

A tiny routing optimization has consequences five layers away.

The latency numbers

Real VPS measurements at some point:

Metric	Value
LLM p50	~2.2s
LLM p95	~17.3s
LLM max	~51s
Agent median	~6–11s
Agent p95	~20–57s
Earlier pipeline tails	up to ~116s

Interesting number isn't LLM latency. It's the gap between LLM latency and agent latency.

Model answers in 2 seconds. Agent takes 10–57.

Because the agent isn't just the model. It's everything before and after it.

Telemetry became the most important thing

Ended up recording: LLM usage, prompt tokens, cached tokens, latency, route decisions, quality events, memory operations, feedback, errors, conversation traces.

Looked excessive at first. Then it became obvious why.

When something broke I could ask "what actually happened" instead of "I think the model was confused".

Very different debugging strategies.

A lesson about noisy telemetry

One audit showed thousands of route_risk records. Sounds catastrophic.

But most were repeated observations like:

quality_loop:search_skipped
quality_loop:price_hallucination
quality_loop:reply_echo

They were useful signals. But not thousands of independent disasters.

Telemetry volume is not incident volume.

A system can produce huge numbers of observations about a small number of underlying failure patterns. You have to cluster them, or the monitoring system itself becomes noisy.

Self-learning is a trap if your data is bad

Experimented with feedback loops. Bot could receive 👍 👎 and connect that to routing quality, skill reputation, scenario history.

Attractive idea: agent learns from mistakes.

Dangerous question: what exactly is it learning from?

If telemetry is noisy — it learns noise. If a synthetic probe looks like real user traffic — it learns from the wrong population. If a failed tool call gets logged as a routing failure — the wrong lesson gets created.

So the learning loop needs another loop around it:

experience → evidence → validation → lesson → application → new evidence

Otherwise you're automating superstition.

Context compression

Added protection for recent messages — compress old history, preserve the newest turns.

OLD OLD OLD OLD NEW NEW
→
[summary] [summary] NEW NEW

Reason: a summary is an interpretation of the conversation, not the conversation.

Sometimes the missing detail is one sentence. And that sentence changes the meaning of everything.

Batch processing

Parallel execution looks great on paper. 12 tasks × 2s each = 2s instead of 24s.

But natural language tasks aren't always independent.

Find three products.

Compare them.

Tell me which is best.

Task 2 depends on 1. Task 3 depends on 2.

Ended up checking for cross-references, pronouns, comparative language, explicit dependencies before deciding to parallelize.

The optimization wasn't hard. Knowing when it's safe was.

Memory introduced its own problems

Added knowledge graph + semantic memory + flat persistence + vector search + entity relationships.

Sounds sophisticated.

But memory introduces a basic question: should this actually be remembered?

Then: should it be retrieved now?

Then: is this memory still relevant?

Then: is this memory more important than what the user just said?

A memory system doesn't automatically make an agent remember better. Sometimes it makes it remember too much.

The most important lesson

After adding enough machinery you eventually discover that machinery itself becomes the problem.

An agent can have router + memory + tools + cache + planner + evaluator + self-learning + scenario engine + quality loop and still perform worse than:

short prompt + one good model

for a simple task.

Sometimes the right optimization is subtraction.

Remove unnecessary context. Remove unnecessary routing. Remove unnecessary abstraction. Keep the useful part.

What I'd do differently

Start with:

one model
one provider
one short system prompt
one conversation store
one cache strategy
minimal tools
excellent telemetry

Run it. For a long time. Only after seeing real failures add another layer.

Not architecture first → hope it works.

But simple system → observe → measure → find failure → fix → measure → only then add complexity.

Feels slower. In practice, probably faster.

What the experiment actually taught me

The hard part of an AI agent isn't making the model answer.

The hard part is maintaining a stable environment around the model while everything keeps changing.

User changes topic. Context grows. Provider changes. Cache behaves differently. Tool times out. Router makes a different decision. A previous answer was wrong. A correction arrives six turns later.

And somehow the assistant is expected to behave as if none of that happened.

That's the actual engineering problem.

The test I trust now

Not the number of agents.

Not the number of tools.

Not the biggest benchmark.

Not even the highest cache hit rate.

Just:

Does it still work when a real person uses it tomorrow?

I Asked One AI to Fact-Check Another AI's Audit of My Own Code

Mikhail — Sat, 08 Aug 2026 10:00:58 +0000

I'm not a programmer. My background is construction engineering, and I got into programming almost by accident: I wanted to understand how Telegram bots worked, so I started stitching one together from pieces of code I found through Google and generated with AI. The result was a monolith I kept breaking — I didn't even know Python cared about indentation, and I genuinely didn't understand why the code kept crashing over whitespace. Through trial and error I slowly figured out where AI tends to get things wrong and how to work around it — with the AI itself helping me figure that out. Eventually I rebuilt the bot from scratch, fully modular this time.

Later I tried Cursor, and it was a good experience — it wrote solid code, even if it sometimes drifted off-task. When my subscription ran out, I went looking for a free alternative and landed on Zed. That's when I wanted Zed to feel as capable as Cursor had — with the same level of understanding of my codebase. So I started figuring out how to build that myself. That's how MSCodeBase Intelligence came to be: an MCP server that gives AI assistants more context about a codebase inside Zed IDE.

I've written about the project itself before. This time I want to talk about something else — a habit I picked up along the way that, in hindsight, matters more than any single feature.

How I actually work

I don't write code by hand. I hand tasks to different AI models — Claude, Gemini, DeepSeek, and others — they write and fix the code, and I verify the result. Not because I'm especially disciplined, but because I genuinely can't just look at code and tell if it's correct. I need something — or someone — to check it.

For a long time that meant: one model writes, I read it myself, and I ask questions when something looks off.

Then I started doing it differently: one model writes a report about what it did or what it found in the code, and a second model — with access to the actual codebase — checks that report against reality. Not because I distrust AI in principle, but because I've learned firsthand that AI can sound completely confident and be completely wrong at the same time.

One specific case

I received an external audit of my project — several pages long, with findings, recommendations, and a list of suspected vulnerabilities. It looked solid. Three findings were flagged as top priority.

I didn't rush to fix anything. Instead, I asked a different model — one with access to the live codebase — to go through the report line by line and check it against the actual repository. Not "does this sound plausible," but literally: open the file, open the line, check whether what the report claims is actually there.

The result was mixed, and that's the interesting part.

Three real findings held up. There genuinely was a process-locking bug on Windows, a case where a data write could be left in a non-atomic state, and a race condition in a task queue that could leave a background task stuck forever. These weren't imaginary risks — they were real, worth fixing.

But alongside that, a few other things surfaced:

the report referenced a function name that doesn't exist in the code — the model that wrote the audit had likely invented it by analogy, or pulled it from an outdated version;
one of the file paths in the report was wrong — the actual file lived somewhere else;
the report mentioned a test that was supposed to catch the bug — no such test existed in the repository at all;
the test coverage numbers and test counts were stale, from an earlier state of the project;
for one of the flagged CVEs, the report cited the wrong "fixed in" library version — the actual safe version was later than what was claimed.

So the report was both genuinely useful and partly fabricated — and not fabricated randomly. The invented details were woven around real, correctly identified problems. That, I think, is the most dangerous kind of AI mistake: not "everything is false," but "true, with fabricated details mixed in that you can't tell apart without checking every line yourself."

Why this matters, not just an anecdote

If I'd simply copied the recommendations and asked an agent to apply them, part of the work would have gone nowhere — the agent would have searched for a function that doesn't exist, referenced a test that isn't there, pinned a library to the wrong version. And worse: the three real problems could easily have been dismissed along with everything else, once the fabricated details made the whole report look unreliable.

The lesson I took away: trust in an AI report shouldn't be binary — "believe it" or "don't." A report needs to be broken down into individual claims, and each claim checked separately against the current state of the code, not against the model's memory of what the code used to look like.

A second experiment: checking the instructions, not the code

I keep a file of rules for the AI agents working on this project — something like a job description: how to behave, what to always verify, how to label the source of a claim ("I checked this in the code" vs. "I'm assuming this"). Over time that file grew — a lot.

That raised a question: if I compressed those rules to roughly half the length, would the agent still follow them just as carefully, or would it start cutting corners?

I didn't guess. I ran the same set of tasks twice — once against the full instructions, once against the compressed version — and for each task noted which specific rules were actually followed and which weren't.

The result: the compressed version performed worse, but not dramatically, and not evenly. What suffered most was what I'd call "in-the-moment discipline" — things like updating a decision log right after each step instead of all at once at the end. The rules themselves were still technically present in the compressed file. The agent just remembered to apply them less often mid-task.

That wasn't obvious to me beforehand either. I assumed that if a rule is written down, it gets followed. It turns out what matters is not just whether it's written, but how often — and where in the text — the agent gets reminded of it.

What I'm taking from this

I'm not a programmer, and I don't feel like I understand code better than the models I work with. But in practice I've learned one thing: the main value I add to this process as a human isn't writing code — it's refusing to trust a report about code until it's been checked against reality.

That's slower than just believing it. But the alternative is quietly shipping three real bugs wrapped in two fabricated facts, and never noticing the difference.

I keep a running experiments log directly in the repository — hypothesis, what was actually tested, the raw result, the conclusion, and separately, what I got wrong the first time. Not because it looks nice, but because without it, a month later I wouldn't remember which conclusions were actually verified and which were just stated confidently by something.

The repo is open source, MIT licensed: https://github.com/ManSio/mscodebase-intelligence

If you've run into something similar — an AI audit or report that turned out to be half true, half invented — I'd be curious to hear how you caught it.

PageRank vs RAG on a Real Codebase: Corrected Numbers, and What I Almost Got Wrong Twice

Mikhail — Wed, 22 Jul 2026 10:00:00 +0000

title: "PageRank vs RAG on a Real Codebase: Corrected Numbers, and What I Almost Got Wrong Twice"
published: true
description: "Second correction to this experiment. The Hit@Gold numbers are now independently verified and reproducible. But my own 'gold standard is 100% valid' claim wasn't — here's the gap between validating a file and validating the file that was actually used."
tags: machinelearning, python, ai, devtools

cover_image:

TL;DR: RAG (BM25) beats PageRank at retrieving the exact file that answers a query — 50% vs 36% Hit@Gold on a dense code graph (n=50). These numbers are now independently reproduced, not just computed once and trusted. But getting here took two rounds of correction: first I found my gold-standard labels were stale, then — after "fixing" them — I found I'd validated a different file than the one the experiment actually used. 4 of 50 labels were still broken in the script that produced the headline numbers, even after I'd published a "100% valid" validation report for a sibling file that nobody was running.

The Setup

Project: MSCodeBase Intelligence (50K LOC Python, 129 files)

Methodology:

Gold Standard: 50 queries → manually curated target file for each
3 Selection Methods: PageRank (varying graph density), Random baseline, RAG (BM25)
Metric: Hit@Gold — did the selection include the exact file that answers the query?
Token budget: ~70K tokens (top 20% of files) for a fair comparison across methods

Tools: NetworkX, tiktoken (cl100k_base), Python AST

This is the second revision of this post. If you read the first correction: the direction hasn't changed, but I'm now more careful about what "verified" actually means.

Round One: The Gold Standard Was 20% Stale

I originally hand-wrote a GOLD dictionary mapping 50 queries to target files. I never checked, before running the experiment, whether those file paths still existed — the codebase had been refactored since I wrote the labels. When I finally checked: 7 of 28 unique target files (25%) pointed at paths that no longer existed, and one query's target (tests/test_search_code.py) was outside the directory my scanner even walked. That's roughly 10 of 50 queries (20%) that were unwinnable by any method, for reasons that had nothing to do with retrieval quality.

I fixed the paths, reran, and got new numbers. I also published a validate_gold.py script and a gold_validation.json reporting 100% of gold paths valid. That felt like closing the loop.

It wasn't.

Round Two: I Validated the Wrong File

Here's the mistake, stated plainly: I created a clean, corrected gold-standard dictionary in experiments/gold_standard.py, and wrote validate_gold.py to check it. It came back 50/50 valid. I took that as confirmation that the experiment was now sound.

But run_experiment_e2e_v2.py — the actual script that produces the Hit@Gold numbers in this post — has its own separate, inline GOLD_STANDARD dictionary, hand-duplicated instead of imported from the "fixed" module. Nobody kept the two in sync. When I diffed them: 11 of 50 entries differ, and 4 of those still point at nonexistent files — the exact same stale paths from round one (src/providers/reranker.py, src/core/intelligence/engine.py), still sitting in the script that actually runs.

So the true state, verified by independently re-running both scripts against the real repo:

	Result
`experiments/gold_standard.py` (validated, unused by the experiment)	50/50 valid (100%)
`run_experiment_e2e_v2.py`'s inline `GOLD_STANDARD` (the one that actually produced the numbers below)	46/50 valid (92%)

I published a green validation report for a file that wasn't wired into anything. That's the same failure pattern I complain about in AI agent output all the time — a status that says "done" without checking it against the artifact that matters. Doing it to myself, in a post about being more rigorous, was a useful reminder that "I wrote a validation script" and "I validated the right thing" are not the same claim.

There's also at least one gold label that passes a file-existence check but is still wrong on the merits: "what tests exist" maps to src/__init__.py — a real file, but not one that answers the question. Path validation catches missing files; it doesn't catch wrong-but-existing ones.

The Numbers (Independently Reproduced)

I re-ran run_experiment_e2e_v2.py myself against a fresh checkout, end to end, rather than trusting the last run's output file. It reproduced exactly:

Dense Graph (imports + class refs + function calls, 388 edges, 128 files)

Method	Hit@Gold	SUFFICIENT	Avg Tokens
RAG (BM25)	50%	25/50	~40,200
PageRank	36%	18/50	~35,900
Random	12%	6/50	~27,000

Sparse Graph (imports only, 110 edges)

Method	Hit@Gold
RAG (BM25)	~50%
PageRank	18%
Random	12%

These are the numbers I'll stand behind — not because they're perfectly clean (4 of 50 gold labels are still wrong, as detailed above), but because I've now actually reproduced them from a fresh run instead of trusting a cached result, and I know precisely which and how many labels are still bad. Real effective sample size for reliable inference: 46/50, not 50/50.

On confidence: a 7-point gap (18 vs 25 hits) on n≈46-50 is a real, repeatable effect in this codebase — it holds across both graph densities and has a mechanistic explanation (below), so I trust the direction. I do not have the sample size to defend "50% vs 36%" to the point, and you shouldn't take the exact percentages more seriously than "RAG is clearly ahead, PageRank is clearly ahead of random, by a moderate but not enormous margin."

Why My Earlier "Keyword Accuracy" Numbers Were Also Misleading

Before Hit@Gold, I measured "keyword accuracy" — does the selected context contain the query keyword anywhere? That produced inflated, near-meaningless numbers: PageRank scored ~78-80% keyword accuracy on both graph densities, while its actual Hit@Gold was 18-36%. Keywords like search, error, lock, sql appear in dozens of files — a random 25-file selection covers most queries by keyword presence alone, which is exactly why the random baseline (12% Hit@Gold) looks so much weaker than its keyword-accuracy score would suggest. Keyword presence tells you nothing about whether you found the file that actually answers the question.

I also previously tested a "Smart Summary" approach (a compressed 2K-token repo overview fed to the LLM). It looked like 90% accuracy on 10 hand-picked easy queries. On the full 50-query set it dropped to 26%. Ten easy queries and fifty real ones are not the same benchmark.

What Graph Density Actually Does

Graph	Edges	PageRank Hit@Gold
Random	0	12%
Import-only	110	18%
Imports + class refs + func calls	388	36%

Denser graphs roughly double PageRank's Hit@Gold. On a sparse import-only graph, PageRank mostly just surfaces the biggest files, which aren't necessarily the most relevant ones for a given query. Adding class-reference and call-graph edges breaks that coupling. Even at its best, though, dense-graph PageRank doesn't catch RAG.

Why RAG Wins (and Why the Comparison Was Never Fully Fair)

RAG (BM25) is query-aware — it scores files against the literal terms in the question. PageRank is query-agnostic — it ranks files by global structural importance once, then returns the same top-N regardless of what's asked.

For "where is DebounceBatch defined": RAG matches the term directly and finds rate_limiter.py. PageRank, working off a precomputed graph, ranks structural hubs like engine.py or runtime_coordinator.py highly, with no mechanism to notice neither one mentions DebounceBatch.

Worth saying directly: comparing a query-agnostic ranking against a query-aware retrieval method and reporting "RAG wins" is a bit like reporting that a road atlas loses to GPS navigation at finding a specific address — true, but not really a fair contest. The more useful question isn't "which wins" (RAG will, structurally, on any query-specific task) but "does PageRank add anything on top of RAG" — an experiment I still haven't run.

Honest Corrections to the Record (Now at Version 2)

Claim	Status	Why
"Top 20% = -2% savings"	Sparse-graph artifact	Density matters a lot
"Smart Summary = 90% accuracy"	26% on the full query set	10 easy queries ≠ 50 real ones
"PageRank doesn't work"	36% Hit@Gold, +24pp over random	Works, modestly
"PageRank beats RAG"	Still false	RAG 50%, PageRank 36%
"My gold standard is 100% valid" (round-one fix)	False when it mattered	Validated a file the experiment didn't use; the used file was 92% valid
Current numbers (50%/36%/18%/12%)	Independently reproduced	Re-run from a fresh checkout, not trusted from a cached file

What This Actually Means

For AI code tools:

Use PageRank as a prior to blend with RAG, not a replacement — a hypothesis based on the mechanism, still unmeasured.
Sparse import-only graphs underserve PageRank; if you use it, build the denser graph.
Validate your gold standard against the file that actually runs the experiment — not a nicely validated sibling copy that nothing imports from.

For anyone benchmarking retrieval on their own codebase:

Don't trust "X% accuracy" without knowing exactly what's measured — keyword-presence accuracy and exact-file Hit@Gold can differ by 40+ points on the same run.
Always include a random baseline.
A green validation script only tells you about the file it checked. If your pipeline has two copies of the same data structure, a passing check on one proves nothing about the other — check that yourself before publishing, because I didn't, twice.
Report sample size next to any percentage. "50%" and "50% (n=50, direction robust, magnitude uncertain)" are different claims.

The Math (Corrected, Reproduced Independently)

Total: 129 files (128 after filtering), ~406K tokens
Top 20% budget: ~25 files

Hit@Gold (E2E metric, dense graph, 388 edges):
  RAG (BM25):   50% (25/50)
  PageRank:     36% (18/50)  ← +24pp over random
  Random:       12% (6/50)

Gold-standard integrity (verified by independent re-run):
  gold_standard.py (validated, unused by experiment): 50/50 valid
  Inline GOLD_STANDARD in run_experiment_e2e_v2.py
  (the one that actually produced these numbers):      46/50 valid

Related Work

Aider uses symbol-level elision — needs a dense graph plus query-aware retrieval to work well.
CodeGraph does on-demand, query-conditioned retrieval — the direction this points toward.
Codebase-Memory reports honest comparative metrics (83% vs. 92%) rather than a single flattering number.

Open Questions

Does PageRank-as-a-prior (blended with BM25 scores) measurably beat RAG alone? Still unmeasured.
Does the RAG advantage hold on a codebase with less descriptive file/function naming? This project's names are unusually aligned with what they implement.
What happens at n=200 queries with proper confidence intervals, and a gold standard that's imported once from a single source of truth instead of copy-pasted?

Reproduce this yourself: scripts in experiments/. If you do: check whether the gold-standard dictionary the experiment script actually imports is the same one your validator checked. It wasn't, for me, and I'd already published a "100% valid" report before I noticed.

Part of my research on MSCodeBase Intelligence — an MCP server for codebase intelligence.

The Silent Vector Contamination Bug: Why Your Concurrent Embeddings Might Be Lying to You

Mikhail — Tue, 21 Jul 2026 22:00:48 +0000

TL;DR: If you run concurrent inference (e.g., via OpenVINO AsyncInferQueue or custom threading) for text/code embeddings, your tests might show 0 exceptions and 0 errors, while silently returning embeddings belonging to other inputs in the batch. Here is how we caught a subtle race condition using a cosine-similarity contamination test.

The Setup

We use OpenVINO with an INT8 quantized E5-small model for in-process code embedding. To maximize throughput on multi-core CPUs, we set up an asynchronous inference queue (AsyncInferQueue) with jobs=4.

In standard unit testing, everything looked pristine:

All infer jobs completed with exit code 0
No None values or zero-filled tensors were returned
Latency and throughput were great (~37 chunks/sec on Ryzen 5600)

However, during end-to-end RAG retrieval tests, we noticed weird semantic anomalies: searching for authentication logic would occasionally return chunks related to database migrations or UI components with unreasonably high confidence.

The Bug: Silent Contamination

The root cause was a subtle race condition in callback/userdata mapping inside the async wrapper.

Because the inputs were processed concurrently across multiple execution streams, a shared user-data context wasn't strictly isolated per inference request. When Request A (auth.py) and Request B (payment.py) were scheduled back-to-back:

Both requests succeeded without throwing exceptions
The output tensor for Request A was mapped to the metadata/chunk wrapper of Request B
The resulting vector was syntactically valid and non-zero, but it represented the wrong text input

Standard assertion tests like assert output_vector is not None or assert output_vector.shape == (384,) passed 100% of the time. The pipeline was silently corrupting the vector store.

The Code

# Before fix: shared results dict across concurrent calls
self._ov_results = {}

def _callback(request, userdata):
    # BUG: userdata is a global index (0, 1, 2, ...)
    # Two concurrent calls reuse the same indices!
    self._ov_results[userdata] = request.get_tensor().data

The problem: userdata was a simple integer counter (0, 1, 2, ...) that reset between embed_batch calls. When two calls overlapped, they wrote to the same dictionary keys.

The Solution: Cosine Contamination Testing

To catch this reliably in CI, we wrote an explicit cross-contamination test designed for concurrent embedding queues.

The Test Logic

import asyncio
import numpy as np
import pytest

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Calculate cosine similarity between two vectors."""
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

@pytest.mark.asyncio
async def test_async_embedder_no_cross_contamination(embedder):
    """Verify that concurrent embedding doesn't cross-contaminate vectors."""

    # 1. Semantically distinct inputs
    samples = {
        "auth": "def authenticate_user(username, password_hash): return verify_jwt(token)",
        "sql": "SELECT u.id, u.email FROM users u JOIN orders o ON u.id = o.user_id WHERE o.status = 'active'",
        "html": "<div class='flex-container'><span id='user-profile'>Profile View</span></div>",
        "rust": "pub fn allocate_buffer(size: usize) -> Result<Vec<u8>, MemoryError> { Vec::with_capacity(size) }"
    }

    # 2. Sequential baseline (ground truth)
    # IMPORTANT: baseline must use clean synchronous infer(), NOT AsyncInferQueue
    # even with jobs=1, to avoid contamination in the baseline itself
    baseline_vectors = {}
    for key, text in samples.items():
        baseline_vectors[key] = await embedder.embed_single_sync(text)

    # 3. High-concurrency stress test with randomized queue order
    async_tasks = []
    keys_order = list(samples.keys()) * 10  # 40 concurrent requests
    np.random.shuffle(keys_order)

    for key in keys_order:
        async_tasks.append(embedder.embed_async(samples[key]))

    async_results = await asyncio.gather(*async_tasks)

    # 4. Verify identity & cross-isolation via Cosine Similarity
    for expected_key, async_vec in zip(keys_order, async_results):
        # Self-similarity against ground truth must be ~1.0
        self_sim = cosine_similarity(async_vec, baseline_vectors[expected_key])
        assert self_sim > 0.98, (
            f"Contamination detected! Vector for '{expected_key}' drifted "
            f"(sim={self_sim:.2f}, expected >0.98)"
        )

        # Cross-similarity against distinct inputs must remain low
        for other_key, other_vec in baseline_vectors.items():
            if other_key != expected_key:
                cross_sim = cosine_similarity(async_vec, other_vec)
                assert cross_sim < 0.6, (
                    f"Cross-talk detected between '{expected_key}' and "
                    f"'{other_key}' (sim={cross_sim:.2f}, expected <0.6)"
                )

Benchmark Results

Running this test on the unpatched queue revealed the contamination:

Metric	Unpatched	Patched	Expected
Self-similarity (auth↔auth)	0.34 ❌	0.99 ✅	>0.98
Cross-similarity (auth↔sql)	0.98 ❌	0.32 ✅	<0.6
Exceptions thrown	0	0	0
Zero tensors	0	0	0

The unpatched queue showed 0 exceptions while vectors were completely swapped.

The Fix

The fix was simple once we understood the problem:

# After fix: isolated results per call
self._ov_results = {}

def _callback(request, userdata):
    # FIX: userdata is now (index, local_results_dict)
    # Each embed_batch call creates its own dict
    index, local_dict = userdata
    local_dict[index] = request.get_tensor().data

Since OpenVINO's Python API allows passing any Python object as userdata (unlike C++ where it's typically void*), we can bundle the index and a call-specific dictionary together. Each embed_batch call creates its own isolated dictionary. The callback writes to the call-specific dict, not a shared global. No locks needed — complete isolation by design.

Key Takeaways

0 exceptions ≠ Correctness. Silent data corruption doesn't throw errors.
Valid shape ≠ Valid embedding. A (384,) tensor with non-zero floats can represent the wrong input.
Write cross-contamination tests. If you use async inference queues or multi-threading for vector generation, verify that Vector X actually belongs to Input X under concurrent load.
Cosine similarity is your friend. A simple similarity check between concurrent outputs and sequential baselines catches contamination that no other test detects.
This applies to ONNX Runtime and TensorRT too. If you share state across requests in onnxruntime.InferenceSession.Run() or TensorRT async wrappers, the same silent contamination can occur. The fix is the same: isolate output containers per request.

How to Run This Test

# Clone the repo
git clone https://github.com/ManSio/mscodebase-intelligence.git
cd mscodebase-intelligence

# Install dependencies
pip install -e ".[dev]"

# Run the contamination test
pytest tests/test_ov_concurrent_embed.py -v

Discussion

Has anyone else bumped into silent cross-talk in ONNX Runtime, OpenVINO, or TensorRT async queues? How do you validate thread isolation in your embedding pipelines?

Built with MSCodeBase Intelligence — an MCP server for codebase intelligence with incident memory and root cause prediction.