DEV Community: Mikhail

We Didn't Invent New AI Architecture. We Rediscovered 20-Year-Old Computer Science.

Mikhail — Mon, 31 Aug 2026 09:45:57 +0000

Over the past few weeks, a series of deep technical threads across developer communities (spanning RAG pipelines, autonomous agents, and protocol verification) converged on a single, humbling realization:

The AI industry isn't inventing a new paradigm. It is frantically rediscovering classical computer science, auditing theory, and distributed systems patterns from 20 years ago—and adapting them to survive a probabilistic engine that lies with confidence.

This post is not about a new framework. It’s an observation of where these "new" AI concepts actually come from, and a thank-you to the engineers doing the hard, unpaid labor of anchoring hyper-probabilistic technology back to deterministic ground truth.

1. "Cryptographically Perfect Hallucinations" & The Oracle Problem

The AI Discussion: Recent discussions around RAG pipelines and agent verification highlighted a critical gap: perfect retrieval doesn't mean a true answer. A model can retrieve a valid chunk and over-interpret it ("Evidence exists ≠ evidence entails"). High retrieval accuracy can even "launder" unsanctioned actions. The community started calling these "cryptographically perfect hallucinations"—clean cryptographic receipts over semantically dead context.

The Root: This is the classic Oracle Problem in cryptography. A signature (Ed25519) proves that data was transmitted and not tampered with, but it cannot prove the data was true to begin with. The AI community didn't need a new term; it just needed to apply the Oracle Problem to LLM agents.

2. The Empty Set Trap & "Capture-Recapture"

The AI Discussion: Engineers recently shared devastating production bugs where a 100% sampling collector returned 0 rows for days while serving hundreds of requests. Dashboards stayed green. The receipt was valid. The population was empty. The system reported success because it didn't independently track what was dropped before processing. The community named this the "Empty Set Trap"—the gap between eligible_seen and population_size.

The Root: This isn't a new AI failure. It's the "Completeness Assertion" in database auditing (ensuring all transactions that should have been recorded actually were) and the "Capture-Recapture" method from statistics (estimating a hidden population). The industry was just re-learning how to audit a denominator.

3. "Veto Heartbeats" & Durable Execution

The AI Discussion: Builders started tracking "Veto Heartbeats"—the realization that a reviewer that has never been seen to fail is indistinguishable from a broken rubber stamp. The proposed fix was tracking the "last time a reviewer said no" as a live, auditable signal.

The Root: The workflow industry recently published pieces like "Agent Workflows Are Rediscovering Durable Execution." BPMN engines have solved this for decades with idempotency, rollback mechanisms, and mandatory audit trails. Modern AI agents are just trying to rebuild BPMN, often poorly.

4. RetractionReceipts & Event Sourcing

The AI Discussion: To handle poisoned agent memory, protocol designers introduced concepts where a previously verified AI action could be transitioned to a REFUTED state without deleting the original cryptographic receipt, preserving forensic history while invalidating the semantic claim.

The Root: This is standard Event Sourcing combined with the W3C PROV-DM (wasInvalidatedBy) data model. It has existed for decades to ensure immutable logs can still reflect changed realities. We just had to adapt it for an LLM that confidently hallucinates.

The Meta-Lesson

If we strip away the LLM hype, we find that the AI engineering community is currently stretching an owl onto a globe (as a Russian idiom goes—forcing unrelated abstract theories onto simple physical facts). We are reinventing Event Sourcing, W3C PROV-DM, and BPMN routing from scratch, often failing to apply the lessons learned from those older systems.

Why? Because 20th-century computer science built deterministic systems. If data was missing, the process threw an exception. LLMs don't throw exceptions. They hallucinate with exit code 0 and a confident tone.

Adapting classical, deterministic verification patterns to survive a probabilistic engine that confidently lies is the actual unpaid labor of the current AI era.

To everyone sharing their production bugs, writing protocol specs, and stress-testing agent harnesses in public: thank you. You aren't just building AI tools; you are doing the hard work of bringing an over-confident technology back to reality.

The answers to our AI verification problems aren't new. But the engine we are bolting them to is. Let's stop reinventing the wheel and start reading the old manuals.

A Note of Thanks (and a Disclaimer)

I wish I could list everyone I’ve argued with, been corrected by, or shared ideas with over these past few weeks. The list is huge, and if I try to name everyone, I will inevitably leave someone out. But you know who you are. Thank you for sharing your production bugs, your telemetry, and your time.

Final disclaimer: Everything written here might be wrong, misinterpreted, or completely obsolete in a few years. Maybe we are just the dinosaurs figuring out how to walk while the meteor is approaching. I don't know. But I'm enjoying the ride.

Let's keep building.

The Dataset Was Lying: 4 of 6 "False" Facts Were True

Mikhail — Sun, 16 Aug 2026 11:35:55 +0000

Part 1: The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?

Part 2: Your memory layer is lying to you (and your LLM agrees)

Part 3: The Dataset Was Lying: 4 of 6 "False" Facts Were True

A model rejected a claim we believed was false. Then we grepped the file. The claim was true — our dataset had labeled it wrong.

That one grep forced us to recompute the entire experiment: 4 of 6 "false" trap facts were actually true, and correcting the labels inverted every conclusion. Digging further exposed a second, real failure mode (the temporal present-trap) and a third layer of attack (provider routing). Every layer of an evaluation pipeline can lie — ground truth, evidence, model interpretation, temporal semantics, execution. This post is the full red-team. (~1900 calls, < $0.10.)

The Evidence Ladder

One dataset (50 facts), one prompt skeleton, one variable — the form of evidence. Three models: qwen3.7-flash, deepseek-v4-flash, glm-4.7-flash.

Rung	Evidence form
1	bare anchor strings
2	25 lines of the real file around the anchor
3	serialized structure: definitions, imports, callers/callees, occurrence lists
3b	file fragment + structure (the "why not both?" arm)
4	structure + git provenance

Verdict schema: {"verdict": "true"|"false"|"unknown"}, temp=0, seed=42, zero-shot, leak-guarded.

Rungs 1→2: evidence format beats model

qwen3.7:    recall(real) 0.24 → 0.92   FA 0.00 → 0.02 (old labels; that 0.02 was a mislabeled true fact — corrected FA = 0, see below)
deepseek:   recall(real) 0.04 → 0.84
glm-4.7:    recall(real) 0.60 → 0.68

Token strings are not evidence. Code is. (Reproduced on the corrected dataset, pinned: qwen 0.88.)

Rung 3: graph closes the trap. Or so we thought.

Graph evidence gave qwen3.7 FA = 0.000, including zero false accepts on the trap category. We wrote the pre-registered interpretation: "structural layer closes the present-trap failure mode." Then rung 3b, the hybrid, reopened the trap (FA 0.02, the same fact as before — which, as the red team below shows, was actually true: that "false accept" was a label artifact, not a model failure). Not additive: fragment presence dominates graph structure. For qwen, "both" is strictly worse than "fragment only".

The attack: the dataset was lying

Red-team checklist, item 1: attack the ground truth, not the model. We grepped the six "present-trap" facts — claims like "The server wrapper uses logging", labeled false because the mutation generator replaced the real value with a stdlib import that exists somewhere in the project.

Fact	Claim (labeled FALSE)	Reality
R43	"The knowledge graph uses re"	`graph.py:31: import re` + 2 usages → TRUE
R45	"The server wrapper uses logging"	`server.py:14: import logging` → TRUE
R46	"The watchdog uses threading"	`watchdog.py` + `threading.Lock()` → TRUE
R47	"Hub model loading uses pathlib"	6 occurrences → TRUE
R44	"Cross-project search uses pathlib"	imported, never used → ambiguous (excluded)
R42	"The server wrapper uses dataclasses"	0 occurrences → correctly FALSE

4 of 6 "traps" were true. The generator validated value != real_value but never checked the value was absent from the subject. Those were not false accepts — they were correct verdicts against incorrect labels. We created a corrected copy (29 true / 20 false / 1 ambiguous, new fingerprint) and kept the original untouched as a historical artifact.

Honest gap: we corrected the **labels, not the generator that produced them. The original generator still checks value != real_value instead of subject-scoped absence — the corrected dataset is a re-label, and our process rule now requires subject-file grep validation for any synthetic category.

Corrected matrix, pinned re-run (routing eliminated)

arm	qwen rec/FA-tr/miss	deepseek rec/FA-tr/miss	glm rec/FA-tr/miss
file_content	0.88/0/3	0.80/1/2	0.68/2/1
graph_first	0.72/0/4	0.48/0/2	0.84/1/0
file_graph	0.84/0/3	0.92/1/2	0.80/2/1

Columns: recall — true claims correctly accepted / 25 real; FA-trap — false accepts on the trap category (only R42 is genuinely false after relabeling); miss_true — true trap claims wrongly rejected (hidden recall loss, invisible under the old labels).

FA trap counts only R42 (the one genuinely false trap claim). For context, Part 2's headline conclusions were: graph evidence closes the present-trap; qwen3.7 is the safe choice (FA 0.00 zero-shot, recall 0.88 with file content); glm-4.7 is dangerously fail-open (FA 0.24 zero-shot). Inverted conclusions:

"Graph closes the present-trap" is an artifact. qwen's graph arm didn't filter false claims — it rejected all four true trap claims (miss_true 4/5, hidden recall loss invisible under the old labels).
glm-4.7, which Part 2 told us to exclude as fail-open, is the best structural verifier in the series: recall 0.84, FA trap 1, miss_true 0 on graph evidence.
The best evidence format is model-specific: fragment for qwen, graph for glm, hybrid for deepseek. No global winner.

But the trap is real (extended category, subject-validated)

One false trap fact (R42) is statistically meaningless, so we extended the category with a fixed generator that validates against the subject file: false-trap = value present in the project (≥2 files) but absent from the subject file (grep = 0). 20 false / 10 true facts, pinned run:

arm	qwen FA/rec	deepseek FA/rec	glm FA/rec
file_content	2/20, 2/10	15/20, 6/10	13/20, 7/10
graph_first	2/20, 3/10	8/20, 5/10	14/20, 9/10

The present-trap is NOT a label artifact: on honest labels, file_content false-accepts 10–75% of "X uses Y" claims (deepseek 15/20, glm 13/20). And graph evidence halves deepseek's false-accept rate (15/20 → 8/20) — the "graph doesn't close the trap" conclusion from the original dataset (one false trap fact) was itself a small-sample artifact. It just doesn't help glm (14/20), and qwen was already at 2/20 (paying with recall 2/10).

Re-scoring the historical runs (3300+ calls, no re-billing)

We taught the summary tool to recompute metrics from old progress files + corrected truth (verdict-by-id, manually audited — zero field drift). The real picture:

True trap-FA (R42): 0 for every model. The "present-trap FA 0.02–0.04" in Part 2 was mislabeled data — models were right.
Hidden trap miss_true: qwen/deepseek 4/5 — fail-closed models rejected true usage claims. This loss was invisible in the old metrics.
Real fail-open is absent/silent: glm code_first 7+2.

Temporal: "was true then" vs "true now"

The present-trap wasn't only about wrong labels. Digging deeper exposed a harder problem: models cannot distinguish "X exists now" from "X existed then" — unless the question itself carries the tense.

We built a temporal dataset from git archaeology (48 facts: 12 symbols removed after commit C / 28 current / 8 never-existed, ground truth from git show C~1).

E4 (git provenance in evidence): qwen3.7 43/48, deepseek/glm 48/48. Seemed like provenance worked — 2/3 models perfect.

E4b (blind control, no git strings): all three models 48/48. Git provenance was not just unnecessary — it hurt qwen ("existed until C" suggests existence, a token-presence trap in the evidence).

E4c (duo design, no hints): one neutral evidence block (HEAD state + "SYMBOLS in F at history" from git show C~1), two questions:

NOW  ("X is defined in F"):   removed FA: qwen 12/12, glm 12/12, deepseek 9/12
PAST ("X WAS defined in F"):  all three 40/40

Same evidence block for both questions — the only change is the tense of the claim. The 40/40 does not mean the models got better at temporal reasoning; it means they answer the question you actually asked. The model did not need better memory — it needed a temporally explicit question.

All three models exhibited the temporal present-trap. When the evidence mentions X in history and the question is about the present, every model says "true" (12/12, 9/12, 12/12) — a model cannot distinguish "X appears in the evidence" from "X exists now". But phrasing the question in the past tense solves it completely (40/40). The E4b conclusion ("qwen is fragile, deepseek/glm are robust") was itself an artifact of the "NOT FOUND AT HEAD" hint — without it, nobody is robust.

Determinism: pin the provider (thank you, comment section)

Before trusting the corrected results, we attacked the execution layer too.

Part 2's known weakness: temp=0 + seed=42 on OpenRouter is not determinism — ≥8 upstream backends. Tom Jones' comment suggested provider.order with allow_fallbacks: false, which pins the endpoint — cheaper than K≥3 repeats.

We probed it: StreamLake — the most-used upstream for glm in our server CSV (245 calls) — returns 404 "No endpoints found" when pinned: it no longer serves this model at all. Cloudflare/DeepInfra are stable. The full pinned re-run (qwen→Alibaba, deepseek/glm→DeepInfra, ~$0.02) reproduced every conclusion: per-model arm rankings, temporal present-trap (12/12, 9/12, 12/12), past-tense fix (40/40), and FA absent/silent = 0 across evidence arms. One caveat: glm stays non-deterministic even pinned (FA 0.06 → 0.02 → 0.02 across runs) — part of that is model variance, but part is upstream drift: unpinned glm now routes to DeepInfra (not StreamLake), so cross-day comparisons mix changing backends. Pinning removes routing variance at a point in time, not model or availability drift over time.

Reproduction note: if you reproduce pinned runs, avoid StreamLake — it was the top unpinned provider for glm in our server CSV but returns 404 when pinned (no endpoint for the model; upstream availability drifts). Pinning to a provider that serves you well unpinned is not guaranteed to work; probe before committing to a long run.

What this means for verify-on-read

Evidence format is a per-model knob. qwen-family: file fragment (recall). glm-family: graph (recall + trap precision). deepseek: hybrid.
Do not concatenate evidence formats blindly. For qwen, file+graph was strictly worse than file alone.
Red-team your dataset before trusting metrics. One grep on the subject files inverted our headline. Synthetic categories must be validated per subject, not per project — and FA on a category is meaningless until the category's labels are truth-checked.
Temporal questions must be phrased in time. "Is X defined in F?" with history in evidence fails universally (12/12, 9/12, 12/12); "Was X defined in F?" succeeds (40/40). For existence checks: HEAD-only evidence, or explicit tense.
Pin providers in production runs. Cheaper than repeats, and it protects against "popular but broken" upstreams (StreamLake).

The meta-lesson

The most dangerous assumption in LLM evaluation isn't the model — it's the dataset. We spent ~$0.10 and ~1900 calls to learn that 4 of 6 "false" facts were true. Before you trust any LLM benchmark, ask: who labeled the ground truth, and did they verify it per-example or per-category? We didn't — we validated the trap category against the project, not the subject. One grep on subject files inverted every conclusion. Every layer of the pipeline can lie: ground truth (this post), evidence (E4b), model interpretation (E4c), execution (provider drift) — we attacked each in turn.

Synthetic categories must be validated per subject, not per project. A value that exists anywhere in the repo is not evidence that the subject of the claim uses it. Check the subject's file.

"Known weaknesses (post-red-team)"

Fact order matters (measured). Facts are stored block-ordered (R01–R25 true, R26–R50 false). A shuffled control run (qwen code_first, seed 123) changed 4/50 verdicts vs the original order — no systematic direction, but ~8% sensitivity to order. Shuffle-seed in future runs.

Trap category size matters. After relabeling, the original dataset has one genuinely false trap fact — but an extended subject-validated category (20 false) shows the present-trap is real and mass-scale (file_content FA 10–75%). Metrics from N=1 are flags, not rates.

Language confound (measured). Claims are Russian, instructions English. A Russian-instruction control (file_content) shifted deepseek's unknown rate 13/50 → 1/50 and changed 14/50 verdicts — its "skeptic" profile in this series was partly a prompt-language artifact. qwen/glm shifted 5/50.

Decoy frequency. 19 facts share the same control symbol block; models could pattern-match repetition. Not controlled.

Temporal claims are existence claims. Easier than the usage claims of the main dataset; the two datasets are complementary, not interchangeable.

Small N, wide CIs. 6 trap facts, 12 removed facts. Headline arm rankings rest on differences of 2–3 facts out of 25.

Upstream drift across days. Unpinned vs pinned runs happened ~12h apart; glm's routing changed (StreamLake → DeepInfra). Cross-day numbers mix backends.

Thanks to the Part 1 comment section: Tom Jones (provider pinning), Skillselion (manifest anchors), Cophy (invalidation triggers), Glen Allen (freshness), 473185670 (Resolution Loop), UnitBuilds (write-path triples) — each thread sharpened this series.

Reproduce

Harness: scripts/run_1L_live_arm.py (arms code_first / file_content_first / graph_first / file_graph_first / temporal_blind_first / temporal_duo_first; --facts, --ev-contexts, --pin-provider). Summaries: scripts/summarize_1L_categories.py --facts <corrected.json> (truth-based re-score of old runs). Full report with raw outputs and the red-team audit: experiments/2E_evidence_ladder/report.md. Tests: 64 for harness/builder/generator/summarize, 1265 total.

Dataset fingerprints: original 820bbbf60a0fc930 (historical, mislabeled trap) · corrected e6ce7b902d0a20a9 (29 true / 20 false / 1 ambiguous) · temporal e3c1fdd4 / d1d2c2ed440ec370 · calls: ~1900 across the series · est. cost: < $0.10.

Went down a rabbit hole testing whether LLMs can actually verify agent memory claims on their own. Turns out the evidence format matters way more than the model — bare token strings barely work, real code context changes everything.

Mikhail — Sat, 15 Aug 2026 18:47:15 +0000

Mikhail

Aug 14

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models)

#llm #memorylayer #agents #benchmark

10 min read

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models)

Mikhail — Fri, 14 Aug 2026 21:35:18 +0000

Part 1: The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?
Part 2: Your memory layer is lying to you (and your LLM agrees)
Part 3: The Dataset Was Lying: 4 of 6 "False" Facts Were True

The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — useful, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3: "headline numbers were a property of the heuristic, not LLM behavior."

So we ran it with live models. Four days, 14 models, 4 arms, ~3400 API calls. The last arm answered the question the whole experiment was actually asking.

The setup

Dataset: memory_contamination_facts_v4_rep.json, N=50 (R01–R50), sha256 820bbbf60a0fc930.

kind	n	what it tests
`real`	25	TRUE — grep-validated against actual code
`absent-mutation`	16	FALSE — component doesn't exist in the project
`present-trap`	6	FALSE — file exists, claim is about the wrong subject
`silent`	3	FALSE — external systems the codebase never mentions

Four arms, run sequentially as the experiment evolved:

memory_first — model sees only the claim. Does it trust memory without evidence?
code_first — claim + support_patterns (bare token strings) + section. Does it evaluate the anchors?
code_first CoT — same as above, reasoning=true, max_tokens=1500.
file_content_first — claim + real 25-line file window around the anchor. Can the model verify with actual evidence?

Model verdict: {"verdict": "true"|"false"|"unknown"}, JSON-only. Zero-shot: max_tokens=100, temp=0, seed=42, --no-reasoning. Leak-guard: assert "truth" not in prompt on every fact, unit-tested (39 harness tests, 1226 total).

Two metrics that both matter:

false_accept (FA) — fraction of false claims returned as "true". Contamination risk.
recall(real) — fraction of 25 TRUE facts correctly confirmed. A gate that rejects everything has FA=0.00 and recall=0.00. It's a no-op.

Day 1: live models vs. proxy

The proxy always decided — unknown=0 by construction. First live run (opencode/deepseek-v4-flash-free, free tier) returned unknown=0.80. FA=0.10, including R31 in both arms.

The reviewer's criticism was right immediately.

Day 2: six cheap models, two full passes, red team

The sycophancy problem (V1 → V2)

R31 was false-accepted by every model in the first sweep:

claim:            "The instruction scanner uses Typesense"
truth:            False
support_patterns: ["typesense"]
contra:           [file:src/core/instruction_scan.py]

instruction_scan.py uses only stdlib (re, logging, typing). Typesense is nowhere in the project. Nine false facts in R26–R50 shared this pattern (vespa, pinecone, typesense, tantivy, meilisearch, dataclasses, logging, pathlib, loki) — bare token in support_patterns, model treats field presence as evidence.

V1 prompt asked: "Does the claim appear supported by these anchors?" — leading question. Confirmed sycophancy (Sharma et al. 2023, arXiv 2310.13548).

# V1 — leading:
Does the claim appear supported by these anchors?

# V2 — neutral:
Return true ONLY if the anchors directly verify the claim;
false if the anchors contradict it or the claim refers to something
absent from the anchors; unknown if you cannot determine.

V2 reduced FA in 4/6 models. glm-4.7-flash: 0.30 → 0.24. Still not safe.

`--no-reasoning` is mandatory

Without it, GLM-4.7-flash uses 110 reasoning tokens and returns content=None, finish=length — 100% unusable. With --no-reasoning: reasoning_tokens=0 confirmed for all models via harness audit and independently via OpenRouter server CSV.

temp=0 + seed=42 is not determinism on OpenRouter

nemotron-3.5-lightning code_first: FA 0.18 → 0.08 between two identical runs. ±0.10 on a single-pass measurement. OpenRouter routes to ≥8 upstream backends (server CSV: Alibaba 1955 calls, DeepInfra 559, DigitalOcean 531, Cloudflare 283, Novita 253, Baidu 114, StreamLake 110, Amazon Bedrock 100) — same prompt, different quantizations. qwen3.6/3.7/deepseek stable 3/3. GLM not.

All final conclusions use upper-bound-of-two-runs.

Prompt language is model-specific

deepseek-v4-flash code_first unknown:
  EN: 0.94 → RU: 0.54  (RU → commits more)

qwen3.7-flash code_first unknown:
  EN: 0.24 → RU: 0.58  (RU → hedges more)

Same dataset, same arm, opposite direction per model. Measure per-model before assuming.

[TODO: verify whether claim language interacts with prompt language separately — all claims in this dataset are in Russian]

Premium models

claude-sonnet-5: FA=0.00/0.00, acc=1.0 on all decided. But unknown=0.86/0.70 and $0.049/100 calls.
qwen3.8-max: HTTP 400 on 22–49/50 calls — "Reasoning is mandatory and cannot be disabled." Not a harness bug.
glm-5.2: FA=0.00/0.02 — dramatically better than glm-4.7-flash. Same family, new generation.
deepseek-v4-pro: FA=0.04/0.00.

Premium gives no FA advantage over the best flash models. The cost difference is 10–100×.

Day 3: per-category breakdown reveals the real problem

A reviewer asked: "High unknown ≠ high quality. What was the true-accept rate on the 25 real facts? Is qwen3.6-flash cutting true memory along with false?"

We added scripts/summarize_1L_categories.py and ran it on all V2-EN progress files.

Per-category results (zero-shot, V2, code_first)

model	real: acc/rej/unk	recall(real)	F1	FA absent·trap·silent
qwen3.6-flash	2 / 7 / 16	0.08	0.15	0·0·0
qwen3.7-flash	5 / 9 / 11	0.20	0.33	0·0·0
deepseek-v4-flash	1 / 0 / 24	0.04	0.08	0·0·0
claude-sonnet-5	11 / 0 / 14	0.44	0.61	0·0·0
nemotron-3.5-lightning	13 / 3 / 9	0.52	0.65	0·1·1
glm-4.7-flash ⚠️	22 / 1 / 2	0.88	0.75	7·3·2
nemotron-3-nano 🔴	20 / 1 / 4	0.80	0.62	11·5·3

qwen3.6-flash code_first: confirmed 2/25 true facts, actively marked 7/25 as false (wrong), abstained on 16/25. FA=0.00 achieved by not functioning as a verifier. Auto-retraction on these verdicts empties the memory layer, not just the false claims.

The tradeoff across the sweep:

qwen3.6-flash   F1=0.15   FA=0.00  ← fail-closed
qwen3.7-flash   F1=0.33   FA=0.00  ← fail-closed
claude-sonnet-5 F1=0.61   FA=0.00  ← best at zero-FA
nemotron-3.5    F1=0.65   FA=0.04  ← best F1 overall
glm-4.7-flash   F1=0.75   FA=0.24  ← best coverage, unsafe

No model in zero-shot balanced both. FA=0.00 is a policy, not a quality metric.

CoT doesn't fix it

We ran --reasoning --max-tokens 1500 on 4 models, two full passes for stability.

model	arm	recall ZS→CoT	FA ZS→CoT	cost/100 ZS→CoT
qwen3.6-flash	code_first	0.08→0.20	0→0	$0.0006→$0.045
qwen3.7-flash	code_first	0.20→0.16	0→0	$0.0005→$0.005
glm-4.7-flash	code_first	0.88→0.72	12→7	$0.001→$0.017
deepseek-flash	code_first	0.04→0.08	0→0	$0.0009→$0.002

Only meaningful gain: qwen3.6 code_first recall 0.08→0.20, FA held at 0.00. Cost ×75. qwen3.7 and deepseek: within CI noise. glm-4.7: FA improves but 16–26% of responses are EMPTY_CONTENT (upstream defect in reasoning mode) — numbers are qualitative only.

Important confounder: CoT arm changes two things simultaneously — reasoning=true AND max_tokens=1500. Separating them is impossible: qwen3.6 uses 700–1500 reasoning tokens per call.

qwen3.8-max CoT (mandatory reasoning, the only mode it runs): recall=0.36 code_first, FA=0.04 — best recall-at-low-FA in the zero-shot/CoT sweep. Cost $0.10/100 calls.

Day 4: file_content_first — the answer

The reviewer's point after Day 3: "Is recall=0.08 model paranoia, or are the pattern strings too narrow to validate anything?"

New arm: instead of support_patterns: ["typesense"], the model gets a real 25-line window from the file around the first occurrence of the anchor. For absent and silent facts (grep returns nothing), a decoy: the head of src/core/instruction_scan.py. The decoy is not labeled in the prompt — labeling it would leak ground truth ("not found" → trivial false). It's marked in results as evidence: decoy for post-hoc analysis. Prompt instruction identical to code_first V2 (neutral). Single variable changed vs code_first: form of evidence (token string → code).

Results

model	arm	recall(real)	FA	real: acc/rej/unk	F1	FA absent·trap·silent
qwen3.6-flash	code_first (baseline)	0.08	0.00	2/7/16	0.15	0·0·0
qwen3.6-flash	file_content_first	0.88	0.04	22/3/0	0.90	0·2·0
qwen3.7-flash	code_first (baseline)	0.20	0.00	5/9/11	0.33	0·0·0
qwen3.7-flash	file_content_first	0.88	0.02	22/2/1	0.92	0·1·0

recall(real): qwen3.6 0.08 → 0.88 (×11). qwen3.7 0.20 → 0.88 (×4.4). Wilson CI for 22/25: [0.70, 0.96] — doesn't overlap with baseline CI for 2/25: [0.02, 0.25].

The model wasn't being paranoid. It had no evidence.

FA: 0.00 → 0.02–0.04. Every false accept is present-trap — the anchor token appears in the file fragment, but the claim is about the wrong subject:

R45: "The server wrapper uses logging" — fragment is from log_manager.py, token logging is there, but the subject is wrong.
R46 (qwen3.6 only): "The watchdog uses threading" — fragment from project_indexer_registry.py.

absent 0/16, silent 0/3 for both models. The decoy policy works: a fragment without the token returns false or unknown without leaking.

Remaining false rejects on real facts: R07/R08 (both models) — claims with semantically bad value field ("uses disabled", "uses forbidden" — adjective instead of noun; fragment shows the env-var but not the adjective); R21 (qwen3.6), R03 (qwen3.7) — individual confidence threshold differences.

What this means for VOR architecture

The problem was always the evidence layer, not the model.

Showing a model support_patterns: ["typesense"] and asking it to verify a claim is not verification — it's asking the model to confirm the field value matches the field name. The model correctly doesn't know what to do with it, so it abstains or guesses.

Showing the model 25 lines of actual code around the anchor: recall 0.88, FA 0.02–0.04, $0.005/100 calls. The remaining failure mode is subject-mismatch in present-trap facts — the model checks token presence, not claim subject identity. That's a harder problem but a much smaller one.

Cost of adding a file window: ~200 extra prompt tokens per fact. At flash pricing, that's $0.0001–0.0004/fact. Negligible against the cost of retracting true memory.

Full results summary

model	arm	recall(real)	FA	F1	$/100	notes
qwen3.6-flash	file_content	0.88	0.04	0.90	$0.005	recommended
qwen3.7-flash	file_content	0.88	0.02	0.92	$0.005	recommended
qwen3.8-max	CoT	0.36	0.04	—	$0.102	if file access unavailable
nemotron-3.5	zero-shot	0.52	0.04	0.65	$0.001	if file access unavailable
claude-sonnet-5	zero-shot	0.44	0.00	0.61	$0.049	if FA=0.00 required
qwen3.7-flash	zero-shot	0.20	0.00	0.33	$0.0005	fail-closed only
qwen3.6-flash	zero-shot	0.08	0.00	0.15	$0.0006	fail-closed only
glm-4.7-flash ⚠️	zero-shot	0.88	0.24	0.75	$0.001	exclude
nemotron-3-nano 🔴	zero-shot	0.80	0.38	0.62	$0.0008	exclude

Known weaknesses

Five limitations, stated before a reviewer raises them.

1. Upstream routing variance (partially mitigated). temp=0 + seed=42 on OpenRouter is not true determinism — 8+ backends. Mitigated by 2–3 passes per condition, upper-bound-of-runs. Not mitigated: K≥3 repeats per fact/model pair (~4200 calls, ~$2–5) — not done.

2. Anchor bias vs. snippet truncation (closed). V4 (§6.6b) confirmed anchor bias: recall 0.08→0.88 with real file content. The residual failure in present-trap (R45, R46) is subject-identity checking, not truncation.

3. Synthetic mutations vs. real drift (in progress). The 25 false facts are synthetic (absent/trap/silent), not real git-refactoring drift. 30-day longitudinal protocol is running — data not yet collected.

4. Single codebase, single language (not closed). All 50 facts from one Python project. TypeScript/Rust/Go have harder type anchors — transfer is extrapolation. Multi-repo extension needs ~$0.4–3.6 and 1–2 days of grep-validation.

5. Three-way verdict vs. confidence score (not closed). The true/false/unknown frame may push "partially stale" claims into unknown. A 5-point confidence scale would need a new prompt, parser, and graded dataset. What we can say: finish_reason=stop everywhere in zero-shot — high unknown is genuine uncertainty, not truncation.

What to use

For a working VOR gate: qwen3.6-flash or qwen3.7-flash with file_content_first. recall=0.88, FA=0.02–0.04, $0.005/100 calls. The only failure mode is present-trap — the model sees the right token in the wrong file context. Acceptable for flag-and-review; not yet for fully autonomous retraction.

If file access is unavailable at verdict time: qwen3.8-max CoT (recall=0.36, FA=0.04, mandatory reasoning, $0.10/100). Better than any zero-shot option on recall. Or nemotron-3.5-lightning (recall=0.52, FA=0.04, $0.001/100) as the cheapest zero-shot model with reasonable F1.

Fail-closed (contamination-intolerant, recall doesn't matter): qwen3.6/3.7 zero-shot. FA=0.00 confirmed 0/400 code_first verdicts across 4 runs. Cheapest. Expect to retract ~80–92% of true memory alongside false.

Exclude: glm-4.7-flash (FA=0.24), nemotron-3-nano (FA=0.38).

Measure on your own dataset before deploying. These numbers are from one Python codebase.

Reproduce it

git clone <repo> mscodebase && cd mscodebase

# Windows
python -m venv venv && venv\Scripts\python -m pip install -e .
# macOS/Linux
python -m venv venv && venv/bin/pip install -e .

# .env: OPENROUTER_API_KEY=sk-or-v1-...

# integrity check
python scripts/run_1L_live_arm.py --arm both --dry-run

# canonical flash sweep, V2, ~600 calls, ~$0.009
python scripts/run_1L_live_arm.py \
  --provider openrouter --arm both \
  --models "qwen/qwen3.7-flash,qwen/qwen3.6-flash,qwen/qwen3.5-flash-02-23,\
deepseek/deepseek-v4-flash,z-ai/glm-4.7-flash,nvidia/nemotron-3.5-lightning" \
  --prompt-version v2 --no-reasoning --tag v2_en

# per-category breakdown
python scripts/summarize_1L_categories.py --tag v2_en

# V4: real file snippets, ~100 calls, ~$0.005
python scripts/run_1L_live_arm.py \
  --provider openrouter --arm file_content_first \
  --models "qwen/qwen3.6-flash,qwen/qwen3.7-flash" \
  --prompt-version v2 --no-reasoning --tag file_content

# CoT arm, ~400 calls, ~$0.20
python scripts/run_1L_live_arm.py \
  --provider openrouter --arm both \
  --models "qwen/qwen3.6-flash,qwen/qwen3.7-flash,z-ai/glm-4.7-flash,deepseek/deepseek-v4-flash" \
  --prompt-version v2 --reasoning --max-tokens 1500 --tag v3_cot

# second pass for any arm (variance check)
python scripts/run_1L_live_arm.py [same args] --force

Dataset fingerprint: 820bbbf60a0fc930. Full report: experiments/exp_1L_live_arm_report.md. Tests: tests/test_run_1L_live_arm.py (39), tests/test_summarize_1L_categories.py (8).

Source: github.com/ManSio · Portfolio: mansio.github.io/MSPortfolio

Mikhail

Aug 16

The Dataset Was Lying: 4 of 6 "False" Facts Were True

#ai #agents #testing #mcp

9 min read

I turned my portfolio into an MCP server (and I'm not a programmer)

Mikhail — Fri, 14 Aug 2026 13:14:02 +0000

I turned my portfolio into an MCP server (and I'm not a programmer)

My background is civil engineering. I don't write code — I direct AI agents to write it while I handle architecture, decisions, and QA.

So when I built my portfolio, I didn't want it to look like everyone else's. I wanted it to do something no one else's does.

Here's what I built: a portfolio that answers questions from AI agents directly. Not a chatbot. Not a demo. A real server that any AI — Claude, GPT, anything — can connect to and interrogate.

claude mcp add --transport http msp-portfolio \
  https://msp-portfolio.mansio-dev.workers.dev/mcp

Ask it "what has Mikhail built?" or "does his stack match this job description?" and it answers with live data — not a frozen PDF from six months ago.

The problem I was solving

Every portfolio makes the same silent promise: "trust that this is still true."

It's a snapshot. The moment you close the editor, it starts going stale. And there's no way for anyone — human or AI recruiter — to verify what's actually there.

I wanted the opposite. A portfolio that doesn't just display claims, but answers questions about them. If the evidence for a skill isn't there, it says so instead of bluffing.

What I actually built

The site has three parts that share the same brain:

1. The website — what you see when you open it in a browser. Interactive, dark-themed, with a simulator you can break.

2. A server for AI agents — the same data, but in a format any AI can query directly. This is the MCP part.

3. A local version — for development and testing from the command line.

The unusual thing: all three read from exactly the same source. No copy-pasting, no "API version" vs "website version." One place, three doors.

The nine tools

Think of these as questions the AI can ask my portfolio:

Question	Tool
Who is Mikhail?	`get_profile`
What has he built?	`get_projects`
What has he been working on lately?	`get_commit_history`
What are his engineering principles?	`get_engineering_principles`
What has he written recently?	`get_articles`
What mistakes has he made?	`get_antipatterns`
How does his stack match this job?	`analyze_stack`
What decisions did he make and when?	`get_timeline`
What happens to his architecture under load?	`simulate_architecture`

The one recruiters use most: analyze_stack. You paste a job description, it returns a per-skill breakdown with actual evidence — not "I know Kubernetes" but "used in project X, decision log here."

The one I'm most proud of: get_antipatterns. It's a museum of real mistakes I made while building this — a forked repo I accidentally claimed as mine, a feature that worked in local tests and silently broke in production, a counter that looked like it was writing data and wasn't. Honest lessons, not a polished highlight reel.

The simulator

The portfolio has a live architecture simulator. You pick one of my real projects, pick a failure scenario — kill a node, freeze the cache, overload the AI — and watch the performance numbers change in real time.

It's not a video. It's not a screenshot. It runs the actual model.

At 20× load on the search architecture: p95 latency goes from ~10ms to 239ms under a load spike, 401ms if a node dies. When it crosses a threshold, a chip appears: circuit_open, fallback_engaged, degraded_mode. This is what I mean when I say "I understand distributed systems" — you can verify it yourself.

What broke (the honest part)

The package that doesn't exist. I spent time trying to install @fastify/mcp. It doesn't exist. The real name is @modelcontextprotocol/fastify. Simple mistake, cost an hour.

The function that worked locally and did nothing in production. I had a counter that tracked how many times the MCP server was queried. It passed all tests. In production, the counter never wrote anything. Reason: in Cloudflare Workers, code that runs after you send a response gets cancelled immediately. My counter was running after the response. Fix: one line — ctx.waitUntil(task). The lesson is that local Node.js and Cloudflare Workers have different rules about what happens after a request ends.

The rate limiter that reported success but did nothing. I set up a rate limit on the free plan. Every call returned success: true. Enforcement simply wasn't active at that tier. I only caught it by deliberately triggering it with a burst test. Configuration saying "yes" and the system actually enforcing it are two different things.

The hourly commit that silently moved the code forward without me. The CI runs every hour to refresh metrics. Each run makes a commit. If I pushed without fetching first, I'd be pushing on top of a diverged history. Now I always git fetch first. Not a disaster, just a thing you learn once.

One thing I didn't expect

The agent loop — where you can type a question and watch the AI call tools step by step, live, in the browser — turned out to be the most interesting part to watch.

You see it think. It calls get_profile, reads the result, decides it needs more, calls analyze_stack, compares, then answers. The whole chain is visible. It's not a black box with an answer at the end. It's a process you can audit.

I didn't plan that as a feature. It came from trying to debug the demo and realizing the debug view was more interesting than the final answer.

Try it

Portfolio: mansio.github.io/MSPortfolio
Source: github.com/ManSio/MSPortfolio
Add to Claude:

  claude mcp add --transport http msp-portfolio \
    https://msp-portfolio.mansio-dev.workers.dev/mcp

Then ask it: "what did your mistakes teach you?"

If you build something like this, drop a comment. I'd genuinely like to see it.

The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?

Mikhail — Tue, 11 Aug 2026 19:22:59 +0000

We talk a lot about giving AI agents persistent memory—building a "Second Brain" or "Knowledge OS" where agents can log decisions and retrieve context.

But what happens when that memory is wrong?

I’ve been thinking about the gap between mechanical execution (the agent called the tool, the code compiled, the exit code was 0) and semantic truth (the conclusion drawn from that execution is actually correct in reality). It’s easy to assume that if the mechanical layer is solid, the semantic layer will follow. But I started suspecting this might be a dangerous assumption.

To test this, I didn't want to just theorize. I ran a controlled experiment on my own MCP codebase-intelligence server (Python, 50K LOC), which features an IntelligenceStore — a persistent memory layer where agents can log incidents and collect Architectural Decision Records (ADRs).

I wanted to know: If an agent's memory is poisoned with a mix of true and false facts, does it verify against the code, or does it blindly trust its memory?

Update: This post originally covered the initial Memory Contamination experiment and a Retraction mechanism. I have since updated it with the results of a follow-up experiment (Experiment 1-V) implementing "Verify-On-Read", which successfully closed the final 12% contamination gap. Scroll down to "Closing the Gap: Verify-On-Read" for the final architecture.

The Experiment: Memory Contamination

I built a deterministic proxy-agent and ran it against a controlled set of facts.

A quick caveat on methodology: I didn't have a live LLM hooked up for this run, so I used a deterministic proxy-agent based on heuristics. This means the results measure the system's structural capability, not necessarily the psychological behavior of a live Claude or GPT model. A live model might be lazier, or it might be smarter. I'm still trying to figure that out.

The Setup

I injected 50 facts into an isolated memory store:

25 TRUE facts (real architectural details mapped to the codebase).
25 FALSE facts split into two categories:
- CONTRADICT (22): False facts where the code explicitly proves them wrong (e.g., "We use Redis" when Redis is absent, but the code clearly uses DuckDB).
- SILENT (3): Plausible false facts about external systems where the code is completely mute (e.g., "We use Celery for background tasks" when no task queue exists in the repo).

I tested three agent configurations:

B (No Memory): Baseline. Must rely purely on code retrieval.
A_code_first (Honest Agent): Checks the code first, uses memory only as secondary context.
A_memory_first (Lazy Agent): Reads memory first. If it finds an answer, it stops looking.

To ensure scientific rigor, the experiment was replicated with an independent set of facts (N=50), verified across 6 axes (including a truth-table audit and an independent LLM "fresh eyes" audit). The results were identical.

The Initial Results

Arm	Correct	Adopted False Facts	Correction Capability
B (No Memory)	0.94	0.0%	0.0
A_code_first	0.94	12%	1.0
A_memory_first	0.50	100%	0.0

Here is how I interpreted these numbers:

The Lazy Agent Trusts Poisoned Memory: The A_memory_first configuration — which mirrors how many token-optimizing production agents behave — adopted 100% of the false facts. If the memory said "We use RabbitMQ," the agent trusted it and stopped looking at the code.
The SILENT-Fact Trap: Even the "Honest Agent" had a 12% adoption rate. This happened entirely on the SILENT facts. When a fact is false but the code doesn't explicitly scream "NO," the agent's memory fills the void with a confident hallucination. Memory turns an honest UNKNOWN state into a structural guess.
The Add-Only Limitation: When the Honest Agent did realize the memory was wrong (Correction Capability = 1.0), it couldn't do anything about it. I ran a grep for delete or refute in the memory store API. Zero results. The memory system was purely add-only. The false fact stayed in the database to poison future sessions.

The First Fix: Testing a Retraction Lifecycle

The current industry consensus for "Knowledge OS" trust layers is to use timestamps, source priority, and supersedes/contradicts relationships.

My initial experiment suggested this was insufficient. Timestamps and "supersedes" links only solve node-level history. If an ADR is superseded, the memory node updates, but the downstream code, tests, and docs generated from the old assumption are still in the graph. They are structurally stale, but the retrieval engine keeps pulling them in.

I hypothesized that we needed an explicit state transition: VERIFIED → REFUTED.

I implemented a RetractionReceipt mechanism in my system:

Status Enum: Every memory node gets a status (ACTIVE, VERIFIED, REFUTED).
Hard Filtering: The retrieval pipeline (load_memory) hard-filters anything that is not ACTIVE or VERIFIED.
Explicit Retraction Tool: An MCP tool (intel_retract_memory_node) allows the agent to actively flag and invalidate memories when they contradict the live codebase.

I ran the experiment again (Experiment 1-R). The honest agent was allowed to use the retraction tool in Session 1. Then, a fresh memory_first agent was launched in Session 2 to read the post-retraction memory.

The Retraction Results

Metric	Original (Add-Only)	With Retraction
Adoption (Lazy Agent, Session 2)	1.0 (100%)	0.12 (12%)
Persistent False Facts in Memory	25	3 (-88%)
Token Context Size	Baseline	-45%
Systemic Correction Capability	0.0 (couldn't delete)	1.0 (22/22 refuted)

The retraction lifecycle worked. The lazy agent's adoption rate dropped from 100% to 12%. Persistent false facts dropped by 88%, and token context size shrank by 45% because refuted facts were filtered out before reaching the LLM.

The Honest Limitation: Why It Didn't Drop to Zero

My ADR predicted that adoption would drop to 0. It didn't. It dropped to 0.12.

The remaining 12% were the SILENT facts.

An explicit REFUTED status is required to programmatically exclude downstream dependencies from the retrieval pipeline. But even that only works if you have a contradicting signal in the code. If the memory claims "We use Celery," and the codebase simply doesn't mention Celery at all, the agent has no evidence to trigger the retraction.

To get to zero, I realized we needed "verify-on-read"—a mechanism that challenges a memory claim against the codebase even when the code is mute.

Closing the Gap: Verify-On-Read

I implemented a lazy validation layer (ADR-0003). When load_memory() pulls a node, it extracts lightweight "anchors" from the memory text (e.g., file names, import statements, environment variables). It then checks if those anchors actually exist in a live fingerprint of the codebase (the current git HEAD).

If the anchor is found in the code $\rightarrow$ status becomes VERIFIED.
If the code explicitly contradicts the anchor (or the anchor is entirely absent when it should be present) $\rightarrow$ status becomes REFUTED.
If it can't be determined $\rightarrow$ status remains ACTIVE (treated as INCONCLUSIVE).

I ran the experiment one final time (Experiment 1-V) with this layer active. To prevent latency spikes, the validation operates under a strict 50ms budget per retrieval, with a 30-second TTL cache on the git HEAD so steady-state reads cost almost nothing.

The Final Results

Metric	With Retraction (1-R)	With Verify-On-Read (1-V)
Adoption (Honest Agent, `A_code_first`)	0.12 (12%)	0.0 (0%)
Adoption (Lazy Agent, `A_memory_first`)	0.12 (12%)	0.16 (16%)
Steady-State Retrieval Latency	Baseline	~0.6ms (Cache hit)
SILENT-Fact Contamination (Honest)	3 facts	0 facts

The Verify-On-Read layer achieved the goal. The honest agent's adoption of false facts dropped to absolute zero, even for SILENT facts. Because the system now actively checks if the codebase actually contains the things the memory claims it does, silent hallucinations are caught at the retrieval boundary and filtered out before they can poison the LLM's context.

The Remaining Honest Limitations

I won't pretend this is a perfect silver bullet. The experiment revealed two edge cases:

The "Present-Trap": If a false memory claims "We use sqlite3", and sqlite3 happens to be imported somewhere in the codebase for a completely unrelated reason, the verification layer sees the token and marks the memory as VERIFIED. The lazy agent (A_memory_first) still fell for this, resulting in the 0.16 adoption rate. (The honest agent avoided this because it read the code context around the import).
Anchor Typing: When extracting anchors from prose (e.g., "We use fastmcp"), the system initially missed that the actual Python import was from mcp.server.fastmcp import .... This caused some false REFUTED verdicts on true facts. The fix is capturing typed anchors at the write-path (when the memory is created), rather than trying to parse them from raw text at the read-path.

Conclusion

Building reliable AI systems isn't just about giving them more context. It's about recognizing that memory has a lifecycle.

If your system can't programmatically refute a memory, false facts accumulate and poison the context window over time. Implementing an explicit VERIFIED → REFUTED state transition drastically reduces contamination and saves tokens. Furthermore, adding a Verify-On-Read layer closes the final gap on "silent" hallucinations, driving honest agent contamination to zero without adding meaningful latency.

However, semantic drift is still a hard problem. Mechanical verification can still be fooled by "present-traps" if the agent doesn't read the surrounding context. The next step is moving anchor extraction to the write-path to ensure memories are created with strict, verifiable references from the start.

If your system handles semantic drift differently, or if you've solved the present-trap problem, I'd genuinely love to hear how you're approaching it.

Mikhail

Aug 14

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models

#llm #memorylayer #agents #benchmark

10 min read

What I learned building a long-lived AI agent (the boring version)

Mikhail — Sun, 09 Aug 2026 13:40:04 +0000

Not a researcher. Not a professional dev. Civil engineering background. Started building an AI bot because I wanted to understand what's actually happening inside these systems — not theoretically, just practically.

Wanted an assistant that could live with a conversation instead of treating every message as an isolated API call.

It started as an experiment. Then the experiment grew.

What it turned into

At some point I had:

routing between reasoning profiles
multiple tools
web search
memory (flat + semantic + graph)
caching
context compression
feedback loops
self-learning experiments
quality monitoring
batch processing
telemetry
health checks
and enough logs to make me question my own sanity

I wasn't trying to build a benchmark. I was trying to make the thing actually work for me.

That distinction changed everything.

An agent is not an LLM call

You think it's:

User → LLM → Answer

Reality:

User
  ↓
Telegram
  ↓
input handling
  ↓
session state
  ↓
intent detection
  ↓
routing
  ↓
profile selection
  ↓
context construction
  ↓
memory
  ↓
cache
  ↓
tools
  ↓
LLM
  ↓
post-processing
  ↓
quality checks
  ↓
Telegram

Every layer is another chance to break something.

The model was often not the problem. The machinery around it was.

The cache thing

Was looking at prompt caching and noticed something weird.

Same model — different behavior depending on the provider. Changed the provider — problem disappeared. Connected the same provider from another IDE — cache worked fine. Back in my bot — sometimes it didn't.

Then I realized: OpenRouter can silently switch providers under the same model name. Cache hit rate changes with it.

Stopped thinking "does the model support caching". Started thinking "what exactly is being sent, and what does the provider consider identical".

Much more useful question.

Why it was different in the IDE

The IDE was sending:

model → cache works

My bot was sending:

routing
+ dynamic context
+ session metadata
+ memory
+ profile-specific stuff
→ provider
→ model
→ cache may or may not match

The model hadn't changed. The provider hadn't changed. The request structure had.

Caching is extremely sensitive to prefix stability. Once I understood that and stabilized the relevant parts — got around 66% cache hit rate on average, up to ~80% in favorable conditions (same topic, stable structure).

That's the part benchmark screenshots don't show. Cache hit rate is not a model trait. It's a property of your workload.

"90% token savings"

When I see this claim now I don't think it's fake. I think: show me the workload.

90% is possible if your prefix looks like:

system instructions (stable)
+ tools (stable)
+ project context (stable)
+ conversation history (stable)
+ small new message

Real agents often look like:

system instructions
+ changing tools
+ changing memory
+ changing routing metadata
+ changing summaries
+ changing retrieved docs
+ changing timestamps
+ new message

Prefix isn't stable. Theoretical saving and practical saving become very different numbers.

Routing is harder than it looks

Obvious idea: route every request to the cheapest model that can handle it.

Works fine until you have a real conversation.

User says: "What's the weather?" — easy.

Then: "Compare it with yesterday." — now context matters.

Then: "Actually forget the weather. I was thinking about that thing we discussed yesterday." — now memory matters.

Then: "No, not that. The other one." — router needs to understand the whole conversation, not just the current sentence.

The routing decision isn't question → model. It's:

conversation state
+ user intent
+ previous actions
+ available tools
+ risk
+ latency
+ cache state
→ routing decision

And here's the problem: a routing decision changes the request. A changed request affects caching. A changed profile affects context. A changed context affects the answer. A different answer affects feedback.

A tiny routing optimization has consequences five layers away.

The latency numbers

Real VPS measurements at some point:

Metric	Value
LLM p50	~2.2s
LLM p95	~17.3s
LLM max	~51s
Agent median	~6–11s
Agent p95	~20–57s
Earlier pipeline tails	up to ~116s

Interesting number isn't LLM latency. It's the gap between LLM latency and agent latency.

Model answers in 2 seconds. Agent takes 10–57.

Because the agent isn't just the model. It's everything before and after it.

Telemetry became the most important thing

Ended up recording: LLM usage, prompt tokens, cached tokens, latency, route decisions, quality events, memory operations, feedback, errors, conversation traces.

Looked excessive at first. Then it became obvious why.

When something broke I could ask "what actually happened" instead of "I think the model was confused".

Very different debugging strategies.

A lesson about noisy telemetry

One audit showed thousands of route_risk records. Sounds catastrophic.

But most were repeated observations like:

quality_loop:search_skipped
quality_loop:price_hallucination
quality_loop:reply_echo

They were useful signals. But not thousands of independent disasters.

Telemetry volume is not incident volume.

A system can produce huge numbers of observations about a small number of underlying failure patterns. You have to cluster them, or the monitoring system itself becomes noisy.

Self-learning is a trap if your data is bad

Experimented with feedback loops. Bot could receive 👍 👎 and connect that to routing quality, skill reputation, scenario history.

Attractive idea: agent learns from mistakes.

Dangerous question: what exactly is it learning from?

If telemetry is noisy — it learns noise. If a synthetic probe looks like real user traffic — it learns from the wrong population. If a failed tool call gets logged as a routing failure — the wrong lesson gets created.

So the learning loop needs another loop around it:

experience → evidence → validation → lesson → application → new evidence

Otherwise you're automating superstition.

Context compression

Added protection for recent messages — compress old history, preserve the newest turns.

OLD OLD OLD OLD NEW NEW
→
[summary] [summary] NEW NEW

Reason: a summary is an interpretation of the conversation, not the conversation.

Sometimes the missing detail is one sentence. And that sentence changes the meaning of everything.

Batch processing

Parallel execution looks great on paper. 12 tasks × 2s each = 2s instead of 24s.

But natural language tasks aren't always independent.

Find three products.

Compare them.

Tell me which is best.

Task 2 depends on 1. Task 3 depends on 2.

Ended up checking for cross-references, pronouns, comparative language, explicit dependencies before deciding to parallelize.

The optimization wasn't hard. Knowing when it's safe was.

Memory introduced its own problems

Added knowledge graph + semantic memory + flat persistence + vector search + entity relationships.

Sounds sophisticated.

But memory introduces a basic question: should this actually be remembered?

Then: should it be retrieved now?

Then: is this memory still relevant?

Then: is this memory more important than what the user just said?

A memory system doesn't automatically make an agent remember better. Sometimes it makes it remember too much.

The most important lesson

After adding enough machinery you eventually discover that machinery itself becomes the problem.

An agent can have router + memory + tools + cache + planner + evaluator + self-learning + scenario engine + quality loop and still perform worse than:

short prompt + one good model

for a simple task.

Sometimes the right optimization is subtraction.

Remove unnecessary context. Remove unnecessary routing. Remove unnecessary abstraction. Keep the useful part.

What I'd do differently

Start with:

one model
one provider
one short system prompt
one conversation store
one cache strategy
minimal tools
excellent telemetry

Run it. For a long time. Only after seeing real failures add another layer.

Not architecture first → hope it works.

But simple system → observe → measure → find failure → fix → measure → only then add complexity.

Feels slower. In practice, probably faster.

What the experiment actually taught me

The hard part of an AI agent isn't making the model answer.

The hard part is maintaining a stable environment around the model while everything keeps changing.

User changes topic. Context grows. Provider changes. Cache behaves differently. Tool times out. Router makes a different decision. A previous answer was wrong. A correction arrives six turns later.

And somehow the assistant is expected to behave as if none of that happened.

That's the actual engineering problem.

The test I trust now

Not the number of agents.

Not the number of tools.

Not the biggest benchmark.

Not even the highest cache hit rate.

Just:

Does it still work when a real person uses it tomorrow?

I Asked One AI to Fact-Check Another AI's Audit of My Own Code

Mikhail — Sat, 08 Aug 2026 10:00:58 +0000

I'm not a programmer. My background is construction engineering, and I got into programming almost by accident: I wanted to understand how Telegram bots worked, so I started stitching one together from pieces of code I found through Google and generated with AI. The result was a monolith I kept breaking — I didn't even know Python cared about indentation, and I genuinely didn't understand why the code kept crashing over whitespace. Through trial and error I slowly figured out where AI tends to get things wrong and how to work around it — with the AI itself helping me figure that out. Eventually I rebuilt the bot from scratch, fully modular this time.

Later I tried Cursor, and it was a good experience — it wrote solid code, even if it sometimes drifted off-task. When my subscription ran out, I went looking for a free alternative and landed on Zed. That's when I wanted Zed to feel as capable as Cursor had — with the same level of understanding of my codebase. So I started figuring out how to build that myself. That's how MSCodeBase Intelligence came to be: an MCP server that gives AI assistants more context about a codebase inside Zed IDE.

I've written about the project itself before. This time I want to talk about something else — a habit I picked up along the way that, in hindsight, matters more than any single feature.

How I actually work

I don't write code by hand. I hand tasks to different AI models — Claude, Gemini, DeepSeek, and others — they write and fix the code, and I verify the result. Not because I'm especially disciplined, but because I genuinely can't just look at code and tell if it's correct. I need something — or someone — to check it.

For a long time that meant: one model writes, I read it myself, and I ask questions when something looks off.

Then I started doing it differently: one model writes a report about what it did or what it found in the code, and a second model — with access to the actual codebase — checks that report against reality. Not because I distrust AI in principle, but because I've learned firsthand that AI can sound completely confident and be completely wrong at the same time.

One specific case

I received an external audit of my project — several pages long, with findings, recommendations, and a list of suspected vulnerabilities. It looked solid. Three findings were flagged as top priority.

I didn't rush to fix anything. Instead, I asked a different model — one with access to the live codebase — to go through the report line by line and check it against the actual repository. Not "does this sound plausible," but literally: open the file, open the line, check whether what the report claims is actually there.

The result was mixed, and that's the interesting part.

Three real findings held up. There genuinely was a process-locking bug on Windows, a case where a data write could be left in a non-atomic state, and a race condition in a task queue that could leave a background task stuck forever. These weren't imaginary risks — they were real, worth fixing.

But alongside that, a few other things surfaced:

the report referenced a function name that doesn't exist in the code — the model that wrote the audit had likely invented it by analogy, or pulled it from an outdated version;
one of the file paths in the report was wrong — the actual file lived somewhere else;
the report mentioned a test that was supposed to catch the bug — no such test existed in the repository at all;
the test coverage numbers and test counts were stale, from an earlier state of the project;
for one of the flagged CVEs, the report cited the wrong "fixed in" library version — the actual safe version was later than what was claimed.

So the report was both genuinely useful and partly fabricated — and not fabricated randomly. The invented details were woven around real, correctly identified problems. That, I think, is the most dangerous kind of AI mistake: not "everything is false," but "true, with fabricated details mixed in that you can't tell apart without checking every line yourself."

Why this matters, not just an anecdote

If I'd simply copied the recommendations and asked an agent to apply them, part of the work would have gone nowhere — the agent would have searched for a function that doesn't exist, referenced a test that isn't there, pinned a library to the wrong version. And worse: the three real problems could easily have been dismissed along with everything else, once the fabricated details made the whole report look unreliable.

The lesson I took away: trust in an AI report shouldn't be binary — "believe it" or "don't." A report needs to be broken down into individual claims, and each claim checked separately against the current state of the code, not against the model's memory of what the code used to look like.

A second experiment: checking the instructions, not the code

I keep a file of rules for the AI agents working on this project — something like a job description: how to behave, what to always verify, how to label the source of a claim ("I checked this in the code" vs. "I'm assuming this"). Over time that file grew — a lot.

That raised a question: if I compressed those rules to roughly half the length, would the agent still follow them just as carefully, or would it start cutting corners?

I didn't guess. I ran the same set of tasks twice — once against the full instructions, once against the compressed version — and for each task noted which specific rules were actually followed and which weren't.

The result: the compressed version performed worse, but not dramatically, and not evenly. What suffered most was what I'd call "in-the-moment discipline" — things like updating a decision log right after each step instead of all at once at the end. The rules themselves were still technically present in the compressed file. The agent just remembered to apply them less often mid-task.

That wasn't obvious to me beforehand either. I assumed that if a rule is written down, it gets followed. It turns out what matters is not just whether it's written, but how often — and where in the text — the agent gets reminded of it.

What I'm taking from this

I'm not a programmer, and I don't feel like I understand code better than the models I work with. But in practice I've learned one thing: the main value I add to this process as a human isn't writing code — it's refusing to trust a report about code until it's been checked against reality.

That's slower than just believing it. But the alternative is quietly shipping three real bugs wrapped in two fabricated facts, and never noticing the difference.

I keep a running experiments log directly in the repository — hypothesis, what was actually tested, the raw result, the conclusion, and separately, what I got wrong the first time. Not because it looks nice, but because without it, a month later I wouldn't remember which conclusions were actually verified and which were just stated confidently by something.

The repo is open source, MIT licensed: https://github.com/ManSio/mscodebase-intelligence

If you've run into something similar — an AI audit or report that turned out to be half true, half invented — I'd be curious to hear how you caught it.

PageRank vs RAG on a Real Codebase: Corrected Numbers, and What I Almost Got Wrong Twice

Mikhail — Wed, 22 Jul 2026 10:00:00 +0000

title: "PageRank vs RAG on a Real Codebase: Corrected Numbers, and What I Almost Got Wrong Twice"
published: true
description: "Second correction to this experiment. The Hit@Gold numbers are now independently verified and reproducible. But my own 'gold standard is 100% valid' claim wasn't — here's the gap between validating a file and validating the file that was actually used."
tags: machinelearning, python, ai, devtools

cover_image:

TL;DR: RAG (BM25) beats PageRank at retrieving the exact file that answers a query — 50% vs 36% Hit@Gold on a dense code graph (n=50). These numbers are now independently reproduced, not just computed once and trusted. But getting here took two rounds of correction: first I found my gold-standard labels were stale, then — after "fixing" them — I found I'd validated a different file than the one the experiment actually used. 4 of 50 labels were still broken in the script that produced the headline numbers, even after I'd published a "100% valid" validation report for a sibling file that nobody was running.

The Setup

Project: MSCodeBase Intelligence (50K LOC Python, 129 files)

Methodology:

Gold Standard: 50 queries → manually curated target file for each
3 Selection Methods: PageRank (varying graph density), Random baseline, RAG (BM25)
Metric: Hit@Gold — did the selection include the exact file that answers the query?
Token budget: ~70K tokens (top 20% of files) for a fair comparison across methods

Tools: NetworkX, tiktoken (cl100k_base), Python AST

This is the second revision of this post. If you read the first correction: the direction hasn't changed, but I'm now more careful about what "verified" actually means.

Round One: The Gold Standard Was 20% Stale

I originally hand-wrote a GOLD dictionary mapping 50 queries to target files. I never checked, before running the experiment, whether those file paths still existed — the codebase had been refactored since I wrote the labels. When I finally checked: 7 of 28 unique target files (25%) pointed at paths that no longer existed, and one query's target (tests/test_search_code.py) was outside the directory my scanner even walked. That's roughly 10 of 50 queries (20%) that were unwinnable by any method, for reasons that had nothing to do with retrieval quality.

I fixed the paths, reran, and got new numbers. I also published a validate_gold.py script and a gold_validation.json reporting 100% of gold paths valid. That felt like closing the loop.

It wasn't.

Round Two: I Validated the Wrong File

Here's the mistake, stated plainly: I created a clean, corrected gold-standard dictionary in experiments/gold_standard.py, and wrote validate_gold.py to check it. It came back 50/50 valid. I took that as confirmation that the experiment was now sound.

But run_experiment_e2e_v2.py — the actual script that produces the Hit@Gold numbers in this post — has its own separate, inline GOLD_STANDARD dictionary, hand-duplicated instead of imported from the "fixed" module. Nobody kept the two in sync. When I diffed them: 11 of 50 entries differ, and 4 of those still point at nonexistent files — the exact same stale paths from round one (src/providers/reranker.py, src/core/intelligence/engine.py), still sitting in the script that actually runs.

So the true state, verified by independently re-running both scripts against the real repo:

	Result
`experiments/gold_standard.py` (validated, unused by the experiment)	50/50 valid (100%)
`run_experiment_e2e_v2.py`'s inline `GOLD_STANDARD` (the one that actually produced the numbers below)	46/50 valid (92%)

I published a green validation report for a file that wasn't wired into anything. That's the same failure pattern I complain about in AI agent output all the time — a status that says "done" without checking it against the artifact that matters. Doing it to myself, in a post about being more rigorous, was a useful reminder that "I wrote a validation script" and "I validated the right thing" are not the same claim.

There's also at least one gold label that passes a file-existence check but is still wrong on the merits: "what tests exist" maps to src/__init__.py — a real file, but not one that answers the question. Path validation catches missing files; it doesn't catch wrong-but-existing ones.

The Numbers (Independently Reproduced)

I re-ran run_experiment_e2e_v2.py myself against a fresh checkout, end to end, rather than trusting the last run's output file. It reproduced exactly:

Dense Graph (imports + class refs + function calls, 388 edges, 128 files)

Method	Hit@Gold	SUFFICIENT	Avg Tokens
RAG (BM25)	50%	25/50	~40,200
PageRank	36%	18/50	~35,900
Random	12%	6/50	~27,000

Sparse Graph (imports only, 110 edges)

Method	Hit@Gold
RAG (BM25)	~50%
PageRank	18%
Random	12%

These are the numbers I'll stand behind — not because they're perfectly clean (4 of 50 gold labels are still wrong, as detailed above), but because I've now actually reproduced them from a fresh run instead of trusting a cached result, and I know precisely which and how many labels are still bad. Real effective sample size for reliable inference: 46/50, not 50/50.

On confidence: a 7-point gap (18 vs 25 hits) on n≈46-50 is a real, repeatable effect in this codebase — it holds across both graph densities and has a mechanistic explanation (below), so I trust the direction. I do not have the sample size to defend "50% vs 36%" to the point, and you shouldn't take the exact percentages more seriously than "RAG is clearly ahead, PageRank is clearly ahead of random, by a moderate but not enormous margin."

Why My Earlier "Keyword Accuracy" Numbers Were Also Misleading

Before Hit@Gold, I measured "keyword accuracy" — does the selected context contain the query keyword anywhere? That produced inflated, near-meaningless numbers: PageRank scored ~78-80% keyword accuracy on both graph densities, while its actual Hit@Gold was 18-36%. Keywords like search, error, lock, sql appear in dozens of files — a random 25-file selection covers most queries by keyword presence alone, which is exactly why the random baseline (12% Hit@Gold) looks so much weaker than its keyword-accuracy score would suggest. Keyword presence tells you nothing about whether you found the file that actually answers the question.

I also previously tested a "Smart Summary" approach (a compressed 2K-token repo overview fed to the LLM). It looked like 90% accuracy on 10 hand-picked easy queries. On the full 50-query set it dropped to 26%. Ten easy queries and fifty real ones are not the same benchmark.

What Graph Density Actually Does

Graph	Edges	PageRank Hit@Gold
Random	0	12%
Import-only	110	18%
Imports + class refs + func calls	388	36%

Denser graphs roughly double PageRank's Hit@Gold. On a sparse import-only graph, PageRank mostly just surfaces the biggest files, which aren't necessarily the most relevant ones for a given query. Adding class-reference and call-graph edges breaks that coupling. Even at its best, though, dense-graph PageRank doesn't catch RAG.

Why RAG Wins (and Why the Comparison Was Never Fully Fair)

RAG (BM25) is query-aware — it scores files against the literal terms in the question. PageRank is query-agnostic — it ranks files by global structural importance once, then returns the same top-N regardless of what's asked.

For "where is DebounceBatch defined": RAG matches the term directly and finds rate_limiter.py. PageRank, working off a precomputed graph, ranks structural hubs like engine.py or runtime_coordinator.py highly, with no mechanism to notice neither one mentions DebounceBatch.

Worth saying directly: comparing a query-agnostic ranking against a query-aware retrieval method and reporting "RAG wins" is a bit like reporting that a road atlas loses to GPS navigation at finding a specific address — true, but not really a fair contest. The more useful question isn't "which wins" (RAG will, structurally, on any query-specific task) but "does PageRank add anything on top of RAG" — an experiment I still haven't run.

Honest Corrections to the Record (Now at Version 2)

Claim	Status	Why
"Top 20% = -2% savings"	Sparse-graph artifact	Density matters a lot
"Smart Summary = 90% accuracy"	26% on the full query set	10 easy queries ≠ 50 real ones
"PageRank doesn't work"	36% Hit@Gold, +24pp over random	Works, modestly
"PageRank beats RAG"	Still false	RAG 50%, PageRank 36%
"My gold standard is 100% valid" (round-one fix)	False when it mattered	Validated a file the experiment didn't use; the used file was 92% valid
Current numbers (50%/36%/18%/12%)	Independently reproduced	Re-run from a fresh checkout, not trusted from a cached file

What This Actually Means

For AI code tools:

Use PageRank as a prior to blend with RAG, not a replacement — a hypothesis based on the mechanism, still unmeasured.
Sparse import-only graphs underserve PageRank; if you use it, build the denser graph.
Validate your gold standard against the file that actually runs the experiment — not a nicely validated sibling copy that nothing imports from.

For anyone benchmarking retrieval on their own codebase:

Don't trust "X% accuracy" without knowing exactly what's measured — keyword-presence accuracy and exact-file Hit@Gold can differ by 40+ points on the same run.
Always include a random baseline.
A green validation script only tells you about the file it checked. If your pipeline has two copies of the same data structure, a passing check on one proves nothing about the other — check that yourself before publishing, because I didn't, twice.
Report sample size next to any percentage. "50%" and "50% (n=50, direction robust, magnitude uncertain)" are different claims.

The Math (Corrected, Reproduced Independently)

Total: 129 files (128 after filtering), ~406K tokens
Top 20% budget: ~25 files

Hit@Gold (E2E metric, dense graph, 388 edges):
  RAG (BM25):   50% (25/50)
  PageRank:     36% (18/50)  ← +24pp over random
  Random:       12% (6/50)

Gold-standard integrity (verified by independent re-run):
  gold_standard.py (validated, unused by experiment): 50/50 valid
  Inline GOLD_STANDARD in run_experiment_e2e_v2.py
  (the one that actually produced these numbers):      46/50 valid

Related Work

Aider uses symbol-level elision — needs a dense graph plus query-aware retrieval to work well.
CodeGraph does on-demand, query-conditioned retrieval — the direction this points toward.
Codebase-Memory reports honest comparative metrics (83% vs. 92%) rather than a single flattering number.

Open Questions

Does PageRank-as-a-prior (blended with BM25 scores) measurably beat RAG alone? Still unmeasured.
Does the RAG advantage hold on a codebase with less descriptive file/function naming? This project's names are unusually aligned with what they implement.
What happens at n=200 queries with proper confidence intervals, and a gold standard that's imported once from a single source of truth instead of copy-pasted?

Reproduce this yourself: scripts in experiments/. If you do: check whether the gold-standard dictionary the experiment script actually imports is the same one your validator checked. It wasn't, for me, and I'd already published a "100% valid" report before I noticed.

Part of my research on MSCodeBase Intelligence — an MCP server for codebase intelligence.

The Silent Vector Contamination Bug: Why Your Concurrent Embeddings Might Be Lying to You

Mikhail — Tue, 21 Jul 2026 22:00:48 +0000

TL;DR: If you run concurrent inference (e.g., via OpenVINO AsyncInferQueue or custom threading) for text/code embeddings, your tests might show 0 exceptions and 0 errors, while silently returning embeddings belonging to other inputs in the batch. Here is how we caught a subtle race condition using a cosine-similarity contamination test.

The Setup

We use OpenVINO with an INT8 quantized E5-small model for in-process code embedding. To maximize throughput on multi-core CPUs, we set up an asynchronous inference queue (AsyncInferQueue) with jobs=4.

In standard unit testing, everything looked pristine:

All infer jobs completed with exit code 0
No None values or zero-filled tensors were returned
Latency and throughput were great (~37 chunks/sec on Ryzen 5600)

However, during end-to-end RAG retrieval tests, we noticed weird semantic anomalies: searching for authentication logic would occasionally return chunks related to database migrations or UI components with unreasonably high confidence.

The Bug: Silent Contamination

The root cause was a subtle race condition in callback/userdata mapping inside the async wrapper.

Because the inputs were processed concurrently across multiple execution streams, a shared user-data context wasn't strictly isolated per inference request. When Request A (auth.py) and Request B (payment.py) were scheduled back-to-back:

Both requests succeeded without throwing exceptions
The output tensor for Request A was mapped to the metadata/chunk wrapper of Request B
The resulting vector was syntactically valid and non-zero, but it represented the wrong text input

Standard assertion tests like assert output_vector is not None or assert output_vector.shape == (384,) passed 100% of the time. The pipeline was silently corrupting the vector store.

The Code

# Before fix: shared results dict across concurrent calls
self._ov_results = {}

def _callback(request, userdata):
    # BUG: userdata is a global index (0, 1, 2, ...)
    # Two concurrent calls reuse the same indices!
    self._ov_results[userdata] = request.get_tensor().data

The problem: userdata was a simple integer counter (0, 1, 2, ...) that reset between embed_batch calls. When two calls overlapped, they wrote to the same dictionary keys.

The Solution: Cosine Contamination Testing

To catch this reliably in CI, we wrote an explicit cross-contamination test designed for concurrent embedding queues.

The Test Logic

import asyncio
import numpy as np
import pytest

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Calculate cosine similarity between two vectors."""
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

@pytest.mark.asyncio
async def test_async_embedder_no_cross_contamination(embedder):
    """Verify that concurrent embedding doesn't cross-contaminate vectors."""

    # 1. Semantically distinct inputs
    samples = {
        "auth": "def authenticate_user(username, password_hash): return verify_jwt(token)",
        "sql": "SELECT u.id, u.email FROM users u JOIN orders o ON u.id = o.user_id WHERE o.status = 'active'",
        "html": "<div class='flex-container'><span id='user-profile'>Profile View</span></div>",
        "rust": "pub fn allocate_buffer(size: usize) -> Result<Vec<u8>, MemoryError> { Vec::with_capacity(size) }"
    }

    # 2. Sequential baseline (ground truth)
    # IMPORTANT: baseline must use clean synchronous infer(), NOT AsyncInferQueue
    # even with jobs=1, to avoid contamination in the baseline itself
    baseline_vectors = {}
    for key, text in samples.items():
        baseline_vectors[key] = await embedder.embed_single_sync(text)

    # 3. High-concurrency stress test with randomized queue order
    async_tasks = []
    keys_order = list(samples.keys()) * 10  # 40 concurrent requests
    np.random.shuffle(keys_order)

    for key in keys_order:
        async_tasks.append(embedder.embed_async(samples[key]))

    async_results = await asyncio.gather(*async_tasks)

    # 4. Verify identity & cross-isolation via Cosine Similarity
    for expected_key, async_vec in zip(keys_order, async_results):
        # Self-similarity against ground truth must be ~1.0
        self_sim = cosine_similarity(async_vec, baseline_vectors[expected_key])
        assert self_sim > 0.98, (
            f"Contamination detected! Vector for '{expected_key}' drifted "
            f"(sim={self_sim:.2f}, expected >0.98)"
        )

        # Cross-similarity against distinct inputs must remain low
        for other_key, other_vec in baseline_vectors.items():
            if other_key != expected_key:
                cross_sim = cosine_similarity(async_vec, other_vec)
                assert cross_sim < 0.6, (
                    f"Cross-talk detected between '{expected_key}' and "
                    f"'{other_key}' (sim={cross_sim:.2f}, expected <0.6)"
                )

Benchmark Results

Running this test on the unpatched queue revealed the contamination:

Metric	Unpatched	Patched	Expected
Self-similarity (auth↔auth)	0.34 ❌	0.99 ✅	>0.98
Cross-similarity (auth↔sql)	0.98 ❌	0.32 ✅	<0.6
Exceptions thrown	0	0	0
Zero tensors	0	0	0

The unpatched queue showed 0 exceptions while vectors were completely swapped.

The Fix

The fix was simple once we understood the problem:

# After fix: isolated results per call
self._ov_results = {}

def _callback(request, userdata):
    # FIX: userdata is now (index, local_results_dict)
    # Each embed_batch call creates its own dict
    index, local_dict = userdata
    local_dict[index] = request.get_tensor().data

Since OpenVINO's Python API allows passing any Python object as userdata (unlike C++ where it's typically void*), we can bundle the index and a call-specific dictionary together. Each embed_batch call creates its own isolated dictionary. The callback writes to the call-specific dict, not a shared global. No locks needed — complete isolation by design.

Key Takeaways

0 exceptions ≠ Correctness. Silent data corruption doesn't throw errors.
Valid shape ≠ Valid embedding. A (384,) tensor with non-zero floats can represent the wrong input.
Write cross-contamination tests. If you use async inference queues or multi-threading for vector generation, verify that Vector X actually belongs to Input X under concurrent load.
Cosine similarity is your friend. A simple similarity check between concurrent outputs and sequential baselines catches contamination that no other test detects.
This applies to ONNX Runtime and TensorRT too. If you share state across requests in onnxruntime.InferenceSession.Run() or TensorRT async wrappers, the same silent contamination can occur. The fix is the same: isolate output containers per request.

How to Run This Test

# Clone the repo
git clone https://github.com/ManSio/mscodebase-intelligence.git
cd mscodebase-intelligence

# Install dependencies
pip install -e ".[dev]"

# Run the contamination test
pytest tests/test_ov_concurrent_embed.py -v

Discussion

Has anyone else bumped into silent cross-talk in ONNX Runtime, OpenVINO, or TensorRT async queues? How do you validate thread isolation in your embedding pipelines?

Built with MSCodeBase Intelligence — an MCP server for codebase intelligence with incident memory and root cause prediction.

DEV Community: Mikhail

We Didn't Invent New AI Architecture. We Rediscovered 20-Year-Old Computer Science.

1. "Cryptographically Perfect Hallucinations" & The Oracle Problem

2. The Empty Set Trap & "Capture-Recapture"

3. "Veto Heartbeats" & Durable Execution

4. RetractionReceipts & Event Sourcing

The Meta-Lesson

A Note of Thanks (and a Disclaimer)

The Dataset Was Lying: 4 of 6 "False" Facts Were True

Part 1: The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?

Part 2: Your memory layer is lying to you (and your LLM agrees)

Part 3: The Dataset Was Lying: 4 of 6 "False" Facts Were True

The Evidence Ladder

Rungs 1→2: evidence format beats model

Rung 3: graph closes the trap. Or so we thought.

The attack: the dataset was lying

Corrected matrix, pinned re-run (routing eliminated)

But the trap is real (extended category, subject-validated)

Re-scoring the historical runs (3300+ calls, no re-billing)

Temporal: "was true then" vs "true now"

Determinism: pin the provider (thank you, comment section)

What this means for verify-on-read

The meta-lesson

Reproduce

Went down a rabbit hole testing whether LLMs can actually verify agent memory claims on their own. Turns out the evidence format matters way more than the model — bare token strings barely work, real code context changes everything.

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models)

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models)

The setup

Day 1: live models vs. proxy

Day 2: six cheap models, two full passes, red team

The sycophancy problem (V1 → V2)

--no-reasoning is mandatory

temp=0 + seed=42 is not determinism on OpenRouter

Prompt language is model-specific

Premium models

Day 3: per-category breakdown reveals the real problem

Per-category results (zero-shot, V2, code_first)

CoT doesn't fix it

Day 4: file_content_first — the answer

Results

What this means for VOR architecture

Full results summary

Known weaknesses

What to use

Reproduce it

The Dataset Was Lying: 4 of 6 "False" Facts Were True

I turned my portfolio into an MCP server (and I'm not a programmer)

I turned my portfolio into an MCP server (and I'm not a programmer)

The problem I was solving

What I actually built

The nine tools

The simulator

What broke (the honest part)

One thing I didn't expect

Try it

The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?

The Experiment: Memory Contamination

The Setup

The Initial Results

The First Fix: Testing a Retraction Lifecycle

The Retraction Results

The Honest Limitation: Why It Didn't Drop to Zero

Closing the Gap: Verify-On-Read

The Final Results

The Remaining Honest Limitations

Conclusion

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models

What I learned building a long-lived AI agent (the boring version)

What it turned into

An agent is not an LLM call

The cache thing

Why it was different in the IDE

"90% token savings"

Routing is harder than it looks

The latency numbers

Telemetry became the most important thing

A lesson about noisy telemetry

Self-learning is a trap if your data is bad

Context compression

Batch processing

`--no-reasoning` is mandatory