Part 1: The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?
Part 2: Your memory layer is lying to you (and your LLM agrees)
Part 3: The Dataset Was Lying: 4 of 6 "False" Facts Were True
The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — useful, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3: "headline numbers were a property of the heuristic, not LLM behavior."
So we ran it with live models. Four days, 14 models, 4 arms, ~3400 API calls. The last arm answered the question the whole experiment was actually asking.
The setup
Dataset: memory_contamination_facts_v4_rep.json, N=50 (R01–R50), sha256 820bbbf60a0fc930.
| kind | n | what it tests |
|---|---|---|
real |
25 | TRUE — grep-validated against actual code |
absent-mutation |
16 | FALSE — component doesn't exist in the project |
present-trap |
6 | FALSE — file exists, claim is about the wrong subject |
silent |
3 | FALSE — external systems the codebase never mentions |
Four arms, run sequentially as the experiment evolved:
- memory_first — model sees only the claim. Does it trust memory without evidence?
-
code_first — claim +
support_patterns(bare token strings) +section. Does it evaluate the anchors? -
code_first CoT — same as above,
reasoning=true, max_tokens=1500. - file_content_first — claim + real 25-line file window around the anchor. Can the model verify with actual evidence?
Model verdict: {"verdict": "true"|"false"|"unknown"}, JSON-only. Zero-shot: max_tokens=100, temp=0, seed=42, --no-reasoning. Leak-guard: assert "truth" not in prompt on every fact, unit-tested (39 harness tests, 1226 total).
Two metrics that both matter:
-
false_accept (FA) — fraction of false claims returned as
"true". Contamination risk. - recall(real) — fraction of 25 TRUE facts correctly confirmed. A gate that rejects everything has FA=0.00 and recall=0.00. It's a no-op.
Day 1: live models vs. proxy
The proxy always decided — unknown=0 by construction. First live run (opencode/deepseek-v4-flash-free, free tier) returned unknown=0.80. FA=0.10, including R31 in both arms.
The reviewer's criticism was right immediately.
Day 2: six cheap models, two full passes, red team
The sycophancy problem (V1 → V2)
R31 was false-accepted by every model in the first sweep:
claim: "The instruction scanner uses Typesense"
truth: False
support_patterns: ["typesense"]
contra: [file:src/core/instruction_scan.py]
instruction_scan.py uses only stdlib (re, logging, typing). Typesense is nowhere in the project. Nine false facts in R26–R50 shared this pattern (vespa, pinecone, typesense, tantivy, meilisearch, dataclasses, logging, pathlib, loki) — bare token in support_patterns, model treats field presence as evidence.
V1 prompt asked: "Does the claim appear supported by these anchors?" — leading question. Confirmed sycophancy (Sharma et al. 2023, arXiv 2310.13548).
# V1 — leading:
Does the claim appear supported by these anchors?
# V2 — neutral:
Return true ONLY if the anchors directly verify the claim;
false if the anchors contradict it or the claim refers to something
absent from the anchors; unknown if you cannot determine.
V2 reduced FA in 4/6 models. glm-4.7-flash: 0.30 → 0.24. Still not safe.
--no-reasoning is mandatory
Without it, GLM-4.7-flash uses 110 reasoning tokens and returns content=None, finish=length — 100% unusable. With --no-reasoning: reasoning_tokens=0 confirmed for all models via harness audit and independently via OpenRouter server CSV.
temp=0 + seed=42 is not determinism on OpenRouter
nemotron-3.5-lightning code_first: FA 0.18 → 0.08 between two identical runs. ±0.10 on a single-pass measurement. OpenRouter routes to ≥8 upstream backends (server CSV: Alibaba 1955 calls, DeepInfra 559, DigitalOcean 531, Cloudflare 283, Novita 253, Baidu 114, StreamLake 110, Amazon Bedrock 100) — same prompt, different quantizations. qwen3.6/3.7/deepseek stable 3/3. GLM not.
All final conclusions use upper-bound-of-two-runs.
Prompt language is model-specific
deepseek-v4-flash code_first unknown:
EN: 0.94 → RU: 0.54 (RU → commits more)
qwen3.7-flash code_first unknown:
EN: 0.24 → RU: 0.58 (RU → hedges more)
Same dataset, same arm, opposite direction per model. Measure per-model before assuming.
[TODO: verify whether claim language interacts with prompt language separately — all claims in this dataset are in Russian]
Premium models
- claude-sonnet-5: FA=0.00/0.00, acc=1.0 on all decided. But unknown=0.86/0.70 and $0.049/100 calls.
- qwen3.8-max: HTTP 400 on 22–49/50 calls — "Reasoning is mandatory and cannot be disabled." Not a harness bug.
- glm-5.2: FA=0.00/0.02 — dramatically better than glm-4.7-flash. Same family, new generation.
- deepseek-v4-pro: FA=0.04/0.00.
Premium gives no FA advantage over the best flash models. The cost difference is 10–100×.
Day 3: per-category breakdown reveals the real problem
A reviewer asked: "High unknown ≠ high quality. What was the true-accept rate on the 25 real facts? Is qwen3.6-flash cutting true memory along with false?"
We added scripts/summarize_1L_categories.py and ran it on all V2-EN progress files.
Per-category results (zero-shot, V2, code_first)
| model | real: acc/rej/unk | recall(real) | F1 | FA absent·trap·silent |
|---|---|---|---|---|
| qwen3.6-flash | 2 / 7 / 16 | 0.08 | 0.15 | 0·0·0 |
| qwen3.7-flash | 5 / 9 / 11 | 0.20 | 0.33 | 0·0·0 |
| deepseek-v4-flash | 1 / 0 / 24 | 0.04 | 0.08 | 0·0·0 |
| claude-sonnet-5 | 11 / 0 / 14 | 0.44 | 0.61 | 0·0·0 |
| nemotron-3.5-lightning | 13 / 3 / 9 | 0.52 | 0.65 | 0·1·1 |
| glm-4.7-flash ⚠️ | 22 / 1 / 2 | 0.88 | 0.75 | 7·3·2 |
| nemotron-3-nano 🔴 | 20 / 1 / 4 | 0.80 | 0.62 | 11·5·3 |
qwen3.6-flash code_first: confirmed 2/25 true facts, actively marked 7/25 as false (wrong), abstained on 16/25. FA=0.00 achieved by not functioning as a verifier. Auto-retraction on these verdicts empties the memory layer, not just the false claims.
The tradeoff across the sweep:
qwen3.6-flash F1=0.15 FA=0.00 ← fail-closed
qwen3.7-flash F1=0.33 FA=0.00 ← fail-closed
claude-sonnet-5 F1=0.61 FA=0.00 ← best at zero-FA
nemotron-3.5 F1=0.65 FA=0.04 ← best F1 overall
glm-4.7-flash F1=0.75 FA=0.24 ← best coverage, unsafe
No model in zero-shot balanced both. FA=0.00 is a policy, not a quality metric.
CoT doesn't fix it
We ran --reasoning --max-tokens 1500 on 4 models, two full passes for stability.
| model | arm | recall ZS→CoT | FA ZS→CoT | cost/100 ZS→CoT |
|---|---|---|---|---|
| qwen3.6-flash | code_first | 0.08→0.20 | 0→0 | $0.0006→$0.045 |
| qwen3.7-flash | code_first | 0.20→0.16 | 0→0 | $0.0005→$0.005 |
| glm-4.7-flash | code_first | 0.88→0.72 | 12→7 | $0.001→$0.017 |
| deepseek-flash | code_first | 0.04→0.08 | 0→0 | $0.0009→$0.002 |
Only meaningful gain: qwen3.6 code_first recall 0.08→0.20, FA held at 0.00. Cost ×75. qwen3.7 and deepseek: within CI noise. glm-4.7: FA improves but 16–26% of responses are EMPTY_CONTENT (upstream defect in reasoning mode) — numbers are qualitative only.
Important confounder: CoT arm changes two things simultaneously — reasoning=true AND max_tokens=1500. Separating them is impossible: qwen3.6 uses 700–1500 reasoning tokens per call.
qwen3.8-max CoT (mandatory reasoning, the only mode it runs): recall=0.36 code_first, FA=0.04 — best recall-at-low-FA in the zero-shot/CoT sweep. Cost $0.10/100 calls.
Day 4: file_content_first — the answer
The reviewer's point after Day 3: "Is recall=0.08 model paranoia, or are the pattern strings too narrow to validate anything?"
New arm: instead of support_patterns: ["typesense"], the model gets a real 25-line window from the file around the first occurrence of the anchor. For absent and silent facts (grep returns nothing), a decoy: the head of src/core/instruction_scan.py. The decoy is not labeled in the prompt — labeling it would leak ground truth ("not found" → trivial false). It's marked in results as evidence: decoy for post-hoc analysis. Prompt instruction identical to code_first V2 (neutral). Single variable changed vs code_first: form of evidence (token string → code).
Results
| model | arm | recall(real) | FA | real: acc/rej/unk | F1 | FA absent·trap·silent |
|---|---|---|---|---|---|---|
| qwen3.6-flash | code_first (baseline) | 0.08 | 0.00 | 2/7/16 | 0.15 | 0·0·0 |
| qwen3.6-flash | file_content_first | 0.88 | 0.04 | 22/3/0 | 0.90 | 0·2·0 |
| qwen3.7-flash | code_first (baseline) | 0.20 | 0.00 | 5/9/11 | 0.33 | 0·0·0 |
| qwen3.7-flash | file_content_first | 0.88 | 0.02 | 22/2/1 | 0.92 | 0·1·0 |
recall(real): qwen3.6 0.08 → 0.88 (×11). qwen3.7 0.20 → 0.88 (×4.4). Wilson CI for 22/25: [0.70, 0.96] — doesn't overlap with baseline CI for 2/25: [0.02, 0.25].
The model wasn't being paranoid. It had no evidence.
FA: 0.00 → 0.02–0.04. Every false accept is present-trap — the anchor token appears in the file fragment, but the claim is about the wrong subject:
- R45: "The server wrapper uses logging" — fragment is from
log_manager.py, tokenloggingis there, but the subject is wrong. - R46 (qwen3.6 only): "The watchdog uses threading" — fragment from
project_indexer_registry.py.
absent 0/16, silent 0/3 for both models. The decoy policy works: a fragment without the token returns false or unknown without leaking.
Remaining false rejects on real facts: R07/R08 (both models) — claims with semantically bad value field ("uses disabled", "uses forbidden" — adjective instead of noun; fragment shows the env-var but not the adjective); R21 (qwen3.6), R03 (qwen3.7) — individual confidence threshold differences.
What this means for VOR architecture
The problem was always the evidence layer, not the model.
Showing a model support_patterns: ["typesense"] and asking it to verify a claim is not verification — it's asking the model to confirm the field value matches the field name. The model correctly doesn't know what to do with it, so it abstains or guesses.
Showing the model 25 lines of actual code around the anchor: recall 0.88, FA 0.02–0.04, $0.005/100 calls. The remaining failure mode is subject-mismatch in present-trap facts — the model checks token presence, not claim subject identity. That's a harder problem but a much smaller one.
Cost of adding a file window: ~200 extra prompt tokens per fact. At flash pricing, that's $0.0001–0.0004/fact. Negligible against the cost of retracting true memory.
Full results summary
| model | arm | recall(real) | FA | F1 | $/100 | notes |
|---|---|---|---|---|---|---|
| qwen3.6-flash | file_content | 0.88 | 0.04 | 0.90 | $0.005 | recommended |
| qwen3.7-flash | file_content | 0.88 | 0.02 | 0.92 | $0.005 | recommended |
| qwen3.8-max | CoT | 0.36 | 0.04 | — | $0.102 | if file access unavailable |
| nemotron-3.5 | zero-shot | 0.52 | 0.04 | 0.65 | $0.001 | if file access unavailable |
| claude-sonnet-5 | zero-shot | 0.44 | 0.00 | 0.61 | $0.049 | if FA=0.00 required |
| qwen3.7-flash | zero-shot | 0.20 | 0.00 | 0.33 | $0.0005 | fail-closed only |
| qwen3.6-flash | zero-shot | 0.08 | 0.00 | 0.15 | $0.0006 | fail-closed only |
| glm-4.7-flash ⚠️ | zero-shot | 0.88 | 0.24 | 0.75 | $0.001 | exclude |
| nemotron-3-nano 🔴 | zero-shot | 0.80 | 0.38 | 0.62 | $0.0008 | exclude |
Known weaknesses
Five limitations, stated before a reviewer raises them.
1. Upstream routing variance (partially mitigated). temp=0 + seed=42 on OpenRouter is not true determinism — 8+ backends. Mitigated by 2–3 passes per condition, upper-bound-of-runs. Not mitigated: K≥3 repeats per fact/model pair (~4200 calls, ~$2–5) — not done.
2. Anchor bias vs. snippet truncation (closed). V4 (§6.6b) confirmed anchor bias: recall 0.08→0.88 with real file content. The residual failure in present-trap (R45, R46) is subject-identity checking, not truncation.
3. Synthetic mutations vs. real drift (in progress). The 25 false facts are synthetic (absent/trap/silent), not real git-refactoring drift. 30-day longitudinal protocol is running — data not yet collected.
4. Single codebase, single language (not closed). All 50 facts from one Python project. TypeScript/Rust/Go have harder type anchors — transfer is extrapolation. Multi-repo extension needs ~$0.4–3.6 and 1–2 days of grep-validation.
5. Three-way verdict vs. confidence score (not closed). The true/false/unknown frame may push "partially stale" claims into unknown. A 5-point confidence scale would need a new prompt, parser, and graded dataset. What we can say: finish_reason=stop everywhere in zero-shot — high unknown is genuine uncertainty, not truncation.
What to use
For a working VOR gate: qwen3.6-flash or qwen3.7-flash with file_content_first. recall=0.88, FA=0.02–0.04, $0.005/100 calls. The only failure mode is present-trap — the model sees the right token in the wrong file context. Acceptable for flag-and-review; not yet for fully autonomous retraction.
If file access is unavailable at verdict time: qwen3.8-max CoT (recall=0.36, FA=0.04, mandatory reasoning, $0.10/100). Better than any zero-shot option on recall. Or nemotron-3.5-lightning (recall=0.52, FA=0.04, $0.001/100) as the cheapest zero-shot model with reasonable F1.
Fail-closed (contamination-intolerant, recall doesn't matter): qwen3.6/3.7 zero-shot. FA=0.00 confirmed 0/400 code_first verdicts across 4 runs. Cheapest. Expect to retract ~80–92% of true memory alongside false.
Exclude: glm-4.7-flash (FA=0.24), nemotron-3-nano (FA=0.38).
Measure on your own dataset before deploying. These numbers are from one Python codebase.
Reproduce it
git clone <repo> mscodebase && cd mscodebase
# Windows
python -m venv venv && venv\Scripts\python -m pip install -e .
# macOS/Linux
python -m venv venv && venv/bin/pip install -e .
# .env: OPENROUTER_API_KEY=sk-or-v1-...
# integrity check
python scripts/run_1L_live_arm.py --arm both --dry-run
# canonical flash sweep, V2, ~600 calls, ~$0.009
python scripts/run_1L_live_arm.py \
--provider openrouter --arm both \
--models "qwen/qwen3.7-flash,qwen/qwen3.6-flash,qwen/qwen3.5-flash-02-23,\
deepseek/deepseek-v4-flash,z-ai/glm-4.7-flash,nvidia/nemotron-3.5-lightning" \
--prompt-version v2 --no-reasoning --tag v2_en
# per-category breakdown
python scripts/summarize_1L_categories.py --tag v2_en
# V4: real file snippets, ~100 calls, ~$0.005
python scripts/run_1L_live_arm.py \
--provider openrouter --arm file_content_first \
--models "qwen/qwen3.6-flash,qwen/qwen3.7-flash" \
--prompt-version v2 --no-reasoning --tag file_content
# CoT arm, ~400 calls, ~$0.20
python scripts/run_1L_live_arm.py \
--provider openrouter --arm both \
--models "qwen/qwen3.6-flash,qwen/qwen3.7-flash,z-ai/glm-4.7-flash,deepseek/deepseek-v4-flash" \
--prompt-version v2 --reasoning --max-tokens 1500 --tag v3_cot
# second pass for any arm (variance check)
python scripts/run_1L_live_arm.py [same args] --force
Dataset fingerprint: 820bbbf60a0fc930. Full report: experiments/exp_1L_live_arm_report.md. Tests: tests/test_run_1L_live_arm.py (39), tests/test_summarize_1L_categories.py (8).
Source: github.com/ManSio · Portfolio: mansio.github.io/MSPortfolio
A note on how this was written.
Every experiment, bug, failure, and idea here is mine — I earned them the hard way, in production, in public. AI worked as my editor: it helped me structure thoughts and polish my English. It did not invent the facts, because it has none of its own.
No AI detectors were consulted in the making of this disclosure. They have enough trouble agreeing on what I am.
Top comments (4)
Your limitations section already closes the objection I arrived with, so here is something that might generalise further than the plus or minus 0.10 you measured.
We hit the same OpenRouter routing effect on a tool-calling benchmark, and what surprised us was how unevenly it landed across models. Same 400 prompts, same scorer, the only change being whether the endpoint was pinned.
On an identical request, Groq emitted the correct tool call and DeepInfra emitted none. Two backends, two behaviours, one prompt.
The reason I think it bears on your table. You measured the band on nemotron-3.5-lightning and mitigated with 2 to 3 passes and upper-bound-of-runs, which is already more care than most people take. If the size of the effect belongs to the model-and-provider pair, then that band may not carry across to the other 13, and the widest-swinging pairs are exactly the ones a cross-model ranking would misplace.
Your headline looks safe to me either way. A jump from 0.08 to 0.88 sits an order of magnitude past any routing swing we have seen, so the arm comparison holds. I would hold the model-to-model gaps more loosely than the arms.
Weigh ours accordingly. BFCL-shaped synthetic tool-calling prompts on our own harness, not customer traffic, n=400 per arm, one run per arm, no confidence intervals computed, and we have only watched it on two pairs. Read it as a shape whose magnitude will differ in your setup.
One practical note in case it helps. OpenRouter accepts provider.order with allow_fallbacks false, which pins the endpoint and takes the variable out entirely. Much cheaper than the K greater than or equal to 3 repeats you costed at around 4200 calls.
By the way, we accidentally stumbled onto something similar in a parallel experiment (memory-verification harness) – using OpenRouter's server CSV, we saw the same prompt landing on 8+ different upstream providers (Alibaba, DeepInfra, DigitalOcean, Cloudflare, Novita, Baidu, StreamLake, Amazon Bedrock).
We also tried your advice with provider.order / allow_fallbacks: false – it works, the parameter really does lock the backend (all responses confirm provider: "Alibaba", verdicts stable 3/3). But here's a concrete data point: one of our models (qwen3.7) was already sitting on a single upstream 100% of the time even without pinning – 248/248 calls on Alibaba, in our own server audit. Meanwhile two other models in the same run were split across 6 different backends each (StreamLake, DeepInfra, Cloudflare, Venice, Baidu, Novita). The single-backend model was also the only one that stayed perfectly stable run-to-run; the multi-backend ones weren't.
So it looks like the ±0.10 spread tracks routing diversity, not the model itself – which might explain why your llama-3.3-70b had such a big swing (17 p.p.) while gpt-oss-120b was near zero: if gpt-oss also happens to sit on one backend by default, pinning wouldn't move the needle much for it.
We've left K≥3 repetition as an open item for now (~4200 calls, $2-5 estimated, not run yet). If you've already got a solid way to measure this per-pair, we'd rather look at that than reinvent it.
The 248/248-on-Alibaba data point is a good one, and it lines up with the swing
Tom measured. One thing I would hold loosely about it though: backend distribution
is itself a moving target. Providers get added, capacity shifts, prices change,
one goes down for an afternoon. So "qwen3.7 sat on a single upstream" reads to me
as an observation about a window rather than a property of the model - the same
kind of statement as a latency number, good until the routing table changes.
Which cuts in a useful direction, I think: if you cannot tell in advance which
pairs are single-backend, and the single-backend property can lapse without any
signal, then pinning everything with
provider.order+allow_fallbacks: falseis cheaper than working out which models need it. It costs one config field. The
audit that establishes a model does not need pinning costs more than just pinning
it.
The place it would still be worth measuring per-pair is the direction Tom's table
points at - pinned-vs-unpinned as a quality question, not just a variance one.
His llama-3.3-70b row is 95.5 vs 78.2, and that is not noise around a mean, that is
one backend emitting the correct tool call and another emitting none. If that also
holds on verification verdicts, then unpinned numbers in a cross-model ranking are
not merely wide, they are measuring a different system per row.
The
file_content_firstarm is the result I will be quoting, and the Wilson CIsnot overlapping makes it hard to argue with. Two questions about the arm itself,
both cheap to settle, and I am asking because I think the answers might make the
result stronger rather than weaker.
1. Is the decoy neutral, or is it the counter-evidence?
For
absentandsilentfacts the model gets the head ofsrc/core/instruction_scan.py. That is also the file listed incontrafor R31,and you describe it as stdlib-only -
re,logging,typing. So for all 19of those facts the model receives the one fragment that most cleanly refutes the
claim, and the result is a clean 0/16 and 0/3.
At verdict time in a real system there is no such fragment. When grep returns
nothing you have either no window at all, or a window from some unrelated file -
and an unrelated file does not show that Typesense is absent from the project,
only that it is absent from that file. Those feel like different questions to
me, and the second one is the one production asks.
Would a decoy drawn per-fact from a random other file (a test helper, a config
module - something with no bearing on the claim) still give 0/19? That is 19
facts across two models, so somewhere under 40 calls. If it holds, the decoy
policy is not doing the work and the result gets cleaner. If it drops, you have
found the boundary of the arm before a reviewer does.
2. Could the present-trap failures be the anchor selection rather than the
model?
The window is centred on the first occurrence of the anchor. For R45 the anchor
is
loggingand for R46 it isthreading- tokens that appear in a lot of placesin any Python project. R45 claims something about "the server wrapper", and the
fragment the model got came from
log_manager.py. So the model never saw theserver wrapper at all.
That makes me wonder whether "the model checks token presence, not claim subject
identity" is the right reading. From where I sit it could equally be "the anchor
search ignores the subject" - which would be a much smaller problem, because the
file choice is deterministic and does not involve the model at all. If the window
came from the file the claim is about, R45 becomes a straightforward
verification: is
loggingin the server wrapper? No. False.Six present-trap facts, two models, twelve calls.
Where this bites us
We ran into the same shape from the other end last week, on retrieval rather than
verification. Our benchmark scored two ranking formulas against 17 hand-written
fixtures and put them in a clear order: 92.3% vs 69.2% precision@1. We then rebuilt
the same benchmark against 499 real records from the running system, and the order
reversed - 15% vs 30%. The formula the fixtures preferred was the worse one, and
we had been shipping against that verdict for four months.
The mechanism was the same one you name: the evidence layer, not the model. The
old benchmark carried its own copy of the scoring maths, so it was in effect
grading itself. What made it survive was that it was green - and a green check
over a fixture is very hard to distrust.
Your "FA=0.00 is a policy, not a quality metric" line is the sentence I wish we
had had on the wall for those four months.