The Dataset Was Lying: 4 of 6 "False" Facts Were True

---

Part 1: The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?

Part 2: Your memory layer is lying to you (and your LLM agrees)

Part 3: The Dataset Was Lying: 4 of 6 "False" Facts Were True

---

A model rejected a claim we believed was false. Then we grepped the file. The claim was true — our dataset had labeled it wrong.

That one grep forced us to recompute the entire experiment: 4 of 6 "false" trap facts were actually true, and correcting the labels inverted every conclusion. Digging further exposed a second, real failure mode (the temporal present-trap) and a third layer of attack (provider routing). Every layer of an evaluation pipeline can lie — ground truth, evidence, model interpretation, temporal semantics, execution. This post is the full red-team. (~1900 calls, < $0.10.)

The Evidence Ladder

One dataset (50 facts), one prompt skeleton, one variable — the form of evidence. Three models: qwen3.7-flash, deepseek-v4-flash, glm-4.7-flash.

Rung	Evidence form
1	bare anchor strings
2	25 lines of the real file around the anchor
3	serialized structure: definitions, imports, callers/callees, occurrence lists
3b	file fragment + structure (the "why not both?" arm)
4	structure + git provenance

Verdict schema: {"verdict": "true"|"false"|"unknown"}, temp=0, seed=42, zero-shot, leak-guarded.

Rungs 1→2: evidence format beats model

qwen3.7:    recall(real) 0.24 → 0.92   FA 0.00 → 0.02 (old labels; that 0.02 was a mislabeled true fact — corrected FA = 0, see below)
deepseek:   recall(real) 0.04 → 0.84
glm-4.7:    recall(real) 0.60 → 0.68

Token strings are not evidence. Code is. (Reproduced on the corrected dataset, pinned: qwen 0.88.)

Rung 3: graph closes the trap. Or so we thought.

Graph evidence gave qwen3.7 FA = 0.000, including zero false accepts on the trap category. We wrote the pre-registered interpretation: "structural layer closes the present-trap failure mode." Then rung 3b, the hybrid, reopened the trap (FA 0.02, the same fact as before — which, as the red team below shows, was actually true: that "false accept" was a label artifact, not a model failure). Not additive: fragment presence dominates graph structure. For qwen, "both" is strictly worse than "fragment only".

The attack: the dataset was lying

Red-team checklist, item 1: attack the ground truth, not the model. We grepped the six "present-trap" facts — claims like "The server wrapper uses logging", labeled false because the mutation generator replaced the real value with a stdlib import that exists somewhere in the project.

Fact	Claim (labeled FALSE)	Reality
R43	"The knowledge graph uses re"	graph.py:31: import re + 2 usages → TRUE
R45	"The server wrapper uses logging"	server.py:14: import logging → TRUE
R46	"The watchdog uses threading"	watchdog.py + threading.Lock() → TRUE
R47	"Hub model loading uses pathlib"	6 occurrences → TRUE
R44	"Cross-project search uses pathlib"	imported, never used → ambiguous (excluded)
R42	"The server wrapper uses dataclasses"	0 occurrences → correctly FALSE

4 of 6 "traps" were true. The generator validated value != real_value but never checked the value was absent from the subject. Those were not false accepts — they were correct verdicts against incorrect labels. We created a corrected copy (29 true / 20 false / 1 ambiguous, new fingerprint) and kept the original untouched as a historical artifact.

Honest gap: we corrected the **labels, not the generator that produced them. The original generator still checks value != real_value instead of subject-scoped absence — the corrected dataset is a re-label, and our process rule now requires subject-file grep validation for any synthetic category.

Corrected matrix, pinned re-run (routing eliminated)

arm	qwen rec/FA-tr/miss	deepseek rec/FA-tr/miss	glm rec/FA-tr/miss
file_content	0.88/0/3	0.80/1/2	0.68/2/1
graph_first	0.72/0/4	0.48/0/2	0.84/1/0
file_graph	0.84/0/3	0.92/1/2	0.80/2/1

Columns: recall — true claims correctly accepted / 25 real; FA-trap — false accepts on the trap category (only R42 is genuinely false after relabeling); miss_true — true trap claims wrongly rejected (hidden recall loss, invisible under the old labels).

FA trap counts only R42 (the one genuinely false trap claim). For context, Part 2's headline conclusions were: graph evidence closes the present-trap; qwen3.7 is the safe choice (FA 0.00 zero-shot, recall 0.88 with file content); glm-4.7 is dangerously fail-open (FA 0.24 zero-shot). Inverted conclusions:

1. "Graph closes the present-trap" is an artifact. qwen's graph arm didn't filter false claims — it rejected all four true trap claims (miss_true 4/5, hidden recall loss invisible under the old labels).
2. glm-4.7, which Part 2 told us to exclude as fail-open, is the best structural verifier in the series: recall 0.84, FA trap 1, miss_true 0 on graph evidence.
3. The best evidence format is model-specific: fragment for qwen, graph for glm, hybrid for deepseek. No global winner.

But the trap is real (extended category, subject-validated)

One false trap fact (R42) is statistically meaningless, so we extended the category with a fixed generator that validates against the subject file: false-trap = value present in the project (≥2 files) but absent from the subject file (grep = 0). 20 false / 10 true facts, pinned run:

arm	qwen FA/rec	deepseek FA/rec	glm FA/rec
file_content	2/20, 2/10	15/20, 6/10	13/20, 7/10
graph_first	2/20, 3/10	8/20, 5/10	14/20, 9/10

The present-trap is NOT a label artifact: on honest labels, file_content false-accepts 10–75% of "X uses Y" claims (deepseek 15/20, glm 13/20). And graph evidence halves deepseek's false-accept rate (15/20 → 8/20) — the "graph doesn't close the trap" conclusion from the original dataset (one false trap fact) was itself a small-sample artifact. It just doesn't help glm (14/20), and qwen was already at 2/20 (paying with recall 2/10).

Re-scoring the historical runs (3300+ calls, no re-billing)

We taught the summary tool to recompute metrics from old progress files + corrected truth (verdict-by-id, manually audited — zero field drift). The real picture:

• True trap-FA (R42): 0 for every model. The "present-trap FA 0.02–0.04" in Part 2 was mislabeled data — models were right.
• Hidden trap miss_true: qwen/deepseek 4/5 — fail-closed models rejected true usage claims. This loss was invisible in the old metrics.
• Real fail-open is absent/silent: glm code_first 7+2.

Temporal: "was true then" vs "true now"

The present-trap wasn't only about wrong labels. Digging deeper exposed a harder problem: models cannot distinguish "X exists now" from "X existed then" — unless the question itself carries the tense.

We built a temporal dataset from git archaeology (48 facts: 12 symbols removed after commit C / 28 current / 8 never-existed, ground truth from git show C~1).

E4 (git provenance in evidence): qwen3.7 43/48, deepseek/glm 48/48. Seemed like provenance worked — 2/3 models perfect.

E4b (blind control, no git strings): all three models 48/48. Git provenance was not just unnecessary — it hurt qwen ("existed until C" suggests existence, a token-presence trap in the evidence).

E4c (duo design, no hints): one neutral evidence block (HEAD state + "SYMBOLS in F at history" from git show C~1), two questions:

NOW  ("X is defined in F"):   removed FA: qwen 12/12, glm 12/12, deepseek 9/12
PAST ("X WAS defined in F"):  all three 40/40

Same evidence block for both questions — the only change is the tense of the claim. The 40/40 does not mean the models got better at temporal reasoning; it means they answer the question you actually asked. The model did not need better memory — it needed a temporally explicit question.

All three models exhibited the temporal present-trap. When the evidence mentions X in history and the question is about the present, every model says "true" (12/12, 9/12, 12/12) — a model cannot distinguish "X appears in the evidence" from "X exists now". But phrasing the question in the past tense solves it completely (40/40). The E4b conclusion ("qwen is fragile, deepseek/glm are robust") was itself an artifact of the "NOT FOUND AT HEAD" hint — without it, nobody is robust.

Determinism: pin the provider (thank you, comment section)

Before trusting the corrected results, we attacked the execution layer too.

Part 2's known weakness: temp=0 + seed=42 on OpenRouter is not determinism — ≥8 upstream backends. Tom Jones' comment suggested provider.order with allow_fallbacks: false, which pins the endpoint — cheaper than K≥3 repeats.

We probed it: StreamLake — the most-used upstream for glm in our server CSV (245 calls) — returns 404 "No endpoints found" when pinned: it no longer serves this model at all. Cloudflare/DeepInfra are stable. The full pinned re-run (qwen→Alibaba, deepseek/glm→DeepInfra, ~$0.02) reproduced every conclusion: per-model arm rankings, temporal present-trap (12/12, 9/12, 12/12), past-tense fix (40/40), and FA absent/silent = 0 across evidence arms. One caveat: glm stays non-deterministic even pinned (FA 0.06 → 0.02 → 0.02 across runs) — part of that is model variance, but part is upstream drift: unpinned glm now routes to DeepInfra (not StreamLake), so cross-day comparisons mix changing backends. Pinning removes routing variance at a point in time, not model or availability drift over time.

Reproduction note: if you reproduce pinned runs, avoid StreamLake — it was the top unpinned provider for glm in our server CSV but returns 404 when pinned (no endpoint for the model; upstream availability drifts). Pinning to a provider that serves you well unpinned is not guaranteed to work; probe before committing to a long run.

What this means for verify-on-read

1. Evidence format is a per-model knob. qwen-family: file fragment (recall). glm-family: graph (recall + trap precision). deepseek: hybrid.
2. Do not concatenate evidence formats blindly. For qwen, file+graph was strictly worse than file alone.
3. Red-team your dataset before trusting metrics. One grep on the subject files inverted our headline. Synthetic categories must be validated per subject, not per project — and FA on a category is meaningless until the category's labels are truth-checked.
4. Temporal questions must be phrased in time. "Is X defined in F?" with history in evidence fails universally (12/12, 9/12, 12/12); "Was X defined in F?" succeeds (40/40). For existence checks: HEAD-only evidence, or explicit tense.
5. Pin providers in production runs. Cheaper than repeats, and it protects against "popular but broken" upstreams (StreamLake).

The meta-lesson

The most dangerous assumption in LLM evaluation isn't the model — it's the dataset. We spent ~$0.10 and ~1900 calls to learn that 4 of 6 "false" facts were true. Before you trust any LLM benchmark, ask: who labeled the ground truth, and did they verify it per-example or per-category? We didn't — we validated the trap category against the project, not the subject. One grep on subject files inverted every conclusion. Every layer of the pipeline can lie: ground truth (this post), evidence (E4b), model interpretation (E4c), execution (provider drift) — we attacked each in turn.

Synthetic categories must be validated per subject, not per project. A value that exists anywhere in the repo is not evidence that the subject of the claim uses it. Check the subject's file.

"Known weaknesses (post-red-team)"

1. Fact order matters (measured). Facts are stored block-ordered (R01–R25 true, R26–R50 false). A shuffled control run (qwen code_first, seed 123) changed 4/50 verdicts vs the original order — no systematic direction, but ~8% sensitivity to order. Shuffle-seed in future runs.
2. Trap category size matters. After relabeling, the original dataset has one genuinely false trap fact — but an extended subject-validated category (20 false) shows the present-trap is real and mass-scale (file_content FA 10–75%). Metrics from N=1 are flags, not rates.
3. Language confound (measured). Claims are Russian, instructions English. A Russian-instruction control (file_content) shifted deepseek's unknown rate 13/50 → 1/50 and changed 14/50 verdicts — its "skeptic" profile in this series was partly a prompt-language artifact. qwen/glm shifted 5/50.
4. Decoy frequency. 19 facts share the same control symbol block; models could pattern-match repetition. Not controlled.
5. Temporal claims are existence claims. Easier than the usage claims of the main dataset; the two datasets are complementary, not interchangeable.
6. Small N, wide CIs. 6 trap facts, 12 removed facts. Headline arm rankings rest on differences of 2–3 facts out of 25.
7. Upstream drift across days. Unpinned vs pinned runs happened ~12h apart; glm's routing changed (StreamLake → DeepInfra). Cross-day numbers mix backends.

Thanks to the Part 1 comment section: Tom Jones (provider pinning), Skillselion (manifest anchors), Cophy (invalidation triggers), Glen Allen (freshness), 473185670 (Resolution Loop), UnitBuilds (write-path triples) — each thread sharpened this series.

Reproduce

Harness: scripts/run_1L_live_arm.py (arms code_first / file_content_first / graph_first / file_graph_first / temporal_blind_first / temporal_duo_first; --facts, --ev-contexts, --pin-provider). Summaries: scripts/summarize_1L_categories.py --facts <corrected.json> (truth-based re-score of old runs). Full report with raw outputs and the red-team audit: experiments/2E_evidence_ladder/report.md. Tests: 64 for harness/builder/generator/summarize, 1265 total.

Dataset fingerprints: original 820bbbf60a0fc930 (historical, mislabeled trap) · corrected e6ce7b902d0a20a9 (29 true / 20 false / 1 ambiguous) · temporal e3c1fdd4 / d1d2c2ed440ec370 · calls: ~1900 across the series · est. cost: < $0.10.

---

A note on how this was written.

Every experiment, bug, failure, and idea here is mine — I earned them the hard way, in production, in public. AI worked as my editor: it helped me structure thoughts and polish my English. It did not invent the facts, because it has none of its own.

No AI detectors were consulted in the making of this disclosure. They have enough trouble agreeing on what I am.

---

Comments

[kgaidev]

We build kgai (decision memory for dev teams) so one thing im curious about. When you flipped the four labels, did the grep evidence for each flip land in the corrected dataset itself, or only in this post? You kept the original as a historical artifact, but the actual story is the link between the two versions, which label moved and what moved it. Thats the part a second red team would want to attack

[Mikhail]

Checked out kgai.dev — the decision graph concept looks interesting.

Quick technical question out of curiosity: since you're capturing team rationale at scale, how do you handle graph mutation determinism without local AST/CallGraph indexing? Relying solely on LLM-extracted decision JSONs usually leads to context drift and heavy token churn on long-living repos. Do you run a custom parser on top of git diffs, or is the decision resolution delegated entirely to the LLM context window at the prompt level?

[kgaidev]

Neither, the split is at the write boundary. Capture is the agents job, the model writes a structured decision event (title, reason, which named elements it touches, what it replaces) at the moment the decision happens, via hooks. Thats the only place an LLM is involved. Everything after is deterministic: events are append only with a total order (lamport plus content hash), the graph is a derived read model rebuilt by replaying the log, and recall is lexical plus graph traversal, no embeddings, no model in the read path. Same store, same query, same context. We dont parse code at all, no AST, no git diff parser. The graph holds named elements and the why behind them, which you cant derive from code anyway. And drift stays bounded because nothing is ever re extracted, the log only grows. Recall returns the current head decisions for the elements in play, token churn is the head set, history stays on disk until someone asks why.

[Mikhail]

Solid architecture, thanks for the detailed breakdown! Event sourcing with Lamport clocks + content hashes for the write log makes total sense for deterministic replays.
​One edge case that comes to mind: how do you keep the named elements in the decision log in sync with the codebase as it evolves through refactoring (e.g., function renames, module restructures)? Without AST/symbol tracking, doesn't the decision graph risk referencing stale 'ghost' symbols over time, or do you rely on the LLM to record rename events explicitly?"

[kgaidev]

The sync comes from where the change is made. The rename or restructure happens in code, through the agent, and the capture hook sits on that same session boundary, so the decision gets recorded the moment the change is made, not reconstructed later. So the main risk reduces to changes made on the side, outside the instrumented workflow. If the team works with kgai enabled that window basically isnt there, the shift is captured when it happens. And if someone does refactor around it, recall on the new name comes back thin, and thats your signal.

[Mikhail]

Got it, thanks for clarifying! Basing sync on session-boundary capture assumes a closed-loop environment where all code mutations flow through instrumented agents.

In reality, developer workflows are hybrid — human edits, IDE refactorings, git merges, and external PRs happen outside agent hooks all the time. Relying on 'thin recall' as a drift signal means the LLM loses decision history precisely when it needs it most (during/after refactoring).

That's exactly why we treat AST/CallGraph indexers as the ground truth layer: capture hooks provide the 'intent', but deterministic AST tracking grounds that intent to the physical reality of the syntax tree no matter who or what modified the code."

[kgaidev]

Fair, and yes, thats the assumption, with one correction: it doesnt need instrumented agents. Capture ships as a skill, so its tied to the LLM, not to an agent harness. Any assistant that can run a CLI records the same way.

kgai is built for teams where the LLM already does most of the work. And it doesnt try to be ground truth for the code, the repo is, the graph sits next to it and the model reads both.

Pure hand edits it wont catch, no argument there. For those you add the decision yourself, one CLI call, or import it from another system, the log takes structured events from anywhere.

[Mikhail]

Appreciate the clarification! Tying capture to an LLM skill/CLI rather than a rigid agent harness definitely makes integration much smoother across different tooling.

The human factor remains the main friction point though — in fast-paced teams, relying on developers to manually invoke a CLI when doing manual refactors or hotfixes usually leads to drift over time.

That's the fundamental trade-off between the two approaches:

• Capturing "intent" at the write boundary gives you the reasoning (the "why"), but risks decay whenever code mutates outside that instrumented loop.
• Deriving context deterministically from AST/CallGraphs gives you guaranteed ground truth for the structure (the "what"), but requires explicit linking to capture rationale.

Both layers are ultimately complementary for long-living codebases!

[Mikhail]

Fair critique — and the honest answer is: half of it landed in the dataset, half didn't.

What's in the corrected copy (memory_contamination_facts_v4_rep_corrected.json): _meta.corrected_from points at the original fingerprint, _meta.correction lists exactly which IDs moved, and each flipped fact carries a label_note with the reason.

What's NOT in the dataset: the exact grep coordinates themselves (graph.py:31, server.py:14, etc.). Those live in the report's red-team section and this blog post. There's no structured, machine-readable flip_log (id → from → to → evidence[]) inside the JSON — the diff between the two versions is text, not data. Reconstructing what moved it from the JSON artifacts alone requires cross-referencing the report. A second red team would absolutely attack that seam, and that's a legitimate known weakness.

What we changed going forward: the new E5 trap generator validates per subject file and embeds its own validation counts (label_validated: "project_files=46, subject_file=0") directly into each fact inside the dataset. New synthetic categories now carry their grep evidence inside the dataset per-fact, so the evidence is never again only in a blog post.