The Negative Test That Passed for the Wrong Reason

Hardcore Engineer — Sat, 19 Sep 2026 15:49:30 +0000

Last february we swapped our embedding model from text-embedding-3-small to bge-large, thinking "better embedder, better RAG." our golden set stayed green. two negative tests that had been failing for months suddenly passed. looked like a win.

When the retriever fails to pull the relevant chunks, the LLM never sees the trap. if it's a polite model, it refuses to answer, and your test passes. but it passed for the wrong reason: the retrieval failed, not the model. the test looked green while the system was
broken.

This is the same shape as Serhiy Kucherenko's finding that "the corpus decides more than the model does," just quieter. his refusal traps worked because the prompt boundary held. ours failed silently because the retrieval boundary broke.

The fix: trap_chunk_id and embedder tags

Every negative test in our golden set now carries two stamps:

trap_chunk_id: the id of the chunk containing the trap, stamped at authoring time. our CI step fails the run if that chunk isn't in the retrieved set, regardless of what the LLM says. "test did not run" becomes a third verdict next to pass and fail.
embedder tag: the name of the embedding model under which the test was last validated. after any swap, lines with the old tag are stale by default — they stay in the file, but the CI summary flags them as "unverified under current embedder" until someone re-runs and re-dates them.

The mechanics are dumb simple:

# golden_set.json
{
  "question_id": "neg_17",
  "question": "what was the q3 2019 revenue?",
  "expected_answer": "not in documents",
  "trap_chunk_id": "chunk_847",
  "embedder_tag": "text-embedding-3-small-2024-02-15",
  "last_validated": "2024-02-15"
}

# ci_eval.py
retrieved_chunk_ids = run_retrieval(question)
if trap_chunk_id not in retrieved_chunk_ids:
    verdict = "test did not run"
elif llm_answer == expected_answer:
    verdict = "pass"
else:
    verdict = "fail"

When we swapped to bge-large in february, the two negative tests that "passed" immediately turned red: trap_chunk_id wasn't in the retrieved set. the new embedder was worse at semantic similarity for those specific traps. we rolled back the swap, fixed the chunking strategy, and only then re-validated the tests under the new embedder.

The maintenance cost nobody talks about

The catch is real: trap_chunk_id moves whenever you re-chunk or re-parse documents. every pipeline change triggers a re-stamping step:

re-run the full document ingestion
for each negative test, find the new chunk id containing the trap
update the golden set with the new trap_chunk_id
re-run eval under the current embedder
update the embedder_tag and last_validated timestamp

It's maybe 20 minutes of work per pipeline change, but it's non-negotiable. skip it and you're trusting verdicts that describe a retrieval that no longer exists.

Same goes for the embedder tag: after any swap, every line with the old tag is stale. our CI summary now has a section called "unverified under current embedder" that lists all the tests waiting for re-validation. cheap bookkeeping, but it stopped us from shipping broken retrieval twice.

Why cosine similarity can't save you

Amritpal singh wrote about this exact problem: his refusal threshold had no effect because the similarity distributions for answerable and unanswerable questions overlapped. cosine similarity can't tell "about the topic" from "answers the question." both get high scores.

Our trap_chunk_id approach sidesteps the overlap problem entirely. instead of trying to find a threshold that separates the distributions (which doesn't exist), we assert that the specific chunk containing the trap must be retrieved. if it's not, the test didn't run.

This is the same insight as hybrid search: single-vector similarity will always fail on negatives because high similarity ≠ correct answer. BM25 catches exact matches, vectors catch semantic similarity, metadata filters catch structural context. but for negative tests, none of that matters if the trap chunk isn't retrieved at all.

The checklist

If you're running negative tests in your RAG eval:

every negative test carries a trap_chunk_id
CI fails if that chunk isn't retrieved (verdict: "test did not run")
every test carries an embedder_tag from when it was last validated
after any embedder swap, tests with the old tag are flagged as stale
re-chunking triggers a re-stamping of all trap_chunk_ids
the CI summary shows "unverified under current embedder" tests prominently

If your eval doesn't have these, you're probably shipping retrieval bugs that look like model bugs.

The broader lesson

Edward Izgorodin's framing is the sharpest I've seen: "writing the correction down is not the fix, because the memory that carried the mistake is still in the store and still eligible." But there's a subtler failure mode: evaluation fails without observability into whether the test actually ran.

A negative test that passes because the LLM refused is not evidence that the retrieval worked. it's evidence that the LLM is polite. the only way to know if the retrieval worked is to check whether the trap chunk was actually retrieved.

This is the same shape as observability in production: you don't trust that a service is healthy because it returns 200 OK. you check that it actually did the work. same logic applies to eval.

Where to go deeper

This post is the mechanics. if you want the full audit checklist for production RAG (parsing, chunking, metadata, hybrid search, reranking, eval), i packaged it as a PDF: "RAG checklist: 10 checks to stop LLM hallucinations on tables and numbers" plus a golden set template with 20 questions covering digits, contracts, dates, comparisons, negatives, and paraphrases.

Both are free in my telegram channel: @llmops_engineering — comment "+" on the pinned post and i'll send you both PDFs. no email signup, no spam, just the files.

The channel is where i post production RAG war stories, vector DB benchmarks, and the boring LLMOps stuff that actually ships. if this post saved you a week of debugging, the channel will save you a month.

What's your eval setup for negative tests? have you hit the "test passed but retrieval was broken" failure mode? drop your approach in the comments — curious how other teams handle this.

Why your RAG hallucinates on tables (and a minimal local GraphRAG starter to fix it)

Hardcore Engineer — Fri, 11 Sep 2026 09:42:03 +0000

If you've ever built a RAG system for corporate documents, you've probably hit this wall: you ask "What was the revenue in Q3?", and the LLM confidently hallucinates a number with three extra zeros.

This isn't a model bug. It's an architectural blind spot. As recent research from Microsoft points out, naive 500-token chunking destroys table structures. Cosine similarity over embeddings just doesn't understand rows and columns.

The Fix: GraphRAG

Instead of flat vector search, we extract entities and relationships into a Knowledge Graph. When a user asks about numbers, the LLM translates the question into a Cypher query, and the graph returns exact data. Zero hallucinated digits.

I open-sourced a minimal local starter

To prove this works without sending sensitive corporate data to cloud APIs, I packaged my local GraphRAG stack into a minimal, production-oriented starter repo:
👉 github.com/kaziava/local-graphrag-starter

The Stack:

PyMuPDF for local PDF parsing
LangChain (LLMGraphTransformer) to extract the graph
Neo4j (via docker-compose) as the graph DB
Ollama (running llama3.1:8b or qwen2.5:3b) for local inference

How it works:

# 1. Start Neo4j
docker compose up -d

# 2. Build the graph from your PDF
python main.py ingest data/report.pdf

# 3. Ask a question (NL -> Cypher -> Exact Answer)
python main.py ask "What was Apple's revenue in Q3 2024?"

Benchmarks (MacBook M2, 16GB RAM)

Parse 50-page PDF: ~45s
Load graph: ~10s
Answer a question: 3–5s
Cost per query: $0 (compared to ~$15–20 for the same volume via GPT-4 API)

Honest Limitations & Early Stage Status

🚧 This is an early-stage reference architecture. Local 8B models are weaker than frontier LLMs on complex multi-hop reasoning, and complex tables might still need parser tuning.

If you try running it on your machine and hit any OS-specific bugs, please open an Issue on GitHub! I'm actively maintaining it and will fix things fast.

Want to dive deeper into the code?

I regularly share raw benchmark scripts, Docker configs, and architectural diagrams from my production LLMOps experience. I document this primarily in Russian on my Telegram channel (@llmops_engineering), but the code snippets, schematics, and engineering discussions are universal. Feel free to join the engineer chat there or reach out.

What graph DB are you using for your RAG setups? Let me know in the comments!

DEV Community: Hardcore Engineer