I was reading a post-mortem on The New Stack this week, "The 'silent hallucination' loop: how our autonomous data pipeline poisoned its own vector store" by Emmanuel Akita (July 2026), and I had to stop halfway through. He'd independently arrived at the thesis behind every post in this Resilient Harness series: reliability lives in the harness around the model, not in the prompt inside it. His takeaway, in his own words:
"Probabilistic systems require deterministic boundaries."
That's the whole idea in seven words. So let me walk through what happened to his RAG pipeline, and then show you where I've reproduced the same lesson from the other direction. That includes the one place his story looks like it contradicts my own tutorial, and why it doesn't.
What went wrong inside the pipeline?
Akita's team built a RAG pipeline for a fintech client: ingest thousands of unstructured financial PDFs, extract the key fields, embed everything, and feed an internal Q&A chatbot. It worked flawlessly at first. Then the chatbot started answering questions about 2022 while citing 2018 data, and attributing competitors' revenue to the client's subsidiaries.
The terrifying part, in his words: the retrieval was working correctly. The dashboard was green, latency was sub-100 ms, the vector search returned exactly what it was asked for. The data it returned was garbage. That's why he calls it silent: no error, no exception, no breach. A perfectly healthy system confidently serving wrong answers.
Here's the chain, and why each link maps onto something I've already written about.
1. The extraction agent hallucinated, and the hallucination got embedded
Their ingestion agent used a frontier LLM to pull metadata from each PDF (document_type, fiscal_year, company_entity, a summary) as JSON, then appended that to the text chunks before embedding. When the LLM hit an illegible fiscal year in a poorly scanned PDF, it didn't raise an exception. It guessed "2024." That guess got embedded alongside the text, so now they had, as Akita puts it, "high-speed searches for documents that didn't exist."
His diagnosis of the root cause: "treating a probabilistic extraction process as deterministic."
This is the silent failure I wrote about in Why AI Agents Fail at Multi-Step Tasks: the agent reports success, every dashboard looks healthy, and the output is quietly wrong. You don't catch it by watching for crashes. You catch it by checking the content of the work against something ground-truth.
2. The LLM-as-a-judge validator rubber-stamped it
This is the part I found most useful, because it kills a pattern a lot of teams reach for by default. Akita's team had a second LLM, a "Validator Agent," checking the extractor's JSON against the raw text before it hit the vector DB. It still let the hallucinations through. Why?
"Using a probabilistic model to police another probabilistic model doesn't give you a firewall; it gives you a confirmation bias loop."
The validator kept agreeing with the extractor out of sycophancy: it would fail to find the year in the text and rationalize "the first model must have seen something I missed." Two probabilistic systems checking each other don't add up to a deterministic guarantee. They add up to consensus, which is not the same thing as correctness.
Now, I've recommended LLM-as-a-judge for evaluation myself. Does that make me wrong? No, and the distinction is the whole point. In my LLM-as-Judge tutorial (and the 3-framework comparison that followed it) I use an LLM judge offline, in batch experiments and test suites, to score quality. Even there it never stands alone: it runs with explicit rubrics, deterministic checks (Contains, ToolCalled), and trajectory evaluation. Akita's mistake wasn't using an LLM judge. It was putting one inline on the write path as the integrity gate, the component that decides what becomes trusted ground truth in production. That's exactly where a probabilistic checker must not be the last line. Use the judge to measure; use code to gate.
| LLM-as-a-judge is the right tool for | It's the wrong tool for |
|---|---|
| Scoring subjective quality offline: helpfulness, tone, rubric grading | The inline integrity gate on your write path |
| Batch evaluation in test suites, tracking quality trends over releases | Anything code can check exactly ("does this year appear in the source?") |
| Comparing agents/prompts where no ground truth exists | Security or data-integrity decisions where one miss poisons production |
3. Prompt engineering made it worse
Their first instinct was to fix it in the prompt: "DO NOT HALLUCINATE", "If you are not sure 100% certain of the metadata, output NULL", "You are a strict financial auditor." The result, per his report: the validator turned overly defensive, started rejecting perfectly good data, and reasoning steps drove API costs up ~40%.
I made almost this exact argument in How to Stop Prompt Injection in AI Agents That Read Untrusted Content: you can't reliably prompt your way out of a trust problem, because the model's cooperation is a mood, not a guarantee. There I was defending against an attacker's injected text; Akita was defending against his own model's hallucination. Same conclusion: the prompt is the wrong layer. The fix belongs in the harness.
The fix: a deterministic boundary before anything becomes trusted state
Akita's team removed all decision-making authority from the validation step and replaced the Validator LLM with plain, boring Python. That's the harness principle this whole series is about: validation lives in the code around the model, not in the prompt inside it. The model proposes; the harness decides. Three moves:
-
Pydantic grounding. A plausible integer isn't enough for
fiscal_yearto be accepted: the year has to physically appear in the raw source text (a regex check). "2024" hallucinated for a 2018 document fails, because "2024" isn't in the text. -
Deterministic cross-referencing. The
company_entitygets fuzzy-matched against a hardcoded SQL table of the client's real entities, with a competitor flag so a legitimate competitor analysis doesn't get wrongly rejected. - Quarantine by default. Nothing from extraction goes straight to the vector store. Everything stages in a PostgreSQL table first, and only payloads that pass the Pydantic + SQL checks get embedded.
His reported outcome: data poisoning stopped immediately, and replacing the validator LLM cut API expenses ~50%. (Those are his numbers from one production post-mortem, not a reproducible benchmark, but the mechanism is the point.)
Read that fix again and it's the same shape as Stop AI Agent Hallucinations: Validate Before the Agent Writes to Memory. That post's entire thesis is validate before the agent writes to memory: a deterministic gate that decides what's allowed to become trusted state, before it becomes trusted state. Akita gates writes to a vector store with Pydantic; I gate writes to agent memory with a Strands BeforeToolCallEvent hook. Different tool, identical shape: the boundary sits on the write path, and it's code, not a model.
Two faces of one bug: self-poisoning vs memory poisoning
Akita's failure had no attacker: the system poisoned itself with its own hallucination. My post on prompt injection and memory poisoning is the mirror image: an attacker plants a malicious instruction in content the agent reads, the agent stores it as a trusted memory, and a brand-new session reloads it from disk and acts on it. Both are "the system trusts a persisted store, and the store is wrong."
| Akita's pipeline (self-inflicted) | The mirror case (adversarial) | |
|---|---|---|
| Who put the bad data in? | The LLM's own hallucination | An attacker's injected instruction |
| Trusted store | Vector database | Agent memory (agent.state) |
| What failed | LLM-as-a-judge validator (sycophancy) | Prompt-level "ignore bad instructions" |
| The fix | Pydantic grounding + SQL, before embedding |
BeforeToolCallEvent gate, before the tool runs |
| Research | Independent post-mortem; echoes Misevolution | Reproduces Zombie Agents |
The academic backing lines up too. Your Agent May Misevolve (Shao et al., Sep 2025) is the first systematic study of agents drifting into unsafe behavior with no external attacker, which is exactly Akita's self-poisoning loop. Zombie Agents (Yang et al., Feb 2026) covers the adversarial side: a one-time injection stored in memory becomes a persistent cross-session compromise.
The one rule that covers all of it
Whether the bad data arrives by attack or by hallucination, the defense is the same and it doesn't live in the prompt:
Let the LLM extract, analyze, and summarize. But the moment data is written to storage as ground truth, it must pass a stricter, deterministically-engineered barrier.
That's Akita's rule, and it's the spine of everything I've been writing:
- Stop AI Agent Hallucinations: Validate Before the Agent Writes to Memory — validate before the agent writes to memory (his Pydantic-before-embedding, in agent form).
- How to Stop Prompt Injection in AI Agents That Read Untrusted Content — gate the dangerous action at the tool boundary, not in the prompt.
- Why AI Agents Fail at Multi-Step Tasks — catch the silent failure by verifying the work, not by trusting a "done."
- How to Evaluate AI Agents: LLM-as-Judge Tutorial — use the LLM judge to measure quality offline, paired with deterministic checks; never as the inline gate.
Different posts, one principle: a probabilistic component can propose; only a deterministic boundary should be allowed to commit.
Questions you might have
Isn't a second LLM ("LLM-as-a-judge") supposed to catch this?
It catches some things, but it's probabilistic, so it can't give you a guarantee. As Akita found, it tends toward sycophantic agreement, rubber-stamping the first model's output. Two models agreeing is consensus, not correctness. Use an LLM judge to measure quality offline (with rubrics and deterministic checks, as I show here); for a hard integrity constraint on the write path ("this year must appear in the source text"), use code.
Can't I fix this with a stricter prompt?
That's the trap Akita hit and the one my prompt injection post is built around. Stricter prompts made his validator reject good data and pushed costs up ~40%. The prompt is the wrong layer: the guarantee has to live somewhere the model's mood can't reach.
Does this only apply to RAG pipelines?
No. Akita's is a RAG ingestion path; my demos are agent tool calls. The shared shape is any point where a probabilistic component's output gets committed as trusted state: memory, a vector store, a database, an outbound action. Put a deterministic gate there.
Where does this go in production?
The same allow/deny moves to a policy layer at the tool or gateway boundary (for example Amazon Bedrock AgentCore), so the rule is centralized and can't be edited away by a poisoned (or self-poisoned) store.
Try it yourself
I built the agent-side versions of this (validate-before-write and the tool-boundary gate) as runnable demos. Clone the repo and run them:
git clone https://github.com/elizabethfuentes12/resilient-agent-harness-sample-for-aws.git
cd resilient-agent-harness-sample-for-aws/01-memory-guardrails
uv venv && source .venv/bin/activate
uv pip install -r requirements.txt
echo "OPENAI_API_KEY=sk-..." > .env
uv run test_memory_guardrails.py
# The adversarial mirror image: poison survives a restart, a tool gate blocks it
cd ../02-memory-poisoning-defense
uv pip install -r requirements.txt
cp ../01-memory-guardrails/.env .env
uv run test_memory_poisoning_defense.py
Full credit to Emmanuel Akita for the original post-mortem. Go read it: it's a clean, honest write-up of a failure most teams never publish.
Have you ever shipped a pipeline that was perfectly healthy and perfectly wrong? Tell me in the comments what finally caught it.
📬 Building reliable AI agents? I write about agent memory, guardrails, evaluation, and multi-agent patterns. Subscribe to my newsletter to get the next one.
Gracias!
🇻🇪 Dev.to Linkedin GitHub Twitter Instagram Youtube
Top comments (1)
excellent material