The RAG Change That Looked Fine — Until Production Users Caught It

Prakash — Sun, 08 Nov 2026 14:42:50 +0000

I thought my RAG pipeline was solid.

We had a notebook that retrieved the right chunks. Answers looked grounded. Ingest scores looked healthy. The demo never failed in front of stakeholders.

Then a small change went out: we tightened chunk size to "improve precision."

CI stayed green. Unit tests stayed green.

Two weeks later support tickets started — answers that skipped the corpus, citations that pointed at the wrong doc, and one nasty case where a low-quality upload slipped ingest because trust_score still looked fine.

That was the moment I stopped trusting notebook confidence.

What actually broke

The failures were not dramatic. They were quiet.

Recall dropped after the chunk tweak, but nothing in CI measured retrieval quality. The model still answered fluently, so grounding gaps hid behind confident prose. Ingest trusted a score that was not the same as "safe to serve."

We were testing code. We were not testing whether the RAG system still behaved like a RAG system.

What I built to catch it next time

I turned those three failure modes into CI checkpoints:

Retrieve — does the retriever still find the right context after a change?

Ground — does the answer stay tied to the corpus, or is it drifting?

Defend — does ingest reject poisoned or low-trust content before it lands in the index?

That became SentinelRAG.

Offline-first, so the gate can run in GitHub Actions with a MockLLM and no GPU. When something slips, you get evidence.json plus JUnit/SARIF — the kind of artifacts reviewers and auditors can actually open.

If you already use promptfoo, Ragas, or DeepEval, you can import those scores into the same gate instead of maintaining a second report pile.

The part that changed my workflow

The win was not a prettier dashboard. The win was a red PR.

After a chunk or prompt change, the gate either passes or it fails the build. That sounds obvious. It was not obvious when my notebook still looked fine and production users were the ones finding the regression.

I also keep a poison demo around. Running it once is a useful gut-check: if your pipeline cannot fail loudly on a known bad ingest, it will not protect you when a real one shows up.

If you are shipping RAG

I open-sourced this because I kept hitting the same gap across projects: plenty of eval notebooks, almost no merge-blocking gate for retrieval, grounding, and ingest trust.

If you maintain a RAG feature in CI, you can try the same pattern on your next change:

pip install sentinelrag

Repo and docs: https://github.com/prakash888kp/SentinelRAG
Package: https://pypi.org/project/sentinelrag/

I would genuinely like to hear what your pipeline still misses — open an issue if a checkpoint does not match how you ship.

DEV Community: Prakash

The RAG Change That Looked Fine — Until Production Users Caught It

The part that changed my workflow

If you are shipping RAG