5 RAG mistakes that looked fine in the demo and broke in production

Nicola Mastromarino — Sat, 03 Oct 2026 05:06:32 +0000

Every RAG demo works. You pick five questions, the system answers them beautifully, and everyone nods.

I've built RAG pipelines over clinical notes and over a knowledge graph, and the gap between "works in the demo" and "works on real questions" is bigger than you'd expect. Real users ask messy questions, use acronyms nobody documented, and expect answers from documents that changed last Tuesday.

Here are five mistakes that pass the demo and fail on real data.

1. Judging quality by vibes

The demo set is five questions you wrote, about documents you know. Of course it works.

Without an evaluation set you can't tell whether a change to chunking or embeddings made things better or just different. On a clinical QA project (about 40K MIMIC-III notes) I tracked Token F1 and hallucination rate against a baseline. That's the only reason I can say QLoRA fine-tuning cut hallucinations by roughly 40% and lifted Token F1 by 190%, instead of "the answers felt better."

The same applies to retrieval. Build a golden set of 30-50 real questions, each mapped to the document that should answer it, and measure retrieval separately from generation. If the right chunk never reaches the model, no prompt will save you.

def hit_rate_at_k(eval_set, retriever, k=5):
    hits = 0
    for item in eval_set:
        docs = retriever(item["question"], k=k)
        if any(d.id in item["relevant_ids"] for d in docs):
            hits += 1
    return hits / len(eval_set)

Twenty lines of code, and it changes how you work.

2. Fixed-size chunking

Splitting every document into 500-token blocks is fast and wrong. It cuts tables in half, separates a list from its heading, and ends chunks mid-sentence. The chunk that says "the limit is 10,000" no longer says what limit.

Chunk along the document's structure (sections, headings, paragraphs), keep the section title attached as metadata, and consider retrieving small chunks but passing the larger parent section to the model.

3. Vector search only

Embeddings are great at meaning and mediocre at exact strings. Error codes, product IDs, contract numbers, internal acronyms: this is exactly what users search for, and exactly where pure semantic search is weakest.

The retriever in my clinical system combines BM25 with a biomedical embedding model, fused with Reciprocal Rank Fusion. Medical text is full of abbreviations and drug names, exactly the kind of string where keyword matching earns its place next to embeddings. Add a reranker on top and you get one of the cheapest upgrades with one of the biggest effects.

4. Letting the model answer no matter what

When retrieval returns nothing useful, most systems still pass the top-k chunks to the LLM, and the LLM still produces a fluent answer. That's the worst failure mode, because it looks like success.

Set a relevance threshold and let the system say "I couldn't find this in the documents."
Require citations, so a wrong answer is at least checkable.
Don't stuff the context with 20 chunks "just in case." More context often means more noise, not more accuracy.

My clinical QA system keeps a human in the loop (it was designed with the EU AI Act in mind). When the domain is medical, a fluent wrong answer isn't a UX bug, it's a risk.

5. Treating the index as finished

The demo index was built once. Real documents change, get deleted, get replaced by newer versions, and some of them shouldn't be visible to every user.

You need an ingestion pipeline, not a script: updates and deletions handled, versions tracked, metadata filters for permissions, and a plan for re-embedding when you change the embedding model. An answer based on a policy that was withdrawn two months ago is worse than no answer.

The pattern behind all five

Every one of these is invisible in a demo and obvious on real data. The demo tests whether the pipeline runs. Real usage tests whether it retrieves the right thing, says no when it should, and stays correct over time.

Build the evaluation set first. Everything else gets easier once you can measure it.

What's the worst RAG failure you've seen that passed the demo? I'd love to hear the war stories in the comments.

I build LLMs for a living. Here's why I wrote a book about slowing down.

Nicola Mastromarino — Mon, 03 Aug 2026 13:30:00 +0000

My job is to make machines think faster.

I work daily with large language models and generative AI systems. I fine-tune, optimize, reduce latency, eliminate friction. My entire professional value is measured in throughput, tokens per second, inference speed.

And then one evening I noticed something: it was 11 PM. I had been "productive" for 14 hours straight. My thumb was scrolling through my phone — screen off. Just muscle memory. My brain was too fried to even choose what to look at, but it kept reaching for stimulation anyway.

That was the moment I realized something was deeply wrong. Not just with me — with how we, as people who build fast systems, have started treating ourselves like fast systems.

The paradox nobody talks about in tech

We spend our careers removing friction, automating repetitive tasks, optimizing pipelines. The promise is always the same: "this will save you time."

But somehow, the more time-saving tools we build and adopt, the less time we feel we have. We're busier than ever. More anxious than ever. More connected than ever, and more exhausted than ever.

I started digging into the neuroscience behind this. What I found changed how I think about work, attention, and time.

The ancient Greeks had two completely different words for time:

Kronos — chronological, measurable, ticking time. The kind you track in Jira tickets and calendar blocks.
Kairos — qualitative, dense, meaningful time. The kind where an hour feels like a minute because you're fully present.

Modern knowledge work, and especially tech culture, has become obsessed with Kronos. We fill every second. We context-switch 50 times a day. We treat rest as wasted clock cycles.

But here's what the neuroscience actually says: your brain does its most important work — creativity, memory consolidation, emotional regulation — precisely when you stop optimizing it.

The Default Mode Network (the part of your brain that activates during rest and "mind wandering") is not idle. It's processing. It's connecting dots you couldn't connect while you were sprinting.

We've been so busy optimizing Kronos that we've completely destroyed our Kairos.

What I did about it

I started writing down protocols for myself. Not productivity hacks. Not "wake up at 5 AM" routines. Just practical, neuroscience-backed ways to protect my attention and reclaim some sense of pace in a job that constantly demands more speed.

Things like:

Monotasking blocks: single-context work sessions where notifications don't exist
Cognitive cooldowns: intentional transition rituals between deep work and shallow work
Digital friction by design: making the default action the slow one, not the fast one

The notes grew. I kept refining them. Eventually they became a short book: The Art of Slowing Down in the Age of Hyper-Speed.

Writing it felt deeply ironic — an AI engineer writing a manifesto against the very culture his work accelerates. But that's exactly the point. We're the ones building the machines. We're the ones who should understand, better than anyone, why the human brain can't run like one.

The one thing I'd leave you with

If you work in AI, ML, or any fast-moving area of tech, you probably already feel this tension. The industry screams FASTER every single day. New model, new framework, new benchmark every week.

You don't have to opt out of all of it. But you do have to protect some space for Kairos. Otherwise, you're just a very expensive server running at 99% CPU with no garbage collection.

And we all know what happens to systems with no garbage collection.

Thanks for reading. If this resonated, I'd love to hear how you deal with the pressure to optimize everything — including yourself.

DEV Community: Nicola Mastromarino