The difference between a working RAG prototype and a reliable AI application isn't the LLM. It's everything around it.
You built a RAG chatbot.
You uploaded some PDFs, generated embeddings, connected a vector database, and plugged everything into your favourite LLM.
You asked five questions.
Five impressive answers.
Your manager loved the demo. The team was excited. Someone suggested deploying it next week.
And that's exactly where the real problems begin.
Because building a Retrieval-Augmented Generation (RAG) prototype and engineering a production-ready RAG system are two very different challenges.
A prototype demonstrates that your architecture can work.
A production system must demonstrate that it works reliably, securely, and predictably—even when users ask unexpected questions, documents change, traffic increases, or upstream services fail.
Here are seven problems that frequently separate impressive RAG demonstrations from dependable AI applications.
1. Your Vector Search Is Finding Similar Text, Not Necessarily the Right Answer
Imagine you're building an internal HR assistant.
An employee asks:
"How many days of parental leave am I eligible for?"
Your retriever returns three highly relevant documents:
- The company's 2023 parental leave policy
- A general HR benefits overview
- The current parental leave policy
All three documents are semantically similar to the question.
But only one contains the current, applicable answer.
Semantic similarity is not the same as factual relevance or policy validity.
Vector search measures similarity in embedding space. It doesn't automatically understand which document is authoritative, up to date, or applicable to a particular user.
What to do instead
Combine semantic retrieval with:
- Keyword or BM25 retrieval where exact terminology matters
- Metadata filters for version, effective date, and department
- Reranking to improve the final context selection
- Explicit document precedence rules
Consider a retrieval pipeline like this:
User query → Access filter → Hybrid retrieval → Reranking → Validated context → LLM
The objective isn't to retrieve the most similar text.
It's to retrieve the best evidence for answering the user's question.
2. Your Chunking Strategy Is Quietly Destroying Information
Most introductory RAG tutorials use something resembling:
"Split documents into chunks of 1,000 characters with an overlap of 200."
That can be a useful starting point.
But what happens when a financial policy contains a table?
Or a regulatory document has exceptions listed several paragraphs after the rule?
Or the applicable condition appears in the heading, while the actual answer appears in the next section?
Naive character-based splitting can separate information that needs to be interpreted together.
The retriever may return a perfectly relevant chunk that is missing a critical qualification.
The LLM then generates an incomplete answer from incomplete evidence.
What to do instead
Experiment with structure-aware chunking:
- Preserve meaningful sections and headings
- Keep relevant table rows and their headers together
- Include useful document metadata
- Test parent-child retrieval for complex documents
- Compare chunk sizes against representative questions
For example, instead of storing this isolated chunk:
"Employees may claim reimbursement up to ₹25,000."
Preserve its essential context:
"Domestic travel policy — Grade B employees — reimbursement limit: ₹25,000 — effective from April 2026."
A small improvement in document preparation can sometimes deliver a larger practical benefit than switching to a more expensive model.
3. Your RAG System Doesn't Know When to Say "I Don't Know"
Ask your chatbot a question that isn't answered anywhere in your knowledge base.
What happens?
Some systems still produce a convincing response.
That's dangerous because a fluent answer can be mistaken for an authoritative one.
Consider this question:
"What is our company's approved policy for using confidential client data in external AI tools?"
If the approved policy isn't available, the correct behaviour is not to invent one based on general best practices.
The correct behaviour is to acknowledge that the approved information is unavailable.
What to do instead
Design an explicit insufficient-evidence workflow.
Question → Retrieval → Evidence assessment → Answer or abstain
Your system should distinguish between:
- Sufficient approved evidence
- Partial or conflicting evidence
- Missing evidence
- Evidence that the current user isn't authorised to access
For the last three cases, the application may need to ask a clarification question, cite the available limitations, or direct the user to an authorised human contact.
And importantly, a similarity threshold alone isn't a trustworthy measure of factual confidence.
Calibrate the decision against labelled examples and representative failure scenarios.
4. You're Evaluating the LLM but Ignoring the Retrieval Pipeline
Here's a common testing approach:
Ask the chatbot 20 questions.
Read the answers.
Decide that most look correct.
Deploy.
Unfortunately, this doesn't tell you enough.
A RAG application can fail at multiple stages:
Retrieval failure: The correct document was never retrieved.
Context failure: The correct document was retrieved, but the relevant passage was omitted.
Generation failure: The model received the correct evidence but misinterpreted it.
Citation failure: The answer sounded correct but referenced an unrelated source.
Each failure requires a different engineering response.
What to measure
At minimum, separate retrieval quality from answer quality.
| Metric | What it tells you |
|---|---|
| Recall@K | Whether required evidence appears in the top-K results |
| Precision@K | How much retrieved material is relevant |
| Faithfulness | Whether claims are supported by the supplied evidence |
| Answer relevance | Whether the response addresses the question |
| Citation correctness | Whether references support the claims attributed to them |
| Task success | Whether the application completes the intended user task |
Create a small, human-reviewed evaluation dataset with expected evidence and acceptable answer criteria.
Include questions about missing information, conflicting policies, permissions, and outdated documents.
Then run those tests whenever you modify the prompt, embedding model, chunking strategy, or retrieval configuration.
A beautiful answer isn't necessarily a correct answer.
If you cannot reproduce and measure quality, you cannot reliably improve it.
5. Your Security Model Ends at the Chatbot Interface
This is one of the most serious architecture mistakes in enterprise RAG.
Imagine your vector database contains documents from HR, finance, engineering, and legal departments.
An employee asks:
"What are the executive compensation details for this year?"
Your application has a perfectly working retrieval system.
But does the retrieval layer know whether that employee is authorised to access those records?
If not, you have a serious security problem.
A well-designed RAG application needs access control at the data retrieval layer—not merely an instruction telling the LLM to avoid confidential information.
What to do instead
- Enforce user identity and permissions before retrieval
- Filter source documents using effective access rules
- Test document- and tenant-level isolation
- Apply controls to retrieved snippets, caches, and citations
- Recheck access when permissions or documents change
- Treat retrieved content as untrusted input
Remember that prompt injection can arrive through retrieved documents, not just the user's message.
A document containing malicious instructions must not gain authority simply because your retriever included it in the prompt.
Prompt instructions are not a substitute for application-level security controls.
6. Your Prototype Has No Observability
Your RAG chatbot works.
Until one morning it suddenly doesn't.
Users complain that answers are slower.
Some citations reference outdated documents.
API costs have increased.
Certain questions now return irrelevant results.
What changed?
Without observability, debugging becomes guesswork.
Production RAG needs visibility into both the application infrastructure and the behaviour of the AI pipeline.
What to monitor
Track signals such as:
- Retrieval latency
- Model response latency
- End-to-end p95 latency
- Token usage and cost per request
- Retrieval errors and timeouts
- Source document and index versions
- Prompt and model versions
- User feedback and recurring failure categories
- Evaluation quality over time
Use request-level tracing to connect each generated answer to the retrieval and generation steps that produced it.
Avoid storing sensitive prompts, personal information, or confidential document contents in unrestricted logs.
The important distinction
Traditional monitoring can tell you whether your service is available.
AI-specific evaluation helps you determine whether the service is producing useful, supported answers.
You need both.
7. You Have No Strategy for Document Changes
Suppose your system was indexed yesterday.
Today, your organisation updates its travel reimbursement policy.
Will your assistant retrieve the new document?
Will it stop citing the old policy?
What if the indexing process fails halfway through?
What if the new version contains a parsing error?
These are operational problems, not prompt engineering problems.
What to do instead
Design a controlled knowledge lifecycle.
Source update → Change detection → Parsing → Validation → Re-indexing → Evaluation → Publication
Consider using:
- Document version identifiers
- Incremental ingestion
- Index versioning
- Controlled publication and rollback
- Freshness monitoring
- Evaluation gates before releasing updated collections
A rollback must not restore access to information a user is no longer authorised to view.
This is why production RAG is better understood as a continuously operated information system than as a one-time chatbot project.
The Architecture Shift: From RAG Demo to Production RAG
A typical prototype looks like this:
PDF → Chunking → Embeddings → Vector Database → LLM → Answer
A more mature implementation requires additional engineering around the same core process.
| Engineering concern | Prototype | Production system |
|---|---|---|
| Data ingestion | Manual upload | Managed, versioned pipeline |
| Retrieval | Basic vector search | Evaluated, permission-aware retrieval |
| Context | Fixed top-K chunks | Filtered and reranked evidence |
| Accuracy | Manual inspection | Repeatable evaluation |
| Security | Basic application login | End-to-end authorisation |
| Operations | Console logs | Traces, metrics, alerting |
| Document updates | Manual rebuild | Managed refresh and release |
| Failure handling | Generic error | Explicit fallback and recovery |
Not every application needs the most complex architecture.
But every production application needs engineering decisions appropriate to its risks, users, and reliability requirements.
For a deeper layer-by-layer breakdown, I've put together a Production RAG System Architecture Guide at Technovids, covering ingestion, retrieval, reranking, citations, evaluation, monitoring, and security.
It's a useful companion reference if you're moving from prototype to deployment.
A Practical Checklist Before You Deploy
Before calling your RAG application production-ready, ask:
- [ ] Have we tested retrieval on representative real-world questions?
- [ ] Can we identify which source supports each important answer?
- [ ] Does the system handle missing or conflicting evidence?
- [ ] Are document permissions enforced before retrieval?
- [ ] Have we tested prompt injection through retrieved content?
- [ ] Can we detect quality regressions after updates?
- [ ] Do we monitor latency, errors, and token cost?
- [ ] Can we trace a problematic answer to its source and configuration?
- [ ] Can document updates be validated and rolled back safely?
- [ ] Do we have a clear operational owner and incident response process?
If you answered "no" to several of these, that doesn't mean your prototype failed.
It means you have identified the engineering work required before production.
Final Thoughts
The hardest part of RAG isn't getting an LLM to answer questions from documents.
It's ensuring the right documents reach the model, the information remains trustworthy, permissions are respected, failures can be diagnosed, and changes don't silently break the system.
A prototype proves possibility. Production engineering establishes reliability.
That distinction matters whether you're building an internal knowledge assistant, a customer support system, a financial document search application, or an enterprise AI agent.
One question for developers building RAG systems
What's been your biggest challenge when moving a RAG application from prototype to production?
Retrieval quality, hallucinations, security, latency, or evaluation?
I'd be interested to hear which problems have been hardest to solve.
Written by Pankaj Rana, Technovids — sharing practical insights into AI engineering, RAG architecture, LLMOps, and enterprise AI systems.
More technical resources: Technovids.
Top comments (0)