The tutorial-to-production gap
Every RAG tutorial follows the same arc. Load some documents. Chunk them. Embed them into a vector database. Wire up a retrieval step before the LLM call. Ask a question, get a grounded answer. Twenty minutes, looks great, feels like the future.
Then you try to run it on real data, with real users, at real scale, and you discover that the AI part of RAG was the easy part. The hard part is everything the tutorial skipped.
The problems that surface after the demo
Document lifecycle management
The tutorial loaded documents once. Production data changes constantly. Documents get updated, versioned, deprecated, deleted. Your vector database now has embeddings for three versions of the same policy document. The model retrieves chunks from the version that was superseded last quarter and confidently presents outdated information as current.
Nobody in the tutorial mentioned that you need a sync pipeline that tracks document state, invalidates stale embeddings, handles partial updates without re-embedding the entire corpus, and deals with the fact that deleting a document from the source doesn't automatically delete its chunks from the vector store.
This isn't an AI problem. This is the same data staleness problem that search engines have solved for decades. If you've never built a search index that handles updates and deletions, your RAG system will serve stale data and nobody will know until a user notices the answer references a policy that changed six months ago.
Chunking that loses context
The default chunking strategy in most tutorials is "split by token count with some overlap." This works on clean, well-structured documents. It fails on real documents in predictable ways.
A table gets split across two chunks. The header is in chunk one, the data is in chunk two. Neither chunk is useful alone. A paragraph references "the conditions listed above" but "above" is in the previous chunk. A legal clause has an exception three paragraphs later that changes the meaning entirely, but the retriever only surfaces the clause, not the exception.
The chunking strategy determines what the model can see. Bad chunking means the model gets fragments that look complete but are missing the context that changes the answer. The model doesn't know it's missing context. It generates a confident response based on incomplete information. The user doesn't know either. Everyone trusts the system until someone manually checks and discovers the answer was based on half a table.
Getting chunking right requires understanding the document structure: section boundaries, table integrity, cross-references, hierarchical relationships between headings and content. This is document engineering, not prompt engineering.
Semantic search isn't enough
Vector similarity search is good at finding conceptually related content. It's bad at finding exact matches. A user asks "what is the maximum liability under contract 2024-0847?" The embedding search returns chunks about liability in general, about similar contracts, about maximum limits in different contexts. The exact contract number might not even surface because embedding similarity doesn't prioritize exact string matches.
Production RAG systems need hybrid retrieval: semantic search for conceptual relevance, keyword/BM25 search for exact matching, and a ranking layer that combines both. Most tutorials only show the semantic path because it's the one that uses embeddings, which is the novel part. The keyword path is "boring traditional search" and gets skipped.
The irony is that the boring traditional search part is what determines whether the system can actually answer specific questions about specific things. Semantic search finds the neighborhood. Keyword search finds the address.
Access control is where it gets really hard
The tutorial assumed everyone can see everything. Production data has access controls. Different users have different permissions. A document that's visible to legal shouldn't surface in retrieval for marketing. Patient records should only be retrievable by authorized clinicians. Financial data has regulatory constraints on who can access it.
This means your retrieval layer needs to enforce the same ACLs as your source systems. At query time. For every request. Without being so slow that the response takes 30 seconds.
Most vector databases weren't designed with fine-grained access control in mind. Bolting ACLs onto a system that doesn't natively support them creates either security gaps (filtering after retrieval, which means unauthorized chunks were still read) or performance problems (filtering before retrieval, which limits the candidate pool and degrades relevance).
Getting this right is a security and access management problem, not an AI problem.
RAG is search infrastructure with an LLM on top
The pattern that keeps emerging: every hard problem in production RAG is a problem that existed before LLMs.
Document lifecycle management is a search indexing problem. Chunking is a document processing problem. Hybrid retrieval is a search relevance problem. Access control is an authorization problem. Monitoring for quality degradation is an observability problem.
The LLM is the last mile. It takes the retrieved chunks and generates a natural language response. That part works remarkably well. Everything upstream of it, the part that determines what chunks the model actually sees, is traditional data engineering and search infrastructure.
If your team doesn't have experience with search indexing, document processing pipelines, and access control at the data layer, the RAG system will struggle. Not because the AI is bad, but because the retrieval is bad, and retrieval is a solved problem in the search world that most AI teams are rediscovering from scratch.
The build vs integrate decision
Before building RAG from scratch, check whether your problem is actually "I need semantic search over my documents" or "I need a search engine." If it's the second one, mature search infrastructure (Elasticsearch, Solr, or even a well-configured database with full-text search) might get you 80% of the way there without the operational complexity of a vector database, embedding pipeline, and chunking strategy.
RAG makes sense when you genuinely need the LLM to synthesize information from multiple sources into a coherent answer. If your users are asking specific questions that have specific answers in specific documents, traditional search with good ranking might be the simpler tool.
Same principle as the last post: start with the friction, not the technology.
What's been the hardest production RAG problem your team has hit? Was it the AI part or the data engineering part?
Top comments (11)
Access control tends to get skipped until an audit forces it, and it's the hardest of your four to bolt on late. Filtering after retrieval still means unauthorized chunks were read into memory first, so for regulated data the permission check has to live in the index, not after it. Do you push that down into the vector store, or keep a separate authz index in front of it?
For regulated data the check has to live before retrieval, full stop. If unauthorized chunks were read into memory during the search, most compliance frameworks consider that a violation regardless of whether the user saw the results. We keep a separate authz layer in front of the vector store that pre-filters the document set per user before the similarity search runs. It limits the candidate pool which can hurt relevance on small permission sets, but that's a better problem to have than explaining to an auditor why restricted medical records were loaded into your retrieval pipeline.
Really interesting! How does the auth layer work? How do you ensure performance?
The part I’d add is that ingestion needs its own acceptance tests, not just a working retrieval demo. I like checking three boring cases before trusting a RAG pipeline: a replaced document, a deleted document, and two valid versions with different effective dates. If any of those can still surface the wrong chunk, the model is just making a data bug sound fluent.
Those three test cases should be mandatory before any RAG pipeline goes to production and I'm going to steal that as a checklist. Replaced document, deleted document, two valid versions with different effective dates. If any of them surface the wrong chunk, the rest of the pipeline doesn't matter because the model will just make the data bug sound authoritative. Most teams skip ingestion testing entirely because the retrieval demo looked good on clean data and nobody thinks to test what happens when the data gets messy.
The document lifecycle point is the one most RAG demos neatly skip. Retrieval quality often looks fine until the corpus starts versioning in place, partial updates arrive, and nobody can answer which chunk was current when the model used it.
At that point you are running search infrastructure with an LLM attached, so sync pipelines, deletion semantics, and freshness receipts matter more than another round of prompt tweaking. Teams usually discover that much later than they should.
Have you seen more teams regret building their own indexing pipeline, or integrating with an existing search stack too early and then fighting its constraints?
Honestly I've seen more teams regret building their own indexing pipeline. The "we'll just write a quick sync script" approach starts simple and then grows into a poorly tested, undocumented search infrastructure that one person understands and nobody wants to maintain. Integrating with an existing search stack (Elasticsearch, OpenSearch) has constraints but at least those constraints are documented, battle-tested, and someone else debugs the edge cases. The teams that built their own usually end up rebuilding on top of an existing stack six months later anyway, having spent the interim rediscovering why search infrastructure is hard.
Answering your closing question from a solo-scale system rather than a team one: my hardest production problem was staleness — but of state, not documents.
My corpus is an append-only project journal that an AI assistant retrieves from. Every entry was true when written. Retrieval kept surfacing fragments that were true-at-the-time and wrong-now, and the model presented them with full confidence — your "superseded policy" failure, exactly.
Better chunking or hybrid search wouldn't have touched it, because the
failure wasn't finding the wrong text; it was that the corpus made no distinction between record and state.
The fix was pure document engineering, upstream of retrieval: a single state block
regenerated at every write, a rule that the snapshot outranks any individual entry, and a
facts registry where each atomic fact — date, amount, URL — carries its source and a
last-verified date. Retrieval didn't get smarter. The corpus stopped letting it lie.
So: agreed, and I'd push it one step further. At small scale you often don't need more
search infrastructure at all. You need corpus discipline.
The retrieval layer can't fix what the document structure breaks.
"The corpus stopped letting it lie" is the best one-liner in this thread. Your fix is the right one and it generalizes: if your data contains temporal facts that change, the retrieval layer can't solve it because the retrieval layer doesn't know what's current and what's historical. The state snapshot that outranks individual entries is basically a materialized view of truth, and the facts registry with last-verified dates gives you staleness detection without requiring the model to figure it out. This is corpus design, not search tuning, and most teams never get there because they keep trying to fix it downstream.
The stale-embedding problem is the one that quietly wrecks trust, because the answer looks confident and grounded while citing a version that was deprecated last quarter. The fix that worked for me was carrying a document version and validity window in the chunk metadata and filtering at retrieval, so superseded chunks never enter the context. How are you handling partial updates, do you re-embed a whole document on any edit or diff at the chunk level?
Version and validity window in metadata with pre-retrieval filtering is the clean approach and it solves the superseded-chunk problem at the right layer. For partial updates we re-embed at the document level, not the chunk level, because chunk-level diffing introduces its own problems: if the document structure changed, the old chunk boundaries don't align with the new ones and you end up with orphaned chunks from the previous version that don't map cleanly to anything in the current version. Full document re-embed is more expensive per update but the index stays consistent. At our scale the cost is tolerable, though I could see chunk-level diffing making sense for very large documents that change frequently in small sections.