An LLM can explain a software feature quite well and still give you the wrong steps to use it.
Consider a support chatbot answering this question:
How can I recover a deleted file in version 3?
It might describe a menu from version 2, confuse the product with another application, or suggest a recovery option that does not exist. The answer can sound convincing even when the model has never seen the relevant manual.
This is the problem I wanted to understand while reading The Survey of Retrieval-Augmented Text Generation in Large Language Models by Yizheng Huang and Jimmy X. Huang. I used the August 2024 version.
Retrieval-Augmented Generation (RAG) combines a search step with an LLM that writes the answer. The paper is a survey of existing research. What I found useful was how it breaks RAG into smaller decisions: how to prepare information, find it, select it, and use it in an answer.
I'll use a hypothetical support chatbot throughout this post. It searches product manuals, troubleshooting documents, and release notes.
What happens without RAG?
Without external retrieval, an LLM answers using what it learned during training and whatever information we include in the conversation.
That is often enough for a general explanation. It becomes less dependable when the question needs private documentation or details from a recent release.
We could paste the relevant manual into the prompt. For a small amount of information, that may be all we need. But someone still has to find the right manual and the right section. Doing that for every question becomes inconvenient as the document collection grows.
RAG automates that step. The application retrieves relevant information and gives it to the model before asking it to answer.
For our chatbot, the benefit is straightforward: it can consult the version 3 documentation instead of relying only on the model's learned knowledge. Updating the documents also does not require retraining the generator each time.
This does not guarantee a correct answer. If retrieval finds the wrong procedure, the model can still explain the wrong procedure very clearly.
The basic RAG workflow
The survey starts with three stages:
Indexing → Retrieval → Generation
Indexing prepares documents for searching. A long manual is usually split into smaller passages, or chunks. The application builds searchable indexes over these passages.
For keyword search, the index helps locate terms. For vector search, an embedding model turns passages into numerical representations that can be compared with a question.
Retrieval happens when the user asks something. The application searches for passages likely to contain the answer.
Generation comes afterward. The LLM receives the question and selected passage text, then writes the response. Embeddings locate the text; the generator reads the text.
Document preparation runs when documents are added or changed. It does not need to run again for every question.
Figure 2, page 3 of Huang and Huang's survey. Cropped from the paper. The upper part shows basic RAG; the lower part groups the areas where it can be improved.
How the basic workflow can be improved
Traditional RAG follows a straightforward process. Documents are split into chunks and indexed. When a question arrives, the application searches for relevant passages and sends the selected text, together with the question, to the LLM. The model then generates an answer. This basic setup often uses one retrieval pass, so its answer depends heavily on what that first search finds.
Enhanced RAG adds ways to improve the evidence before it reaches the generator. Documents can be cleaned, summarized, or enriched with useful context before indexing. The question can be expanded or reformulated to help search. Retrieved passages can then be filtered and re-ranked. If the evidence is still insufficient, the application can refine the question and search again. Once enough supporting information is available, the LLM combines it into an answer that can be tailored to the reader.
That is the broader workflow illustrated in the paper's detailed diagram below: prepare the data, refine the query, retrieve, check the results, and generate. The survey groups these ideas into pre-retrieval, retrieval, post-retrieval, and generation. Here, “enhanced RAG” describes a set of possible improvements, not one fixed model or a requirement to use every step. For example, hybrid search can also be part of a simple pipeline.
For our support chatbot, these changes address concrete problems: a chunk missing its version number, a question phrased differently from the manual, or a search that finds the recovery steps but misses a separate retention policy.
1. Give each chunk enough context
A chunk containing “This option is available for 30 days” is not very useful by itself. Which option? For which product?
For our chatbot, I would try keeping the section heading, recovery steps, and associated limitations together. Metadata such as the product version and document title can provide additional context.
Small chunks can lose meaning; large ones can include unrelated material. The right size depends on the documents.
Before changing models, I would read a few chunks and check whether they still make sense.
2. Make the question easier to search
Users do not always use the terminology found in documentation. Query manipulation helps bridge that gap.
The survey distinguishes three ideas:
- Reformulation: rewrite “get my file back in v3” as “recover a deleted file in version 3.”
- Expansion: add related terms such as “restore” or “removed document.”
- Normalization: standardize wording such as “v3” and “version 3.”
The rewrite must preserve the user's intent. Dropping the version number might retrieve more results while making them less useful.
I would add this step when test questions reveal a wording problem, rather than assume every question needs an extra LLM call.
3. Use keyword and semantic search together
BM25 ranks passages using indexed terms. It considers factors such as term frequency, how common a term is across documents, and document length. It is not whole-sentence exact matching: depending on the query settings, matching some terms can be enough to return a result. Elastic explains the mechanics here.
Dense retrieval compares learned representations of the question and passages. It can help connect “recover a removed file” with “restore a deleted document,” although it can still miss relevant evidence.
Hybrid search combines these approaches. In a support application, lexical search can help with a specific error code, while semantic search can help with a user's description of the problem.
The results are combined into a ranked list. Whether this improves our chatbot is something to test against a simpler baseline.
4. Re-rank and filter what comes back
The first search finds candidates efficiently. A re-ranker takes a closer look at those candidates and changes their order.
A cross-encoder, for example, processes a question and passage together to score their relevance. Doing this across a large collection is expensive, so it is usually applied to a smaller retrieved set. The Sentence Transformers documentation illustrates this separation.
For our chatbot, re-ranking could move the actual recovery procedure above a passage that only mentions deleted files.
Filtering removes unsuitable material, such as duplicate passages or instructions explicitly limited to another version. Re-ranking changes priority; filtering changes what remains.
A BERT-based embedding retriever and a BERT-based cross-encoder have different jobs here. BERT itself is not a vector database or search engine.
5. Search again when something is missing
Suppose the manual says recovery depends on a separate retention policy. The first search has found part of the answer, but not enough to finish it.
An iterative approach can use that result to search for the policy before generating the final response. Section 4.2 of the survey discusses this along with recursive, conditional, and adaptive retrieval strategies.
Extra searches need a reason and a stopping condition. Otherwise, they add delay and cost without adding evidence.
Figure 4, page 10 of the survey, reproduced as a crop. The paper uses a climate-change example here. Its displayed scores are illustrative.
Follow the retrieval-quality decision near the bottom left: insufficient evidence leads back to query refinement; sufficient evidence proceeds toward generation.
This particular figure filters before re-ranking. That is one arrangement, not a universal order for all RAG systems.
6. Turn the evidence into a clear answer
The survey describes enhancing as combining and organizing retrieved information. Our chatbot might combine recovery steps from a manual with a limitation from the release notes.
Customization adapts the explanation to the reader. A beginner may need numbered steps; an experienced user may prefer a short answer. The underlying facts should remain the same.
If the documents do not establish a recovery method, the answer should say so rather than fill the gap with invented steps.
What is used in current systems?
These October 2026 documentation examples show implemented capabilities, rather than industry-wide adoption figures.
Hybrid retrieval and re-ranking: Azure AI Search documents a classic RAG approach using hybrid search and semantic ranking. Amazon Bedrock also supports reranker models with Knowledge Bases. These techniques are available as managed features, not only as research implementations.
Contextual retrieval: Anthropic's approach adds a short, chunk-specific explanation before creating embeddings and a BM25 index. For our example, that explanation could clarify which product version a passage describes. This method was published in September 2024, after the survey version used here.
Agentic retrieval: Azure's documented approach can use an LLM to break a complex question into focused searches, run them in parallel, and combine the results. Its LLM-based query-planning capability is currently marked preview. It can help with harder questions, but adds work and latency compared with a single-query pipeline.
For this support chatbot, I would start by testing good chunks, version metadata, hybrid retrieval, and re-ranking. I would add query rewriting or repeat retrieval when failures show why they are needed.
How would I check whether it improved?
The survey's evaluation discussion gives three useful questions:
- Context relevance: Did we retrieve useful evidence?
- Answer relevance: Did the response address the question?
- Faithfulness: Are the answer's claims supported by the retrieved material?
Factual correctness still matters. An answer can faithfully repeat an outdated manual and be wrong for the user's situation.
The main lesson I took from the survey is to inspect where the failure happened. If the right passage never reached the generator, a larger generator is not the first thing I would try.
Primary reference: Yizheng Huang and Jimmy X. Huang, The Survey of Retrieval-Augmented Text Generation in Large Language Models, arXiv:2404.10981v2, 23 August 2024. Figures 2 and 4 belong to the paper's authors.
Top comments (0)