Advanced RAG Techniques: Lessons from a Chatbot Builder's Playbook

Most RAG tutorials stop at "chunk, embed, retrieve, generate." But once you've shipped a basic RAG chatbot, you run into a different set of problems: retrieval that's technically working but returning the wrong chunk, users asking vague follow-ups that depend on earlier context, and responses that feel generic instead of personalized.

These are techniques from someone who's spent close to a decade building conversational AI and chatbots — practical, implementation-level ideas for improving retrieval quality, handling conversation history well, and making responses feel personalized rather than templated.

Bring Back Intent Detection

Before RAG was the default approach, chatbot builders relied heavily on intent detection — classifying incoming questions into categories (salary question, support request, etc.) and mapping each to a pre-written, subject-matter-expert-approved answer. That approach hasn't gone away just because RAG exists — it's worth combining with it.

The pattern: take the user's question, predict its intent, and if it matches a known category, serve the pre-written answer directly — skipping the LLM and vector search entirely for high-confidence, frequently-asked questions. Only fall back to standard RAG (vector search over your document store) when no intent is confidently predicted. This gets you speed, cost savings, and answer consistency on your most common questions, while RAG handles the long tail.

Function Calling as an Intent Layer

A more modern variant of the same idea: use function/tool calling (available in most major LLM SDKs — OpenAI's function calling, tool calling in Claude, etc.) as your intent detection mechanism instead of a dedicated NLU model.

If a function is predicted — say, the user says "I want to schedule an interview" and the model predicts a schedule_interview function — you can trigger a real action: call a scheduling API, fetch fresh data from an external source, or store information about the user for later. If no function is predicted, you fall through to standard retrieval. This blurs the line between "RAG" and "agent" — retrieval becomes just one of several possible paths the system can take, alongside API calls, database queries, or other tool invocations.

Index Questions, Not Just Answers

This is one of the highest-leverage, easiest-to-implement tricks for retrieval quality: store FAQs indexed by question, not by raw answer text.

Here's the intuition. If you store raw facts like "the salary for developers is 80k," and a user asks "what's the salary for developers?", the semantic distance between the question and that raw fact — while reasonably close — isn't as close as it could be. But if you instead store the question itself ("what is the salary for developers?") as the indexed vector, with the answer attached as metadata, an incoming user question that's phrased similarly maps almost directly onto your stored question — a much tighter match, and a meaningfully more reliable retrieval.

Building this at scale: manually writing FAQs doesn't scale well. Instead, generate them automatically: take your existing document chunks (treated as "answers"), and for each one, use an LLM to generate the question it answers. Store the generated question as your retrieval key and the original chunk as the associated answer.

You can also layer this into a fallback hierarchy: check FAQs first (since they're curated and reliable), and only fall back to raw document retrieval if no FAQ match is found.

On chunking specifically: there's no universally "correct" chunk size — it's less about chunk size and more about consistent data formatting, and chunks ideally covering a single coherent topic rather than mixing several.

Query Rephrasing and Multi-Query Expansion

Two techniques that work in opposite directions, both aimed at improving what actually gets searched for.

Query rephrasing uses chat history to disambiguate a vague follow-up. If a user asks "what is the salary?" in isolation, that's too abstract to retrieve well — but if the conversation history shows the bot previously asked "what's your position?" and the user answered "engineer," you can have an LLM rewrite the query into a chat-history-aware version: "what is the salary for engineers?" That narrows retrieval to exactly the relevant chunk instead of returning a jumble of salary figures for every role.

Multi-query expansion goes the other direction: instead of narrowing a single query, generate multiple variations of the user's question ("how much do they pay?", "what is the salary?", "how much will I earn?") and retrieve for all of them. This increases your odds of surfacing relevant chunks that a single phrasing might have missed — at the cost of an extra LLM call and more retrieval overhead. It's a solid technique to reserve as a fallback: try the direct/rephrased query first, and only fan out to multi-query expansion if that initial attempt comes up empty.

Handling Chat History (Three Approaches, Plus a Bonus)

Incorporating conversation history well is one of the trickier parts of building a real RAG chatbot. There are a few approaches, roughly in order of complexity:

1. Raw history injection. The simplest approach: paste the full chat history (or the last 5–10 turns, if the full history isn't practical) directly into the prompt alongside retrieved context, and instruct the model to use it. Works, but scales poorly as conversations get long.

2. Chat history summarization. Run an extra LLM pass that reads the full chat history and produces a concise summary, which gets inserted into the prompt instead of the raw transcript. This keeps prompts smaller and — critically — is highly customizable: you can tune the summarization prompt to extract exactly what matters for your use case (e.g., building a user profile over time) rather than a generic recap.

3. RAG over past conversations. Rather than summarizing, treat past conversation history (or summaries of it) as its own retrievable corpus. When a new question comes in, search your document store and search past conversations for relevant prior context, then pass both into the prompt. Useful when a user might reference something discussed in an earlier session, not just earlier in the current one.

Bonus: structured data extraction from chat. While chatting, run an extraction step that pulls out specific, stable facts about the user — e.g., for a language-learning chatbot, extracting and storing that a user's English level is B2 — and persist that as structured data (SQL, JSON) rather than free text. That becomes a durable property you inject into every future prompt for that user, without needing to re-derive it from the raw conversation each time.

Stable vs. Fluctuating User Properties

A particularly useful mental model for personalization, borrowed from a BMW chatbot case study: split what you know about a user into two categories.

• Stable properties — things that hold true throughout the conversation (and often across sessions): technical sophistication, role, language level, preferences. Extract these once via an LLM pass over conversation history and inject them into every prompt going forward.
• Fluctuating properties — things that are true only in the current moment: the user is frustrated right now, they're in a hurry, they seem unsure. Predict these per-turn (not persisted long-term) and use them to adjust just the current response — tone, pacing, level of detail.

You can even attach a lightweight rules layer to fluctuating properties — e.g., "if the user seems frustrated, do X; if they seem unsure, do Y" — encoding customer-support and sales best practices as retrievable guidelines rather than hardcoded logic. Those guidelines themselves can even be extracted at scale by having an LLM analyze a corpus of past successful support conversations.

Ask Clarifying Questions Instead of Guessing

A technique worth stealing directly: instead of always forcing an answer from ambiguous retrieval results, add a check that decides whether the system actually has enough information to answer confidently — and if not, ask a clarifying question.

The flow:

1. Rewrite the user's query for clarity (e.g., "how much" → "what's the salary")
2. Retrieve chunks as normal
3. Run an additional LLM check: given the user's question and the retrieved context, is this specific enough to answer confidently, or is something missing?
4. If information is missing (e.g., retrieved chunks cover both "engineer" and "doctor" salaries, but the user never specified which), generate a clarifying question ("Are you an engineer or a doctor?") instead of guessing
5. Otherwise, answer directly

This is especially valuable in domains with highly specific, branching documentation (telecom pricing that varies by country and plan is a good example) where a wrong guess is worse than a follow-up question.

The One Piece of Advice That Matters Most

If you can only take one thing away: data quality beats every retrieval trick. Garbage in, garbage out — no amount of clever query rewriting, FAQ indexing, or clarifying-question logic compensates for source data that's outdated, contradictory, or poorly organized. Invest in cleaning, deduplicating, and checking your source data for contradictions before optimizing anything downstream.

Beyond that, the highest-leverage, easiest-to-implement techniques from this list are the FAQ-indexing approach (question-as-key, generated at scale) and the clarifying-questions pattern — both are relatively simple to bolt onto an existing RAG system and tend to produce an outsized improvement in perceived answer quality.

---

None of these techniques are mutually exclusive — most production RAG systems end up combining several: intent/function-call routing as a first pass, FAQ-style retrieval for common questions, query rewriting for ambiguous follow-ups, and chat-history summarization or extraction for personalization over time.