Retrieval quality often fails at the query, not the index. Users ask with jargon, underspecified pronouns, or wording that never appears in the docs. Query rewriting is a dedicated stage that produces better search strings (or search vectors) before you read chunks into the generator.
Clarify the boundaries used in this series:
| Technique | What it changes | Typical shape |
|---|---|---|
| Query rewriting (this article) | The text/vector used to search | Fixed or lightly branched retrieve → fuse |
| Agentic RAG | Which tools run and whether to retry | Dynamic loop with budgets |
| Multi-hop retrieval | How evidence links across hops | Iterative retrieve conditioned on prior facts |
Rewriting can feed an agent or a multi-hop pipeline; by itself it is still “improve the query, then retrieve.”
1) Multi-query expansion
Generate several paraphrases of the user question, retrieve for each, and fuse results (RRF or score normalization).
Why it helps: one embedding neighborhood rarely covers every phrasing (“PTO” vs “paid time off” vs “vacation balance”).
def multi_query_retrieve(question: str, k_per_q=10, n_variants=4):
variants = llm_list(
f"Write {n_variants} diverse search queries for a company knowledge base.\n"
f"Keep meaning, vary wording. Question: {question}"
)
queries = [question] + variants
rankings = [hybrid_search(q, top=k_per_q) for q in queries]
return reciprocal_rank_fusion(rankings)[:k_per_q]
Tradeoffs: n times retrieval cost; noisy paraphrases can drag in junk. Cap variants (3–5) and dedupe near-identical strings.
2) Decomposition
Split a complex question into sub-questions, retrieve per sub-question, then answer (or merge evidence) in order.
Example: “Compare the refund window for monthly vs annual plans and cite the policy versions.”
Sub-questions:
- Refund window for monthly plans + policy version
- Refund window for annual plans + policy version
Decomposition is still query rewriting / planning of search strings. If each sub-answer becomes a bridge entity for a later search, you are crossing into multi-hop territory—keep the handoff explicit in code.
3) HyDE: hypothetical document embeddings
HyDE (Gao et al.) prompts an LLM to write a hypothetical answer document, embeds that document, and retrieves real corpus neighbors of the hypothetical embedding. The synthetic text captures “what a relevant document might look like,” which can improve zero-shot dense retrieval when query and doc phrasing diverge.
Critical rule for production RAG: the hypothetical document is not evidence. Never paste it into the generator context as if it were a source. Only retrieved corpus chunks (and their IDs) are evidence.
def hyde_retrieve(question: str, k=8):
hypothetical = llm(
"Write a short paragraph that answers the question as if it were an internal wiki page. "
"It may be wrong; it is only for search.\nQuestion: "
+ question
)
vec = embed(hypothetical) # synthetic → vector only
hits = vector_search(vec, top=k) # REAL chunks
# optional: also run hybrid_search(question) and fuse
return hits
Hallucinated details in the hypothetical text are expected; the embedding bottleneck and the real corpus neighborhood are what ground the search. Still, HyDE can bias toward fluent-but-wrong regions if your corpus lacks good coverage—always measure.
Distinguishing rewriting from agentic and multi-hop
- Rewriting: prepare better queries; retrieval pattern is mostly predetermined.
- Agentic: choose tools and whether to continue based on observations.
- Multi-hop: condition the next retrieve on extracted facts from prior chunks (entity linking), and grade the whole evidence chain.
If your “rewriter” starts calling tools recursively without budgets, rename it—it became an agent.
Worked example: pick a strategy
Question: “How do we rotate secrets for the payments API in prod?”
- Try multi-query if docs mix “secret rotation”, “key rollover”, “credential refresh”.
- Add decomposition if the answer needs both rotation steps and ownership/on-call.
- Try HyDE if user language is informal and docs are formal runbooks.
- Escalate to agentic if you must fetch a specific runbook by ID after search, then a second system’s wiki.
- Use multi-hop if hop-1 finds “payments API uses Vault path X” and hop-2 must open the Vault runbook for path X.
Limitations
- Every rewrite burns LLM tokens before retrieval even starts.
- Bad rewrites amplify retrieval errors (especially decomposition that drops a constraint).
- HyDE hypotheticals can reinforce corpus gaps; they do not create missing documents.
- Fusion without reranking may surface “average” chunks that match none of the intents well—pair with a cross-encoder when quality matters.
Related reading
- Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE) — also ACL Anthology and reference code.
- Companion drafts: Agentic RAG; Multi-Hop Retrieval; Hybrid Search (beginner).
About the author
I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.



Top comments (0)