DEV Community

Cover image for Advanced Query Rewriting for RAG: Multi-Query, Decomposition, and HyDE
Hu Jin
Hu Jin

Posted on

Advanced Query Rewriting for RAG: Multi-Query, Decomposition, and HyDE

Retrieval quality often fails at the query, not the index. Users ask with jargon, underspecified pronouns, or wording that never appears in the docs. Query rewriting is a dedicated stage that produces better search strings (or search vectors) before you read chunks into the generator.

Clarify the boundaries used in this series:

Technique What it changes Typical shape
Query rewriting (this article) The text/vector used to search Fixed or lightly branched retrieve → fuse
Agentic RAG Which tools run and whether to retry Dynamic loop with budgets
Multi-hop retrieval How evidence links across hops Iterative retrieve conditioned on prior facts

Rewriting can feed an agent or a multi-hop pipeline; by itself it is still “improve the query, then retrieve.”

Rewriting produces variants, then retrieval uses only the real corpus

1) Multi-query expansion

Generate several paraphrases of the user question, retrieve for each, and fuse results (RRF or score normalization).

Why it helps: one embedding neighborhood rarely covers every phrasing (“PTO” vs “paid time off” vs “vacation balance”).

def multi_query_retrieve(question: str, k_per_q=10, n_variants=4):
    variants = llm_list(
        f"Write {n_variants} diverse search queries for a company knowledge base.\n"
        f"Keep meaning, vary wording. Question: {question}"
    )
    queries = [question] + variants
    rankings = [hybrid_search(q, top=k_per_q) for q in queries]
    return reciprocal_rank_fusion(rankings)[:k_per_q]
Enter fullscreen mode Exit fullscreen mode

Tradeoffs: n times retrieval cost; noisy paraphrases can drag in junk. Cap variants (3–5) and dedupe near-identical strings.

2) Decomposition

Split a complex question into sub-questions, retrieve per sub-question, then answer (or merge evidence) in order.

Example: “Compare the refund window for monthly vs annual plans and cite the policy versions.”

Sub-questions:

  1. Refund window for monthly plans + policy version
  2. Refund window for annual plans + policy version

Decomposition is still query rewriting / planning of search strings. If each sub-answer becomes a bridge entity for a later search, you are crossing into multi-hop territory—keep the handoff explicit in code.

3) HyDE: hypothetical document embeddings

HyDE (Gao et al.) prompts an LLM to write a hypothetical answer document, embeds that document, and retrieves real corpus neighbors of the hypothetical embedding. The synthetic text captures “what a relevant document might look like,” which can improve zero-shot dense retrieval when query and doc phrasing diverge.

HyDE embeds a synthetic doc, then retrieves real chunks only

Critical rule for production RAG: the hypothetical document is not evidence. Never paste it into the generator context as if it were a source. Only retrieved corpus chunks (and their IDs) are evidence.

def hyde_retrieve(question: str, k=8):
    hypothetical = llm(
        "Write a short paragraph that answers the question as if it were an internal wiki page. "
        "It may be wrong; it is only for search.\nQuestion: "
        + question
    )
    vec = embed(hypothetical)          # synthetic → vector only
    hits = vector_search(vec, top=k)   # REAL chunks
    # optional: also run hybrid_search(question) and fuse
    return hits
Enter fullscreen mode Exit fullscreen mode

Hallucinated details in the hypothetical text are expected; the embedding bottleneck and the real corpus neighborhood are what ground the search. Still, HyDE can bias toward fluent-but-wrong regions if your corpus lacks good coverage—always measure.

Distinguishing rewriting from agentic and multi-hop

Query rewriting versus agentic loops versus multi-hop chains

  • Rewriting: prepare better queries; retrieval pattern is mostly predetermined.
  • Agentic: choose tools and whether to continue based on observations.
  • Multi-hop: condition the next retrieve on extracted facts from prior chunks (entity linking), and grade the whole evidence chain.

If your “rewriter” starts calling tools recursively without budgets, rename it—it became an agent.

Worked example: pick a strategy

Question: “How do we rotate secrets for the payments API in prod?”

  • Try multi-query if docs mix “secret rotation”, “key rollover”, “credential refresh”.
  • Add decomposition if the answer needs both rotation steps and ownership/on-call.
  • Try HyDE if user language is informal and docs are formal runbooks.
  • Escalate to agentic if you must fetch a specific runbook by ID after search, then a second system’s wiki.
  • Use multi-hop if hop-1 finds “payments API uses Vault path X” and hop-2 must open the Vault runbook for path X.

Limitations

  • Every rewrite burns LLM tokens before retrieval even starts.
  • Bad rewrites amplify retrieval errors (especially decomposition that drops a constraint).
  • HyDE hypotheticals can reinforce corpus gaps; they do not create missing documents.
  • Fusion without reranking may surface “average” chunks that match none of the intents well—pair with a cross-encoder when quality matters.

Related reading


About the author

I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.

Top comments (0)