Some questions cannot be answered from one passage. Example: “Which team owns the service that stores EU customer invoices?” You may need hop 1 to find which service stores those invoices, then hop 2 to find which team owns that service. That is multi-hop retrieval: iterative evidence gathering where later searches are conditioned on earlier findings.
This is not the same as:
- Query rewriting, which rewrites the user text up front (multi-query, HyDE) without necessarily depending on retrieved facts.
- Agentic RAG, which is a general tool-calling control loop (the agent might perform multi-hop, or might just retry the same search).
Multi-hop is specifically about chained evidence and the failure mode where hop 1 is wrong.
Datasets like HotpotQA popularized evaluation for multi-hop QA with supporting-fact annotations; production systems need the same discipline around provenance even when your corpus is private.
The iterative pattern
- Retrieve with the original question (hybrid search).
- Extract bridge elements: entities, IDs, dates, constraints supported by chunks.
- Formulate the next query from bridges + remaining unmet slots.
- Retrieve again (optionally constrained by metadata).
- Stop when slots are filled or hop budget is hit.
- Generate using the union of supporting chunks; cite each hop.
Entity linking between hops
Bridges should be explicit objects in code, not a vague refreshed prompt:
@dataclass
class Bridge:
kind: str # "service", "team", "doc_id", ...
value: str
support_chunk_ids: list[str]
confidence: float # from ranker or LLM judge
def extract_bridges(question, chunks) -> list[Bridge]:
# LLM or rules: only emit bridges entailed by chunk text
...
def next_query(question, bridges, open_slots):
return llm(
"Write a search query to fill open slots, using bridge values.\n"
f"Question: {question}\nBridges: {bridges}\nOpen: {open_slots}"
)
Linking quality matters: prefer identifiers that appear verbatim in text (svc-invoice-eu) over ambiguous names (“the billing service”).
Provenance: the evidence chain
Store a chain, not only the final context window:
hop=1 query="EU customer invoices storage service"
chunk:C12 → bridge service=invoice-eu
hop=2 query="team owns service invoice-eu"
chunk:C88 → team=Payments Platform
answer cites C12 + C88
If you only keep hop-2 chunks, you cannot justify why invoice-eu was chosen—and you cannot debug cascades.
Preventing error cascades
Problem: hop 1 returns a plausible but wrong service; hop 2 “succeeds” with a confident owner for that wrong service; the final answer looks coherent.
Mitigations that help in practice:
- Keep top-k bridges, not top-1. Branch 2–3 hop-2 queries; fuse or ask the generator to compare support.
- Verify bridges against chunk text (string match or entailment) before using them.
- Require dual support for critical slots when possible (two independent chunks).
- Abstain when hop-1 confidence is low or hop-2 contradicts hop-1 constraints (region, product).
- Cap hops (often 2–3). More hops multiply error rates.
- Separate faithfulness checks on the final answer against the union of cited chunks.
def multihop(question, max_hops=2, bridges_per_hop=3):
evidence = []
bridges = []
q = question
for hop in range(max_hops):
hits = hybrid_search(q, top=8)
evidence.extend(hits)
new_bridges = extract_bridges(question, hits)[:bridges_per_hop]
new_bridges = [b for b in new_bridges if verify(b, hits)]
if not new_bridges:
break
bridges.extend(new_bridges)
if slots_filled(question, bridges):
break
# branch on bridges instead of committing to one
rankings = [hybrid_search(next_query(question, [b], open_slots(question, bridges)), top=8)
for b in new_bridges]
evidence.extend(reciprocal_rank_fusion(rankings)[:8])
return generate_with_citations(question, dedupe(evidence), bridges)
When multi-hop is the right tool
Use it when answers structurally need intermediate facts. Do not use it as a generic “make RAG smarter” switch—latency grows roughly with hops, and cascades create confident failures.
Pairing tips:
- Use query rewriting to improve hop-1 recall.
- Use an agent only if hop-2 needs a different tool (SQL, ticket system), with budgets.
- Use GraphRAG when intermediate links are already materialized as edges (ownership graphs), which can collapse some hops into a neighborhood read.
Limitations
- Public multi-hop scores do not imply your private corpus will behave the same.
- LLM bridge extraction can invent entities not in the chunks—verification is mandatory.
- Branching multiplies cost; measure p95 latency with production-like hop budgets.
- Comparison questions (“which of A or B …”) need explicit compare logic, not only chained lookup.
Related reading
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering — project site: hotpotqa.github.io.
- Companion drafts: Agentic RAG; Advanced Query Rewriting; GraphRAG in Practice.
About the author
I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.



Top comments (1)
The point about keeping top-k bridges instead of committing to the top-1 is the part I'd stress most, since a confident wrong hop 1 is what quietly poisons everything after it. One thing I'd add is a cheap check that the bridge value actually co-occurs with the original constraint (like the EU region in your invoice example) in the hop-2 chunks, and abstain when it doesn't. That catches the plausible-but-wrong service case without needing a full entailment pass on every hop.