Your RAG Pipeline Is Probably Failing Multi-Hop Questions (Here's Why)

The Problem You're Not Measuring

If you've built a RAG system in the last year, you've probably reached for a vector database. Pinecone, Weaviate, Qdrant—doesn't matter which. You chunk your docs, generate embeddings, toss them in, and suddenly your LLM can "remember" things. Shipped.

But here's the uncomfortable truth: vector-only RAG falls apart the moment queries require reasoning across multiple pieces of information. We're talking 32% accuracy on multi-hop questions. That's not a tuning problem—it's an architectural one.

Why Vector Search Isn't Enough

Vector databases are brilliant at semantic similarity. Ask "What's our refund policy?" and they'll surface the right doc chunk every time. But ask something like:

"Which projects led by engineers who reported to Sarah are now blocked by infrastructure issues?"

Now you need:

1. Find engineers reporting to Sarah
2. Find projects led by those engineers
3. Filter for blocked status
4. Check blocking reason is infrastructure

Vector search treats each chunk independently. There's no concept of "reports to" or "blocks" as traversable relationships. You're hoping the right context happens to land in the same embedding neighbourhood. Sometimes it does. Often it doesn't.

This is precisely why the real bottleneck isn't compute or model size—it's how we structure memory.

Graph Memory: Relationships as First-Class Citizens

Graph-based memory stores knowledge as nodes (entities) and edges (relationships). Think Neo4j, but purpose-built for agentic systems.

Instead of:

Chunk 437: "Sarah manages the Platform team..."
Chunk 891: "Project Apollo is blocked due to..."
Chunk 1203: "The Platform team owns..."

You get:

(sarah:Person)-[:MANAGES]->(platform:Team)
(alice:Engineer)-[:MEMBER_OF]->(platform)
(alice)-[:LEADS]->(apollo:Project)
(apollo)-[:BLOCKED_BY]->(infra:Issue {type: "infrastructure"})

Now that multi-hop query becomes a graph traversal. You're not praying the embeddings align—you're following explicit paths through structured knowledge.

What This Looks Like in Practice

For a Python agent using something like LangGraph or a custom orchestrator:

def answer_complex_query(question: str):
    # LLM extracts entities and relationships from question
    query_graph = llm.parse_to_graph_query(question)

    # Execute graph traversal
    results = graph_db.traverse(
        start_nodes=query_graph.entities,
        relationships=query_graph.relations,
        constraints=query_graph.filters
    )

    # LLM synthesises answer from structured results
    return llm.generate_answer(question, results)

The key difference: your retrieval layer now understands structure, not just semantics.

Multi-Agent Systems Amplify the Problem

Single-agent RAG is one thing. But enterprise workflows increasingly involve multiple specialised agents coordinating:

• Agent A researches customer context
• Agent B pulls relevant contract terms
• Agent C generates a response draft
• Agent D validates against compliance rules

If each agent is independently fumbling with vector search over the same corpus, you're compounding retrieval errors at every step. Graph memory gives agents a shared, queryable mental model of the domain.

For teams working on AI automation and software development, this isn't theoretical—it's the difference between a demo that impresses and a system that ships.

Hybrid Is the Pragmatic Play

Don't rip out your vector DB tomorrow. Hybrid architectures are where the smart money is:

• Vector search for broad semantic retrieval ("find documents about billing")
• Graph traversal for structured reasoning ("find Sarah's team's blocked projects")
• LLM orchestration to decide which retrieval mode to use

You can even use vector embeddings as initial filtering before graph traversal—shrink the search space, then reason precisely.

# Hybrid retrieval pattern
candidates = vector_db.search(query, top_k=50)  # Cast wide net
entities = extract_entities(candidates)          # Identify key nodes
results = graph_db.traverse_from(entities)       # Reason through relationships

What to Do Next

If you're building agentic systems today:

1. Audit your queries. How many actually require multi-hop reasoning? If it's >20%, you need structure.
2. Start small. Build a domain graph for one use case—customer support tickets, project dependencies, org structure.
3. Instrument retrieval. Log when your RAG system fails. You'll spot the pattern fast.
4. Prototype hybrid. Keep your vector pipeline, add graph traversal for complex queries.

The architecture shift isn't about chasing novelty. It's about building systems that actually reason instead of just pattern-match.

Because 32% accuracy doesn't ship.

Comments

[Tae Kim]

The 32% multi-hop accuracy is the real number that should end the vector-only debate. The entity resolution step before graph construction is what most teams skip — if two source mentions of the same person resolve to separate nodes, your MANAGES edge never fires. In the financial news corpus I built Graph RAG on, we went from 52% phantom nodes down to 8% after running a two-stage resolver (bi-encoder blocking plus cross-encoder scoring) before any entity touched the graph, and multi-hop recall jumped considerably as a direct result. Graph traversal gives you deterministic relationship paths that embedding neighborhoods simply cannot guarantee.