Your RAG system can't answer "why" questions. Here's what you're missing.

The problem with semantic search

You've built a RAG system. User asks a question, you embed it, run a vector similarity search, retrieve the top 5 chunks, shove them into context, and let the LLM answer. Works brilliantly for simple lookup queries.

Then someone asks: "Why did the deployment fail after the database migration?"

Your agent retrieves a chunk about the deployment error. Maybe another about the migration. But it completely misses that the migration changed a column type, which broke a dependency in the service layer, which caused the deployment to fail. That's three hops of reasoning your vector store can't connect.

The gap between 32% and 86% accuracy on multi-hop questions isn't a prompt engineering problem. It's an architecture problem.

What vector search actually gives you

Vector embeddings are phenomenal at semantic similarity. They'll find documents about the same topic, even if the words differ. But they're terrible at representing relationships between facts.

When you embed a sentence like "Service A depends on Service B", that directional relationship gets flattened into a 1536-dimensional float array. The embedding knows these services are related, but it doesn't know how or which direction.

Ask "what services depend on Service B?" and you might retrieve chunks mentioning both services. Ask "what does Service A depend on?" and you'll get similar chunks. The vector store can't distinguish between these queries because the relationship isn't preserved—just the proximity.

The retrieval gap in production

Here's where it breaks in real systems:

Scenario: Your AI agent manages a microservices architecture. Documentation lives in Notion, incident reports in Jira, config in GitHub.

Query: "What's the blast radius if we take down the auth service?"

What vector retrieval gets you:

• Chunks about the auth service
• Maybe some chunks about services that mention auth
• Possibly incident reports that mention auth failures

What you actually need:

• Services that directly call auth
• Services that depend on those services (second hop)
• Downstream effects on user-facing features (third hop)
• Historical incidents showing actual impact patterns

Vector search retrieves documents. But reasoning about systems requires traversing a graph of relationships.

Why hybrid retrieval matters

The solution isn't to abandon vector search—it's to stop treating it as your only retrieval mechanism.

Graph-based memory stores information as entities and edges. When your agent processes documentation, it extracts:

• Entities: services, APIs, databases, teams, incidents
• Relationships: depends_on, calls, deploys, owns, caused_by

Now that "blast radius" query becomes a graph traversal:

# Pseudocode for hybrid retrieval
def answer_query(query):
    # Step 1: Use vector search for initial recall
    candidate_entities = vector_search(query, top_k=10)

    # Step 2: Expand via graph traversal
    related_nodes = graph.traverse(
        start_nodes=candidate_entities,
        relationships=['depends_on', 'calls', 'impacts'],
        max_depth=3
    )

    # Step 3: Retrieve detailed content for final context
    context = fetch_content(related_nodes)

    return llm.generate(query, context)

You use vectors for semantic recall ("find anything related to auth service"), then use the graph to expand outward along explicit relationships.

When you actually need this

Not every agent needs a graph. If you're building a documentation Q&A bot that answers "how do I configure X?", vector search is probably fine.

You need hybrid retrieval when:

• Questions require connecting multiple facts ("why did X happen?")
• You're reasoning about systems with explicit relationships (dependencies, hierarchies, workflows)
• Accuracy on complex queries is more valuable than simplicity
• You're seeing high retrieval scores but wrong final answers

Quick diagnostic: Ask your agent three questions that require connecting information from different documents. If it can retrieve all relevant chunks but still gives incomplete answers, you have a relationship problem, not a retrieval problem.

What this looks like in practice

Implementing hybrid retrieval isn't trivial, but it's not exotic either:

1. Extract entities and relationships during document ingestion (use an LLM or NLP pipeline)
2. Store vectors in your existing vector DB (Pinecone, Weaviate, etc.)
3. Store the graph separately (Neo4j, or even a relational DB with recursive CTEs)
4. Query both during retrieval and merge results

The teams seeing that 86% accuracy boost on multi-hop reasoning aren't using magic—they're just stopped expecting embeddings to preserve information they were never designed to capture. The research on vector-only retrieval makes this gap clear.

If you're building production AI systems that need to reason about relationships, not just retrieve similar text, you'll eventually hit this wall. Graph memory isn't a nice-to-have—it's how you bridge the gap between semantic similarity and actual reasoning.

For more on building robust agent architectures, check out resources on AI automation and software development that cover these hybrid approaches in depth.

The bottom line

Vector search is brilliant at finding relevant documents. Terrible at understanding how facts connect. If your agent needs to answer "why" or "what happens if", you need more than cosine similarity. You need a graph.