DEV Community

Marc Newstead
Marc Newstead

Posted on

Why Your RAG Agent Can't Connect the Dots (And How to Fix It)

#ai #rag #database #architecture

The Problem You've Probably Hit

You've built a RAG agent. It answers questions from your docs brilliantly... until someone asks something that requires connecting information across multiple sources. Then it falls apart.

"Who worked on projects related to the component that failed in production last week?"

Your vector-based agent returns documents about the failure, documents about team members, and documents about projects. But it can't connect them. That's multi-hop reasoning, and it's where vector embeddings hit their ceiling.

What Multi-Hop Actually Looks Like in Code

Let's be concrete. Single-hop reasoning is straightforward:

# Single hop: "What does the auth service do?"
query_embedding = embed("auth service functionality")
results = vector_db.similarity_search(query_embedding, k=5)
# Returns relevant docs about auth service ✓
Enter fullscreen mode Exit fullscreen mode

Multi-hop reasoning chains multiple steps:

# Multi-hop: "Which engineer should fix the auth service bug?"
# Step 1: What is the auth service?
# Step 2: Who maintains it?
# Step 3: Who's currently available?
# Step 4: Who has fixed similar bugs before?
Enter fullscreen mode Exit fullscreen mode

With vectors alone, you're either:

  • Hoping all that context lives in one chunk (unlikely)
  • Re-querying multiple times and losing the thread
  • Jamming everything into the LLM context window and burning tokens

None of these scale.

Why We Defaulted to Vectors

Vectors were the pragmatic choice in 2023. The tooling was mature, implementation was straightforward, and for 80% of use cases—document retrieval, FAQ matching, semantic search—they worked brilliantly.

Pinecone, Weaviate, Chroma: all excellent tools. The problem isn't the technology; it's the architectural assumption that every knowledge retrieval problem is a similarity search problem.

It isn't.

What Graphs Do Differently

Graph databases store knowledge as entities and relationships:

// Neo4j example
(Alice:Engineer)-[:MAINTAINS]->(AuthService:Component)
(AuthService)-[:DEPENDS_ON]->(UserDB:Database)
(UserDB)-[:HOSTED_ON]->(ProdServer:Infrastructure)
(Bob:Engineer)-[:ON_CALL_FOR]->(ProdServer)
Enter fullscreen mode Exit fullscreen mode

Now that multi-hop query becomes traversable:

MATCH (bug:Issue {component: "AuthService"})
      -[:AFFECTS]->(service:Component)
      <-[:MAINTAINS]-(engineer:Engineer)
      -[:FIXED]->(similar:Issue)
WHERE similar.type = bug.type
RETURN engineer.name, COUNT(similar) as experience
ORDER BY experience DESC
Enter fullscreen mode Exit fullscreen mode

The graph encodes the connections. You're not asking an LLM to infer relationships from unstructured text—you're traversing explicit edges.

The Practical Hybrid Architecture

Here's what's actually working in production:

Use vectors for:

  • Initial document retrieval
  • Semantic similarity matching
  • Unstructured content search

Use graphs for:

  • Entity relationships
  • Multi-hop queries
  • Traversing connected data

Implementation pattern:

  1. Ingest: Extract entities and relationships from your documents (using NER, LLMs, or structured parsers)
  2. Store: Documents as vectors, entities and relationships in a graph
  3. Query: Use vectors to find candidate documents, use graphs to find connected context
  4. Combine: Pass both to your LLM as enriched context

This isn't theoretical. Teams have measured 40%+ accuracy improvements on multi-hop tasks by adding graph memory to existing vector pipelines. The research on why graph beats vectors shows the performance gap clearly.

Actually Building This

You don't need to rip out your existing stack. Start small:

  1. Identify multi-hop queries in your logs (anything requiring "and then" logic)
  2. Extract key entities from those query domains
  3. Add a graph layer (Neo4j, Amazon Neptune, or even PostgreSQL with recursive CTEs)
  4. Build a hybrid retriever that queries both systems

If you're working with a team that specialises in AI automation and software development, they'll likely be working through similar architectural decisions right now.

The Real Tradeoff

Graphs add complexity. You need:

  • Entity extraction pipelines
  • Relationship modelling
  • Graph database expertise
  • More complex query logic

But if your agent needs to reason across connected information—and most interesting agents do—the accuracy gains justify the overhead.

Vectors are brilliant for similarity. Graphs are brilliant for connectivity. Use both.

Top comments (0)