Building a Hybrid RAG System: Combining Neo4j Graph Memory with Vector Search

Most Retrieval-Augmented Generation (RAG) tutorials stop at "chunk your docs, embed them, throw them in a vector store." That works fine for simple Q&A over a PDF. But the moment your data has relationships entities that connect to other entities, hierarchies, dependencies pure vector search starts falling short. It can find semantically similar chunks, but it has no concept of how those chunks relate to each other.

That's the problem I set out to solve while building a RAG system that combines Neo4j's graph database with vector memory (ChromaDB/FAISS), orchestrated using LangChain.

The Core Problem with Vector-Only RAG

Vector search is great at answering "what's similar to this query?" but weak at answering "how is A connected to B?" If a user asks something that requires multi-hop reasoning for example, connecting a person to a project to a client to a contract clause a flat vector index has no native way to traverse that chain. It just returns whatever chunks score highest on similarity, even if the actually relevant answer requires linking two or three separate pieces of information together.

Why a Hybrid Architecture

The system I built uses two memory layers working together instead of one:

Vector Memory (ChromaDB/FAISS) handles semantic similarity. Given a query, it quickly narrows down the most relevant candidate chunks/entities based on embedding distance.
Neo4j Graph Memory handles structured relationships. Once candidate entities are identified via vector search, the graph layer traverses actual relationships (e.g., WORKS_ON, DEPENDS_ON, PART_OF) to pull in connected context that a pure vector lookup would miss.

The retrieval flow looks roughly like this:

User Query
│
▼
Embed query → Vector Search (top-k candidate nodes/chunks)
│
▼
Use candidate entities as anchor nodes in Neo4j
│
▼
Graph traversal (1–2 hop) → pull connected context
│
▼
Merge vector-retrieved chunks + graph-retrieved context
│
▼
Pass combined context to LLM for final answer generation

This way, the vector store does what it's good at (fast semantic narrowing), and the graph does what it's good at (structured, multi-hop relationship reasoning) instead of forcing one system to do both jobs.

Key Engineering Decisions

1. Entity extraction before storage. Instead of just chunking and embedding raw text, incoming data is first parsed to extract entities and relationships, which get written into Neo4j as nodes and edges. The raw text chunks still go into the vector store, but now they're linked to graph nodes via shared IDs — so a vector hit can be used to "jump into" the graph.

2. Bounded graph traversal. Multi-hop graph traversal can explode in size very quickly if left unbounded. I capped traversal depth at [1–2] hops and applied relevance filtering on returned nodes, keeping the injected context focused instead of dumping the whole graph neighborhood into the prompt.

3. Context assembly and token budgeting. Since LLM context windows are finite, the merge step prioritizes: (a) the top vector-matched chunks, (b) the most relevant graph-derived facts, and trims the rest rather than naively concatenating everything.

4. LangChain as the orchestration layer. LangChain handled chaining the retrieval steps together (query embedding → vector search → graph query construction → context merge → LLM call), which made it much easier to swap out individual pieces (e.g., trying a different embedding model or LLM provider) without rewriting the whole pipeline.

What This Actually Improved

In practice, this hybrid approach noticeably improved answers for queries that required connecting multiple pieces of information — the kind of questions that a plain vector-only RAG setup would either answer incompletely or hallucinate a connection for. [Add your own concrete example here e.g., "For queries like 'which projects does X depend on that were delayed,' the hybrid system correctly traversed dependency chains that vector search alone missed."]

What I'd Explore Next
Automating entity/relationship extraction with an LLM-based extraction step, instead of relying on manually defined schemas
Adding a re-ranking step after the vector+graph merge, to further prioritize the most relevant combined context before it hits the LLM
Benchmarking retrieval quality with and without the graph layer on a held-out query set, to quantify the actual improvement in relevance/accuracy

If you're building RAG systems and hitting the limits of pure vector search, a graph layer is worth the extra complexity especially once your data has real relationships worth preserving.

You can check out more of my work at https://rajanpanwar.netlify.app/ or connect with me on LinkedIn / GitHub.