Most Retrieval-Augmented Generation (RAG) tutorials stop at "chunk your docs, embed them, throw them in a vector store." That works fine for simple Q&A over a PDF. But the moment your data has relationships entities that connect to other entities, hierarchies, dependencies pure vector search starts falling short. It can find semantically similar chunks, but it has no concept of how those chunks relate to each other.
That's the problem I set out to solve while building a RAG system that combines Neo4j's graph database with vector memory (ChromaDB/FAISS), orchestrated using LangChain.
The Core Problem with Vector-Only RAG
Vector search is great at answering "what's similar to this query?" but weak at answering "how is A connected to B?" If a user asks something that requires multi-hop reasoning for example, connecting a person to a project to a client to a contract clause a flat vector index has no native way to traverse that chain. It just returns whatever chunks score highest on similarity, even if the actually relevant answer requires linking two or three separate pieces of information together.
Why a Hybrid Architecture
The system I built uses two memory layers working together instead of one:
Vector Memory (ChromaDB/FAISS) handles semantic similarity. Given a query, it quickly narrows down the most relevant candidate chunks/entities based on embedding distance.
Neo4j Graph Memory handles structured relationships. Once candidate entities are identified via vector search, the graph layer traverses actual relationships (e.g., WORKS_ON, DEPENDS_ON, PART_OF) to pull in connected context that a pure vector lookup would miss.
The retrieval flow looks roughly like this:
User Query
│
▼
Embed query → Vector Search (top-k candidate nodes/chunks)
│
▼
Use candidate entities as anchor nodes in Neo4j
│
▼
Graph traversal (1–2 hop) → pull connected context
│
▼
Merge vector-retrieved chunks + graph-retrieved context
│
▼
Pass combined context to LLM for final answer generation
This way, the vector store does what it's good at (fast semantic narrowing), and the graph does what it's good at (structured, multi-hop relationship reasoning) instead of forcing one system to do both jobs.
Key Engineering Decisions
Entity extraction before storage. Instead of just chunking and embedding raw text, incoming data is first parsed to extract entities and relationships, which get written into Neo4j as nodes and edges. The raw text chunks still go into the vector store, but now they're linked to graph nodes via shared IDs — so a vector hit can be used to "jump into" the graph.
Bounded graph traversal. Multi-hop graph traversal can explode in size very quickly if left unbounded. I capped traversal depth at [1–2] hops and applied relevance filtering on returned nodes, keeping the injected context focused instead of dumping the whole graph neighborhood into the prompt.
Context assembly and token budgeting. Since LLM context windows are finite, the merge step prioritizes: (a) the top vector-matched chunks, (b) the most relevant graph-derived facts, and trims the rest rather than naively concatenating everything.
LangChain as the orchestration layer. LangChain handled chaining the retrieval steps together (query embedding → vector search → graph query construction → context merge → LLM call), which made it much easier to swap out individual pieces (e.g., trying a different embedding model or LLM provider) without rewriting the whole pipeline.
What This Actually Improved
In practice, this hybrid approach noticeably improved answers for queries that required connecting multiple pieces of information — the kind of questions that a plain vector-only RAG setup would either answer incompletely or hallucinate a connection for. [Add your own concrete example here e.g., "For queries like 'which projects does X depend on that were delayed,' the hybrid system correctly traversed dependency chains that vector search alone missed."]
What I'd Explore Next
Automating entity/relationship extraction with an LLM-based extraction step, instead of relying on manually defined schemas
Adding a re-ranking step after the vector+graph merge, to further prioritize the most relevant combined context before it hits the LLM
Benchmarking retrieval quality with and without the graph layer on a held-out query set, to quantify the actual improvement in relevance/accuracy
If you're building RAG systems and hitting the limits of pure vector search, a graph layer is worth the extra complexity especially once your data has real relationships worth preserving.
You can check out more of my work at https://rajanpanwar.netlify.app/ or connect with me on LinkedIn / GitHub.
Top comments (5)
The multi-hop example is the right one, and the sharper way to put the split is that vector search answers "what looks like this" while a graph answers "what is connected to this". No amount of better embeddings converts one into the other; person to project to client to contract clause is a traversal, not a similarity.
The thing that decides whether this holds up in production is entity resolution at ingest. The graph is only as good as your ability to recognise that two mentions are the same node, and a duplicate entity silently breaks the traversal while the vector side keeps returning plausible chunks, so nothing looks broken from the outside. Worth measuring separately from retrieval quality.
Solid callout. Entity resolution is the part everyone treats as a one-time ETL step, but it's really an ongoing consistency problem and the failure mode you described is exactly why it's dangerous: retrieval keeps looking healthy while the traversal quietly walks the wrong node. I've started tracking entity resolution accuracy as its own metric now, separate from retrieval relevance. Good catch.
Separating that metric makes the failure visible before it becomes an answer-quality regression. I would split resolution errors into false merges and missed merges: they have different consequences for the traversal. A missed merge loses a path; a false merge can manufacture a relationship that never existed.
For the ongoing-consistency side, keep a small set of previously resolved mentions and replay it whenever extraction rules change. That makes unintended changes to established entity IDs detectable alongside accuracy on new documents.
The retrieval flow diagram here nails the actual value of the hybrid approach. Vector search for semantic narrowing to anchor nodes, then graph traversal for relationship-aware context that vector similarity can't infer — the two systems are doing genuinely different jobs rather than overlapping.
The bounded traversal decision (1–2 hops + relevance filtering) is critical and often glossed over in hybrid RAG writeups. Unbounded graph traversal with a large, well-connected graph will flood your context window with tangentially related nodes faster than you can budget tokens. The token budgeting step that prioritizes vector-matched chunks first, then graph-derived facts, and trims the rest is the right order of operations.
Your next steps list is also the right prioritization. LLM-based entity extraction instead of manually defined schemas is the key unlock for making this generalizable — hand-crafted schemas don't scale across varied domains. And the held-out benchmark comparison between vector-only and hybrid retrieval is what would make the improvement claim quantitative rather than qualitative. That data would be worth a follow-up post.
Appreciate the detailed read. You're right that the benchmark is the missing piece right now — the improvement is real in practice but still qualitative in the post, and that's the gap worth closing before I'd call it a solid claim. Planning to run vector-only vs. hybrid on a held-out set and write that up separately once I have numbers, not just impressions. LLM-based extraction is next on the list for exactly the scalability reason you mentioned, hand-written schemas were already showing their limits outside the domain I built this for.