Standard Retrieval-Augmented Generation (RAG) solved the initial LLM hallucination problem by grounding outputs in vector embeddings. However, pure vector search falls short when queries require multi-hop reasoning, structural context, or relational aggregation across disparate data points.
Moving from passive retrieval to deterministic reasoning requires combining structured knowledge graphs (Neo4j), reasoning-optimized LLMs (Gemini), and standardized tool interfaces via the Model Context Protocol (MCP).
Introduction: The Ceiling on Retrieval-Augmented Generation
Retrieval-Augmented Generation solved a real problem. It gave large language models a way to answer questions about information they were never trained on, by fetching relevant text chunks from a vector store and stuffing them into a prompt. For a few years, this was enough to make chatbots feel genuinely useful over private documents, wikis, and support tickets.
But most teams that have shipped RAG in production have run into the same wall. Ask a RAG system "What did we ship last quarter?" and it retrieves a handful of semantically similar paragraphs — usually the ones that happen to contain the words "shipped" and "quarter" close together. Ask it "Which customers are exposed to the outage caused by the vendor we just switched away from, and who owns the renewal for each?" and it falls apart. That question isn't about semantic similarity. It's about relationships: outage → root cause → vendor → customers → account owners → renewal dates. No amount of embedding similarity search will reliably walk that chain, because the chain isn't encoded in the meaning of any single chunk of text , it's encoded in the structure connecting many pieces of data.
This is the gap between retrieval and reasoning. Retrieval finds things that look related. Reasoning follows things that are related, hop by hop, and can explain the path it took. Closing that gap is what's driving the current shift toward graph-powered agentic AI systems where a large language model like Gemini doesn't just read a static context window, but actively queries a knowledge graph, inspects the results, decides what to query next, and assembles an answer (or a multi-step action) out of a real chain of evidence.
This post is a practical architecture guide for building that kind of system using three components that fit together unusually well: Google's Gemini as the reasoning engine, Neo4j as the structured memory the agent reasons over, and the Model Context Protocol (MCP) as the standardized wiring between them.
Why Graphs Succeed Where Vector Search Falls Short
Vector databases are excellent at one thing: finding text that is semantically similar to a query. They are not built to represent explicit, typed relationships between entities, and they have no native concept of traversal — following a chain of connections several steps deep.
Knowledge graphs flip this around. In a graph database like Neo4j, data is stored as nodes (entities — a customer, a product, an incident, a person) and relationships (typed, directional edges — PURCHASED, CAUSED_BY, REPORTS_TO, DEPENDS_ON). Because relationships are first-class citizens rather than something inferred from proximity, a graph can answer questions that require:
Multi-hop traversal — "Show me every service that depends, even indirectly, on the database we're deprecating."
Path explanation — "Why is this transaction flagged?" answered as a literal path of connected entities, not a paragraph that merely sounds plausible.
Aggregation over structure — "Which supplier has the most single points of failure across our product lines?"
Temporal and causal chains — sequences of events where order and causality matter, not just topical similarity.
Crucially, a graph doesn't replace vector search it complements it. Many production architectures now combine both: vector indexes (which Neo4j supports natively) handle fuzzy, semantic entry points into the graph, while graph traversal handles the precise, multi-hop reasoning once an entry point is found. This hybrid is often called GraphRAG, and it's the bridge between the RAG era and the reasoning era this post is about.
The next step beyond GraphRAG is giving the model agency over the graph letting it decide, turn by turn, what to query, rather than fetching a fixed set of graph results once and stuffing them into a prompt. That's where Gemini and MCP come in.
The Architecture at a Glance
At a high level, the system has four moving parts:
Gemini — the reasoning and orchestration layer. It interprets the user's intent, decides which tools to call, evaluates intermediate results, and decides whether it has enough information to answer or needs another hop.
An agent runtime — typically Google's Agent Development Kit (ADK), which handles the orchestration loop, tool-calling protocol, and session/memory management around Gemini.
MCP — the standardized interface between the agent and its tools. Instead of writing bespoke integration code for every data source, the agent talks to an MCP server that exposes a consistent set of callable tools, resources, and prompts.
Neo4j — the knowledge graph itself, exposed to the agent through an MCP server for Neo4j, which translates natural-language-derived tool calls into Cypher queries and returns structured, serialized graph results (nodes, relationships, and their properties).
MCP: The Connective Tissue
A Neo4j MCP server typically exposes tools such as:
get_schema — introspects the graph's node labels, relationship types, and property keys, so the agent knows what it can ask before it asks it.
read_cypher — executes a read-only Cypher query and returns serialized results.
write_cypher — executes a write query, usually gated behind stricter permissions.
This schema-introspection step matters enormously for reliability. A common failure mode in naive "LLM writes Cypher" setups is the model hallucinating a relationship type or property that doesn't exist in your graph. By having the agent call get_schema first and by keeping that step as part of the agent's standard operating procedure , dramatically cut down on malformed queries.
Agentic Reasoning Loop
The system operates in a multi-step cycle rather than a single retrieval shot:
Schema Inspection: The agent queries the MCP server for the latest graph schema metadata.
Path Decomposition: Gemini breaks complex multi-part questions into discrete graph questions.
Targeted Query Execution: Gemini issues Cypher queries via MCP to traverse relevant paths and isolate subgraphs.
Context Synthesis & Verification: The model evaluates whether the retrieved relationships fully satisfy the objective, executing follow-up traversals if dependency gaps remain.
Combining Gemini’s reasoning with Neo4j and MCP replaces fragile vector search with an explainable, auditable, and modular enterprise architecture.
Beyond Read-Only: Agentic Actions on the Graph
Once an agent can reliably read from a graph, a natural next step is letting it write to it carefully. Examples that show up in production systems:
An agent that ingests new support tickets, extracts entities and relationships with Gemini, and writes them into the graph as new nodes and edges, effectively building the knowledge graph incrementally as an ongoing task.
An agent that, after diagnosing an incident's blast radius via traversal, calls a second MCP server (say, one connected to a ticketing system) to open remediation tasks for each affected account owner it found in the graph.
A memory-augmented agent that stores conversation-derived facts about a user as graph nodes, so future sessions can traverse a persistent, structured memory instead of re-summarizing raw chat history.
This is also where multi-server orchestration becomes valuable: a single Gemini-based agent can hold connections to a Neo4j MCP server and a Jira MCP server and a Slack MCP server simultaneously, using the graph as the reasoning substrate and the other servers as the hands that take action once reasoning concludes.
Design Considerations and Failure Modes
Schema-first, always. Never let the agent write ad hoc Cypher against a schema it hasn't inspected in the current session. The get_schema step is cheap and prevents most hallucinated-query failures.
Separate read and write permissions at the MCP server layer, not just in the prompt. Prompt instructions like "only read, never write" are guidance, not security. If an agent shouldn't be able to mutate the graph, the MCP server's credentials should not have write access, full stop.
Cap traversal depth and result size. An agent given free rein to traverse can construct expensive, unbounded queries. Constrain Cypher templates or add server-side limits so a single tool call can't return or scan an unreasonable amount of data.
Keep the graph model close to the questions you'll actually ask. A graph that's a literal 1:1 mirror of your relational schema often isn't much more useful for reasoning than the relational schema itself. The payoff comes from modeling the relationships that actually matter for the questions your agent needs to answer causal links, ownership chains, dependency graphs even if that means a deliberate, curated schema rather than an automatic dump of every table.
Evaluate on multi-hop questions specifically. Standard RAG evaluation sets (single-fact lookup) won't tell you whether your agent can actually chain reasoning steps. Build an eval set of questions that require two, three, and four hops, and check not just the final answer but whether the traversal path itself was correct.
Watch for combinatorial explosion in graph results. A query returning thousands of matching paths will blow past context limits and confuse the model. Aggregate, paginate, or ask the agent to narrow its own queries when result sets get large.
Where This Pattern Fits Best
This architecture earns its complexity when the underlying questions genuinely require relationship traversal, not just topic matching:
Root-cause and impact analysis — IT operations, incident response, supply chain disruption.
Fraud and risk investigation — tracing money, identity, or account relationships across many hops.
Enterprise knowledge management — org charts, project dependencies, compliance chains, where "who's affected by this decision" is a graph question.
Recommendation and personalization — reasoning over how entities relate (co-purchases, shared attributes, social graphs) rather than pure content similarity.
Regulatory and compliance auditing — where an answer needs to be explainable as a literal, inspectable path of evidence, not a plausible-sounding summary.
Top comments (0)