Throughout the AI System Design series we follow Mailmind, a fictional AI email assistant that reads an inbox, triages messages, and drafts replies. It is our running example.
Before you read: Why RAG Exists: The Context Problem and Embeddings and Vector Search, Demystified.
RAG (retrieval-augmented generation) means finding relevant material and handing it to the model so it can answer from it. The finding step can be a keyword search, a vector search, a database query or a graph traversal. This article compares the two retrieval styles people argue about most, vector search and knowledge graph traversal, both of which can feed RAG. Vector search uses embeddings to answer fuzzy questions about meaning across unstructured text. Graph traversal answers precise multi-hop questions about how specific entities connect. Mailmind, our AI inbox assistant, combines them: the graph narrows the candidates, then vector search ranks by meaning within that narrowed set.
Search is fast. Graph traversal is fast. But they break on each other's questions. The smarter system isn't the one with the better index. It's the one that knows when to use which lookup.
After the last episode's production-grade prompt engineering, you can make Mailmind behave consistently. But even a well-instructed agent can only answer from what it can find. What does Mailmind actually know about your inbox? And when a question goes beyond the last few messages in the chat window, how does it find the right memory?
When does semantic search beat a database query?
Vector search retrieves passages by meaning, not by exact words. That makes it the right choice when you can describe what you need but cannot name it.
Think of it as a search engine that indexes ideas, not keywords. You type "emails where the customer was unhappy about a missed deadline," and it surfaces messages that never say "unhappy" or "missed deadline" but convey frustration with phrasing like "This took longer than expected." That ability comes from how the text is stored and searched.
Mailmind breaks every email into overlapping chunks of a few hundred tokens. Each chunk is turned into an embedding, a list of numbers that captures its meaning, once, when the email arrives. Those embeddings live in a vector database, which can find the nearest neighbours to any vector you give it. When a question comes in, Mailmind embeds the question and asks the vector database for the chunks whose vectors are closest, usually by cosine similarity. The top results go to the LLM (large language model) as context. The name RAG comes from Lewis et al., who paired a pre-trained neural retriever over a vector index of Wikipedia with a text generator and fine-tuned the retriever's query side and the generator together (Lewis et al. 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks). Today's pipelines apply the same idea with off-the-shelf embedding models and a vector database, usually without training anything.
That retrieval path is perfect for open-ended discovery. It is the wrong tool for exact facts, though, and the next sections explain what to use instead.
Three lookups, not two
Before reaching for a graph, notice that many "structural" questions are not graph questions at all. An email already carries metadata: sender, recipients, thread, date, labels. Three kinds of lookup cover almost everything:
- Exact metadata, with SQL or filters. Single-hop facts stored on the email itself: "emails from Sam this month", "has Sam replied in the Stratos thread?" An ordinary database query answers these exactly. Many vector databases also store metadata next to each vector and can filter on it during a search, so this often lives in the same system.
- Vector search. Meaning: "threads where a customer sounds frustrated about a delay."
- Graph traversal. Multi-hop relationships between entities: "people on the Falcon project whose emails I have not answered." The question chains project to people to their emails to your replies, and the first link, who works on Falcon, is not written on any single email.
A relational database can follow such chains too, with several joins. A graph database makes chains of relationships the natural thing to query, and the query stays readable as the chain grows.
What kind of question does a knowledge graph answer?
A knowledge graph answers questions about how specific real-world entities connect. You ask for the relationship between things, and it walks a network of facts to give a precise answer.
Picture a map of your inbox's people, projects, threads, emails and orders. You can trace a path from one point to another and count the steps. Mailmind builds this map from nodes (Person, Project, Thread, Email, Order) and relationships such as WORKS_ON, SENT, IN_THREAD, ABOUT, ORDERED and REPLIED_TO.
When a user asks "which people from the Falcon project have I not replied to in the last seven days?", the graph starts at the Falcon project node, follows WORKS_ON edges to Person nodes, follows SENT edges to the emails they sent in that window, and drops every email that already has a reply from you. This is a graph traversal, usually written in a query language such as Cypher:
MATCH (:Project {name: "Falcon"})<-[:WORKS_ON]-(p:Person)-[:SENT]->(e:Email)
WHERE e.sent_at >= datetime() - duration("P7D")
AND NOT EXISTS { (:Email {from_me: true})-[:REPLIED_TO]->(e) }
RETURN DISTINCT p.name
In words: start at Falcon, walk to everyone who works on it, then to the emails they sent in the last seven days, keep only the emails with no reply from you, and return the senders. The database does not scan every row. It walks edges, which gives it a different cost model and failure mode from a vector search. Graph Databases for AI: When Relationships Are the Data covers how graph databases work, later in this series.
The answer is exact, as long as the graph itself is right: if a person was never linked to Falcon, or appears twice under two names, the query cannot know. And ask a graph for "complaint-like threads" and it returns nothing at all. That failure points straight at the limits of structured data, and at why Mailmind combines lookups.
Why can't either lookup handle every question alone?
Vector search stumbles on exact facts. "Has Sam replied to the Stratos thread?" is not a question about meaning; it is a lookup: is there a message in that thread whose sender is Sam? Vector search might retrieve a chunk where Sam mentions Stratos in another context, and the model may then wrongly assume a reply. A metadata query, or a check for a REPLIED_TO edge, returns a yes or no.
The graph stumbles on anything shaped like a description. "Find emails where the sender was upset about shipping" fails because "upset" is not a node or a relationship. No edge links Emotion to Email. The graph has no way to compute tone from raw text, so it returns nothing, even when the right emails exist.
graph TD
A["User question"] --> B["Vector search"]
A --> C["Graph traversal"]
B --> D["Misses exact facts: did Sam reply?"]
C --> E["Misses meaning: upset tone"]Vector search and the graph cover two different axes of memory. One axis is structure: who did what to whom. The other is meaning: what kind of message this is. Metadata covers the plain facts on each email. Combining them lets Mailmind answer questions that mix all three.
How does Mailmind combine both kinds of lookup?
Mailmind does not choose between vector search and the graph. It sends the structural part of a question to the graph to get a precise candidate list, then uses vector search to rank by meaning within that list. The graph handles "who, where, when"; vector search handles "what kind of".
Consider the question "Show me Falcon project threads this month that discuss budget." The flow looks like this:
- The system parses the question into a structural part (Falcon project, this month) and a semantic part (discussing budget).
- The graph finds every thread ABOUT the Falcon project with activity this month. Say that yields 14 threads.
- Mailmind embeds only the semantic part of the question ("discussing budget") and runs a vector search over the chunk embeddings it stored when those emails arrived, filtered to the 14 thread IDs. Nothing is re-embedded at query time except the question.
- The top-scoring chunks go into the LLM's context.
The graph shrinks the search space from everything in the inbox to exactly the relevant threads. Vector search then contributes what a keyword filter would miss: budget might be called "spend," "allocation," or "runway."
flowchart LR
A["User question"] --> B{"Structural filter?"}
B -->|Yes| C["Graph traversal: narrow to candidate threads"]
B -->|No| D["Vector search over all chunks"]
C --> E["Vector search over stored chunks, filtered to candidates"]
D --> F["Rank by similarity"]
E --> F
F --> G["LLM with top results"]There is one crack in this stack. None of it works if one person exists as two separate nodes. Entity resolution, knowing that "Sam K." and "Samuel Kim" are the same person, is what keeps the graph honest. Entity Resolution: One Thing, Many Names covers it later in this series.
Quick Reference: three lookups
| Property | Metadata or SQL | Vector search | Graph traversal |
|---|---|---|---|
| Data shape | Fields on each email (sender, thread, date) | Text chunks stored as embeddings | Entities and the relationships between them |
| Query type | Exact single-hop facts | Semantic: "find things like this" | Structural: "how are A and B connected?" |
| Best for | Filters, counts, "did X reply?" | Open-ended discovery, tone, topics | Multi-hop paths across entities |
| Mailmind example | "Emails from Sam this month" | "Threads where a customer was upset about a delay" | "Falcon people I haven't replied to" |
| Failure mode | Cannot judge meaning | Finds similar text, can miss or invent exact facts | Returns nothing when the question is about meaning |
| Underlying store | A relational database, or the filters in a vector database | A vector database (such as Pinecone or pgvector) | A graph database (such as Neo4j) |
Frequently Asked Questions
Q: If I already have a vector database, why add a graph?
Often you don't need one. Exact facts like "emails from Sam in April" come from metadata filters, which many vector databases support, or from an ordinary database. A graph earns its place when questions chain several relationships, such as "which Falcon contacts did I email in April?", which needs to know who is on the Falcon project before it can look at emails. A graph stores those relationships directly and answers in one traversal, exactly, provided the entities were extracted and resolved correctly.
Q: Can a knowledge graph store full email text, avoiding vector search entirely?
You could store text as a property on nodes, but graph traversal cannot judge how similar two passages are. If the user asks "find any thread where the tone matches this complaint," traversal has no way to compare tone; it would return nothing or everything. You still need vector search to score by meaning. Some graph databases now include a vector index of their own, as Neo4j's documentation on vector indexes describes, which runs the same similarity search inside the graph store.
Q: How do you keep a knowledge graph in sync with live email traffic in Mailmind?
Every incoming email triggers a small pipeline: extract entities with an NER (named entity recognition) model, resolve identities, then upsert nodes and relationships into the graph. Updates and deletions are trickier. Mailmind uses a micro-batch approach that replays events from a write-ahead log. The graph is not a static dump; it is a continually updated reflection of the inbox.
Q: Doesn't building a knowledge graph require a rigid schema?
Modern property graph databases let you mix schemaless flexibility with lightweight constraints. Mailmind's graph defines a handful of core node labels (Person, Project, Thread, Email, Order, Meeting) and relationship types, and you can add new ones without downtime. Uniqueness constraints on identity fields, like email address, keep the graph useful without locking you in.
Q: When does vector search fail on a question that seems simple?
When the answer depends on a specific fact or relationship, vector search can retrieve chunks that mention both entities without showing the connection. For example, "Did Alice reply to the strategy doc thread?" might pull an email where Alice mentions the strategy doc in a different context, and the model may then claim a reply that never happened. A metadata lookup, or a check for a REPLIED_TO edge in the graph, answers it definitively.
Test yourself
User asks Mailmind: "Find threads from this week that have a similar frustrated tone to this complaint email about a delayed refund." You run a pure knowledge graph query and get back nothing useful. Why does the graph return nothing, and what do you do to actually answer the question?
Answer: The graph knows timestamps, participants and orders. "Frustrated tone" is not stored anywhere; it is a quality of the text, and traversal can only follow edges, so it has nothing to match. To answer, first narrow the candidates to this week's threads with a date filter. That is plain metadata; no graph is needed. If you like, use the graph to boost threads linked to a refund order, since the complaint is about a refund, but do not drop the others, because the user asked about tone, not refunds. Then embed the complaint email, run a vector search over this week's stored chunk embeddings, and return the threads whose chunks score highest. Embeddings capture meaning and some of the tone, so this is a good first pass; for a sharper signal, have a model label frustration on the shortlist. The filter keeps the search small; vector search turns "similar tone" into a ranking.
Next: Hallucination Detection: Trust, but Verify. It covers how to catch the facts an agent invents, even after it has found the right memories.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
- Vector indexes (Neo4j Cypher Manual)
Originally published at Internals Decoded. AI internals, explained conversationally.
Top comments (0)