An LLM alone is like a lawyer with a photographic memory, but without understanding of chains of paragraphs. It knows that § 823 BGB exists, but not necessarily how it relates to § 254 BGB. A knowledge graph closes this gap – by mapping relationships that a pure language model does not intrinsically understand.
TL;DR: Knowledge graphs are not competition for LLMs, but their structured complement. For legal applications, they are often superior – but not without costs.
The Core Conflict: Vector Search vs. Graph Structure
Modern AI knowledge systems are based on two fundamentally different approaches:
Vector RAG (the standard)
Documents are cut into chunks, embedded as vectors, and queried via similarity search. Works well for "Find me a document that looks like this." Works poorly for "Which paragraphs are relevant for this case, and how do they relate?"
Strength: Easy to set up, no manual modeling, good results for fact-based questions.
Weakness: No understanding of relationships between entities. A vector search might find § 823 BGB, but not automatically the associated commentaries, case law references, and exceptions from other paragraphs.
Knowledge Graph RAG (GraphRAG)
Documents are modeled as nodes (entities) and edges (relationships). A knowledge graph for law could contain nodes for paragraphs, judgments, commentaries, and legal terms – and edges that map "references", "was cited in", "restricts", or "supplements".
Strength: Relationships are explicitly modeled. Multi-step queries ("Which judgments refer to § 823 and were issued after 2020?") deliver precise results.
Weakness: High initial effort, labor-intensive maintenance, more complex infrastructure.
Practical Example: Legal Research
Imagine an LLM is supposed to answer the following question:
"A tenant has reduced the rent due to a defective heating system. The landlord contests this. Which paragraphs, judgments, and commentaries are relevant?"
Vector RAG: The system finds documents containing "rent defect", "rent reduction", and "heating". But it cannot recognize that § 536 BGB governs rent reduction, while § 569 BGB concerns extraordinary termination – and that a specific 2023 BGH ruling clarifies the balance between the two paragraphs.
GraphRAG: The knowledge graph contains the nodes § 536, § 569, BGH ruling VIII ZR 123/23 and the edges BGH ruling VIII ZR 123/23 interprets § 536 and § 569 references § 536 for heating defects. The system navigates along these edges and delivers a structured, context-aware answer.
| Criterion | Vector RAG | GraphRAG |
|---|---|---|
| Setup effort | Low (upload documents) | High (model graph) |
| Relationship understanding | None explicit | Explicitly modeled |
| Multi-hop questions | Weak | Strong |
| Updates | Re-upload | Extend graph |
| Data quality | Dependent on embedding | Dependent on graph design |
| Interpretability | Black box (embedding) | Traceable (edges) |
| Cost (operation) | Low | Higher (graph DB + query) |
| Scaling | Linear with documents | Complex with node count |
Advantages and Disadvantages in Detail
Advantages of Knowledge Graphs in AI Use
1. Multi-Hop Reasoning
A question like "Which exceptions apply to liability under § 823 BGB in road traffic?" requires multiple steps: find § 823 → discover § 254 (contributory negligence) → include § 7 StVG (strict liability) → find case law on this interplay. A knowledge graph navigates this chain along its edges. Vector RAG would have to hope that all relevant documents land in the same chunk.
2. Explainability
Every answer from a knowledge graph can be traced back via the edges. "I recommend § 536 BGB because BGH ruling VIII ZR 123/23 interpreted this paragraph in the context of heating defects." With Vector RAG, it remains unclear which chunk contributed to which part of the answer.
3. Domain-Specific Ontologies
Legal terms have precise, often ambiguous relationships. "Lawsuit" can be a "declaratory action", "performance action", or "formative action". A graph maps this hierarchy. A vector embedding does not.
4. Consistency
A well-maintained knowledge graph is free of contradictions. LLMs with pure Vector RAG can contradict themselves depending on which chunks end up in the context.
Disadvantages of Knowledge Graphs
1. Initial Modeling Effort (massive)
Building a legal knowledge graph means: manually or semi-automatically capturing paragraphs, judgments, commentaries, legal terms, and their relationships. That is months of work for a complete domain. Vector RAG is ready to use in hours.
2. Maintenance and Updates
Laws change. New judgments are added. A knowledge graph must be continuously maintained – new nodes, new edges, sometimes new relationship types. Vector RAG: simply load new documents into the vector database.
3. Cost/Benefit Ratio
The Atlan blog put it succinctly in 2026: "Vector RAG wins for single-hop questions. GraphRAG adds costs without adding accuracy for simple questions." For 90% of everyday AI questions, Vector RAG is perfectly sufficient.
4. Infrastructure Complexity
Vector DB + LLM is a standard stack (OpenAI + Pinecone, or local Chroma + Ollama). GraphRAG additionally requires a graph database (Neo4j, ArangoDB), a graph query planner, and often more computing power.
Hybrid Approach: The Best of Both Worlds
Practice in 2026 increasingly relies on hybrid architectures:
- Vector Search for breadth: Find relevant documents, paragraphs, judgments
- Knowledge Graph for depth: Navigate relationships between the found entities
- LLM for synthesis: Formulate the answer in natural language
An example from research is LegalGraphRAG (ACL 2026), which pursues exactly this approach for legal corpora: vector search for the initial retrieval phase, graph navigation for multi-hop refinement.
Query → Vector Search (find relevant documents)
→ Graph Traversal (explore relationships)
→ LLM (synthesize answer)
Who Benefits from GraphRAG?
| Scenario | Vector RAG | GraphRAG | Hybrid |
|---|---|---|---|
| General knowledge base (FAQ, manual) | ✅ Optimal | ❌ Overkill | ❌ |
| Legal research (individual case + sources) | ⚠️ Moderate | ✅ Strong | ✅ Optimal |
| Compliance checks (many interlinked rules) | ❌ Weak | ✅ Strong | ✅ Optimal |
| Scientific literature research | ✅ Good | ⚠️ Moderate | ✅ Optimal |
| Customer service (standard questions) | ✅ Optimal | ❌ Overkill | ❌ |
Conclusion
Knowledge graphs are not a silver bullet, but for domain-specific, relationship-rich applications like law, they are the decisive advantage over raw vector search. The price is high initial effort – which pays off when questions are complex and relationships are critical.
For most other applications, Vector RAG is perfectly sufficient. The art is knowing when the graph is worth the additional complexity.
Top comments (1)