What This Article Is About
The previous installments built the test set and benchmarked classic vector RAG (QAnything vs LightRAG).
This one tests "graph-enhanced RAG" — the idea being that a knowledge graph should help with multi-hop reasoning: A points to B, B points to C, and while traditional vector retrieval often stops at B, graph traversal can follow edges to reach C.
Two representative frameworks:
- GraphRAG 3.1.1 (Microsoft — the one that put "graph RAG" on the map in 2024)
- HippoRAG 2.0 (a 2025 academic framework inspired by hippocampus memory models)
Same 89 questions (50 single-hop / 20 multi-hop / 19 boundary), same LLM (GLM-4-flash). Numbers talk.
What These Frameworks Actually Do
GraphRAG: LLM-Driven Knowledge Graph Extraction
GraphRAG's pipeline is heavy: it uses an LLM to extract entities and relationships from documents, builds an explicit knowledge graph, groups nodes into communities (Louvain algorithm), and has the LLM generate a summary report per community. At query time, local search does nearest-neighbor retrieval on the graph; global search aggregates community reports.
Documents → LLM extracts entities/relations → Knowledge graph → Community clustering → Reports
↓
Query ─── local search: graph neighbors + vector retrieval ───────────────────────→ LLM answer
└── global search: aggregate community reports ─────────────────────────────→ LLM answer
The trade-off: extremely expensive to index (LLM call per text chunk for relation extraction), but the result supports structured graph queries.
HippoRAG: Hippocampus-Inspired Graph Retrieval
HippoRAG models the retrieval process after human memory's PPR (Personalized PageRank) mechanism: an LLM extracts triples (entity, relation, entity) from documents; a large embedding model (NV-Embed-v2, Mistral-7B based, 1024-dim) vectorizes all entities; at query time, a vector recall finds seed nodes, and PPR diffusion spreads across graph edges to find related nodes.
Documents → LLM extracts triples (head, rel, tail) → Knowledge graph
→ NV-Embed-v2 vectorizes entities → Vector index
↓
Query → Vector recall for seed entities → PPR graph diffusion → Retrieve passages → LLM answer
The trade-off: requires a local 7B model (~14 GB VRAM), but avoids API-based embedding and theoretically achieves higher embedding quality.
Deployment and Pitfalls
GraphRAG: The Pipeline Is More Fragile Than It Looks
GraphRAG is driven by a YAML config; graphrag index runs workflows in sequence. The documentation is fairly complete — but several landmines await.
Pitfall 1: Structured output failures crash the pipeline
The community report generation step (create_community_reports) expects JSON from the LLM. GLM-4-flash's structured output is inconsistent — many prompts return empty JSON, producing a DataFrame with no "community" column, which causes a KeyError during the merge step and kills the run.
The fix: override the workflows: list in settings.yaml to skip create_community_reports, then manually create an empty community_reports.parquet with the correct schema. Local search reads this file but handles an empty table gracefully.
Pitfall 2: Vector dimension mismatch
GraphRAG defaults to vector_size: 3072 (for OpenAI text-embedding-3-large), but BGE-large-en-v1.5 produces 1024-dimensional vectors. Without explicit configuration, the run fails with Vector has dimension 1024, but index configured with 3072. Solution: add vector_size: 1024 under vector_store: in settings.yaml.
Pitfall 3: Re-running re-consumes LLM quota
After a failed run, graphrag index re-runs everything from scratch — including the already-completed extract_graph step, which issues LLM calls for all 4000+ text chunks. You need a separate script that runs only generate_text_embeddings against the existing intermediate artifacts.
Pitfall 4: SiliconFlow API rejects certain content
The test documents include .env configuration snippets containing strings that look like API keys. SiliconFlow's content filter returns HTTP 400 (code 20015) for these inputs, causing batch embedding failures. Restricting embed_text.names to only entity_description (skipping text_unit_text) sidesteps the filter — local search doesn't need text-unit embeddings anyway.
HippoRAG: Local Model Dependency Hell
HippoRAG's biggest dependency is NV-Embed-v2 — a Mistral-7B-based embedding model.
Pitfall 1: Offline loading fails
modeling_nvembed.py calls AutoTokenizer.from_pretrained("nvidia/NV-Embed-v2") — a string ID, not a local path — in __init__, which causes the transformers library to attempt a HuggingFace Hub lookup on every startup. In an air-gapped environment, this crashes immediately.
The fix: monkey-patch AutoTokenizer.from_pretrained at the top of the script to redirect the string "nvidia/NV-Embed-v2" to the local model directory.
Pitfall 2: Out of memory on a 12 GB GPU
On an RTX 3060 (12 GB), loading NV-Embed-v2 (Mistral-7B bf16, ~14 GB weights) with device_map="auto" offloads some layers to CPU, but forward pass activations still need GPU scratch space. The default batch_size=16 and max_seq_len=2048 causes OOM.
Solution: set embedding_max_seq_len=512, embedding_batch_size=1, and explicitly embedding_model_dtype="bfloat16" to prevent auto from choosing fp32.
Benchmark Results
Numbers
| Metric | GraphRAG 3.1.1 | HippoRAG 2.0 |
|---|---|---|
| Index build time | 31.1 min | 36.5 min |
| Boundary refusal rate | 15.8% | 5.3% |
| P90 latency | 32.4 s | 84.0 s |
| Average latency | 26.6 s | 30.1 s |
| Single-hop match | 0.107 | 0.122 |
| Multi-hop match | 0.211 | 0.163 |
| Boundary match | 0.030 | 0.021 |
Answer matching uses Jaccard keyword overlap, not LLM judge; LLM is GLM-4-flash throughout
Multi-hop reasoning: GraphRAG wins, but not by a landslide
GraphRAG's multi_hop match rate is 0.211 vs HippoRAG's 0.163 — about 30% higher. The graph structure does help with multi-hop reasoning, consistent with Microsoft's original claims.
But the gap isn't transformative. Given GraphRAG's indexing cost (31 minutes, thousands of LLM calls for entity extraction), this margin feels modest.
For reference, the vector RAG results from the previous article: LightRAG scored 0.152 on multi-hop, QAnything 0.125. GraphRAG's 0.211 does beat vector RAG meaningfully, but HippoRAG's 0.163 barely edges LightRAG.
Single-hop retrieval: graph adds no benefit
On single-hop factual queries ("what is the default value of X?"), HippoRAG (0.122) slightly outperforms GraphRAG (0.107), but both fall short of LightRAG (0.156) from the previous article.
The takeaway: knowledge graph structure doesn't help — and may actually hurt — direct factual retrieval. The graph traversal path is longer, introduces more "graph noise" (community summaries, relation paths), and dilutes the direct answer signal.
Boundary refusal: GraphRAG surprisingly better
GraphRAG's boundary refusal rate is 15.8% vs HippoRAG's 5.3%. Unexpected.
GraphRAG's local search does say "based on the knowledge base, I cannot answer..." occasionally. HippoRAG's PPR diffusion almost always finds "related" nodes, so the LLM tends to attempt an answer regardless.
That said, 15.8% is still far below QAnything's 52.6% from the previous article. Graph-enhanced RAG frameworks aren't designed for refusal — and it shows.
Latency: HippoRAG is slower than expected
HippoRAG's P90 latency is 84 seconds, average 30 seconds. This defied expectations — local model inference should be faster.
Two reasons:
-
embedding_batch_size=1to avoid OOM kills batch efficiency. On adequate hardware (A100 or 3090 Ti) with batch_size=16, embedding throughput would be 10-20× faster — this is a hardware-constrained artifact, not a framework flaw. - HippoRAG's query path is multi-step: vector recall → PPR diffusion → passage assembly → LLM generation. Each step has overhead.
GraphRAG's 26.6 second average is also slow, driven by GLM-4-flash's rate limits forcing concurrent_requests: 1.
Full Comparison: All Four Frameworks
Combining both articles:
| Framework | Type | Index time | Refusal rate | P90 latency | Single-hop | Multi-hop |
|---|---|---|---|---|---|---|
| QAnything v2 | Vector RAG | ~5 min | 52.6% | 49.2 s | 0.132 | 0.125 |
| LightRAG 1.5.6 | Graph+Vector | ~8 min | 21.1% | 17.9 s | 0.156 | 0.152 |
| GraphRAG 3.1.1 | Graph RAG | 31.1 min | 15.8% | 32.4 s | 0.107 | 0.211 |
| HippoRAG 2.0 | Graph RAG | 36.5 min | 5.3% | 84.0 s | 0.122 | 0.163 |
Patterns that emerge:
- Index time scales with how "heavy" the graph construction is: LightRAG asynchronously extracts graph structure in ~8 minutes; GraphRAG and HippoRAG do per-chunk LLM extraction, landing in the 30-minute range.
- Graph RAG genuinely beats vector RAG on multi-hop: GraphRAG 0.211 vs LightRAG 0.152 — statistically visible advantage.
- Vector RAG beats graph RAG on single-hop: LightRAG 0.156 outperforms all graph RAG approaches.
- Refusal capability is a design choice, not an emergent property: QAnything has confidence-threshold filtering; graph RAG frameworks don't — and show near-zero refusal rates as a result.
Which One Should You Choose?
Based on this benchmark:
Prefer GraphRAG if:
- Multi-hop reasoning is your core use case (policy cross-referencing, causal chain analysis)
- You have LLM budget to absorb the indexing cost
- You're using a well-behaved LLM for structured output (GPT-4o, Claude) to enable community reports
- 30-minute build times are acceptable
Prefer HippoRAG if:
- You have local GPU (>12 GB, ideally 24 GB) and want to avoid API-based embedding
- The knowledge base updates frequently (HippoRAG's incremental update is faster than GraphRAG's)
- Privacy requirements rule out sending documents to external LLM APIs for graph extraction
Skip both graph RAG frameworks if:
- Your primary use case is fast factual lookup (use LightRAG)
- You need strong refusal capability out of the box (use QAnything, or add confidence post-processing)
- Deployment cost matters (graph RAG indexing LLM cost is roughly 5-10× vector RAG)
Caveats in this benchmark:
- GLM-4-flash's inconsistent structured output disabled GraphRAG's community reports — those power global search, which is where GraphRAG often shines most. Local search only was tested here.
- HippoRAG's latency is hardware-constrained (batch_size=1); on 24+ GB VRAM, expect significantly better numbers.
- Jaccard keyword matching is conservative — an LLM judge would likely score higher across the board.
Code
Full code in llm-in-action/kb-03-graphrag-eval/ and llm-in-action/kb-03-hipporag-eval/.
GraphRAG key configuration:
# settings.yaml
workflows: # Skip community reports if LLM structured output is unreliable
- create_base_text_units
- create_final_documents
- extract_graph
- finalize_graph
- extract_covariates
- create_communities
- create_final_text_units
- generate_text_embeddings
vector_store:
type: lancedb
vector_size: 1024 # Must match embedding dimensions
embed_text:
names:
- entity_description # Embed only entity descriptions, skip text_unit
batch_max_tokens: 300
batch_size: 4
concurrent_requests: 1 # Required under GLM-4-flash rate limits
HippoRAG key initialization:
from hipporag.utils.config_utils import BaseConfig
global_config = BaseConfig(
embedding_max_seq_len=512, # Reduce VRAM pressure
embedding_batch_size=1,
embedding_model_dtype="bfloat16",
)
hipporag = HippoRAG(
global_config=global_config,
save_dir=SAVE_DIR,
llm_model_name=LLM_MODEL,
llm_base_url=LLM_BASE_URL,
embedding_model_name=EMBEDDING_MODEL_PATH, # Local path
)
Next: HyperGraphRAG Benchmark — Multi-hop Reasoning with Hypergraph Structure. Same 89 questions, measuring how much upgrading from binary edges to hyperedges improves multi-hop reasoning.
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage
Top comments (0)