What This Article Is About
This is the fourth benchmark in the series. The previous three articles tested four frameworks:
- QAnything v2 (vector RAG, article 02)
- LightRAG 1.5.6 (graph + vector hybrid, article 02)
- GraphRAG 3.1.1 (LLM-driven knowledge graph, article 03)
- HippoRAG 2.0 (hippocampus-inspired PPR graph diffusion, article 03)
This one tests the last framework: HyperGraphRAG 1.0.6 (NeurIPS 2025).
What HyperGraphRAG Actually Does
Traditional knowledge graph edges are binary: entity A → entity B. This means a relationship like "A, B, and C jointly participate in an event" must be decomposed into multiple binary edges — and the decomposition loses the joint semantic context.
HyperGraphRAG uses hyperedges to address this: a single hyperedge can connect any number of entities at once.
Traditional graph: A → B, B → C, A → C (3 separate edges, context fragmented)
Hypergraph: {A, B, C} connected by a single hyperedge (joint context preserved)
The pipeline from documents to hypergraph:
Documents → LLM extracts entity sets (co-occurring entities per passage, not pairwise)
→ Build hypergraph (each entity set = one hyperedge)
→ BGE vectorizes entities
→ Query: vector recall for seed entities → hypergraph diffusion
→ extract relevant passages → LLM answer
The theoretical expectation: for "multi-entity joint participation" relationships, hypergraph extraction should surface more signal than binary graph extraction.
Deployment
Installation
HyperGraphRAG is not on PyPI. There is no pip install hypergraphrag, and pip install -e . fails because there is no setup.py or pyproject.toml. The working approach: add the source directory to PYTHONPATH and install the requirements from the README:
pip install -r requirements.txt # graspologic, nano-vectordb, etc.
# Run with:
PYTHONPATH=/path/to/HyperGraphRAG python run_eval.py
SiliconFlow API: Two Hidden Constraints
Constraint 1: Content filter blocks environment variable patterns (HTTP 400, code 20015)
The test documents contain many .env example snippets like:
EMBEDDING_MODEL=jina-embeddings-v4
EMBEDDING_BINDING=jina
EMBEDDING_ASYMMETRIC=true
The ALLCAPS=value format triggers SiliconFlow's content safety filter. Fix: regex-strip any lines matching sensitive patterns before sending embedding requests.
Constraint 2: bge-large-en-v1.5 has a ~512-token input limit
Testing revealed that inputs over roughly 1000 characters return 400. Dense markdown tables (every | and % consumes a token) can exceed 512 tokens at just 900 characters.
Final setting: truncate each text to 700 characters (word-boundary truncation), giving a buffer for high-density content.
Network Stability During Long Runs
GLM-4-flash and SiliconFlow both produced intermittent ConnectTimeout and HTTP 500 errors during the 7-hour indexing run. Two necessary changes:
- Set LLM client timeout to 180s, with 6 retry attempts (10/20/30/40/50/60s backoff)
- Switch from one-shot
rag.insert(all_docs)to per-document insert with a progress file: each successfully indexed document is immediately checkpointed; a crash and restart continues from where it left off
Without both changes, finishing a 443-minute index build in a single run is unrealistic.
Benchmark Results
Numbers
| Metric | HyperGraphRAG 1.0.6 |
|---|---|
| Index build time | 443.2 min |
| Boundary refusal rate | 26.3% |
| P90 latency | 49.3 s |
| Average latency | 29.6 s |
| Single-hop match | 0.081 |
| Multi-hop match | 0.171 |
| Boundary match | 0.021 |
Answer matching uses Jaccard keyword overlap, not LLM judge; LLM is GLM-4-flash throughout
Multi-hop: 0.171 — Tied with LightRAG, Not the Breakout
The theoretical advantage of hyperedges didn't materialize here. HyperGraphRAG's 0.171 on multi-hop is within noise of LightRAG's 0.178.
A few plausible explanations:
- The test documents are technical reference materials (configuration guides, API docs). Multi-entity joint relationships are rare in this genre — that's precisely where hyperedges help most, but the signal isn't there to exploit
- Hyperedge quality depends on LLM extraction quality; GLM-4-flash's performance limits what the graph can represent
- BGE truncation to 700 characters may have dropped some semantic signal from longer entity descriptions
Index Build Time: 443 Minutes
GraphRAG took 31 minutes, HippoRAG took 36 minutes. HyperGraphRAG took 443 minutes — roughly 7.4 hours.
The cause: HyperGraphRAG issues multiple LLM calls per chunk to extract entity sets, and llm_model_max_async=1 (GLM rate limit). With 124 chunks, ~4 LLM calls each, plus the accumulated timeout retries, the total inflates significantly.
In a production setup with a higher-concurrency LLM (GPT-4o), the indexing time would likely drop to 30–60 minutes. GLM's low concurrency is the amplifying factor here.
Boundary Refusal: 26.3% — Better Than Expected for Graph RAG
5 of 19 boundary questions (26.3%) were correctly declined. This is higher than GraphRAG's 15.8% and HippoRAG's 5.3%.
Likely reason: HyperGraphRAG's hybrid mode (local + global search) may be more willing to return "not found in the knowledge base" when neither local graph traversal nor global context returns relevant hits.
Full Five-Framework Comparison
Combining all four articles:
| Framework | Type | Index time | Refusal rate | P90 latency | Single-hop | Multi-hop |
|---|---|---|---|---|---|---|
| QAnything v2 | Vector RAG | ~5 min | 26.3% | 52.2 s | 0.111 | 0.162 |
| LightRAG 1.5.6 | Graph+Vector | ~8 min | 10.5% | 19.4 s | 0.082 | 0.178 |
| GraphRAG 3.1.1 | Graph RAG | 31 min | 15.8% | 32.4 s | 0.107 | 0.211 |
| HippoRAG 2.0 | Graph RAG | 36 min | 5.3% | 84.0 s | 0.122 | 0.163 |
| HyperGraphRAG 1.0.6 | Hypergraph RAG | 443 min | 26.3% | 49.3 s | 0.081 | 0.171 |
Patterns across the five frameworks:
Index cost and multi-hop performance are decoupled: GraphRAG achieved the highest multi-hop score (0.211) with a 31-minute build. HyperGraphRAG spent 443 minutes and got 0.171. More indexing time does not buy better retrieval.
LightRAG has the best cost-effectiveness: 8-minute build, second-best multi-hop (0.178), lowest P90 latency (19.4s). Its only weakness is low refusal rate (10.5%) — the graph tends to produce an answer regardless of relevance.
Refusal capability is a design decision, not an emergent property: QAnything's 26.3% refusal rate comes from explicit confidence-threshold filtering. Graph RAG frameworks don't ship with a refusal mechanism and show near-zero rates as a result.
Single-hop: graph structure adds nothing: HippoRAG's 0.122 is the best among graph RAG approaches, but vector RAG and graph RAG both cluster in the 0.08–0.12 range. For direct factual retrieval, knowledge graph overhead doesn't help.
Which Framework Should You Choose?
Based on the five-framework data:
Default to LightRAG for most use cases: fast indexing, strong multi-hop, lowest latency. Add a confidence post-processing step to improve refusal behavior if needed.
Choose GraphRAG if:
- Multi-hop reasoning is your primary use case (policy cross-referencing, causal chain analysis)
- You have LLM budget for the indexing cost
- You're using a model with reliable structured output (GPT-4o, Claude) to enable community reports; the GLM structured output issues in this test disabled that feature entirely
- 30-minute build times are acceptable
Choose QAnything if:
- You need strong out-of-the-box refusal capability with minimal configuration
- You want a self-contained deployment without managing graph infrastructure
Wait on HyperGraphRAG: the hyperedge concept is theoretically compelling, but version 1.0.6 has real production friction:
- No PyPI package; source-only installation
- 7-hour build time in this configuration
- Multi-hop performance didn't surpass LightRAG in a technical document domain
- Check back when a production-grade release lands
Code Highlights
Full code at llm-in-action/kb-04-hypergraphrag-eval/.
Text truncation + sanitization
import re
_SENSITIVE_LINE_RE = re.compile(
r'[^\n]*(?:'
r'api[_\-]?key|secret|token|password|Authorization'
r'|sk-[A-Za-z0-9]{10,}'
r'|[A-Z][A-Z0-9_]{3,}=[^\s\n]+' # ENV_VAR=value
r')[^\n]*',
re.IGNORECASE,
)
_MAX_CHARS = 700 # bge-large-en-v1.5 via SiliconFlow ~512 tokens
def _truncate(text: str) -> str:
text = _SENSITIVE_LINE_RE.sub("[REDACTED]", text)
if len(text) <= _MAX_CHARS:
return text
truncated = text[:_MAX_CHARS]
last_space = truncated.rfind(" ")
return truncated[:last_space] if last_space > 0 else truncated
Per-document insert with checkpoint resume
def build_index(rag, doc_files: list[Path]) -> float:
progress = load_progress() # load completed list
for fpath in doc_files:
if fpath.name in progress["indexed"]:
continue # skip already done
content = fpath.read_text().strip()
rag.insert([content]) # insert one at a time
progress["indexed"].append(fpath.name)
save_progress(progress) # checkpoint immediately
Next: Series Wrap-Up — benchmark methodology limitations, what this test set measures (and doesn't), and how to design a production-grade RAG evaluation from scratch.
Check out PrimeSkills — a curated marketplace of AI agents and skills validated in real-world, enterprise-grade workflows. Not demos — things that actually work in production.
Find more on my Homepage
Top comments (0)