Why your RAG hallucinates on tables (and a minimal local GraphRAG starter to fix it)

Hardcore Engineer — Fri, 11 Sep 2026 09:42:03 +0000

If you've ever built a RAG system for corporate documents, you've probably hit this wall: you ask "What was the revenue in Q3?", and the LLM confidently hallucinates a number with three extra zeros.

This isn't a model bug. It's an architectural blind spot. As recent research from Microsoft points out, naive 500-token chunking destroys table structures. Cosine similarity over embeddings just doesn't understand rows and columns.

The Fix: GraphRAG

Instead of flat vector search, we extract entities and relationships into a Knowledge Graph. When a user asks about numbers, the LLM translates the question into a Cypher query, and the graph returns exact data. Zero hallucinated digits.

I open-sourced a minimal local starter

To prove this works without sending sensitive corporate data to cloud APIs, I packaged my local GraphRAG stack into a minimal, production-oriented starter repo:
👉 github.com/kaziava/local-graphrag-starter

The Stack:

PyMuPDF for local PDF parsing
LangChain (LLMGraphTransformer) to extract the graph
Neo4j (via docker-compose) as the graph DB
Ollama (running llama3.1:8b or qwen2.5:3b) for local inference

How it works:

# 1. Start Neo4j
docker compose up -d

# 2. Build the graph from your PDF
python main.py ingest data/report.pdf

# 3. Ask a question (NL -> Cypher -> Exact Answer)
python main.py ask "What was Apple's revenue in Q3 2024?"

Benchmarks (MacBook M2, 16GB RAM)

Parse 50-page PDF: ~45s
Load graph: ~10s
Answer a question: 3–5s
Cost per query: $0 (compared to ~$15–20 for the same volume via GPT-4 API)

Honest Limitations & Early Stage Status

🚧 This is an early-stage reference architecture. Local 8B models are weaker than frontier LLMs on complex multi-hop reasoning, and complex tables might still need parser tuning.

If you try running it on your machine and hit any OS-specific bugs, please open an Issue on GitHub! I'm actively maintaining it and will fix things fast.

Want to dive deeper into the code?

I regularly share raw benchmark scripts, Docker configs, and architectural diagrams from my production LLMOps experience. I document this primarily in Russian on my Telegram channel (@llmops_engineering), but the code snippets, schematics, and engineering discussions are universal. Feel free to join the engineer chat there or reach out.