Why your RAG hallucinates on tables (and a minimal local GraphRAG starter to fix it)

If you've ever built a RAG system for corporate documents, you've probably hit this wall: you ask "What was the revenue in Q3?", and the LLM confidently hallucinates a number with three extra zeros.

This isn't a model bug. It's an architectural blind spot. As recent research from Microsoft points out, naive 500-token chunking destroys table structures. Cosine similarity over embeddings just doesn't understand rows and columns.

The Fix: GraphRAG

Instead of flat vector search, we extract entities and relationships into a Knowledge Graph. When a user asks about numbers, the LLM translates the question into a Cypher query, and the graph returns exact data. Zero hallucinated digits.

I open-sourced a minimal local starter

To prove this works without sending sensitive corporate data to cloud APIs, I packaged my local GraphRAG stack into a minimal, production-oriented starter repo:
👉 github.com/kaziava/local-graphrag-starter

The Stack:

• PyMuPDF for local PDF parsing
• LangChain (LLMGraphTransformer) to extract the graph
• Neo4j (via docker-compose) as the graph DB
• Ollama (running llama3.1:8b or qwen2.5:3b) for local inference

How it works:

# 1. Start Neo4j
docker compose up -d

# 2. Build the graph from your PDF
python main.py ingest data/report.pdf

# 3. Ask a question (NL -> Cypher -> Exact Answer)
python main.py ask "What was Apple's revenue in Q3 2024?"

Benchmarks (MacBook M2, 16GB RAM)

• Parse 50-page PDF: ~45s
• Load graph: ~10s
• Answer a question: 3–5s
• Cost per query: $0 (compared to ~$15–20 for the same volume via GPT-4 API)

Honest Limitations & Early Stage Status

🚧 This is an early-stage reference architecture. Local 8B models are weaker than frontier LLMs on complex multi-hop reasoning, and complex tables might still need parser tuning.

If you try running it on your machine and hit any OS-specific bugs, please open an Issue on GitHub! I'm actively maintaining it and will fix things fast.

Want to dive deeper into the code?

I regularly share raw benchmark scripts, Docker configs, and architectural diagrams from my production LLMOps experience. I document this primarily in Russian on my Telegram channel (@llmops_engineering), but the code snippets, schematics, and engineering discussions are universal. Feel free to join the engineer chat there or reach out.

What graph DB are you using for your RAG setups? Let me know in the comments!