The Problem
I wanted to ask questions about my own paperwork and photos without sending them to a cloud AI service. Simple things like:
- "How much did I spend on electricity last year?"
- "What documents do I have about my car insurance?"
- "How many photos did I take in Spain in 2023?"
But I didn't want my invoices, contracts, and family photos ending up in someone else's training data.
The Stack
| Component | Purpose |
|---|---|
| Paperless-ngx | Document archive, OCR, structured metadata |
| Immich | Photos and video backup |
| ChromaDB | Semantic search index |
| Ollama | Local LLM inference |
| Open WebUI | Unified chat interface |
| MCP | Deterministic tools for structured data |
| LiteLLM | Model routing |
Everything runs on a single Windows 11 machine with an RTX 5080 16GB. An Orange Pi 5 Plus handles backups.
Two Hard Lessons
1. Deterministic tools beat LLM reasoning for anything countable
I started by asking the model to tally my invoices. It gave wrong answers. Every time.
The fix was a 20-line MCP tool that queries the Paperless API and returns a number. No reasoning, no hallucination — just the correct count.
2. Routing discipline matters
For structured invoice fields (supplier, total, VAT, invoice number), I added a strict rule to the system prompt:
> "Always query Paperless for these fields. Never use the semantic index."
This eliminated most of the wrong answers I was getting. The semantic index is great for "find me something about car insurance." It's terrible for "what's the exact total of invoice #1234?"
Why Not Just Use Paperless 3.0's Native AI?
I'm on Paperless 3.0.3, so I do have the native AI features including document chat. This setup doesn't replace them — it complements them:
- Broader scope: Same interface covers documents, photos, and video
- Deterministic operations: MCP tools guarantee accuracy for structured queries
- Unified experience: One chat interface for everything
What Didn't Work
I documented the failures too. Some highlights:
- Early attempts at letting the model reason about invoice totals
- Trying to use the semantic index for exact field extraction
- Initial confusion about which data source to query for which task
Full write-up (Spanish and English): github.com/fwinchi/ia-local-casa
Who This Is For
- People who want to query their own data without cloud AI
- Self-hosting enthusiasts
- Anyone curious about MCP as a practical tool for data accuracy
Not a developer. Built with AI assistance, cross-checking suggestions, and testing before applying.
Feedback welcome — especially on what I've got wrong.
Top comments (0)