The pattern: Self-hosted models (Ollama, llama.cpp, vLLM) made private inference affordable. Retrieval-augmented generation still fails the same ways it did two years ago: bad chunks, stale indexes, and answers that cite documents the model never actually used.
Why it matters: Local RAG is a product decision, not a model decision. Chunking strategy, embedding choice, and refresh cadence decide whether users trust the system. A stronger model on a rotten index just hallucinates with better grammar.
Checklist that survives first contact
- Chunk for questions — structure-aware splits (headings, tables, code fences) beat fixed 512-token windows for most enterprise docs.
- Hybrid retrieval — dense vectors plus keyword/BM25 for IDs, error codes, and proper nouns.
- Freshness SLA — document versions and reindex jobs. Stale PDFs are silent outages.
- Citation discipline — require the model to quote retrieved spans; reject answers with empty retrieval sets.
When local wins
Local wins on privacy, predictable cost, and air-gapped or regulated environments. It loses when you need the absolute frontier for one-shot reasoning and you lack GPU headroom. Many teams run a hybrid: local RAG for corp knowledge, cloud for hard reasoning spikes.
What we'd watch next
Eval sets built from your tickets and SOPs — not public QA benchmarks. If retrieval precision@5 is weak, no amount of prompt tuning will save the experience.
Originally published on ayraix.com, practical AI for enterprise builders.
Top comments (0)