shakti tiwari

Posted on Jul 28

RAG for Local AI — Private Memory On-Device (No Cloud, No Retraining)

#rag #local #ai #retrieval

Quick answer: RAG (Retrieval-Augmented Generation) lets your local model answer from your private documents without retraining. You embed your files, retrieve the top-k most relevant chunks at query time, and stuff them into the prompt. It runs fully on-device — even on a phone — and updates instantly when your data changes. This is the cheapest path to a "personal" assistant.

Educational disclaimer: Developer tooling guide. Not legal or data-security advice. If you handle regulated data, add encryption-at-rest and access controls beyond what's shown here.

Why RAG beats fine-tuning for private data

Fine-tuning (QLoRA) bakes knowledge into weights. That's slow, needs 200+ clean rows, and every data update means another training run. RAG keeps the base model frozen and feeds it context at inference.

Per AIMultiple's 2026 RAG analysis, production RAG needs hybrid search (dense vector + sparse BM25, fused with RRF) and independent retrieval/generation monitoring — faithfulness above 0.85 is the regulated baseline. The win for local: you get that without a vendor seeing your corpus.

	RAG	QLoRA
Update data	re-embed (seconds)	retrain (hours)
Privacy	corpus stays local	weights carry data
Hardware	CPU OK	GPU preferred
Best for	facts that change	style/format

The RAG pipeline in 4 parts

Chunk documents into ~500-token pieces.
Embed each chunk with a small model (e.g. nomic-embed-text, the 2026 community pick per LMSA's local RAG guide).
Retrieve top-k by similarity at query time (hybrid: vector + BM25).
Generate — stuff context into the prompt, instruct the LLM to use only retrieved text.

A working RAG that runs on the phone

No vector DB server needed for a few hundred docs. sentence-transformers + numpy:

import numpy as np
from sentence_transformers import SentenceTransformer

embed = SentenceTransformer("all-MiniLM-L6-v2")  # ~80 MB
docs = ["your note 1", "your note 2", "market thesis 2026-07"]
idx = np.array([embed.encode(d) for d in docs])

def ask(model, q, k=4):
    qv = embed.encode(q)
    sims = idx @ qv                      # cosine-ish (assumes normalized)
    top = np.argsort(-sims)[:k]
    ctx = "\n".join(docs[i] for i in top)
    return model(f"Use ONLY the context. If unsure, say 'not in docs'.\n\nContext:\n{ctx}\n\nQ: {q}")

The prompt line "Use ONLY the context" is the anti-hallucination guard Cloudian's 2026 RAG guide recommends — explicitly tell the LLM to rely on retrieved context.

Chunking: the part people skip

Bad chunking = bad retrieval. Rules:

Align granularity with context limit. Too big → truncation; too small → lost connections (Cloudian 2026).
Keep chunks ~300–600 tokens with 10% overlap.
For tables/code, keep them whole — don't split a function across chunks.
Store metadata (source, date) so you can filter ("only 2026 notes").

Hybrid retrieval (dense + sparse)

Dense vectors catch meaning; BM25 catches exact keywords (ticker symbols, IDs). Fuse with RRF (Reciprocal Rank Fusion):

def rrf(dense_rank, sparse_rank, k=60):
    score = {}
    for rank, doc in enumerate(dense_rank):
        score[doc] = score.get(doc, 0) + 1/(k + rank)
    for rank, doc in enumerate(sparse_rank):
        score[doc] = score.get(doc, 0) + 1/(k + rank)
    return sorted(score, key=score.get, reverse=True)

On-device you can run BM25 with rank_bm25 (pure Python) alongside your numpy dense index. No server.

Monitoring retrieval and generation separately

Cloudian 2026 stresses: if an answer is wrong, diagnose whether the retriever surfaced junk or the generator failed to synthesize. Log:

retrieval hit-rate (did top-k contain the answer?)
query→doc mappings
output faithfulness score

For a local setup, a simple log file per query is enough. You don't need a dashboard to start.

Embedding model choice (2026)

The community coalesces on nomic-embed-text for local RAG (LMSA 2026). Alternatives:

bge-small-en — fast, decent.
all-MiniLM-L6-v2 — tiny, good enough for <10k docs.
For code: codesage or a code-trained embedder.

Pick by your corpus size. Phone? Use the smallest that holds faithfulness > 0.85.

RAG + tuned Modelfile = your assistant

Wire RAG in front of the model you built in How to Tune Your Local AI Model. The Modelfile sets persona; RAG supplies facts. Neither touches weights.

Example: a trading aide that always answers from your annotated NIFTY notes, in Hinglish, with citations. Persona = SYSTEM prompt; knowledge = RAG context. Fully offline.

Privacy: why on-device RAG wins

Your corpus never leaves the device. No vendor sees your client data, market theses, or drafts. AIMultiple notes most open-source vector engines don't encrypt at rest natively — so on a phone, use disk encryption (Android/iOS already do) and skip network entirely. That's stronger than any "we don't train on your data" promise.

Common RAG failures

"It ignored my doc" → chunk too big/small, or retriever returned wrong chunk. Tune chunk size + add hybrid.
Hallucinated a source → prompt didn't forbid out-of-context answers. Add "if not in context, say so."
Slow on phone → embedding 1000s of docs on CPU is slow once; do it offline, cache the index.
Stale answers → you changed data but didn't re-embed. Re-embed is seconds; automate it.

Reranking: the quality multiplier

Retrieval gives candidates; a cross-encoder reranker re-scores the top 20 and keeps the best 4. This catches cases where the embedder ranked a slightly-wrong chunk first. On-device options exist (ms-marco-MiniLM cross-encoders run on CPU). Adds latency but lifts faithfulness noticeably when docs are similar.

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(q, docs[i]) for i in top20]
scores = reranker.predict(pairs)
best = [top20[i] for i in np.argsort(-scores)[:4]]

For a phone, skip reranking until you see bad retrieval — the base retriever is often enough under 1k docs.

Scaling past 10k docs

When your corpus grows:

Move from numpy to a real engine (Qdrant/Milvus/Chroma) — but AIMultiple warns most open-source engines ship no auth (Chroma 1.x) and none encrypt at rest natively. Run them on a trusted LAN box, not exposed.
Add metadata filtering ("only 2026", "only NIFTY") before similarity search.
Use IVF/HSNW indexes to keep latency flat as docs grow.

On a phone you rarely hit this — keep the corpus lean (your active notes, not the internet).

Evaluation: put it in CI

Don't ship RAG blind. Build a 20-question gold set ("from my notes, what was the stop on INFY 2026-07-27?"). Score:

retrieval hit-rate (did top-k contain the answer?)
faithfulness (does the answer use only context?)
answer correctness vs gold

Run on every corpus update. This is the "evaluate in CI" step AIMultiple recommends — catches drift before users do.

A real trading-use example

Say you keep annotated NIFTY notes: "2026-07-27: INFY slumped 7%, PCR rose, I'd avoid calls." Wire RAG so the model answers from these:

notes = load_markdown_folder("~/nifty_notes")   # your private corpus
idx = embed_corpus(notes)
def trading_qa(q):
    ctx = retrieve(idx, q, k=4)
    prompt = f"""You are a disciplined NIFTY aide. Use ONLY the notes.
If the answer isn't in notes, say 'not in my notes'. Cite the date.
Notes:\n{ctx}\n\nQ: {q}"""
    return ollama(prompt)   # your tuned local model

Now the assistant quotes your past calls, in Hinglish if you set the SYSTEM prompt that way, with zero cloud. That's a personal trading memory — the exact "customized experience" people pay APIs for, built free on-device.

Security checklist for on-device RAG

[ ] Disk encryption on (Android/iOS default) — covers the index at rest.
[ ] No network calls in the retrieve/generate path (air-gap verified).
[ ] Prompt forbids out-of-context answers (anti-hallucination).
[ ] If serving others, add auth + row-level access (pgvector offers this; plain numpy doesn't).
[ ] Log queries locally only; never ship telemetry.

RAG vs just using a long context window

Modern models have 1M-token contexts (GLM-5.2, Kimi K3). Why RAG at all? Three reasons:

Cost — stuffing 1M tokens every query burns RAM/VRAM; RAG sends only top-k (~2k tokens).
Precision — a huge context dilutes attention; retrieval focuses the model on the 4 relevant chunks.
Freshness — update the index, not the prompt. Long context means re-pasting everything.

RAG is the pragmatic middle: small model, small context, big knowledge via retrieval.

The cost of on-device RAG

Embedding: one-time CPU cost per doc (seconds for hundreds).
Storage: ~1 KB per chunk embedding (float32). 10k docs ≈ 40 MB index. Trivial on a phone.
Inference: same as your base model (no extra cost).
vs API RAG (e.g. hosted vector DB + LLM): you pay per query + egress. On-device = zero.

For a personal assistant this is the whole economic case: build once, query free forever.

FAQ

Q: Do I need a GPU for RAG?
A: No. Embedding a few hundred docs on CPU takes seconds. Generation uses your existing local model.

Q: Chroma/Qdrant or numpy?
A: Under ~10k docs, numpy + BM25 is fine and serverless. Above that, a proper engine helps.

Q: Can RAG lie?
A: Only if the generator ignores context. The "use only context" prompt + faithfulness check controls it.

Q: How is this different from fine-tuning?
A: RAG = frozen model + live context. QLoRA = data baked into weights. RAG updates instantly; QLoRA needs retraining.

Q: What embedder for a phone?
A: all-MiniLM-L6-v2 (~80 MB) or nomic-embed-text. Both run CPU-only; pick by faithfulness on your corpus.

Q: Why not just paste everything into the prompt?
A: 1M-token contexts exist, but RAG is cheaper (sends 2k not 1M tokens), more precise (focused chunks), and fresher (re-embed vs re-paste). See above.

Key takeaways

RAG gives local models private memory with zero retraining.
Hybrid search (dense + BM25 + RRF) beats either alone.
Chunking and the "use only context" prompt decide quality.
Rerank + evaluate in CI once corpus grows past 1k docs.
Runs on a phone — numpy + small embedder, no server.
Privacy is the killer feature: corpus never leaves the device.
Cheaper than long-context or API RAG — build once, query free.

Written by **Shakti Tiwari* — Nifty Option Trader, XGBoost Expert. More on-device AI + systematic trading at optiontradingwithai.in.*

Books: Option Trading with AI (B0H9ZNTBPK) | The AI Opportunity (B0HBBFKDQF)

DEV Community