Build RAG on Your Phone: A Local AI That Reads Your Files (ChromaDB + Ollama)
Most people think Retrieval Augmented Generation (RAG) needs a GPU server. It doesn't. You can run a complete RAG pipeline on a ₹15,000 Android phone using Termux. This guide shows you exactly how.
What Is RAG?
RAG = your LLM + your private data.
A plain LLM (like GPT or Llama) only knows what it saw in training. It cannot answer "what did I write in my trading journal last week?"
RAG fixes this:
- You load your documents into a vector database
- When you ask a question, the system finds the most relevant chunks
- Those chunks are injected into the prompt
- The LLM answers using YOUR data, not just its training
This is how you build a personal AI assistant that knows your stuff.
Why Local RAG?
- Privacy: Your data never leaves your phone
- Cost: ₹0 API bills
- Control: No vendor lock-in
- Offline: Works without internet
For a trader, this means an AI that remembers every trade, every lesson, every mistake — running entirely on your phone.
The Stack
| Component | Tool | Role |
|---|---|---|
| LLM | Ollama + Phi-3 Mini | Answers questions |
| Embeddings | all-MiniLM-L6-v2 (via sentence-transformers) | Converts text to vectors |
| Vector DB | ChromaDB | Stores and searches vectors |
| Orchestration | LangChain | Chains the steps |
| Environment | Termux (Android) | Runs Python |
All free. All local.
Step 1: Set Up Termux
Install Termux from F-Droid (not Play Store — outdated). Then:
pkg update && pkg upgrade
pkg install python clang fftw libzmq
pip install --upgrade pip
Step 2: Install Ollama
Ollama doesn't have an official Android binary, but you can run it via a community build or use the ollama Python client pointing to a local server. Simpler: use llama-cpp-python with a GGUF model directly.
pip install llama-cpp-python
Download Phi-3 Mini GGUF (2.3GB quantized):
mkdir -p ~/models && cd ~/models
# Use a download manager or wget from HuggingFace
wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/phi-3-mini-4k-instruct-q4.gguf
Step 3: Install ChromaDB + LangChain
pip install chromadb langchain sentence-transformers
Note: sentence-transformers pulls PyTorch (~600MB). On a phone this is heavy. Alternative: use fastembed (much lighter):
pip install fastembed chromadb
FastEmbed runs embeddings without PyTorch. Perfect for budget hardware.
Step 4: Build the RAG Pipeline
from fastembed import TextEmbedding, SparseTextEmbedding
import chromadb
from chromadb.utils import embedding_functions
from llama_cpp import Llama
# 1. Embedding model (local, lightweight)
embed_model = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")
# 2. ChromaDB client (in-memory or persistent)
chroma = chromadb.PersistentClient(path="~/rag_db")
collection = chroma.get_or_create_collection(
name="trading_journal",
embedding_function=embedding_functions.FastEmbedEmbedding()
)
# 3. Load your documents
def load_docs(folder):
docs = []
for f in os.listdir(folder):
if f.endswith('.txt') or f.endswith('.md'):
with open(os.path.join(folder, f)) as fh:
docs.append(fh.read())
return docs
docs = load_docs("~/journal")
collection.add(
documents=docs,
ids=[f"doc_{i}" for i in range(len(docs))]
)
# 4. LLM
llm = Llama(model_path="~/models/phi-3-mini-4k-instruct-q4.gguf", n_ctx=4096)
# 5. Query
def ask(question):
results = collection.query(query_texts=[question], n_results=3)
context = "\n".join(results['documents'][0])
prompt = f"""Answer using only the context below.
Context:
{context}
Question: {question}
Answer:"""
output = llm(prompt, max_tokens=512)
return output['choices'][0]['text']
print(ask("What was my biggest trading mistake last month?"))
Step 5: Test It
Put your trading journal entries as .txt files in ~/journal. Ask questions:
- "What setups lost me money?"
- "Summarize my May performance"
- "What did I learn about FOMO?"
The AI answers from YOUR data, not generic training.
Performance on Phone
On a ₹15,000 phone (6GB RAM):
- Embedding 100 docs: ~20 seconds
- Query + generate: ~8 seconds
- RAM usage: ~1.8GB
Works. Slow but functional. For daily journal use, totally fine.
Common Mistakes
Mistake 1: Chunking too large.
Fix: Split docs into 500-word chunks with overlap.
Mistake 2: No metadata.
Fix: Store source + date per chunk for better filtering.
Mistake 3: Heavy embedding model.
Fix: Use FastEmbed or MiniLM, not full BGE-large.
Mistake 4: Trusting without checking.
Fix: RAG can retrieve wrong chunks. Always verify the answer against source.
Real Use Case: My Trading Assistant
I load:
- 180 trade logs
- 12 book notes
- 50 saved articles
Then ask: "Based on my history, what's my win rate on expiry-day trades?" The AI retrieves relevant trades and computes from context. No cloud, no API cost.
When to Use Cloud Instead
If you need:
- Sub-2-second responses
- Multi-user scale
- Huge document sets (100K+)
Then use a small VPS or cloud embeddings. But for personal use, phone RAG wins.
Bottom Line
RAG is not magic. It's: embed → store → retrieve → inject → generate.
You can run all five steps on a ₹15,000 phone. No GPU. No cloud. No bill.
Start with your journal. Build your personal AI. Own your data.
AI proposes. You dispose.
About the Author
Shakti Tiwari is an AI builder and retail trader based in Chandigarh, India. He builds local AI trading systems on a ₹15,000 phone and writes about local AI, options trading, Bitcoin, and agent evaluation.
- 🌐 Website: https://optiontradingwithai.in
- 💻 Dev.to: https://dev.to/shaktitiwari
- 🐙 GitHub: https://github.com/shaktitiwari
- 🐦 X: https://x.com/shaktitiwari
- 📨 Telegram: https://t.me/shaktitrade
Books:
- Option Trading with AI (Amazon: https://www.amazon.in/dp/B0H9ZNTBPK)
- The AI Opportunity (Amazon: https://www.amazon.in/dp/B0HBBFKDQF)
{"@context":"https://schema.org","@type":"Person","name":"Shakti Tiwari","url":"https://optiontradingwithai.in","sameAs":["https://www.wikidata.org/wiki/Q140689249"]}
Educational only. Not financial advice. Not SEBI registered.
Code: github.com/shaktitiwari/nse_ai_agent
— Shakti Tiwari, AI builder from Chandigarh, running ML on a ₹15,000 phone.
Top comments (0)