You want to chat with your own documents without sending sensitive data to OpenAI's servers. That's not paranoia — that's smart business. Whether it's client contracts, internal wikis, or proprietary research, keeping your RAG pipeline local gives you control that cloud-only solutions simply can't match.
Here's exactly how to build one using Claude, Ollama, and Pinecone — and where each tool actually earns its place.
The Stack and Why Each Piece Matters
Ollama handles local model inference. You download it, run ollama pull llama3 or ollama pull mistral, and you've got a capable LLM running entirely on your machine. No API costs, no data leaving your network. The tradeoff: you need decent hardware. A MacBook M2 with 16GB RAM handles 7B models comfortably. Anything bigger and you're looking at a dedicated GPU setup.
Claude (via Anthropic's API) comes in when you need heavier reasoning or better instruction-following than local models can reliably deliver. Claude 3.5 Sonnet runs at $3 per million input tokens and $15 per million output tokens. For a document Q&A system handling moderate traffic, you're realistically looking at $20–80/month. That's not free, but for tasks requiring nuanced synthesis across complex documents, Claude often outperforms comparably-sized local models by a noticeable margin.
Pinecone handles vector storage. The free tier gives you one index with 100K vectors — enough for a few hundred documents. The Starter plan at $70/month unlocks multiple indexes and higher throughput. If you're just prototyping, the free tier will last you weeks. For production with real users, plan for paid.
The architecture is straightforward: chunk your documents, embed them (use nomic-embed-text through Ollama for free local embeddings), push vectors to Pinecone, then at query time pull relevant chunks and feed them as context to whichever model you're using.
Local vs. Cloud: Making the Real Decision
Run Ollama exclusively when: your documents contain anything you'd hesitate to paste into a public chatbot. Financial projections, client data, personnel files, unreleased product specs. Latency is higher (2–8 seconds on consumer hardware), quality varies by model, but you have genuine privacy.
Use Claude when: you need reliable, consistent output for customer-facing features, or when local models keep hallucinating on your specific document type. Technical documentation and legal-adjacent content (not actual legal advice — just document formats) tend to expose local model weaknesses faster.
The hybrid approach that actually works in production: route sensitive internal queries to Ollama, route customer-facing or high-stakes queries to Claude. Build a simple classifier or just hardcode the routing logic based on document type.
Building the Pipeline Without Losing Your Mind
Your chunking strategy matters more than your model choice. Most people chunk too large (losing precision) or too small (losing context). Start with 512 tokens per chunk, 50-token overlap, and adjust based on your retrieval quality.
For the embedding model, nomic-embed-text via Ollama is free and solid. If you're already paying for Claude, Anthropic's embedding API is another option, though most teams don't need it.
Keep your orchestration simple. LangChain works but adds complexity you often don't need. A few hundred lines of Python with direct Pinecone SDK calls and the Anthropic or Ollama Python client gets you 90% of the functionality with half the debugging overhead.
Store your document metadata in Notion alongside your pipeline docs — it's genuinely the best place to track which document versions are in your index, chunking configs you've tested, and retrieval quality notes. Notion's free tier handles this comfortably.
When you're ready to expose this as an internal tool or prototype to stakeholders, Webflow can front a simple chat interface faster than you'd expect, with no backend required if you're hitting an API endpoint.
My Actual Recommendation
Start local-only with Ollama + Pinecone free tier. Spend two weeks with real documents. When you hit the ceiling — inconsistent answers, poor reasoning on complex queries — add Claude for the specific query types where local models fail. Don't pay for Claude across the board until you've identified where it's actually necessary.
Before you build anything, rough out your business case. LexProtocol's free business plan builder is worth a pass-through to pressure-test your assumptions about who needs this and why they'd pay for it.
Build the simple version first. The stack is straightforward — the hard part is knowing what questions your documents actually need to answer.
Top comments (0)