Chat With Your Documents Locally Using AnythingLLM and Ollama

A private RAG system where you drop in PDFs, Word docs, and code files and ask questions. Runs on any machine, no cloud dependency.

What You Need

• Any computer (GPU optional - CPU works fine)
• Ollama installed
• About 10 minutes

Architecture

Component	Role
AnythingLLM	Desktop/server app with RAG, agents, built-in vector DB
Ollama	Serves local LLM for chat + embeddings
Qwen3 14B	Default model for answering questions

Setup

1. Install Ollama

# Install from ollama.com, or run with Docker:
docker run -d --gpus all -p 11434:11434 --name ollama \
  -v ollama:/root/.ollama ollama/ollama

# Pull a model:
ollama pull qwen3:14b
# Pull an embedder:
ollama pull nomic-embed-text

2. Install AnythingLLM

Desktop app (easiest): Download from anythingllm.com

Docker:

docker run -d -p 3001:3001 --name anythingllm \
  --add-host host.docker.internal:host-gateway \
  -v anythingllm:/app/server/storage \
  mintplexlabs/anythingllm

3. Connect & Use

1. Open AnythingLLM (http://localhost:3001 or desktop app)
2. Settings > LLM Provider > Select Ollama, model qwen3:14b
3. Settings > Embedder > Select Ollama, model nomic-embed-text
4. Create a workspace, drop in documents, start asking questions

What You Can Do

• Chat with PDFs, Word docs, code files, web pages
• Create isolated workspaces per project
• Use built-in agent skills (web search, summarization)
• Works on CPU-only machines like a mini PC

Cost vs Cloud

	Local	ChatGPT + GPTs
Monthly	$0	$20-200
Hardware	$0-300	$0
Privacy	Stays on your machine	Sent to cloud
Documents	Unlimited	Token-limited

---

Full guide with troubleshooting: https://everylocalai.com/stack/anythingllm-ollama-rag