Running a Whole RAG Agent Offline: LangGraph + Ollama + Embedded Qdrant (Zero API Keys)

Most RAG tutorials open with "set your OPENAI_API_KEY." This one doesn't need it. In Part 1 I claimed the LLM and embeddings are behind a swappable boundary — "switch providers via config, not code." Part 3 is me cashing that claim: running the entire RAG agent — ingestion, retrieval, the ReAct loop, source citations — on a laptop with zero API keys and no Docker, just Ollama and an embedded Qdrant.

Everything below is real output from an actual run. Including the one thing that broke.

---

What "offline" actually requires

Three pieces, all local:

• Ollama running two models — one for chat, one for embeddings:

  ollama pull qwen3.5:9b   # chat / reasoning
  ollama pull bge-m3       # embeddings (1024-dim, multilingual)

• Embedded Qdrant — no server, no container. The vector store writes to a local directory.
• A one-line config flip so chat goes to Ollama instead of the gateway:

  CHAT_PROVIDER=ollama

That's it. No OPENAI_API_KEY, no docker compose up. The reason this is a flip and not a rewrite is the provider-swap design from Part 1 — let's look at the three factories that make it work.

---

The embeddings factory — swap by config

# app/llm/embeddings.py
@lru_cache
def get_embeddings() -> Embeddings:
    s = get_settings()
    provider = s.embedding_provider.lower()

    if provider == "ollama":
        from langchain_ollama import OllamaEmbeddings
        return OllamaEmbeddings(model=s.embedding_model, base_url=s.ollama_url)

    if provider == "openai":
        from langchain_openai import OpenAIEmbeddings
        return OpenAIEmbeddings(base_url=f"{s.litellm_url}/v1",
                                api_key=s.litellm_key, model=s.embedding_model)

    raise ValueError(f"unknown embedding_provider: {s.embedding_provider!r}")

Both branches return the same LangChain Embeddings interface, so the ingestion and retrieval code never knows which one it got. Local dev → Ollama (offline). Production → OpenAI via the gateway. One caveat that matters later: the two providers produce different vector dimensions, so you can't mix vectors ingested with one and queried with the other. More on that in the gotchas.

The vector store — embedded vs. remote, also by config

# app/rag/store.py
@lru_cache
def get_client() -> QdrantClient:
    s = get_settings()
    if s.qdrant_url:
        return QdrantClient(url=s.qdrant_url, api_key=s.qdrant_api_key)  # remote (prod)
    return QdrantClient(path=s.qdrant_path)                             # embedded (local)

No QDRANT_URL? You get an embedded client that persists to s.qdrant_path — a plain directory. Set QDRANT_URL in prod and the same code talks to a real Qdrant service. The trade-off of embedded mode: it locks the directory to a single process, which becomes gotcha #2.

---

Ingestion: docs → chunks → vectors

The ingest script is the whole pipeline in ~30 lines: load files, split them, probe the embedding dimension, create the collection, upsert.

# scripts/ingest.py (trimmed)
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
chunks = splitter.split_documents(documents)

# probe the embedding dimension so the collection matches the provider
dim = len(get_embeddings().embed_query("probe"))
ensure_collection(dim)
get_vector_store().add_documents(chunks)

The embed_query("probe") trick is worth pausing on: instead of hard-coding 1024 for bge-m3 (or 1536 for OpenAI), it asks the active embedder for one vector and measures it. Swap the provider and the collection is created with the right size automatically.

Running it for real:

$ python scripts/ingest.py --reset
[ingest] source=docs  collection=docs  embed=ollama:bge-m3
[ingest] 5 documents → 53 chunks
[ingest] embedding dim = 1024
[ingest] done — 53 points in collection

Five markdown files, 53 chunks, 1024-dim vectors from bge-m3, written to the local Qdrant directory. No network calls left the machine.

---

Running the agent — no server needed

You can hit the FastAPI endpoint, but to see the graph think you can also invoke it directly. Here's a real run, asking about something that lives in the docs:

res = await graph.ainvoke({"messages": [HumanMessage(content=
    "How is short-term vs long-term memory implemented in this project?")]})

print([type(m).__name__ for m in res["messages"]])
# ['HumanMessage', 'AIMessage', 'ToolMessage', 'AIMessage']

That message sequence is the ReAct loop, visible in the state:

1. HumanMessage — the question
2. AIMessage with tool_calls=[search_docs(...)] — the model decides to retrieve
3. ToolMessage — the retrieved chunks come back
4. AIMessage — the final synthesized answer

And the answer itself, generated entirely by a 9B model on the laptop:

Short-term memory: PostgreSQL (PostgresSaver) stores per-thread
  conversation state; swappable to Redis (RedisSaver) if needed.
Long-term memory: Zep manages the user's persistent knowledge,
  recalled by the app on later turns.

Sources: <doc-a>.md, <doc-b>.md

Grounded in the actual docs, with source attribution, zero API keys. That's the win. Now the part the tutorials skip.

---

Gotchas (the part that's actually worth reading)

1. The empty synthesis turn — the local model, not the pipeline

On one run, the exact same question produced this:

[1] AIMessage   content=''   tool_calls=[search_docs(...)]   finish_reason='tool_calls'
[2] ToolMessage content='[1] (source: ...) ## memory layers ...'   ← retrieval worked
[3] AIMessage   content=''   tool_calls=[]   finish_reason='stop'  ← empty answer

Retrieval succeeded. The chunks were right there in step 2. But step 3 — the model's job to read the chunks and answer — came back empty. finish_reason='stop', no tokens, no error. Re-running the same question gave a perfectly good 280-character answer with citations. So it's intermittent: a small local model occasionally produces an empty turn after a tool call.

Two things to take away:

• It's the model, not your graph. The pipeline (routing → retrieval → state) was flawless; the synthesis step just whiffed.
• The saw_token fallback from Part 2 won't save you here — that fallback calls ainvoke when no tokens stream, but here ainvoke is the empty result. The real mitigations are a larger/better tool-tuned local model, or accepting some flakiness as the price of fully offline. Worth knowing before you demo it live.

2. Embedded Qdrant locks the directory

Embedded mode keeps the store in one process. Run the ingest script while the server is up and you'll get a lock error. Order matters: ingest first → let it exit → then start the server. The ingest script even closes the client explicitly to avoid a noisy shutdown traceback.

3. Embedding dimensions must match end to end

bge-m3 is 1024-dim; OpenAI's text-embedding-3-small is 1536. If you ingest with one provider and query with another, the dimensions don't line up and search breaks. Switching embedding_provider means re-ingesting (--reset). The embed_query("probe") dimension check is exactly what keeps the collection honest per provider.

4. The first call is slow

Ollama loads the model into memory on first use. The first request eats that cost; subsequent ones are fast. Don't benchmark the cold start.

---

Why this matters

You can build, debug, and demo the entire RAG agent — graph, retrieval, citations — on a plane with no wifi. Then, for production, you flip two config values (CHAT_PROVIDER, QDRANT_URL) and the same code talks to a hosted model and a real Qdrant cluster. Part 1 claimed the provider boundary; Part 3 ran on both sides of it.

The flip side is honesty about local models: retrieval is rock-solid, but a 9B model's synthesis step is the weak link, and it'll occasionally hand you an empty answer. Know that going in.

Next: persisting conversation threads with a checkpointer — so the agent remembers across requests — and what that adds to the message log you just saw.

---

Part 3 of a series on running LangGraph in production. Part 1 · Part 2.