Everyone has the same week with Ollama.
On Monday you run ollama run llama3.2 and it feels like magic: a real model on your own hardware, no API bill, nothing leaving the machine. By Wednesday you want the model to answer questions about your own documents. By Friday your "local AI" is five pieces:
- Ollama, for inference.
- A vector database, because Ollama can create an embedding but has nowhere to keep it.
- Postgres or SQLite, for the actual data.
- LangChain or LlamaIndex, to glue the three together.
- An nginx config, because your teammate wants to use it too.
None of that is Ollama's fault. Ollama set out to run models well, and it does. But if you've ever sat in that Friday mess wondering why "local AI" needs so much plumbing, this post is for you.
Where a model runner stops
These come straight from Ollama's own docs, and each one is a reasonable design choice for a model runner. Each one is also a job that falls to you.
1. It doesn't keep anything. /api/embed returns a vector, and then it's your problem. There's no storage, index or search, so every RAG tutorial starts with "now install a vector database".
2. It can't see your data. The model knows what it was trained on, plus whatever you paste into the prompt. Finding the right rows, deciding what goes into the context window and citing where an answer came from is all code you write and maintain.
3. It lets anyone in, or no one. By default Ollama listens only on 127.0.0.1:11434. Its FAQ shows how to open it up with OLLAMA_HOST=0.0.0.0, nginx, ngrok or a Cloudflare Tunnel, and none of those sections mentions authentication, because the server doesn't have any. Most teams either keep it on one laptop or put it behind a proxy and hope.
Keep your Ollama code, change the server
SynapCores is a single-binary database for AI apps: SQL, vectors, graph, agent memory and local inference in one process. It also serves the Ollama API (/api/chat, /api/embed, /api/tags, /api/ps, /api/version) and pulls models from the same Ollama registry, so your model names stay the same.
This is the official ollama Python client (0.6.3), unchanged except for the host and an API key. I ran it this morning against SynapCores 2.5:
import ollama
client = ollama.Client(
host="http://localhost:8080", # SynapCores
headers={"Authorization": "Bearer aidb_..."}, # an API key
)
print([m.model for m in client.list().models])
# ['all-minilm:latest', 'd1:latest', 'qwen2.5-coder:7b']
client.chat(model="qwen2.5-coder:7b",
messages=[{"role": "user", "content": "Reply with just the word OK."}])
# → "OK."
Look again at that auth header. Every request carries a key, and a call without one gets a 401. Keys can be read-only, scoped to specific databases and set to expire, so "this key can read the support docs but not the payroll table" is a setting, not a proxy project.
Answers grounded in your data
Here's a three-row support knowledge base, stored as an ordinary SQL table:
CREATE TABLE kbtext (id INT PRIMARY KEY, body TEXT);
INSERT INTO kbtext VALUES
(1, 'Orbit Pro refunds: customers can request a full refund within 45 days of purchase.'),
(2, 'Orbit Pro support hours are 7am to 7pm Eastern, Monday through Saturday.'),
(3, 'Error code E-221 means the sync token expired; sign out and sign back in.');
Now call the Ollama chat endpoint the way you always have, and add one field, rag:
curl localhost:8080/api/chat -H "Authorization: Bearer $KEY" \
-H "content-type: application/json" -d '{
"model": "qwen2.5-coder:7b", "stream": false,
"rag": {"source": "collection:kbtext", "mode": "inject", "top_k": 2},
"messages": [{"role":"user","content":"How many days do I have to get a refund on Orbit Pro?"}]
}'
"You have 45 days to request a full refund on Orbit Pro after your purchase."
The response headers say where the answer came from:
x-synapcores-grounding: grounded
x-synapcores-citations: ["collection:kbtext/row-0","collection:kbtext/row-1"]
It's the same model and the same endpoint. The difference is that the model now runs inside the database that holds the answer: no vector database to deploy, no glue code and no second copy of your data.
Your model, used directly in SQL
Because inference runs inside the engine, the model is also available in a query:
SELECT id, GENERATE('Summarize in 6 words: ' || body) AS s
FROM kbtext WHERE id = 3;
-- 3 | 'Sync token expired. Sign out, then sign back in.'
That turns "summarize every ticket", "tag every row" or "draft a reply per customer" into a single SELECT statement.
Before you switch
- SynapCores is a database, not a lighter Ollama. If all you need is a model in a terminal, Ollama is great. SynapCores is for when the model needs to work with your data.
- Point clients at port 8080 and add a bearer key. Auth is on by default, so that's the only change in your client code.
Try it in five minutes
curl -fsSL https://get.synapcores.com | sh
# or
docker pull synapcores/community:v2.5.1-ce
It's the free, self-hosted Community Edition.
The full write-up is on synapcores.com. Install options are at synapcores.com/download.
What's the piece of your local-AI stack you'd most like to delete: the vector DB, the glue code or the proxy?
Top comments (0)