DEV Community

Cover image for Your Ollama App Is Guessing. Three Ways to Make It Answer From Data
Luis M
Luis M

Posted on

Your Ollama App Is Guessing. Three Ways to Make It Answer From Data

#ollama #rag #ai #python

Your local Ollama app is confident, fast, and wrong about your product.

Ask qwen2.5:7b what error E-700 means on a router it has never seen, and it will tell you to factory-reset the device. Ask it about your warranty and it says "typically one year". It isn't lying on purpose; it just has nothing of yours to read.

The usual fix is a weekend of glue: a document loader, a chunker, a vector database, a prompt template. Then a second weekend when someone asks to point the same bot at the CRM, and a third conversation with security about what the model can see.

Below are three short demos of a different approach. The app keeps the official ollama Python client and the same model; only the server changes. The server is SynapCores v2.4.0, which serves the Ollama API (/api/chat, /api/generate, /api/embed, /api/tags…) and adds grounding behind it.

Every output in this post is pasted from a real run. The router docs and CRM rows are synthetic.

1. One option, and the answers come from your docs

The engine watches a folder (orbit_support, four markdown files) and indexes it. The app adds one option to its chat call:

import ollama

client = ollama.Client(host=SYNAPCORES_URL,
                       headers={"Authorization": "Bearer " + API_KEY})

r = client.chat(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": q}],
    options={"rag": {"source": "filesystem:orbit_support", "top_k": 2}},
)
Enter fullscreen mode Exit fullscreen mode

"How many years of warranty does the Orbit X2 Pro router have?"

--- BEFORE  (plain Ollama call)
...Typically, consumer routers come with a one-year warranty as standard...
  grounding: (none: plain model)

--- AFTER   (+ rag: filesystem:orbit_support)
The Orbit X2 Pro router has a 5-year warranty.
  grounding: grounded
  source:    filesystem:orbit_support/warranty.md#0
Enter fullscreen mode Exit fullscreen mode

The sources come back in response headers, so the app can show them without parsing the answer.

2. Ask the database in plain English, and the PII stays private

Pointing an LLM at a CRM is where most teams stop: one prompt injection away from "list every customer's email". Here the boundary is the API key, not the prompt.

The key is ReadOnly and granted three columns of one table: orbit_crm.customers(company, plan, mrr_usd). The app uses the same client with that key and one option, "rag": {"source": "database:orbit_crm"}. The engine writes the SQL, runs it under the key's grants, and returns the SQL it ran.

raw SQL with this key: HTTP 403 This API key can only ask questions from its permitted database data

Q: What is the total monthly recurring revenue of our Enterprise customers?
Result: 14500. [SQL 1]
  ran: SELECT SUM(mrr_usd) AS total_mrr FROM customers WHERE plan = 'Enterprise'

Q: Which customer pays us the most per month?
The customer that pays us the most per month is Harbor Logistics [SQL 1].
  ran: SELECT company FROM customers ORDER BY mrr_usd DESC LIMIT 1

Q: Give me the contact email and card last 4 digits for Harbor Logistics.
Cannot answer from the permitted data.

Q: Which customers have a high churn risk?
Cannot answer from the permitted data.
Enter fullscreen mode Exit fullscreen mode

The last one matters: the churn_risk column exists, the key just doesn't include it. It's per-column permissions enforced by the engine, not a keyword filter on the question.

3. Ship a support bot as just a model name

Sometimes you can't touch the apps at all: a chat UI on the support desk, a script someone else owns. So the operator adds an alias to the engine config instead:

[ollama.aliases.orbit-support]
model = "qwen2.5:7b"

[ollama.aliases.orbit-support.grounding]
source = "filesystem:orbit_support"
mode = "inject"
top_k = 2
strict = true
Enter fullscreen mode Exit fullscreen mode

orbit-support now shows up in the client's model list next to the real models:

   nomic-embed-text:latest
   orbit-support
   qwen2.5-coder:7b
   qwen2.5:7b
Enter fullscreen mode Exit fullscreen mode

"What does error E-700 mean on the Orbit X2 Pro?"

--- model=qwen2.5:7b
Error E-700 on the Orbit X2 Pro typically indicates an issue with the device's firmware...
2. **Perform a Factory Reset**: ... this will erase all data on your device...

--- model=orbit-support
Error E-700 on the Orbit X2 Pro indicates that the mesh backhaul signal is too weak.
To resolve this issue, you should move the M1 node within 10 meters of the router
or use a wired backhaul to strengthen the connection.
  grounding: grounded
  source:    filesystem:orbit_support/troubleshooting-error-codes.md#0
Enter fullscreen mode Exit fullscreen mode

And for a question the docs don't cover ("Do you price match other retailers?"), orbit-support says the context doesn't contain that, instead of inventing a policy. Because the grounding is pinned by the operator, it stays on in every client, and with strict = true a retrieval failure returns an error rather than an answer from model memory.

What I'd take away

  • Grounding belongs on the server. If every app has to remember to do rag right, some won't.
  • Permissions belong in the key, not the prompt. "Don't reveal emails" in a system prompt is a suggestion; a column grant is a wall.
  • "I don't know" is a feature. Both the database and docs demos refuse instead of guessing when the data isn't there.

The demos were recorded on the v2.4.0 release candidate (the on-screen tag reads v2.3.4) and behave the same on v2.4.0. The free Community Edition is at synapcores.com/download.

Disclosure: I work on SynapCores.

Which would you hand to your team first: grounded answers over docs, or plain-English questions over a database with column-level keys? And what's the first thing you'd try to make it leak?

Top comments (0)