🛠️ MyZubster: Moving from Generative RAG to Evidence-First AI
Today we made an important step forward in the MyZubster AI stack.
During an end-to-end test, we found a concrete failure mode:
The knowledge base clearly contained:
status = RECORDED
success = true
paymentRequired = false
onchainRecorded = false
Yet when asked:
“Is the handover recorded on blockchain?”
the LLM could still answer incorrectly:
“It is recorded on blockchain.”
The problem wasn't Qdrant. The correct source was being retrieved.
The problem was letting an LLM make the final decision about a structured boolean that the system already knows with certainty.
🔎 The solution
We introduced structured metadata into MyZubster observations:
{
"status": "RECORDED",
"success": true,
"paymentRequired": false,
"onchainRecorded": false,
"handoverId": "6aab8faca70ce84f926d0b41",
"method": "HAND_DELIVERY"
}
The metadata is preserved in the observation and indexed alongside the RAG content.
We then changed the /api/ai/ask flow to:
question
↓
Qdrant retrieval
↓
authoritative metadata?
├── yes → deterministic answer
└── no → LLM / RAG
So questions such as:
“What is the status?”
“Is it recorded on blockchain?”
“Is payment required?”
“Was the operation successful?”
can be answered directly from authoritative structured data.
For example:
Status: RECORDED.
Blockchain recording: no.
No interpretation by the model.
No probability involved.
No false accidentally becoming true.
🤖 What about Ollama?
We didn't remove the LLM.
Ollama is still used when the question requires natural-language reasoning and there isn't enough structured metadata to answer deterministically.
We also made the RAG context more explicit:
[SOURCE 1]
description=...
metadata={"onchainRecorded": false, ...}
and strengthened the prompt with an evidence-first approach: the model should use only facts explicitly present in the retrieved sources.
⚙️ Local inference optimization
For the local test environment we also moved to:
qwen2.5:0.5b
with:
AI_CONTEXT_LIMIT=1
This reduced unnecessary context and improved response time during the test.
🧪 Testing
We added coverage for:
structured metadata persistence;
metadata inclusion in the RAG prompt;
authoritative metadata responses;
existing AI API behavior.
Result:
27 passed
The changes were committed and pushed to main:
e680f54 Harden grounded AI metadata answers
💡 The bigger lesson
A RAG system shouldn't ask an LLM to interpret something the software already knows with certainty.
If the database says:
onchainRecorded = false
the answer should come from that fact—not from the model's probability distribution.
LLMs for interpretation.
Structured data for authoritative decisions.
RAG for evidence.
That's the direction we're taking with MyZubster:
AI that is not only grounded, but verifiable.
Top comments (0)