Ask an agent "what's our refund total this month?" and it will answer. Fluently, confidently, and sometimes without ever touching your database. The number sounds checked because the sentence around it is well written.
The fix is not a better prompt. It's a rule: the agent never states a figure from memory; it calls a tool and quotes what comes back.
This post builds that tool in plain Python. No framework: the same shape works with any model that supports tool calling.
The contract
A data tool an agent can be trusted with returns three things:
- an answer in words, written from the rows,
- the rows themselves, so the agent can quote instead of recall,
- the query that ran, for your logs and a "show details" link.
And when there's no data answer, it returns a sentence to say, not a number.
The tool
It calls FactQL, which runs the query before it answers and returns the rows. Without a key of your own, the script borrows the public sample key (a demo film-rental shop), so you can run it as is.
# pip install requests
import json, os, requests
FACTQL = "https://api.factql.ai"
KEY = os.environ.get("FACTQL_API_KEY") or requests.get(f"{FACTQL}/sample", timeout=30).json()["api_key"]
# 1. The tool description your model sees. The wording matters: it tells the
# model when to stop guessing and call the tool.
ASK_DATA_TOOL = {
"name": "ask_data",
"description": (
"Answer a question about the company's data. Use this for ANY number, "
"total, count, ranking or date that comes from the database. Never state "
"such a figure from memory."
),
"input_schema": {
"type": "object",
"properties": {"question": {"type": "string", "description": "The full question, in words."}},
"required": ["question"],
},
}
# 2. The function that runs when the model calls the tool.
def ask_data(question: str) -> str:
r = requests.post(f"{FACTQL}/query", headers={"X-API-Key": KEY}, json={"q": question}, timeout=120)
if r.status_code == 429:
return json.dumps({"ok": False, "say": "The data service is busy. Ask the user to try again in a minute."})
r.raise_for_status()
a = r.json()
if a.get("result") != "success":
# No data answer. Hand the model the user-facing explanation, not a number.
return json.dumps({"ok": False, "say": a.get("explanation", "No answer found.")})
rows = a["data"]["rows"]
return json.dumps({
"ok": True,
"answer": a["explanation"],
"columns": a["data"]["columns"],
"rows": rows[:20], # enough to quote, small enough for the context window
"row_count": len(rows),
"query": a["sql"], # keep it for your logs and for a "show details" link
})
if __name__ == "__main__":
print(json.dumps(json.loads(ask_data("Which 5 films have earned the most rental revenue?")), indent=2))
What the agent receives
A real run against the sample data (17 Nov 2026):
{
"ok": true,
"answer": "Amounts are in the store's currency.",
"columns": ["title", "total_revenue"],
"rows": [["CAT CONEHEADS", 640.78], ["WIFE TURN", 587.96], ["TORQUE BOUND", 567.98], ["BUCKET BROTHERHOOD", 529.96], ["ZORRO ARK", 519.08]],
"row_count": 5,
"query": "SELECT f.title, SUM(p.amount) AS total_revenue FROM payment p JOIN rental r ON p.rental_id = r.rental_id JOIN inventory i ON r.inventory_id = i.inventory_id JOIN film f ON i.film_id = f.film_id GROUP BY f.film_id, f.title ORDER BY total_revenue DESC LIMIT 5"
}
(Amounts rounded here; the raw values are floats like 640.7800000000009.)
Wiring it into your loop
Pass ASK_DATA_TOOL in your model call's tool list. When the model asks for ask_data, run the function and send the string back as the tool result. Then add one line to your system prompt:
Every figure you state must appear in a tool result from this conversation. If it doesn't, call
ask_datafirst.
Four details that matter
- The description does the steering. "Use this for ANY number… Never state such a figure from memory" is what makes the model reach for the tool.
-
Cap the rows. Twenty rows is enough to quote and cheap in context. Send
row_countso the model knows when it's seeing a slice. -
On failure, give words.
{"ok": false, "say": "..."}leaves the model nothing numeric to improvise from. - Log the query with the conversation id. When someone asks "where did that come from?", you can answer in one lookup.
What this doesn't solve
The agent can still ask the wrong question. A tool makes the number traceable; it doesn't make the question right. Show the rows to the user when the stakes are high.
Where do your agents get their numbers today?
Top comments (0)