Show "can't verify" instead of a wrong number: handling declines in an AI data app

#llm #ai #python #javascript

Your AI app answers a question about your data. The answer sounds right, the number looks plausible, and nothing errors. A week later someone in finance notices it was wrong.

The fix most teams reach for is a better prompt. The fix that actually protects users is simpler: when the system can't back an answer, it should say so, and your app should show that clearly instead of a number.

This post shows how to wire that into an app, in Python and Node, using FactQL. The pattern applies to any AI data layer that can decline.

What a decline looks like

FactQL runs the query before it answers. When it can back the answer, you get the explanation, the query that ran, the rows and the lineage:

{
  "result": "success",
  "sql": "SELECT region, SUM(revenue) AS total_revenue FROM orders GROUP BY region ORDER BY total_revenue DESC",
  "data": { "columns": ["region", "total_revenue"], "rows": [["North", 43148.0], ["South", 41223.0]] },
  "explanation": "North leads with 43,148, followed by South (41,223).",
  "trace_id": "c5f50cff-..."
}

When it can't, it declines instead of guessing. Note this is still HTTP 200:

{
  "result": "error",
  "decline_reason": "no_safe_sql",
  "explanation": "I couldn't find data that answers this. Try naming the table or column you mean.",
  "trace_id": "c5f50cff-..."
}

That status code matters. If your code only checks resp.ok, a decline sails through and your UI renders an empty or half-filled answer. You have to check result.

The four rules

Check result before you show any number.
On a decline, show the explanation. It's written for the end user and tells them what to change.
Don't retry the same question. It gets the same answer.
Retry only on HTTP 429, 500 and 503, after Retry-After.

Python

import os, sys, time
import requests

RETRYABLE = {429, 500, 503}

def ask(question: str, attempts: int = 3) -> dict:
    for attempt in range(attempts):
        resp = requests.post(
            "https://api.factql.ai/query",
            headers={"X-API-Key": os.environ["FACTQL_API_KEY"]},
            json={"q": question},
            timeout=120,
        )
        if resp.status_code in RETRYABLE and attempt < attempts - 1:
            time.sleep(int(resp.headers.get("Retry-After", 2 ** (attempt + 1))))
            continue
        resp.raise_for_status()
        return resp.json()

def render(answer: dict) -> str:
    if answer.get("result") == "success":
        return answer["explanation"]
    print(f"[log] declined: {answer.get('decline_reason')} trace={answer.get('trace_id')}", file=sys.stderr)
    return f"I can't verify that from your data. {answer.get('explanation', '')}".strip()

print(render(ask("What was our ad spend last quarter?")))

Ask about something your data doesn't contain (ad spend on a sales-only database is the easiest) and you'll see the decline path.

Node

const RETRYABLE = new Set([429, 500, 503]);

async function ask(question, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    const resp = await fetch("https://api.factql.ai/query", {
      method: "POST",
      headers: { "X-API-Key": process.env.FACTQL_API_KEY, "Content-Type": "application/json" },
      body: JSON.stringify({ q: question }),
    });
    if (RETRYABLE.has(resp.status) && i < attempts - 1) {
      const wait = Number(resp.headers.get("Retry-After")) || 2 ** (i + 1);
      await new Promise((r) => setTimeout(r, wait * 1000));
      continue;
    }
    if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
    return resp.json();
  }
}

function render(answer) {
  if (answer.result === "success") return answer.explanation;
  console.error(`[log] declined: ${answer.decline_reason} trace=${answer.trace_id}`);
  return `I can't verify that from your data. ${answer.explanation ?? ""}`.trim();
}

If an agent is calling it

When FactQL is a tool for an LLM agent, the decline has to survive the agent too. Two things help:

Return the decline to the agent with an unmistakable prefix, like DECLINED:.
Tell the agent in its system prompt: only state numbers that come back from the tool; if it returns DECLINED, say you can't verify that and pass on the explanation; never estimate.

Without that instruction, a helpful model will often "fill the gap" with an estimate, which puts you right back where you started.

Logging declines is free product research

Every decline_reason you log is a question your users asked that your data couldn't answer. Grouped weekly, that list tells you which tables, columns or definitions to add next.

Full code

All of this, plus a LangChain / LangGraph tool, OpenAI-compatible function calling, a Next.js route and an embedded chat, is in factql/factql-examples (MIT).

We build FactQL, so we're biased: it has a free tier with no card. The pattern works with any AI data layer that can say "I can't verify that".

DEV Community