If you've wired GPT, Claude or Gemini into a reporting pipeline (GA4 + Google Ads in, monthly client report out), you've probably seen this. The writing is good, and some of the numbers are wrong.
This isn't anecdotal. A study of LLM summaries of financial reports (arXiv 2404.06162) measured numeric hallucination rates of roughly 4–6.5%: numbers in the output that contradict, or can't be found in, the source data. If a similar rate held for a marketing report with 80–100 numbers, that would be several wrong figures per report, and a wrong number is exactly what a client is likely to notice.
What goes wrong
These are the error types a checker has to catch:
- Direction flips: "Conversions fell 13.9%" when they rose 13.9%. The number is right; the sentence is wrong.
- Rounding drift: a 25.58% CTR becomes "25%", or "about 26%" in one paragraph and "25%" in another.
- Averaged ratios: the model averages daily CTRs or ROAS values instead of recomputing them from totals. This gives a number that looks plausible and is wrong.
- Invented totals: a figure that appears nowhere in the data.
Better prompts can reduce these errors, but they can't guarantee the numbers are right. A more dependable approach: stop asking the LLM to do arithmetic, and check what it writes.
The pattern: compute in code, verify in code
- Compute every metric deterministically: totals, % change, CTR = clicks / impressions from totals, CPA, ROAS, goal progress. Give the LLM those numbers, already formatted.
- Let the LLM write.
-
Verify the draft:
- Extract every number, with its unit (%, x, currency, plain count) and the precision the writer used.
- Match each one against the computed values, with a tolerance that depends on that precision. "12.3%" matches 12.34; "12k" matches 12,345.
- Check direction words ("rose", "fell", "down") against the sign of the actual change.
- Return PASS, or FAIL with the line, the number and the closest real value.
- If FAIL, ask the model to rewrite only the flagged sentences, and verify again.
Step 3 is the part people skip. It's also where the value is, because it turns "probably right" into "checked".
A runnable example
Try it on your own data, no sign-up: https://reportproof-api.onrender.com/try
I packaged this as an API, ReportProof, which has a free plan. It accepts CSV exports, plain JSON rows, a GA4 runReport response or a Google Ads search response.
import json, os, urllib.request
URL = "https://reportproof.p.rapidapi.com/v1/check"
HEADERS = {"X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"], "X-RapidAPI-Host": "reportproof.p.rapidapi.com",
"Content-Type": "application/json"}
rows = [{"date": "2026-09-01", "campaign": "Brand", "impressions": 2000, "clicks": 100, "cost": 200, "conversions": 10},
{"date": "2026-10-01", "campaign": "Brand", "impressions": 3000, "clicks": 150, "cost": 240, "conversions": 15}]
draft = "In October 2026 clicks fell 50.0% to 150 and cost per conversion was 16."
body = {"data": [{"name": "google_ads", "rows": rows}], "currency": "USD",
"start": "2026-10-01", "end": "2026-10-31", "report": draft}
req = urllib.request.Request(URL, data=json.dumps(body).encode(), headers=HEADERS, method="POST")
v = json.loads(urllib.request.urlopen(req).read())["verification"]
print(v["status"], "-", v["checked"], "numbers checked")
for i in v["issues"]:
print(f" line {i['line']}: {i['number']} -> {i['problem']}")
Output (from a real run of this request against the live API):
FAIL - 3 numbers checked
line 1: 50.0% -> direction: says down but data says up
Clicks did go to 150 and cost per conversion really is 16 (240 / 15), so those pass. The "fell" is caught.
In an agent
If your agent pulls data with Google's official GA4 MCP or Google Ads MCP server, add reportproof-mcp and tell the agent:
Write the report, then call
check_reportwith the raw data and your draft. If it fails, fix only the flagged numbers and check again.
Limits (honestly)
- It checks that each number exists in the data with the right direction. It doesn't check that it's used in the right context (e.g. a correct number attached to the wrong metric).
- In a planted-error test (one number changed by about 7%, 40 runs) it caught 37 of 40. The misses were small whole numbers that happened to match other values in the data.
- CSV/JSON input only; no direct account connections.
Even with those limits, a deterministic check catches most wrong numbers before the client sees them, and it tells you which lines need a human look instead of every line.
I built ReportProof; the free plan is 50 calls a month. Questions and failure cases are welcome in the comments.
Top comments (0)