DEV Community

Vikyath J
Vikyath J

Posted on AI-assisted

Your AI-written marketing report has wrong numbers. Here's a deterministic way to catch them.

If you've wired GPT, Claude or Gemini into a reporting pipeline (GA4 + Google Ads in, monthly client report out), you've probably seen this. The writing is good, and some of the numbers are wrong.

This isn't anecdotal. A study of LLM summaries of financial reports (arXiv 2404.06162) measured numeric hallucination rates of roughly 4–6.5%: numbers in the output that contradict, or can't be found in, the source data. If a similar rate held for a marketing report with 80–100 numbers, that would be several wrong figures per report, and a wrong number is exactly what a client is likely to notice.

What goes wrong

These are the error types a checker has to catch:

  1. Direction flips: "Conversions fell 13.9%" when they rose 13.9%. The number is right; the sentence is wrong.
  2. Rounding drift: a 25.58% CTR becomes "25%", or "about 26%" in one paragraph and "25%" in another.
  3. Averaged ratios: the model averages daily CTRs or ROAS values instead of recomputing them from totals. This gives a number that looks plausible and is wrong.
  4. Invented totals: a figure that appears nowhere in the data.

Better prompts can reduce these errors, but they can't guarantee the numbers are right. A more dependable approach: stop asking the LLM to do arithmetic, and check what it writes.

The pattern: compute in code, verify in code

  1. Compute every metric deterministically: totals, % change, CTR = clicks / impressions from totals, CPA, ROAS, goal progress. Give the LLM those numbers, already formatted.
  2. Let the LLM write.
  3. Verify the draft:
    • Extract every number, with its unit (%, x, currency, plain count) and the precision the writer used.
    • Match each one against the computed values, with a tolerance that depends on that precision. "12.3%" matches 12.34; "12k" matches 12,345.
    • Check direction words ("rose", "fell", "down") against the sign of the actual change.
    • Return PASS, or FAIL with the line, the number and the closest real value.
  4. If FAIL, ask the model to rewrite only the flagged sentences, and verify again.

Step 3 is the part people skip. It's also where the value is, because it turns "probably right" into "checked".

A runnable example

Try it on your own data, no sign-up: https://reportproof-api.onrender.com/try

I packaged this as an API, ReportProof, which has a free plan. It accepts CSV exports, plain JSON rows, a GA4 runReport response or a Google Ads search response.

import json, os, urllib.request

URL = "https://reportproof.p.rapidapi.com/v1/check"
HEADERS = {"X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"], "X-RapidAPI-Host": "reportproof.p.rapidapi.com",
           "Content-Type": "application/json"}

rows = [{"date": "2026-09-01", "campaign": "Brand", "impressions": 2000, "clicks": 100, "cost": 200, "conversions": 10},
        {"date": "2026-10-01", "campaign": "Brand", "impressions": 3000, "clicks": 150, "cost": 240, "conversions": 15}]
draft = "In October 2026 clicks fell 50.0% to 150 and cost per conversion was 16."

body = {"data": [{"name": "google_ads", "rows": rows}], "currency": "USD",
        "start": "2026-10-01", "end": "2026-10-31", "report": draft}
req = urllib.request.Request(URL, data=json.dumps(body).encode(), headers=HEADERS, method="POST")
v = json.loads(urllib.request.urlopen(req).read())["verification"]
print(v["status"], "-", v["checked"], "numbers checked")
for i in v["issues"]:
    print(f"  line {i['line']}: {i['number']} -> {i['problem']}")
Enter fullscreen mode Exit fullscreen mode

Output (from a real run of this request against the live API):

FAIL - 3 numbers checked
  line 1: 50.0% -> direction: says down but data says up
Enter fullscreen mode Exit fullscreen mode

Clicks did go to 150 and cost per conversion really is 16 (240 / 15), so those pass. The "fell" is caught.

In an agent

If your agent pulls data with Google's official GA4 MCP or Google Ads MCP server, add reportproof-mcp and tell the agent:

Write the report, then call check_report with the raw data and your draft. If it fails, fix only the flagged numbers and check again.

Limits (honestly)

  • It checks that each number exists in the data with the right direction. It doesn't check that it's used in the right context (e.g. a correct number attached to the wrong metric).
  • In a planted-error test (one number changed by about 7%, 40 runs) it caught 37 of 40. The misses were small whole numbers that happened to match other values in the data.
  • CSV/JSON input only; no direct account connections.

Even with those limits, a deterministic check catches most wrong numbers before the client sees them, and it tells you which lines need a human look instead of every line.

I built ReportProof; the free plan is 50 calls a month. Questions and failure cases are welcome in the comments.

Top comments (0)