<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Vikyath J</title>
    <description>The latest articles on DEV Community by Vikyath J (@vikyathhjui).</description>
    <link>https://dev.to/vikyathhjui</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4253985%2F166c1a41-3453-44dd-9090-6371ded5fe3b.jpg</url>
      <title>DEV Community: Vikyath J</title>
      <link>https://dev.to/vikyathhjui</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/vikyathhjui"/>
    <language>en</language>
    <item>
      <title>Your AI-written marketing report has wrong numbers. Here's a deterministic way to catch them.</title>
      <dc:creator>Vikyath J</dc:creator>
      <pubDate>Thu, 19 Nov 2026 14:20:47 +0000</pubDate>
      <link>https://dev.to/vikyathhjui/your-ai-written-marketing-report-has-wrong-numbers-heres-a-deterministic-way-to-catch-them-3ei3</link>
      <guid>https://dev.to/vikyathhjui/your-ai-written-marketing-report-has-wrong-numbers-heres-a-deterministic-way-to-catch-them-3ei3</guid>
      <description>&lt;p&gt;If you've wired GPT, Claude or Gemini into a reporting pipeline (GA4 + Google Ads in, monthly client report out), you've probably seen this. The writing is good, and &lt;strong&gt;some of the numbers are wrong.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This isn't anecdotal. A study of LLM summaries of financial reports (&lt;a href="https://arxiv.org/pdf/2404.06162" rel="noopener noreferrer"&gt;arXiv 2404.06162&lt;/a&gt;) measured &lt;em&gt;numeric&lt;/em&gt; hallucination rates of roughly &lt;strong&gt;4–6.5%&lt;/strong&gt;: numbers in the output that contradict, or can't be found in, the source data. If a similar rate held for a marketing report with 80–100 numbers, that would be several wrong figures per report, and a wrong number is exactly what a client is likely to notice.&lt;/p&gt;

&lt;h2&gt;
  
  
  What goes wrong
&lt;/h2&gt;

&lt;p&gt;These are the error types a checker has to catch:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Direction flips:&lt;/strong&gt; "Conversions fell 13.9%" when they rose 13.9%. The number is right; the sentence is wrong.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rounding drift:&lt;/strong&gt; a 25.58% CTR becomes "25%", or "about 26%" in one paragraph and "25%" in another.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Averaged ratios:&lt;/strong&gt; the model averages daily CTRs or ROAS values instead of recomputing them from totals. This gives a number that looks plausible and is wrong.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Invented totals:&lt;/strong&gt; a figure that appears nowhere in the data.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Better prompts can reduce these errors, but they can't guarantee the numbers are right. A more dependable approach: &lt;strong&gt;stop asking the LLM to do arithmetic, and check what it writes.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern: compute in code, verify in code
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Compute&lt;/strong&gt; every metric deterministically: totals, % change, CTR = clicks / impressions &lt;em&gt;from totals&lt;/em&gt;, CPA, ROAS, goal progress. Give the LLM those numbers, already formatted.&lt;/li&gt;
&lt;li&gt;Let the LLM &lt;strong&gt;write&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify&lt;/strong&gt; the draft:

&lt;ul&gt;
&lt;li&gt;Extract every number, with its unit (%, x, currency, plain count) and the precision the writer used.&lt;/li&gt;
&lt;li&gt;Match each one against the computed values, with a tolerance that depends on that precision. "12.3%" matches 12.34; "12k" matches 12,345.&lt;/li&gt;
&lt;li&gt;Check direction words ("rose", "fell", "down") against the sign of the actual change.&lt;/li&gt;
&lt;li&gt;Return &lt;strong&gt;PASS&lt;/strong&gt;, or &lt;strong&gt;FAIL&lt;/strong&gt; with the line, the number and the closest real value.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;If FAIL, ask the model to rewrite only the flagged sentences, and verify again.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Step 3 is the part people skip. It's also where the value is, because it turns "probably right" into "checked".&lt;/p&gt;

&lt;h2&gt;
  
  
  A runnable example
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Try it on your own data, no sign-up:&lt;/strong&gt; &lt;a href="https://reportproof-api.onrender.com/try" rel="noopener noreferrer"&gt;https://reportproof-api.onrender.com/try&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I packaged this as an API, &lt;a href="https://rapidapi.com/VikyathJ/api/reportproof" rel="noopener noreferrer"&gt;ReportProof&lt;/a&gt;, which has a free plan. It accepts CSV exports, plain JSON rows, a GA4 &lt;code&gt;runReport&lt;/code&gt; response or a Google Ads &lt;code&gt;search&lt;/code&gt; response.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="no"&gt;URL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"https://reportproof.p.rapidapi.com/v1/check"&lt;/span&gt;
&lt;span class="no"&gt;HEADERS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"X-RapidAPI-Key"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"RAPIDAPI_KEY"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"X-RapidAPI-Host"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"reportproof.p.rapidapi.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
           &lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-09-01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"campaign"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Brand"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"impressions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"clicks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cost"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"conversions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-10-01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"campaign"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Brand"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"impressions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"clicks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;150&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cost"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;240&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"conversions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;draft&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"In October 2026 clicks fell 50.0% to 150 and cost per conversion was 16."&lt;/span&gt;

&lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"data"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"google_ads"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rows"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"currency"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"USD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-10-01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-10-31"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"report"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;draft&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;HEADERS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"POST"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"verification"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"checked"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"numbers checked"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"issues"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"  line {i['line']}: {i['number']} -&amp;gt; {i['problem']}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output (from a real run of this request against the live API):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;FAIL - 3 numbers checked
  line 1: 50.0% -&amp;gt; direction: says down but data says up
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Clicks did go to 150 and cost per conversion really is 16 (240 / 15), so those pass. The "fell" is caught.&lt;/p&gt;

&lt;h2&gt;
  
  
  In an agent
&lt;/h2&gt;

&lt;p&gt;If your agent pulls data with Google's official &lt;strong&gt;GA4 MCP&lt;/strong&gt; or &lt;strong&gt;Google Ads MCP&lt;/strong&gt; server, add &lt;a href="https://github.com/vikyathhj-ui/reportproof-mcp" rel="noopener noreferrer"&gt;reportproof-mcp&lt;/a&gt; and tell the agent:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Write the report, then call &lt;code&gt;check_report&lt;/code&gt; with the raw data and your draft. If it fails, fix only the flagged numbers and check again.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Limits (honestly)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;It checks that each number &lt;strong&gt;exists in the data with the right direction&lt;/strong&gt;. It doesn't check that it's used in the right &lt;em&gt;context&lt;/em&gt; (e.g. a correct number attached to the wrong metric).&lt;/li&gt;
&lt;li&gt;In a planted-error test (one number changed by about 7%, 40 runs) it caught &lt;strong&gt;37 of 40&lt;/strong&gt;. The misses were small whole numbers that happened to match other values in the data.&lt;/li&gt;
&lt;li&gt;CSV/JSON input only; no direct account connections.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Even with those limits, a deterministic check catches most wrong numbers before the client sees them, and it tells you which lines need a human look instead of every line.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I built ReportProof; the free plan is 50 calls a month. Questions and failure cases are welcome in the comments.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>marketing</category>
      <category>python</category>
    </item>
  </channel>
</rss>
