<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: AICostbudget.com</title>
    <description>The latest articles on DEV Community by AICostbudget.com (@linqiangmax).</description>
    <link>https://dev.to/linqiangmax</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4215637%2F28f47195-4309-43f7-a9bc-9754ceffe82c.jpg</url>
      <title>DEV Community: AICostbudget.com</title>
      <link>https://dev.to/linqiangmax</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/linqiangmax"/>
    <language>en</language>
    <item>
      <title>Where an agent's tokens go: 9 real GitHub bugs, two Gemini Flash-Lite models, 1 fix each</title>
      <dc:creator>AICostbudget.com</dc:creator>
      <pubDate>Sat, 21 Nov 2026 16:11:01 +0000</pubDate>
      <link>https://dev.to/linqiangmax/where-an-agents-tokens-go-9-real-github-bugs-two-gemini-flash-lite-models-1-fix-each-32j1</link>
      <guid>https://dev.to/linqiangmax/where-an-agents-tokens-go-9-real-github-bugs-two-gemini-flash-lite-models-1-fix-each-32j1</guid>
      <description>&lt;p&gt;I gave two Gemini Flash-Lite models the same 9 real bug reports from GitHub and let each one work in a loop: search, read a file, edit, run the tests, submit. Each model fixed one bug. Between them they made 964 model calls.&lt;/p&gt;

&lt;p&gt;The tasks come from SWE-bench-Live (github.com/microsoft/SWE-bench-Live, MIT). I picked small pure-Python repos I could run on a Windows machine without Docker: attrs, faker and cfn-lint. A task only counts if the original tests fail before the fix and pass with the real fix applied. The agent never sees the new tests. When it submits, I apply them and run them. If they fail, it gets one more attempt and sees the tail of the failing output, like a CI log. Each attempt is capped at 30 steps.&lt;/p&gt;

&lt;p&gt;The models were gemini-3.5-flash-lite and gemini-3.1-flash-lite, no thinking, on the free tier, so nothing was billed. The dollar figures below are token counts multiplied by the paid list prices ($0.30 / $0.03 cached / $2.50 per million tokens for 3.5, $0.25 / $0.025 / $1.50 for 3.1). Read them as what this trace would cost, not as a bill.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;3.5 Flash-Lite&lt;/th&gt;
&lt;th&gt;3.1 Flash-Lite&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Bugs fixed&lt;/td&gt;
&lt;td&gt;1 of 9 (first try)&lt;/td&gt;
&lt;td&gt;1 of 9 (second try)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Attempts&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model calls&lt;/td&gt;
&lt;td&gt;483&lt;/td&gt;
&lt;td&gt;481&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Input tokens&lt;/td&gt;
&lt;td&gt;4.29M&lt;/td&gt;
&lt;td&gt;3.00M&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output tokens&lt;/td&gt;
&lt;td&gt;16K&lt;/td&gt;
&lt;td&gt;24K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Input read from cache&lt;/td&gt;
&lt;td&gt;66%&lt;/td&gt;
&lt;td&gt;53%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Total at list price&lt;/td&gt;
&lt;td&gt;$0.561&lt;/td&gt;
&lt;td&gt;$0.426&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Per task&lt;/td&gt;
&lt;td&gt;$0.062&lt;/td&gt;
&lt;td&gt;$0.047&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Per fixed bug&lt;/td&gt;
&lt;td&gt;$0.561&lt;/td&gt;
&lt;td&gt;$0.426&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spent on tasks that never passed&lt;/td&gt;
&lt;td&gt;96%&lt;/td&gt;
&lt;td&gt;88%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spent on second attempts&lt;/td&gt;
&lt;td&gt;49%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Per task it is about 5 or 6 cents. Per fixed bug it is 43 to 56 cents, because one success has to carry all the failures. That gap is the whole idea of cost per successful task. It is also a single data point per model: the two models fixed different bugs, and a different task would have moved these numbers a lot.&lt;/p&gt;

&lt;p&gt;Output was tiny, 16K and 24K tokens against millions of input. An agent loop sends the whole conversation back on every step. The first call of an attempt was about 1,060 tokens. The last call was 10,000 to 15,000. The cache helped, but fresh input was still 78% and 82% of the cost.&lt;/p&gt;

&lt;p&gt;What did they do with their steps? 3.5 made 306 searches, 124 file reads, 33 test runs and 5 edits in 483 calls. 14 of its 17 attempts ran out of steps before it submitted anything. 3.1 searched less (160 times), edited more (50 edits) and submitted in 7 of 18 attempts, but 11 attempts still hit the cap, and only one of the submitted attempts passed the hidden tests. The only retry that paid off was 3.1 on attrs-1320: the second attempt saw the failing test output and fixed it.&lt;/p&gt;

&lt;p&gt;I also put the measured numbers into three calculators on my site, to see which ones hold up on an agent loop.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Importing the trace into the team API cost report gave $0.5611 and $0.4256, the same as my per-call sums. The trace format has no field for thinking tokens, latency or tool calls, so I keep those in a side file.&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://aicostbudget.com/en/llm-retry-cost-calculator?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=agent-cost-case-free-gemini&amp;amp;utm_content=devto-2026-11" rel="noopener noreferrer"&gt;retry/fallback calculator&lt;/a&gt;, given about 2 attempts per task, 89% of tasks unresolved and the cached share, came to $0.5616 and $0.4258.&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://aicostbudget.com/en/ai-workflow-cost-calculator?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=agent-cost-case-free-gemini&amp;amp;utm_content=devto-2026-11" rel="noopener noreferrer"&gt;workflow calculator&lt;/a&gt;, which takes calls per task and tokens per call, said $1.33 and $0.78. That is 2.4 and 1.8 times too high. It has no cache input, so it bills every re-sent token at the full input price. For a loop that re-reads its context 50 times per task, that was the biggest error I found, and it is a gap in my own tool.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Things that make this weaker than it looks. Nine tasks, one fix each, on different bugs, so I can't say one model is better. The issues date from July 2024 to March 2025, so both models may have seen them. Google's pricing page lists context caching as unavailable on the free tier for 3.5 Flash-Lite, yet cache tokens were reported, and paid-tier hit rates could differ. The 30-step cap and my prompt probably cost both models some fixes. 3.1 hit its free daily limit of 500 requests after 9 of 12 tasks, so I left the other three out for both models. And 3.1 needed 14 API retries where 3.5 needed 1.&lt;/p&gt;

&lt;p&gt;I built this to compare GPT-6.2 Sol and Gemini 4 Neon on the same tasks. I don't have an API budget for those yet, and Neon has no free tier, so I ran what I could. The harness has adapters for both, but I have only tested them against fake servers so far. There is also a &lt;a href="https://aicostbudget.com/en/blog/cheapest-llm-cost-per-successful-task?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=agent-cost-case-free-gemini&amp;amp;utm_content=devto-2026-11" rel="noopener noreferrer"&gt;write-up of cost per successful task&lt;/a&gt; if you want the definition I used.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>python</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>Agent trace replay: 3 tasks, 2 successes, $0.009 per delivered result</title>
      <dc:creator>AICostbudget.com</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:00:49 +0000</pubDate>
      <link>https://dev.to/linqiangmax/agent-trace-replay-3-tasks-2-successes-0009-per-delivered-result-1893</link>
      <guid>https://dev.to/linqiangmax/agent-trace-replay-3-tasks-2-successes-0009-per-delivered-result-1893</guid>
      <description>&lt;p&gt;Three tasks ran. Two passed. The tasks that passed averaged $0.006 each, but the run cost $0.009 per delivered result. The extra $0.003 per result came from a task that ran twice and still failed.&lt;/p&gt;

&lt;p&gt;I built a trace replay in AICostBudget to see where that spend goes. Here's the small synthetic example I used to check the math:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;What happened&lt;/th&gt;
&lt;th&gt;Model cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;Passed on the second attempt&lt;/td&gt;
&lt;td&gt;$0.006&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B&lt;/td&gt;
&lt;td&gt;Passed on one call marked as a fallback&lt;/td&gt;
&lt;td&gt;$0.006&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C&lt;/td&gt;
&lt;td&gt;Failed after two attempts&lt;/td&gt;
&lt;td&gt;$0.006&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2 successes across 5 model calls&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.018&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All five calls use GPT-5.4 mini. At $0.75 per million input tokens and $4.50 per million output tokens, four calls cost $0.003 each (1,000 input + 500 output tokens). The fallback call used twice as many tokens, so it cost $0.006.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cost per successful task = $0.018 total spend ÷ 2 successes = $0.009.&lt;/strong&gt; Looking only at A and B gives $0.006, which leaves a third of the run's spend out of the calculation.&lt;/p&gt;

&lt;p&gt;The two retries cost $0.003 each. A's retry got the task through; C's didn't. C's second call shows up under both &lt;strong&gt;Retry tax&lt;/strong&gt; and &lt;strong&gt;Failed-work cost&lt;/strong&gt;. Those categories overlap, so adding them would count that call twice.&lt;/p&gt;

&lt;p&gt;To reproduce it, open the &lt;a href="https://aicostbudget.com/en/team-api-cost-report#agent-trace-title" rel="noopener noreferrer"&gt;trace cost report&lt;/a&gt; and click &lt;strong&gt;Load synthetic example&lt;/strong&gt;. You should see $0.006 in failed-work cost and $0.009 per successful task. You can also import Agent Cost Evidence JSON v0.1 or a supported OpenInference / OTLP JSON export with a separate task outcomes file. The trace stays in your browser, and the estimate uses current standard rates.&lt;/p&gt;

&lt;p&gt;For your own workflow, decide what counts as a pass before comparing retry rules. For invoice extraction, I'd count the invoice being accepted downstream. Then run both rules on the same tasks and compare how many passed and what each success cost.&lt;/p&gt;

&lt;p&gt;If you're already measuring this, where do you record whether a task passed: in the trace itself, or in a separate eval or job record?&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>softwaredevelopment</category>
    </item>
    <item>
      <title>What a 70% vs 90% prompt cache hit rate does to an AI agent’s bill (72 models priced)</title>
      <dc:creator>AICostbudget.com</dc:creator>
      <pubDate>Mon, 09 Nov 2026 14:37:59 +0000</pubDate>
      <link>https://dev.to/linqiangmax/what-a-70-vs-90-prompt-cache-hit-rate-does-to-an-ai-agents-bill-72-models-priced-573d</link>
      <guid>https://dev.to/linqiangmax/what-a-70-vs-90-prompt-cache-hit-rate-does-to-an-ai-agents-bill-72-models-priced-573d</guid>
      <description>&lt;p&gt;An agent sends the same long prefix on every step: system prompt, tool definitions, the files it has read, the conversation so far. Most of it is already in the provider's prompt cache from the step before. How much of it is, the cache hit rate, ends up deciding most of the bill.&lt;/p&gt;

&lt;p&gt;I wanted a number for that, so I fixed one task and priced it on every current model that publishes a cached-input price: 72 models from 13 providers, list prices as of November 9, 2026.&lt;/p&gt;

&lt;p&gt;The task: 20 calls, each sending 40,000 input tokens and getting 1,000 back. Only the hit rate changes.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Hit rate&lt;/th&gt;
&lt;th&gt;Median cost per task&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;No cache&lt;/td&gt;
&lt;td&gt;$1.05&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;td&gt;$0.63&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;70%&lt;/td&gt;
&lt;td&gt;$0.46&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;$0.29&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  70% to 90% is worth more than it sounds
&lt;/h2&gt;

&lt;p&gt;Twenty points of hit rate cuts the median task cost by 38%. At 70% the 30% of input you miss is billed at full price; at 90% only 10% is.&lt;/p&gt;

&lt;p&gt;For scale, I paired each model with the next cheaper model from the same provider, both at 70%, and compared that saving with keeping the model and reaching 90%. In 35 of 59 pairs the better hit rate saved more. That's price only, not quality. But it's worth checking your cache before you start a model migration.&lt;/p&gt;

&lt;h2&gt;
  
  
  Same list price, different bill
&lt;/h2&gt;

&lt;p&gt;Claude Sonnet 5.5, GPT-6.1 Sol and Gemini 4 Argon all list $2 input, $10 output and $0.10 cached input per million tokens. With no cache, 100 tasks cost $180.00 on each.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Cache write per 1M&lt;/th&gt;
&lt;th&gt;100 tasks, 50% hit&lt;/th&gt;
&lt;th&gt;100 tasks, 90% hit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Claude Sonnet 5.5&lt;/td&gt;
&lt;td&gt;$2.50&lt;/td&gt;
&lt;td&gt;$124.00&lt;/td&gt;
&lt;td&gt;$47.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6.1 Sol&lt;/td&gt;
&lt;td&gt;$2.50&lt;/td&gt;
&lt;td&gt;$124.00&lt;/td&gt;
&lt;td&gt;$47.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 4 Argon&lt;/td&gt;
&lt;td&gt;none listed&lt;/td&gt;
&lt;td&gt;$104.00&lt;/td&gt;
&lt;td&gt;$43.20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every miss on Sonnet or Sol gets written to the cache at $2.50 instead of $2. That's 19% more than Argon at a 50% hit rate and 9% more at 90%. (Gemini charges an hourly storage fee if you create caches explicitly; that isn't counted here.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Below about 21%, caching costs you money
&lt;/h2&gt;

&lt;p&gt;22 models in the set charge 25% more than input to write the cache: Claude, and OpenAI's GPT-5.6 and GPT-6 families. So there's a break-even:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;break-even hit rate = (write - input) / (write - cached read)
Claude Sonnet 5.5: (2.50 - 2.00) / (2.50 - 0.10) = 20.8%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;For these 22 it lands between 20% and 22%. An agent rereading the same context clears that easily. A workload where every request has a different prefix may not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cheapest model with no cache isn't the cheapest at 90%
&lt;/h2&gt;

&lt;p&gt;With no cache, Grok 4.3 ($105.00 per 100 tasks) is cheaper than DeepSeek V4 Pro ($113.52). At 90%, DeepSeek costs $21.65 and Grok $29.40, about a quarter less, because DeepSeek takes 96.7% off a cache read and Grok takes 84% off.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to raise and measure it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Put what never changes first: tools, system prompt, reference docs. Append each step at the end. The cache matches from the start of the prompt.&lt;/li&gt;
&lt;li&gt;Keep timestamps, request IDs and per-user details out of the system prompt.&lt;/li&gt;
&lt;li&gt;Don't reorder the tool list between steps.&lt;/li&gt;
&lt;li&gt;On Claude, check the cache lifetime. The default is 5 minutes; if your steps are slower, the 1-hour cache costs more to write but stops you rewriting the prefix every step.&lt;/li&gt;
&lt;li&gt;Measure it: cached input tokens divided by total input tokens. OpenAI reports &lt;code&gt;cached_tokens&lt;/code&gt; in the token details, Anthropic reports &lt;code&gt;cache_read_input_tokens&lt;/code&gt; and &lt;code&gt;cache_creation_input_tokens&lt;/code&gt; next to &lt;code&gt;input_tokens&lt;/code&gt;, Gemini reports &lt;code&gt;cachedContentTokenCount&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;List prices only. The task shape and hit rates are my scenarios, not measurements of real agents. Not included: Gemini's cache storage fee, Batch, DeepSeek off-peak rates, cache expiry between calls, minimum cacheable prompt sizes.&lt;/p&gt;

&lt;p&gt;The full write-up, with a 12-model table and the method, is here: &lt;a href="https://aicostbudget.com/en/blog/agent-prompt-cache-hit-rate-cost" rel="noopener noreferrer"&gt;https://aicostbudget.com/en/blog/agent-prompt-cache-hit-rate-cost&lt;/a&gt;. The prices come from an open dataset (CC BY 4.0) where every price links to the provider's own page: &lt;a href="https://github.com/aicostbudget/ai-api-pricing-data" rel="noopener noreferrer"&gt;https://github.com/aicostbudget/ai-api-pricing-data&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>agents</category>
      <category>finops</category>
    </item>
    <item>
      <title>Google cut a Gemini price in half in August. It doubles back in January.</title>
      <dc:creator>AICostbudget.com</dc:creator>
      <pubDate>Thu, 05 Nov 2026 14:46:52 +0000</pubDate>
      <link>https://dev.to/linqiangmax/google-cut-a-gemini-price-in-half-in-august-it-doubles-back-in-january-4of8</link>
      <guid>https://dev.to/linqiangmax/google-cut-a-gemini-price-in-half-in-august-it-doubles-back-in-january-4of8</guid>
      <description>&lt;p&gt;On August 29 Gemini 3.6 Flash dropped from $1.50 to $0.75 per million input tokens, and from $7.50 to $3.75 for output. Two months later a new line showed up on Google's pricing page: from January 1, 2027, it goes back to $1.50 and $7.50.&lt;/p&gt;

&lt;p&gt;That made me curious how often this kind of thing happens, so I counted. I took every LLM API price change I could prove from the providers' own pricing pages since 2024. Older ones come from Internet Archive copies of the page from before and after the change. Newer ones come from a monitor I run that checks the official pages. I ended up with 47 changes across OpenAI, Anthropic, Google, DeepSeek, Mistral and xAI.&lt;/p&gt;

&lt;p&gt;Short version: prices mostly go down. But cuts come early or not at all, some of them have an end date, and the increases land in the parts of the price list people skip.&lt;/p&gt;

&lt;h2&gt;
  
  
  29 cuts, 8 increases
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Provider&lt;/th&gt;
&lt;th&gt;Changes&lt;/th&gt;
&lt;th&gt;Cuts&lt;/th&gt;
&lt;th&gt;Increases&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Google Gemini&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mistral AI&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;xAI&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The other 10 were neither, for example a cached-input price added for the first time, or an alias moved to a newer snapshot.&lt;/p&gt;

&lt;p&gt;OpenAI and Anthropic never raised a listed rate on an existing model. They just launch the next one at whatever price they like. Google raised prices five times, more than every other provider in the list put together.&lt;/p&gt;

&lt;h2&gt;
  
  
  When prices drop, they drop a lot
&lt;/h2&gt;

&lt;p&gt;Three models had their input price cut by 80% in one step: Codestral (Sep 2024, $1.00 → $0.20), o3 (Jun 2025, $10 → $2) and GPT-5.6 Luna (Jul 2026, $1.00 → $0.20). Gemini 1.5 Flash lost 79%, and DeepSeek V4 Pro and Mistral Large 3 lost 75%.&lt;/p&gt;

&lt;p&gt;Here's what that looks like on an invoice. A team spending $3,000 a month on Luna on July 29 was spending about $600 for the same traffic two days later, and nobody emailed them about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  If a cut comes, it's in the first two months
&lt;/h2&gt;

&lt;p&gt;This was the part I didn't expect. I matched each 2026 model's first cut against its launch date:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Launched&lt;/th&gt;
&lt;th&gt;First cut&lt;/th&gt;
&lt;th&gt;Days&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek V4 Pro&lt;/td&gt;
&lt;td&gt;Apr 24&lt;/td&gt;
&lt;td&gt;Apr 26&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Sonnet 5.5&lt;/td&gt;
&lt;td&gt;Sep 28&lt;/td&gt;
&lt;td&gt;Oct 7&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.6 Luna&lt;/td&gt;
&lt;td&gt;Jul 9&lt;/td&gt;
&lt;td&gt;Jul 30&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.6 Terra&lt;/td&gt;
&lt;td&gt;Jul 9&lt;/td&gt;
&lt;td&gt;Jul 30&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Sonnet 4.6&lt;/td&gt;
&lt;td&gt;Feb 17&lt;/td&gt;
&lt;td&gt;Mar 13&lt;/td&gt;
&lt;td&gt;24&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 3.6 Flash&lt;/td&gt;
&lt;td&gt;Jul 21&lt;/td&gt;
&lt;td&gt;Aug 29&lt;/td&gt;
&lt;td&gt;39&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.6 Sol&lt;/td&gt;
&lt;td&gt;Jul 9&lt;/td&gt;
&lt;td&gt;Aug 22&lt;/td&gt;
&lt;td&gt;44&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Grok 4.20&lt;/td&gt;
&lt;td&gt;Mar 10&lt;/td&gt;
&lt;td&gt;May 6&lt;/td&gt;
&lt;td&gt;57&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every one landed within 57 days, and the median was 24. Plenty of models never got a cut at all, so this is about when, not whether. It still means the first two months after a launch are when you're most likely to be overpaying. That's a bad time to buy prepaid credits or sign a volume commitment.&lt;/p&gt;

&lt;h2&gt;
  
  
  The increases are in rows you don't read
&lt;/h2&gt;

&lt;p&gt;Four of the eight increases were the same change on four Gemini models in September 2025. The Batch-tier cached-input price went from half the standard cached price to the full price. That's a 100% increase, in one row of the pricing table, and I couldn't find any announcement.&lt;/p&gt;

&lt;p&gt;The rest were mostly launch discounts ending. DeepSeek-V3's output went from $0.28 to $1.10 (+293%) in February 2025, though the end date had been on the page the whole time. Mistral Large 4 launched at 50% off for two weeks, doubled, and then got cut again ten days later.&lt;/p&gt;

&lt;p&gt;If you use prompt caching, a big chunk of your bill can sit in that cached row. It's the row that changes without a press release.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I do now
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Budget at the full list price. If what you see is a launch offer, look for the end date. It's usually on the page in small print.&lt;/li&gt;
&lt;li&gt;Read the cached-input and Batch rows, not just input and output.&lt;/li&gt;
&lt;li&gt;Don't prepay for a model in its first two months.&lt;/li&gt;
&lt;li&gt;When a cut lands on a model you're &lt;em&gt;not&lt;/em&gt; using, re-price your workload. An 80% cut can make it cheaper than the one you are using.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The full list, where every change links to the archived pages it came from, is the &lt;a href="https://aicostbudget.com/en/llm-api-price-history?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=llm-price-changes&amp;amp;utm_content=devto-2026-11-b" rel="noopener noreferrer"&gt;LLM API price history&lt;/a&gt;. The longer writeup, with the scheduled Gemini increases and the method, is &lt;a href="https://aicostbudget.com/en/blog/do-llm-api-prices-go-down?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=llm-price-changes&amp;amp;utm_content=devto-2026-11-b" rel="noopener noreferrer"&gt;here&lt;/a&gt;. The raw data is a JSON file in a &lt;a href="https://github.com/aicostbudget/ai-api-pricing-data" rel="noopener noreferrer"&gt;public GitHub dataset&lt;/a&gt; if you'd rather count it yourself.&lt;/p&gt;

&lt;p&gt;If you know of a price change I missed, especially an older OpenAI or Mistral one, leave a comment and I'll check the archives.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I used AI tools to help draft this post. The numbers and sources are mine and I checked each one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>api</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Stop dividing by 4. I counted 951 texts and the rule is wrong both ways.</title>
      <dc:creator>AICostbudget.com</dc:creator>
      <pubDate>Tue, 03 Nov 2026 14:11:34 +0000</pubDate>
      <link>https://dev.to/linqiangmax/stop-dividing-by-4-i-counted-951-texts-and-the-rule-is-wrong-both-ways-58l7</link>
      <guid>https://dev.to/linqiangmax/stop-dividing-by-4-i-counted-951-texts-and-the-rule-is-wrong-both-ways-58l7</guid>
      <description>&lt;p&gt;Most token guides say one token is about four characters, and a lot of code runs &lt;code&gt;Math.ceil(text.length / 4)&lt;/code&gt; right before an API call without anyone checking it.&lt;/p&gt;

&lt;p&gt;I build an LLM cost calculator, so this number ends up inside every estimate I make. This week I finally counted. I ran 951 text samples through OpenAI's o200k_base tokenizer (GPT-4o and newer) with tiktoken 0.14.0.&lt;/p&gt;

&lt;h2&gt;
  
  
  Characters per token
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Content&lt;/th&gt;
&lt;th&gt;Characters per token&lt;/th&gt;
&lt;th&gt;
&lt;code&gt;length / 4&lt;/code&gt; gives you&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;English blog writing&lt;/td&gt;
&lt;td&gt;4.98&lt;/td&gt;
&lt;td&gt;1.24x the real count&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;English technical docs&lt;/td&gt;
&lt;td&gt;4.38&lt;/td&gt;
&lt;td&gt;1.09x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Python source&lt;/td&gt;
&lt;td&gt;4.19&lt;/td&gt;
&lt;td&gt;1.05x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JSON&lt;/td&gt;
&lt;td&gt;2.11&lt;/td&gt;
&lt;td&gt;0.53x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CSV&lt;/td&gt;
&lt;td&gt;2.10&lt;/td&gt;
&lt;td&gt;0.52x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;English with emoji&lt;/td&gt;
&lt;td&gt;2.08&lt;/td&gt;
&lt;td&gt;0.52x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;API keys, UUIDs, hashes&lt;/td&gt;
&lt;td&gt;1.55&lt;/td&gt;
&lt;td&gt;0.39x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chinese&lt;/td&gt;
&lt;td&gt;1.37&lt;/td&gt;
&lt;td&gt;0.34x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For plain English the rule guesses too high, so it errs on the safe side. For JSON, CSV, IDs and emoji it guesses about half. If you're building agents or RAG, the second group is most of what goes into your prompts: tool results, API responses, exported tables. That's the error that shows up on the bill.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why JSON comes out so badly
&lt;/h2&gt;

&lt;p&gt;Tokenizers learn mostly from prose, so a common English word and the space before it are often a single token. Numbers get split up. &lt;code&gt;4821.37&lt;/code&gt; or &lt;code&gt;2026-07-14&lt;/code&gt; turns into several short pieces, every quote, colon, comma and brace counts, and JSON repeats every key on every record.&lt;/p&gt;

&lt;p&gt;Random strings are worse still. An API key or a SHA hash has nothing the tokenizer has seen before, so it comes out at about 1.5 characters per token.&lt;/p&gt;

&lt;p&gt;The averages hide a lot of spread, too. My JSON files went from 1.7 to 2.8 characters per token depending on how many numbers they had in them, and CSV from 1.6 to 3.9.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does to a budget
&lt;/h2&gt;

&lt;p&gt;Take an agent that reads 40,000 characters of JSON tool output per call, 10,000 calls a month.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;length / 4&lt;/code&gt; says 100 million input tokens a month.&lt;/li&gt;
&lt;li&gt;At 2.11 characters per token it's about 190 million.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At Claude Sonnet 5.5's input price ($2 per million tokens) you'd budget $200 and pay about $379, before any output tokens.&lt;/p&gt;

&lt;p&gt;Context limits go wrong the same way. A payload you think is 100K tokens is closer to 190K, so it doesn't fit in a 128K window anymore. Usually you find out from a truncation error, or from an agent that has quietly dropped the start of its instructions.&lt;/p&gt;

&lt;h2&gt;
  
  
  I built a better estimator, and it missed too
&lt;/h2&gt;

&lt;p&gt;The fix I tried first: stop using one divisor and count words, digits, punctuation and CJK characters separately, each at its own rate. I fitted the rates on the 951 samples and got a 26-line JavaScript function that was within about 2% on every type.&lt;/p&gt;

&lt;p&gt;That 2% was measured on the same samples I fitted it to, so it didn't prove much. To get a fair test I took 832 real files none of the estimators had seen, the first 4,000 characters of README, package.json and index.js files from npm packages, and ran mine, &lt;code&gt;length / 4&lt;/code&gt; and the &lt;a href="https://www.npmjs.com/package/tokenx" rel="noopener noreferrer"&gt;tokenx&lt;/a&gt; library on them.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Unseen files&lt;/th&gt;
&lt;th&gt;Files&lt;/th&gt;
&lt;th&gt;&lt;code&gt;length / 4&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;tokenx 2.2.0&lt;/th&gt;
&lt;th&gt;mine&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;README.md&lt;/td&gt;
&lt;td&gt;325&lt;/td&gt;
&lt;td&gt;0.94&lt;/td&gt;
&lt;td&gt;1.07&lt;/td&gt;
&lt;td&gt;1.02&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;package.json&lt;/td&gt;
&lt;td&gt;338&lt;/td&gt;
&lt;td&gt;0.71&lt;/td&gt;
&lt;td&gt;0.99&lt;/td&gt;
&lt;td&gt;0.89&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;index.js&lt;/td&gt;
&lt;td&gt;169&lt;/td&gt;
&lt;td&gt;0.89&lt;/td&gt;
&lt;td&gt;1.01&lt;/td&gt;
&lt;td&gt;1.02&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;1.00 is exact. 0.89 means the estimate counted 89% of the tokens you'd actually be billed for.&lt;/p&gt;

&lt;p&gt;Mine undercounted real package.json files by 11.5%. All my JSON training samples were tables of numbers. Real package.json files are full of version strings like &lt;code&gt;^7.21.0&lt;/code&gt;, &lt;code&gt;@scoped/package-names&lt;/code&gt; and two-space indentation, and my samples had none of that. tokenx, which did worse than mine on my own samples, was within 1.4% here.&lt;/p&gt;

&lt;p&gt;tokenx has a weak spot as well. On my samples it counted only 38% of the tokens in API keys and hashes and 43% in messages with a lot of emoji.&lt;/p&gt;

&lt;p&gt;So I don't think any estimator is safe in general. Each one is about as good as the text it was tested on, and most of us never test it on our own text.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd use
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On a server, or anywhere a wrong count breaks a request or a budget, run the real tokenizer. &lt;a href="https://www.npmjs.com/package/gpt-tokenizer" rel="noopener noreferrer"&gt;gpt-tokenizer&lt;/a&gt; and &lt;a href="https://www.npmjs.com/package/js-tiktoken" rel="noopener noreferrer"&gt;js-tiktoken&lt;/a&gt; both matched Python tiktoken exactly on all 951 samples. The usage numbers the provider returns work too.&lt;/li&gt;
&lt;li&gt;In the browser it's a size question. The exact tokenizers are about 1.1 MB gzipped because the vocabulary has 200,000 entries. tokenx is 1.2 kB and mine is 0.5 kB. For a live counter on English, code or Markdown, either is fine if you label the number as approximate.&lt;/li&gt;
&lt;li&gt;If the text has keys, hashes, UUIDs, base64 or emoji, I wouldn't rely on tokenx alone.&lt;/li&gt;
&lt;li&gt;For real-world JSON, leave at least 15% headroom whichever estimator you use.&lt;/li&gt;
&lt;li&gt;I'd stop using &lt;code&gt;length / 4&lt;/code&gt; for JSON, CSV or Chinese. It missed 29% to 65% of the tokens on each of them.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you want something you can do in your head: divide characters by 5 for plain English, 4.2 for code, 2 for JSON, CSV and logs, 1.5 for keys and hashes, and multiply Chinese or Japanese characters by 0.73.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits of this test
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;It only covers OpenAI's o200k_base. Claude, Gemini, Llama and DeepSeek use their own tokenizers, and the same text can come out at a different count. If the number matters, use the provider's token counting endpoint.&lt;/li&gt;
&lt;li&gt;My CSV and JSON samples are generated tables, and the unseen files all come from one project's npm dependencies, so they're mostly English and code. Business documents or chat logs could rank the estimators differently.&lt;/li&gt;
&lt;li&gt;The Chinese set is 11 paragraphs, and Japanese, Korean and Russian are four sentences each. Don't read too much into those rows.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it on your own text
&lt;/h2&gt;

&lt;p&gt;The samples, the counting script, the comparison script and the estimator are all in &lt;a href="https://github.com/aicostbudget/tokenizer-ratios" rel="noopener noreferrer"&gt;aicostbudget/tokenizer-ratios&lt;/a&gt; (code MIT, data CC BY 4.0).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/aicostbudget/tokenizer-ratios
&lt;span class="nb"&gt;cd &lt;/span&gt;tokenizer-ratios
npm &lt;span class="nb"&gt;install
&lt;/span&gt;npm run compare
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To check your own data, add a few lines to &lt;code&gt;data/samples.jsonl&lt;/code&gt; (one JSON object per line with a &lt;code&gt;category&lt;/code&gt; and a &lt;code&gt;text&lt;/code&gt;) and run it again. If you find a kind of text where all three estimators are badly off, please open an issue with a sample and I'll add it.&lt;/p&gt;

&lt;p&gt;The longer versions, with charts, are on my site: &lt;a href="https://aicostbudget.com/en/blog/how-many-characters-is-a-token?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=tokenizer-ratios&amp;amp;utm_content=devto-2026-11" rel="noopener noreferrer"&gt;how many characters a token really is&lt;/a&gt; and &lt;a href="https://aicostbudget.com/en/blog/token-estimator-accuracy?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=tokenizer-ratios&amp;amp;utm_content=devto-2026-11" rel="noopener noreferrer"&gt;which token estimator to trust&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>javascript</category>
      <category>llm</category>
      <category>opensource</category>
    </item>
    <item>
      <title>I paid 9x more for the “smarter” LLM. It got exactly the same answers right.</title>
      <dc:creator>AICostbudget.com</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:53:31 +0000</pubDate>
      <link>https://dev.to/linqiangmax/i-paid-9x-more-for-the-smarter-llm-it-got-exactly-the-same-answers-right-1hdk</link>
      <guid>https://dev.to/linqiangmax/i-paid-9x-more-for-the-smarter-llm-it-got-exactly-the-same-answers-right-1hdk</guid>
      <description>&lt;p&gt;Most of us pick an LLM by looking at a pricing table: dollars per million input tokens, dollars per million output tokens. Then we pick the model that sounds smart enough and hope the bill is fine.&lt;/p&gt;

&lt;p&gt;That number answers the wrong question. You don't pay for tokens. You pay for &lt;strong&gt;a task that came out right&lt;/strong&gt;, plus everything it took to get there:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the retries when the first answer failed your check,&lt;/li&gt;
&lt;li&gt;the reasoning tokens a "thinking" model writes and you never see,&lt;/li&gt;
&lt;li&gt;the answers that come back empty or malformed, which you pay for anyway.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So I built a small open benchmark that measures the number you actually pay:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cost per successful task = everything you paid for all attempts ÷ results that passed the check
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  How it works
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;16 tasks that look like production work&lt;/strong&gt;: invoice field extraction, receipt images, strict JSON for a schema, text-to-SQL over three SQLite databases, intent classification, small bug fixes, tool calling, Q&amp;amp;A over a ~30K-token handbook sent as a cacheable prefix, contact-record normalisation, and regex from examples. Ten are "standard" and six are a harder tier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every check is code, not an LLM judge.&lt;/strong&gt; Extracted fields are compared after normalisation, SQL is run and its result set compared, bug fixes run against hidden unit tests, regexes are tested on hidden strings.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Up to 3 attempts per item.&lt;/strong&gt; When a check fails, the validator's reason goes back to the model as the next message, the way a real app would retry.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Provider defaults.&lt;/strong&gt; No temperature, no reasoning-effort setting, no fallbacks, because that's what most people run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every input is generated from a fixed seed&lt;/strong&gt; by code in the repo (invented names, companies, databases), so none of it can be in a training set, and &lt;code&gt;generate.py --check&lt;/code&gt; proves the data matches byte for byte.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prices come from an open pricing dataset&lt;/strong&gt;, pinned to a commit, so every cost in the report can be traced to an official price page.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  First run: DeepSeek flash vs DeepSeek V4 Pro
&lt;/h2&gt;

&lt;p&gt;I ran both models on every text task, both tiers: 720 graded answers per model. The image task was skipped because neither model takes images.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Per 1M successful tasks (list price)&lt;/th&gt;
&lt;th&gt;deepseek-flash&lt;/th&gt;
&lt;th&gt;deepseek-v4-pro&lt;/th&gt;
&lt;th&gt;V4 Pro ÷ flash&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Standard tier (9 tasks)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$347&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$1,698&lt;/td&gt;
&lt;td&gt;4.9x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hard tier (6 tasks)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$596&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$5,478&lt;/td&gt;
&lt;td&gt;9.2x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Success within 3 attempts&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Both models eventually got &lt;strong&gt;every&lt;/strong&gt; item right. The flagship cost 4.9x as much per success on everyday tasks and 9.2x on the hard ones.&lt;/p&gt;

&lt;p&gt;Per task, the gap ranged from 2.3x (intent classification, where answers are a single label) to 14x (hard regex):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Hard-tier task&lt;/th&gt;
&lt;th&gt;flash $/success&lt;/th&gt;
&lt;th&gt;V4 Pro $/success&lt;/th&gt;
&lt;th&gt;ratio&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Invoice extraction (10 fields, VAT, deposits)&lt;/td&gt;
&lt;td&gt;$0.00079&lt;/td&gt;
&lt;td&gt;$0.00667&lt;/td&gt;
&lt;td&gt;8.4x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Strict JSON (nested schema, unit conversions)&lt;/td&gt;
&lt;td&gt;$0.00057&lt;/td&gt;
&lt;td&gt;$0.00281&lt;/td&gt;
&lt;td&gt;4.9x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Text-to-SQL (window functions, HAVING)&lt;/td&gt;
&lt;td&gt;$0.00053&lt;/td&gt;
&lt;td&gt;$0.00505&lt;/td&gt;
&lt;td&gt;9.5x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool calling (relative dates, decoys)&lt;/td&gt;
&lt;td&gt;$0.00024&lt;/td&gt;
&lt;td&gt;$0.00117&lt;/td&gt;
&lt;td&gt;4.9x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Long-document multi-hop QA&lt;/td&gt;
&lt;td&gt;$0.00027&lt;/td&gt;
&lt;td&gt;$0.00150&lt;/td&gt;
&lt;td&gt;5.6x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Regex (leap years, backreferences, lookaheads)&lt;/td&gt;
&lt;td&gt;$0.00129&lt;/td&gt;
&lt;td&gt;$0.01800&lt;/td&gt;
&lt;td&gt;14.0x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Where the money goes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Hidden reasoning.&lt;/strong&gt; V4 Pro's output price is 3.3x flash's. If that were the whole story, it would cost about 3–4x per task. It cost up to 9x, because it also writes far more output: 1.3x to 4.3x as many tokens per attempt on the hard tier, almost all of it reasoning. On the hardest regex tasks V4 Pro averaged 4,294 output tokens per attempt against flash's 1,003. The harder the task, the more it "thinks", and the wider the gap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Paying for nothing.&lt;/strong&gt; Three times, V4 Pro reasoned until it hit the 16,000-token output limit and returned an empty answer. Every one of those tokens was billed. The retry passed, and was billed again.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cheap retries.&lt;/strong&gt; Flash made three genuine mistakes in 720 answers: an empty reply on a bug fix, a phone-number field, and a regex that missed one case. Each was fixed by a single retry with the validator's feedback. A retry on flash costs less than a first attempt on V4 Pro.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Caching does the heavy lifting on long prompts.&lt;/strong&gt; The long-document task sends a ~30K-token handbook with every question. About 98% of that input was served from DeepSeek's cache, so it cost about the same per answer as the short-prompt tasks.&lt;/p&gt;

&lt;h2&gt;
  
  
  The benchmark had bugs too
&lt;/h2&gt;

&lt;p&gt;Running real models against a new benchmark finds problems in the benchmark. I fixed them before publishing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The first run surfaced ground-truth errors: an invoice layout where the "No." label could be read as part of the invoice number, and three product descriptions that fit two categories. I regenerated the data and re-ran.&lt;/li&gt;
&lt;li&gt;The tool-calling hard tier failed on &lt;em&gt;both&lt;/em&gt; models at first. The cause was the runner, not the models: it decided how to grade a reply by checking whether the task id was exactly &lt;code&gt;tool_calling&lt;/code&gt;, so the hard variant was graded as plain text. It now reads a &lt;code&gt;REPLY&lt;/code&gt; attribute on the task, and the self-test fails if a task that sends tools doesn't set it.&lt;/li&gt;
&lt;li&gt;Three questions turned out ambiguous: the SQL databases contain two different people with the same name, and the questions didn't say whether to list that name once or twice; and one tool-calling request asked about "this coming Thursday" on a Thursday. Those were reworded. The expected answers didn't change.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Of the nine first-attempt failures across both models, three were genuine model errors, three were the output-limit cut-offs, and three were these ambiguous questions.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does &lt;em&gt;not&lt;/em&gt; show
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;That small models are always enough.&lt;/strong&gt; This is one provider, synthetic tasks, and one run per model. On your workload the flagship may well earn its price.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;That these numbers transfer to other providers.&lt;/strong&gt; That's the next thing worth measuring, which is why the runner supports OpenAI, Anthropic and Gemini too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What you'll actually be billed.&lt;/strong&gt; Costs here are at DeepSeek's peak list price so the two models compare like for like. I ran off-peak, where DeepSeek bills half, and the console charged about 46% of the logged list cost. The whole experiment, smoke tests included, cost about $1.40.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The point isn't "always use the cheap model". It's that &lt;strong&gt;price per token tells you almost nothing about cost per result&lt;/strong&gt;, and measuring the real number is cheap.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run it on your own models
&lt;/h2&gt;

&lt;p&gt;Python 3.11+, bring your own keys. Nothing is sent anywhere except to the providers you choose.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/aicostbudget/llm-cost-per-task-benchmark
&lt;span class="nb"&gt;cd &lt;/span&gt;llm-cost-per-task-benchmark
pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; requirements.txt &lt;span class="nt"&gt;-r&lt;/span&gt; requirements-runner.txt

python runner.py &lt;span class="nt"&gt;--estimate&lt;/span&gt;                      &lt;span class="c"&gt;# what a run would cost, no API calls&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;ANTHROPIC_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;...                     &lt;span class="c"&gt;# only the providers you use&lt;/span&gt;
python runner.py &lt;span class="nt"&gt;--run&lt;/span&gt; &lt;span class="nt"&gt;--models&lt;/span&gt; claude-haiku-5-5 &lt;span class="nt"&gt;--limit&lt;/span&gt; 1 &lt;span class="nt"&gt;--budget&lt;/span&gt; 1   &lt;span class="c"&gt;# smoke test&lt;/span&gt;
python runner.py &lt;span class="nt"&gt;--run&lt;/span&gt; &lt;span class="nt"&gt;--tier&lt;/span&gt; hard &lt;span class="nt"&gt;--models&lt;/span&gt; claude-haiku-5-5 &lt;span class="nt"&gt;--budget&lt;/span&gt; 3 &lt;span class="nt"&gt;--run-id&lt;/span&gt; haiku-hard
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;--budget&lt;/code&gt; is a hard cap: the runner stops before any attempt that could push spend past it, and the same &lt;code&gt;--run-id&lt;/code&gt; resumes where it stopped. No key at all? The Gemini free tier works with &lt;code&gt;--pace&lt;/code&gt; to stay under the rate limit.&lt;/p&gt;

&lt;p&gt;Each run writes a report with the cost per success, the retry rate and the cached-input share. You can paste those into a &lt;a href="https://aicostbudget.com/en/model-switch-cost-calculator?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=cost-per-task-benchmark&amp;amp;utm_content=devto-2026-10" rel="noopener noreferrer"&gt;model switch cost calculator&lt;/a&gt; to see what a switch would save per month (I built that too).&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Repo: &lt;a href="https://github.com/aicostbudget/llm-cost-per-task-benchmark" rel="noopener noreferrer"&gt;https://github.com/aicostbudget/llm-cost-per-task-benchmark&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Full results and caveats: &lt;a href="https://github.com/aicostbudget/llm-cost-per-task-benchmark/blob/main/results/2026-10-29-deepseek.md" rel="noopener noreferrer"&gt;results/2026-10-29-deepseek.md&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you run it on GPT, Claude or Gemini models, I'd love to add your results. Post them in &lt;a href="https://github.com/aicostbudget/llm-cost-per-task-benchmark/issues/1" rel="noopener noreferrer"&gt;issue #1&lt;/a&gt;. And if you have a task where you're sure the flagship clearly wins, tell me in &lt;a href="https://github.com/aicostbudget/llm-cost-per-task-benchmark/issues/2" rel="noopener noreferrer"&gt;issue #2&lt;/a&gt; and I'll build it.&lt;/p&gt;

&lt;p&gt;Longer write-up with per-task tables and the reasoning-token breakdown: &lt;a href="https://aicostbudget.com/en/blog/deepseek-v4-pro-vs-flash-cost?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=cost-per-task-benchmark&amp;amp;utm_content=devto-2026-10" rel="noopener noreferrer"&gt;DeepSeek V4 Pro vs V4.1 Flash&lt;/a&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>opensource</category>
      <category>benchmark</category>
    </item>
  </channel>
</rss>
