<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Alex Chen</title>
    <description>The latest articles on DEV Community by Alex Chen (@qisuancloud).</description>
    <link>https://dev.to/qisuancloud</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4165462%2F86ddb891-b316-48b1-8fbc-09bb2f64cdcd.png</url>
      <title>DEV Community: Alex Chen</title>
      <link>https://dev.to/qisuancloud</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/qisuancloud"/>
    <language>en</language>
    <item>
      <title>A Blackwell Fingerprint in Open Weights: What It Proves, and What It Doesn't</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Tue, 17 Nov 2026 07:07:04 +0000</pubDate>
      <link>https://dev.to/qisuancloud/a-blackwell-fingerprint-in-open-weights-what-it-proves-and-what-it-doesnt-4h8j</link>
      <guid>https://dev.to/qisuancloud/a-blackwell-fingerprint-in-open-weights-what-it-proves-and-what-it-doesnt-4h8j</guid>
      <description>&lt;h1&gt;
  
  
  A Blackwell Fingerprint in Open Weights: What It Proves, and What It Doesn't
&lt;/h1&gt;

&lt;p&gt;This week the timeline stopped arguing about which model is cheapest and started arguing about evidence again. On November 12, Will DePue at Unemployment Labs published a forensic report claiming that six open-weight frontier models from Chinese labs carry the fingerprint of NVIDIA Blackwell GPUs — chips that need a U.S. export license for China, and for which no licenses have been reported. The report's own words are careful: the fingerprint "supports, without proving" that Blackwell also did the training.&lt;/p&gt;

&lt;p&gt;That careful sentence is the whole post. I have spent the last month writing about evidence standards for LLM routing — how many closed findings before you trust a number, what counts as closed, how a headline can move on one observation. This report is the same discipline, applied to the biggest question in AI hardware this year. So let us do what we do with routing claims: separate the measurement from the inference, count what the number is standing on, and say plainly what is still unknown.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a fingerprint is
&lt;/h2&gt;

&lt;p&gt;Every large model starts training from billions of pseudo-random initial values. In the PyTorch stacks most frontier labs use, those values come from Philox, a deterministic random-number generator, run through NVIDIA's cuRAND library. The work is split across the GPU's streaming multiprocessors (SMs): a sufficiently large initialization uses 2,048 threads per SM. Change the SM count and the same seed lands the same numbers in different positions.&lt;/p&gt;

&lt;p&gt;The three counts that matter here:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Hardware class&lt;/th&gt;
&lt;th&gt;SMs&lt;/th&gt;
&lt;th&gt;Threads&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hopper H100 SXM / H200 / H800&lt;/td&gt;
&lt;td&gt;132&lt;/td&gt;
&lt;td&gt;270,336&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Blackwell B200 / B300&lt;/td&gt;
&lt;td&gt;148&lt;/td&gt;
&lt;td&gt;303,104&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Blackwell GB200 / GB300&lt;/td&gt;
&lt;td&gt;152&lt;/td&gt;
&lt;td&gt;311,296&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Training changes most weights, but some survive untouched — rows tied to tokens the model rarely saw — and even trained weights keep a faint trace of their starting order. The report recovers the seed and the SM layout that reproduce those surviving values, then tests that combination on weights it held out of the search. A match on held-out weights is the difference between a finding and a coincidence.&lt;/p&gt;

&lt;p&gt;Two details make this more than a clever search. First, DeepSeek reused one seed, 953049583, for both V3 and V4-Flash — so the same seed reproduces Hopper layout in the old model and Blackwell layout in the new one. Changing the seed by one reproduces nothing on either machine. Second, the authors verified on physical hardware: unmodified PyTorch on a B300 reproduced 461 of 469 selected rows in V4-Flash, 445 of them bit for bit; the same code on an H100 reproduced all 57,344 preserved values in V3 bit for bit, and neither chip reproduced the other model's rows. A rented GB200 reproduced all 13 preserved rows of Kimi K3. And engineer Peter Liu has since published a CPU-only script that recovers the same SM peaks and seeds for V4-Flash, Kimi K3, and MiniMax-M3 — no Blackwell rental required to check the check.&lt;/p&gt;

&lt;h2&gt;
  
  
  The table, and what each number is standing on
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Lab&lt;/th&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Released&lt;/th&gt;
&lt;th&gt;Fingerprint&lt;/th&gt;
&lt;th&gt;Evidence&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek&lt;/td&gt;
&lt;td&gt;V4-Flash&lt;/td&gt;
&lt;td&gt;Apr 2026&lt;/td&gt;
&lt;td&gt;148 SM, B200/B300&lt;/td&gt;
&lt;td&gt;Exact copy, 1,385σ&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek&lt;/td&gt;
&lt;td&gt;V4-Pro&lt;/td&gt;
&lt;td&gt;Apr 2026&lt;/td&gt;
&lt;td&gt;152 SM, GB200/GB300&lt;/td&gt;
&lt;td&gt;Partial survival, 49σ&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Moonshot&lt;/td&gt;
&lt;td&gt;Kimi K3&lt;/td&gt;
&lt;td&gt;Jul 2026&lt;/td&gt;
&lt;td&gt;152 SM, GB200/GB300&lt;/td&gt;
&lt;td&gt;Exact copy, 305σ&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Moonshot&lt;/td&gt;
&lt;td&gt;Kimi K3.1-Pro&lt;/td&gt;
&lt;td&gt;Nov 2026&lt;/td&gt;
&lt;td&gt;152 SM, GB200/GB300&lt;/td&gt;
&lt;td&gt;Exact copy, 305σ, same seed as K3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MiniMax&lt;/td&gt;
&lt;td&gt;MiniMax-M3&lt;/td&gt;
&lt;td&gt;Jun 2026&lt;/td&gt;
&lt;td&gt;148 SM, B200/B300&lt;/td&gt;
&lt;td&gt;Exact copy, 323σ&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ant Group&lt;/td&gt;
&lt;td&gt;Ling-3.1-flash&lt;/td&gt;
&lt;td&gt;Oct 2026&lt;/td&gt;
&lt;td&gt;152 SM, GB200/GB300&lt;/td&gt;
&lt;td&gt;Untrained rows, 244σ&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every lab's earlier model still reads as Hopper: DeepSeek-V3 (Dec 2024), Kimi K2/K2.5 (through Jan 2026), MiniMax-M2.7 (Apr 2026), Ling-3.0-flash (Aug 2026). And not every lab flipped — the newest models checked from Alibaba (Qwen), Zhipu (GLM), Tencent, Xiaomi, and StepFun still read as Hopper. The switch is recent, it is visible inside each lab's own lineage, and it is not universal. That pattern is much harder to explain away than a single hit.&lt;/p&gt;

&lt;p&gt;The sigma numbers need the same treatment I give success rates: ask what they are standing on. The search covered roughly six trillion seed-and-layout combinations. In a search that large, the strongest accidental match would be expected to reach about 8 sigma. The strongest evidence for each of the six models ranges from 49 to 1,385 sigma, and even the weakest individual test reported reaches 20 sigma. That is the hold-out logic from my last post, applied at industrial scale: the seed and layout were chosen on one set of weights and tested on another that played no part in choosing them.&lt;/p&gt;

&lt;p&gt;The method also gets known answers right, which is the calibration check that matters most. In 24 cases where a developer publicly named the training hardware, the method agrees every time: Meta's Llama 3.1 on H100s reads as 132 SMs, EleutherAI's Pythia on A100s reads as 108, NVIDIA's own Nemotron-3 Ultra — whose technical report names the GB200 for one phase — reads as 152, the same as Kimi K3. A method that reproduces 24 known answers earns the right to be heard on the unknown ones.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it proves
&lt;/h2&gt;

&lt;p&gt;It proves, for DeepSeek-V4-Flash conclusively in the authors' words, that Blackwell-class hardware generated the starting weights. The evidence is strongest exactly where it should be: an exact copy, reproduced on a physical B300, with the seed reused from the previous generation, at 1,385 sigma against an 8-sigma chance ceiling.&lt;/p&gt;

&lt;p&gt;It also proves the AMD alternative does not explain V4-Flash. An MI300X can be split into two groups of 152 processing units, or limited to 148, and its random-number software can generate the same underlying sequences — so the layout alone does not settle the question. The authors tested it on a physical MI300X: in the 377 values where AMD's and NVIDIA's arithmetic differ, the published weight matches NVIDIA in 363 and AMD in none. For the other five models the same test could not settle it, and the authors call the AMD explanation "very unlikely" rather than excluded — the distinction is stated, not smoothed over. Note the irony for anyone who runs AMD hardware: the chip that could theoretically fake the fingerprint is the one we measured DeepSeek-V4.1-Flash on ourselves, at $0.016 per million tokens on a real MI300X (SemiAnalysis InferenceX, November 6 run, vLLM, FP4).&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does not prove
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;It does not prove Blackwell did the training.&lt;/strong&gt; The fingerprint is set at step zero. A lab could initialize weights on one system and train on another, or continue training an existing model on new hardware. Kimi K2.7-Code, posted six weeks before K3, is the live example: mostly Hopper, with a small part matching the 148-SM layout. Mixed clusters blur the read the same way. The report's sentence is the honest one: the fingerprint "supports, without proving" that Blackwell also performed the training.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It does not prove where the chips were, who owned them, or that any law was broken.&lt;/strong&gt; Blackwell systems can be rented from data centers outside China. The weights cannot reveal geography or ownership, and the report states plainly that its findings "do not establish where the chips were located, how access was obtained, or whether export controls were violated."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It does not prove anything about DeepSeek-V4.1-Flash — and that is the most instructive part.&lt;/strong&gt; The authors searched exhaustively and found no fingerprint in V4.1-Flash. That is a null result, not a clean bill of health. The method needs a recognizable, hardware-dependent pattern in the starting weights. Weights initialized on a CPU leave no GPU trace — several Alibaba Qwen3 models were initialized that way. Baidu's PaddlePaddle framework leaves none either; the authors recovered ERNIE-5.5's seed (2760) but could not identify its hardware. V4.1-Flash's hardware is an open question until DeepSeek documents its seeding stack or the authors publish Ascend generator tests. Any claim that the report "confirmed" Ascend training for DeepSeek is unverified — the report's only Ascend reference is a deleted fine-tuning configuration in Meituan's LongCat-2.0, read as Huawei Ascend 910.&lt;/p&gt;

&lt;p&gt;In the language of my last post, V4.1-Flash sits in the third bucket: timed out, still unclosed. An unclosed finding is not a success and not a failure. It is an unknown, and it must be displayed as unknown — never folded into a column to make a story look complete.&lt;/p&gt;

&lt;h2&gt;
  
  
  The scale math — and the revision that makes it credible
&lt;/h2&gt;

&lt;p&gt;From model sizes and token counts, the report estimates the final training runs at roughly 1,500 Blackwell GPUs for V4-Flash (50 days), 3,000 for V4-Pro (70 days), and 4,500 for Kimi K3 (70 days) — about 12,000 GPUs across the five runs, with the largest cluster costing roughly $200–300 million to buy. The ranges are wide: K3's is 1,000 to 9,500 GPUs. The authors say so themselves, and list the assumptions the weights cannot confirm: utilization, token counts, run lengths, and that training stayed on the initialization hardware.&lt;/p&gt;

&lt;p&gt;Here is the detail that actually increased my trust: the authors revised these numbers on the same day. The first version put 2,000–3,000 GPUs on the largest runs and $150–200 million on the largest cluster; the revision raised both after the authors lowered their assumed efficiency, and the revision is dated on the page — "Revised November 12, 2026" — with the old figures still visible. That is the backfill discipline I argued for two weeks ago: backfill corrects the record; it does not rewrite the published numbers. A dated revision with the old numbers preserved is a correction. A silent edit is a rewrite. The report does the former, in public, on day one.&lt;/p&gt;

&lt;p&gt;What would renting that run cost at today's prices? V4-Flash's final run — 1,500 GPUs for 50 days — is 1.8 million GPU-hours. At Lambda's H100 SXM rate, which rose to $4.59 per GPU-hour on November 16 (from $3.99), the same hours on B200-class hardware at $7.69 run about &lt;strong&gt;$13.8 million&lt;/strong&gt;; the report's range for the run, 250 to 5,000 GPUs, maps to roughly $2.3–46 million at that rate. Renting the run is an order of magnitude cheaper than owning the cluster the report prices at $200–300 million — which is exactly why "access to hardware" and "ownership of hardware" are different claims, and why the report refuses to collapse them. (For scale: the cheapest H100 hour we track is still RunPod's Community Cloud at $2.69, re-checked November 13. Rental prices are moving weekly; check the day you decide, not the day you read this.)&lt;/p&gt;

&lt;h2&gt;
  
  
  The same rule, applied to ourselves
&lt;/h2&gt;

&lt;p&gt;This post is written on closed findings, in the sense my series has been building toward. The report is public, reproducible, and dated; the per-model numbers are in its appendix; the limits are in its own limitations section; the revision history is on the page. As of today, no lab, NVIDIA, or the Commerce Department has responded to the report specifically — that silence is a fact about the record, not evidence for either side, and the next forensic check is the next DeepSeek release.&lt;/p&gt;

&lt;p&gt;Our own funnel gets the same treatment. Google Search Console, 28 days to November 16: &lt;strong&gt;0 clicks, 47 impressions&lt;/strong&gt;, and our GPU cost calculator still records zero impressions — it does not appear in the page table at all. That zero is a fact about the window, not a verdict on the page. The discussion that produced this series is the same discipline applied to our own writing: the question that started it arrived when the thread under my November 6 post had 9 comments; by today that article has 66, and across our 10 published articles we have 95 comments and 6 reactions. This morning a reader, Tom Jones, pushed us one step further — every threshold should carry its source, its reading index, and its expiry, so a guess never hardens into a constant — and we answered it the same day. A post written on 9 comments would have been a median of three.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bottom line
&lt;/h2&gt;

&lt;p&gt;Do not ask "did they train on Blackwell?" Ask three smaller questions, in order: what hardware generated the starting weights, how strong is the evidence for that, and what would change your mind? For V4-Flash the answers are: Blackwell, 1,385 sigma on held-out weights reproduced on a physical B300, and a successful independent reproduction — which already exists, on a laptop, from Peter Liu's CPU-only script. For the training claim itself, the answer is still: strongly suggested, not proven. And for V4.1-Flash, the answer is: unknown, and anyone who tells you otherwise is selling you a conclusion the evidence does not support.&lt;/p&gt;

&lt;p&gt;That is the standard I hold our routing numbers to — a 30-closed-finding floor before a routing change, a flip count instead of a percentage, an attribution window that has to close before a number counts. It is the same standard the best forensic work in AI hardware is holding itself to this week. The fingerprint is real. The rest is still evidence, not verdict.&lt;/p&gt;

&lt;p&gt;If you want to price the trade-off yourself — what a routing change costs at today's GPU rates before you make it — our free GPU cost calculator runs the same arithmetic: &lt;a href="https://qisuanai.com/gpu-cost-calculator/" rel="noopener noreferrer"&gt;https://qisuanai.com/gpu-cost-calculator/&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Sources, all checked November 17, 2026: Will DePue, "Top Chinese Frontier AI Models Carry the Fingerprint of Export-Controlled NVIDIA Blackwell GPUs," Unemployment Labs, November 12, 2026, &lt;a href="https://unemploymentlabs.com/checkpoint-forensics" rel="noopener noreferrer"&gt;https://unemploymentlabs.com/checkpoint-forensics&lt;/a&gt; (report page, including the November 12 revision note; Table 1, Table 2, Appendix 1, and the limitations section). Explainx.ai write-up with the November 13 updates (TechCrunch recap, Peter Liu's CPU-only reproduction, V4.1-Flash null result), &lt;a href="https://explainx.ai/blog/chinese-open-weights-blackwell-gpu-checkpoint-forensics-november-2026" rel="noopener noreferrer"&gt;https://explainx.ai/blog/chinese-open-weights-blackwell-gpu-checkpoint-forensics-november-2026&lt;/a&gt;. No statement from DeepSeek, Moonshot, MiniMax, Ant Group, NVIDIA, or the Commerce Department in response to the report had been found as of today. Prices: Lambda H100 SXM $4.59 (8x) / $4.89 (1x) and B200 $7.69, effective November 16, 2026, verified from Lambda's pricing page this morning; RunPod Community Cloud H100 $2.69, re-checked November 13. Our own measurement: DeepSeek-V4.1-Flash on MI300X, $0.016 per million tokens (hyperscale own price; $0.022 retail rental), 16,368 tokens/sec per card at 50 tokens/sec per user, SemiAnalysis InferenceX run of November 6, 2026, vLLM FP4 on real MI300X — see our November 10 video. DeepSeek API prices per DeepSeek's Models &amp;amp; Pricing page as reported November 9: deepseek-flash $0.30 input / $1.20 output per million tokens at peak, half off-peak; cache hits $0.006 / $0.003. The rental arithmetic (1,500 GPUs × 50 days × 24 hours × $7.69) is mine, on the report's point estimate — the report's own range is 250–5,000 GPUs, and GB200-class rental is not the B200 rate, so treat it as an order-of-magnitude comparison, not a quote. GSC numbers are our own Search Console pull for sc-domain:qisuanai.com, 28 days October 20–November 16, 2026. The 30-finding floor, 10-finding watchlist threshold, and 7-day attribution window are our operating rules, stated as such; the 8-week replay that would calibrate them has not run yet. For the fuller cost picture, our GPU cost guide: &lt;a href="https://qisuanai.com/blog/gpu-cost-guide-2026/" rel="noopener noreferrer"&gt;https://qisuanai.com/blog/gpu-cost-guide-2026/&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>deeplearning</category>
      <category>ai</category>
      <category>llm</category>
      <category>gpu</category>
    </item>
    <item>
      <title>How Many Closed Findings Before You Trust a Routing Change</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Fri, 13 Nov 2026 07:10:04 +0000</pubDate>
      <link>https://dev.to/qisuancloud/how-many-closed-findings-before-you-trust-a-routing-change-1n4e</link>
      <guid>https://dev.to/qisuancloud/how-many-closed-findings-before-you-trust-a-routing-change-1n4e</guid>
      <description>&lt;h1&gt;
  
  
  How Many Closed Findings Before You Trust a Routing Change
&lt;/h1&gt;

&lt;p&gt;This week X spent its energy arguing about whether a headline number is real. A vLLM post claims DeepSeek V4.1 Flash runs 58× the throughput of GB300 NVL72 on Vera Rubin hardware. The replies immediately asked how it was measured, and at what concurrency. Another post pointed out that DeepSeek's cheapest price is a caching artifact: input the model has already seen costs $0.003 per million tokens off-peak instead of $0.15 — fifty times less — so a "cheap" agent run may just be a well-cached one. A third thread argues over forensic evidence about what hardware trained the newest models at all.&lt;/p&gt;

&lt;p&gt;Nobody on that timeline is arguing about the models. They are arguing about the evidence. That is the same question, one level down, that decides whether your routing change is real: your dashboard says a task type got better, or that the small model can take more work. How many closed findings do you need before you believe it — and act on it?&lt;/p&gt;

&lt;p&gt;A reader asked the version of this question that stuck with me all week. Under my November 6 post, Agentic Affairs asked: before each category of conclusions accumulates 20 entries, does the median mean anything at all? We spent the week stress-testing that question in the comments. This post is the answer we arrived at — and the rule we now run.&lt;/p&gt;

&lt;h2&gt;
  
  
  The median of three is a rumor
&lt;/h2&gt;

&lt;p&gt;Take a task type with three closed findings, measured as cost per completed task: $0.01, $0.02, $0.90. The median is $0.02. It looks cheap. One more finding arrives at $0.85 — a second expensive case, the kind that shows up once the small model meets the edge of its competence. The four values are $0.01, $0.02, $0.85, $0.90, and the median is now $0.44. One observation moved it more than twentyfold.&lt;/p&gt;

&lt;p&gt;That is what "before 20 entries" means in practice. With a handful of findings, the median is not a summary of the category. It is one observation wearing a summary's clothes. It will move again with the next finding, and the direction of the move tells you nothing about the category — it tells you about the finding that just arrived.&lt;/p&gt;

&lt;p&gt;The median is not special here. Any statistic computed on a handful of observations has the same disease: the next observation can rewrite it. The median is just the statistic people reach for when they want to sound robust, which makes its early instability more dangerous, not less. A mean that jumps is obviously unstable. A median that jumps looks like a contradiction — and teams resolve contradictions by picking the version they already believed.&lt;/p&gt;

&lt;h2&gt;
  
  
  One flip moves the number more than you think
&lt;/h2&gt;

&lt;p&gt;The same arithmetic governs success rates, which is where routing decisions actually get made. One closed finding flipping from success to failure moves the rate by 1/n, where n is the number of closed findings:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;5 closed findings: one flip moves the rate &lt;strong&gt;20 points&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;10 closed findings: one flip moves it &lt;strong&gt;10 points&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;20 closed findings: one flip moves it &lt;strong&gt;5 points&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;30 closed findings: one flip moves it &lt;strong&gt;3.3 points&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;100 closed findings: one flip moves it &lt;strong&gt;1 point&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read that table as a decision rule. If your bar for promoting a task type to the cheaper tier is a success rate above 95%, and you have 5 closed findings, a single late failure takes you from 100% to 80% — twenty points below the bar, on one observation. You did not learn the task type is bad. You learned you never had a measurement.&lt;/p&gt;

&lt;p&gt;Our rule, after a week of arguing about it in the comments, is a &lt;strong&gt;30-closed-finding floor&lt;/strong&gt; before any routing change: no task type moves up or down a tier until 30 of its findings are closed. I will be explicit that 30 is our operating rule, not a universal constant — it is the point where one flip stops being able to move the rate by more than a rounding error, and where the median stops jumping by multiples. Your floor will differ with your volume and your cost of being wrong. What should not differ is having a floor, written down, before the decision arrives. A floor chosen in the middle of a debate is not a floor. It is a preference.&lt;/p&gt;

&lt;h2&gt;
  
  
  Count flips, not percentages
&lt;/h2&gt;

&lt;p&gt;There is a second trap that a floor alone does not fix. When a task type has a near-zero failure baseline, one late failure doubles the failure rate. One failure in a hundred closed findings is a 1% failure rate; a second is 2%. The percentage doubled. The evidence grew by one finding.&lt;/p&gt;

&lt;p&gt;This is the flip-count rule we argued out in the thread: at low baselines, ask how many flips moved, not what percentage moved. The percentage is the headline; the flip count is the evidence. A routing change justified by "the failure rate doubled" that rests on a single additional failure is a decision made on one observation — the same mistake as the median of three, wearing a more alarming costume.&lt;/p&gt;

&lt;p&gt;The same logic applies in reverse. A task type that goes from 2 failures in 100 to 1 in 100 has "halved its failure rate" on the strength of one finding not failing. Do not promote on that either.&lt;/p&gt;

&lt;h2&gt;
  
  
  What counts as "closed" — and what never does
&lt;/h2&gt;

&lt;p&gt;A floor of 30 means nothing if the 30 are not actually closed. A finding is closed when its real outcome is known — not when the model answered, not when a cheap check passed. That is the attribution-window argument from my last post, and it has a practical consequence here: the count that matters is closed findings, not total findings.&lt;/p&gt;

&lt;p&gt;We split every interval into three buckets, a distinction a reader (Tom Jones) pushed us to make explicit:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Verified clean&lt;/strong&gt; — the outcome is confirmed good.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verified failed&lt;/strong&gt; — the outcome is confirmed bad, with a reason.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Timed out, still unclosed&lt;/strong&gt; — the window passed with no outcome signal.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The third bucket is the one that corrupts the count. An unclosed finding is not a success. It is an unknown, and it must be displayed as unknown — never folded into the clean column to make the numbers look better. A category with 25 verified-clean findings and 5 unclosed ones has 25 closed findings, not 30. It has not met the floor.&lt;/p&gt;

&lt;p&gt;Backfill — findings whose outcome arrives late — gets the same discipline. A reader (Emily Woods) put it precisely in a comment this morning: backfill corrects the record; it does not rewrite the published numbers. Keep both columns — the state at the original deadline, and the final result — and judge a late backfilled success against the original deadline's timestamp, not against today's. And if the same task type keeps producing late backfills, that is itself a signal: the deadline is wrong, and it should be recalibrated openly and recorded as a decision — not quietly absorbed by letting each backfill rewrite history.&lt;/p&gt;

&lt;h2&gt;
  
  
  The quiet types never graduate — and that is fine
&lt;/h2&gt;

&lt;p&gt;Every pipeline has task types that almost never run. A rare document shape, a seasonal request, a customer-specific format. They will never accumulate 30 closed findings, and most will never reach 10. The question is what happens to them, because the wrong answer is to either promote them on thin evidence or let them clog the review queue forever.&lt;/p&gt;

&lt;p&gt;Our answer, worked out in the thread: a &lt;strong&gt;10-finding threshold&lt;/strong&gt; gates graduation out of the watchlist, and the watchlist itself has a &lt;strong&gt;25-item hard cap&lt;/strong&gt; with a quarterly manual review. A quiet type sits on the watchlist until it either accumulates 10 closed findings or the quarterly review retires it. It never gets promoted on 3 findings because someone is impatient, and it never blocks a decision about the types that do have evidence. The cap matters: without it, the watchlist grows until nobody reviews it, and the review stops being a review.&lt;/p&gt;

&lt;p&gt;The honest caveat, which I gave in the thread and repeat here: the 10-finding threshold is a starting guess, and the quarterly review is a process, not a measurement. We have not run the 8-week replay that would calibrate these numbers — re-scoring the same tasks at 3, 7, and 14 days to see where the numbers stop moving. Until that replay runs, treat every number in this post that describes our own thresholds as a documented guess with a calibration plan, not a measured constant. That is the difference between a rule and a superstition: the rule says how it will be checked.&lt;/p&gt;

&lt;h2&gt;
  
  
  Slow drift beats the two-window rule
&lt;/h2&gt;

&lt;p&gt;The two-window rule from my first post in this series — require the success rate to stay below the bar for two consecutive windows before moving a task type — catches steps. It does not catch slopes. A task type whose failure rate creeps from 1% to 2% to 3% across six weeks never trips a two-window alarm, because no single window looks bad enough. By the time it does, the redos have been quietly compounding for a month and a half.&lt;/p&gt;

&lt;p&gt;That was another reader question from the week — the slow-drift alert — and the answer is that drift needs its own detector, separate from the window rule: a trend line on the closed-finding rate per task type, with an alert when the slope, not the level, crosses a bar. We do not have a good slope bar yet, and I will not invent one here. The point is structural: a rule that only looks at levels will miss every failure that arrives gradually, and most real failures arrive gradually.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a thin-evidence decision costs
&lt;/h2&gt;

&lt;p&gt;Keep the same pipeline from the last two posts, so the numbers stay comparable: a day's work needs 10 GPU-hours — 8 hours of chores on an RTX 4090 at $0.89 per GPU-hour ($7.12) and 2 hours of final answers on an H100 at $2.69 ($5.38). Total: &lt;strong&gt;$12.50 per day&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Now promote a task type to the cheap tier on 5 closed findings, all successful — a 100% provisional rate. The window closes and the 30-finding truth is 27 successes in 30: a 90% final rate. Out of 1,000 tasks a day, 100 now need a large-model redo. Assume a redo takes about 2 minutes of H100 time — your number will differ, measure yours: 100 × 2/60 hours × $2.69 = &lt;strong&gt;$8.97 a day&lt;/strong&gt; in redo cost, on top of the $12.50 day. A decision made on 5 findings added 72% to the daily bill.&lt;/p&gt;

&lt;p&gt;And the redo price is moving against you this week. RunPod's own pricing page now lists the H100 SXM at $4.39 per GPU-hour on Secure Cloud, up from $3.99 — the Community Cloud rate is unchanged at $2.69. Lambda's published rate rises to $4.59 on Monday, November 16, from $3.99. At the new Secure Cloud rate, those same 100 redos cost 100 × 2/60 × $4.39 = &lt;strong&gt;$14.63 a day&lt;/strong&gt; — more than the entire original day's work. The cost of deciding on thin evidence is not fixed. It rises with the price of the model you redo on, and that price rose this week.&lt;/p&gt;

&lt;p&gt;That is the whole argument in one number: the evidence threshold is not bureaucracy. It is the cheapest insurance you can buy against a redo bill that now costs more than the work itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  We measure our own funnel the same way
&lt;/h2&gt;

&lt;p&gt;This is not theory we exempt ourselves from — and this post is itself the example. Our site recorded 5 clicks and 190 impressions in Google Search Console over the latest 28-day window (October 16 to November 12, 2026), and our GPU cost calculator page recorded zero impressions again — it does not appear in the Search Console page table at all. That zero is a fact about the window, not a verdict on the page.&lt;/p&gt;

&lt;p&gt;The discussion that produced this post is the same discipline applied to our own writing. The median question arrived when the thread under my November 6 post had 9 comments. We did not write this post then. We waited while readers stress-tested it — the flip-count rule, the quiet-types threshold, the watchlist cap, the backfill discipline, the slow-drift alert — and by today that article has 66 comments, up from 27 on the morning of November 9 and 42 on November 10. Across our 9 published articles we now have 90 comments and 5 reactions. This post is written on closed findings: questions that were asked, answered, and left standing after a week of replies. A post written on 9 comments would have been a median of three.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bottom line
&lt;/h2&gt;

&lt;p&gt;Do not ask "what is our success rate?" Ask "how many closed findings is that rate standing on, and how far would one more finding move it?" If the answer is fewer than your floor, you do not have a measurement. You have a rumor with a confidence interval wider than the decision — and this week, on X and everywhere else, rumors about numbers are exactly what everyone is arguing about.&lt;/p&gt;

&lt;p&gt;If you want to skip the spreadsheet, our free GPU cost calculator prices the same trade-off — workload, GPU tier, and hourly rates — so you can see what a routing change costs before you make it: &lt;a href="https://qisuanai.com/gpu-cost-calculator/" rel="noopener noreferrer"&gt;https://qisuanai.com/gpu-cost-calculator/&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Prices: RunPod and Lambda public pricing pages, re-checked November 13, 2026 — RunPod Secure Cloud RTX 4090 $0.89, H100 SXM $4.39 (was $3.99), Community Cloud H100 SXM $2.69; Lambda H100 $4.59 and A100 (80 GB) $3.19 effective November 16, 2026, current rate behind login. DeepSeek API prices per DeepSeek's Models &amp;amp; Pricing page as reported November 9, 2026: deepseek-flash $0.30 input / $1.20 output per million tokens at peak, half off-peak; cache hits $0.006 / $0.003. X posts cited as discussion, not facts: vLLM's 58× throughput claim and the replies questioning its measurement conditions; the cache-hit price question; the hardware-forensics thread. The worked example is arithmetic on GPU-hours, not a production benchmark — your redo times and true rates will differ, and they are the numbers that decide your actual bill. The 30-finding floor, 10-finding threshold, and 25-item watchlist cap are our operating rules, stated as such; the 8-week replay that would calibrate them has not run yet, and the 7-day attribution window is a starting guess. Thread data: discussion under "Don't Scale Your LLM Routing on Request Volume" (November 6, 2026) and "Your Routing Dashboard Lies Until the Attribution Window Closes" (November 10, 2026), including Agentic Affairs' median question, the flip-count rule, the quiet-types and watchlist-cap discussion, Tom Jones' three-bucket split, and Emily Woods' backfill discipline.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>gpu</category>
      <category>inference</category>
    </item>
    <item>
      <title>Your Routing Dashboard Lies Until the Attribution Window Closes</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Tue, 10 Nov 2026 07:05:33 +0000</pubDate>
      <link>https://dev.to/qisuancloud/your-routing-dashboard-lies-until-the-attribution-window-closes-17df</link>
      <guid>https://dev.to/qisuancloud/your-routing-dashboard-lies-until-the-attribution-window-closes-17df</guid>
      <description>&lt;p&gt;Last week I argued you should scale your LLM routing on task success rate, not request volume. A reader, Alex Shev, asked the question that breaks that rule — politely, and correctly: what counts as "success" when the truth arrives late?&lt;/p&gt;

&lt;p&gt;Plenty of tasks only reveal their real outcome days after the model answers. A human downstream corrects the extracted fields. A customer reopens the ticket. A refund lands. An invoice the model "completed" gets rejected by the finance system on Thursday. If your dashboard scores the task the moment the model answers, it is grading a provisional outcome and calling it final. Until the attribution window closes, every number on that dashboard is a draft.&lt;/p&gt;

&lt;h2&gt;
  
  
  The outcome is not the answer
&lt;/h2&gt;

&lt;p&gt;A routing dashboard usually records one of two things and calls it success: the model returned something, or a cheap check passed. Both happen at answer time. Neither is the outcome.&lt;/p&gt;

&lt;p&gt;Real outcomes arrive through channels your dashboard may never see:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Downstream human correction.&lt;/strong&gt; An operator fixes the fields the model extracted. The task looked complete at 10:04. It was actually wrong, and the correction is logged in a different system, or not logged at all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Customer action.&lt;/strong&gt; The user reopens the ticket, rephrases the question, or churns. Silence after an answer is not confirmation — it is the absence of a signal, and it gets scored as success by default.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Late system rejection.&lt;/strong&gt; The draft summary passes your format check, then fails the compliance review two days later. The failure belongs to the task, but your dashboard already closed the book on it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is exotic. It is the normal shape of real work. And it means the success rate you alert on is systematically optimistic — not because anyone is cooking the numbers, but because the failures haven't arrived yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  The math: what a 4-point lie costs
&lt;/h2&gt;

&lt;p&gt;Take the same pipeline from my last post, so the numbers stay comparable. A day's work needs 10 GPU-hours: 8 hours of chores on an RTX 4090 at $0.89 per GPU-hour ($7.12) and 2 hours of final answers on an H100 at $2.69 ($5.38). Total: &lt;strong&gt;$12.50 per day&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Say 1,000 tasks run through it in a day. Scored at completion time, 970 succeed:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Provisional cost per completed task: $12.50 / 970 = &lt;strong&gt;$0.0129&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now let the 7-day attribution window close. Forty of those 970 "successes" come back — a corrected field here, a reopened ticket there. Final completions: 930.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Final cost per completed task, before redos: $12.50 / 930 = &lt;strong&gt;$0.0134&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And the 40 late failures each need a large-model redo. Assume a redo takes about 2 minutes of H100 time — your number will differ, measure yours: 40 × 2/60 hours × $2.69 = &lt;strong&gt;$3.59&lt;/strong&gt; of redo cost. True daily cost: $16.09.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;True cost per completed task: $16.09 / 930 = &lt;strong&gt;$0.0173&lt;/strong&gt; — &lt;strong&gt;34% above&lt;/strong&gt; the number the dashboard showed on day one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Four points of success rate, hidden by timing, turned into a third more cost per completed task. And the damage compounds: if you promoted a task type to a cheaper tier — or demoted one to a bigger model — on the provisional 97%, you made a routing decision on a number that was never true. The dashboard didn't lie on purpose. It just reported before the truth arrived.&lt;/p&gt;

&lt;h2&gt;
  
  
  The gap is itself a signal
&lt;/h2&gt;

&lt;p&gt;Run two numbers in parallel and watch the gap between them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Provisional success rate&lt;/strong&gt; — scored at answer time. Fast, useful for alerting, never a basis for a routing change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Final success rate&lt;/strong&gt; — scored after the attribution window closes. Slow, and the only number a routing decision should rest on.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A narrow gap means your fast checks are honest. A wide gap means your routing is fast but fragile: it looks good on the day and quietly bleeds redos all week. That gap is the most useful number on the dashboard, and most dashboards don't show it at all.&lt;/p&gt;

&lt;p&gt;Two guardrails from the discussion under my last post, both earned the hard way:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A window that hasn't closed is not a window.&lt;/strong&gt; The two-window rule — require the success rate to stay below the bar for two consecutive windows before moving a task type up a tier — only works if both windows are closed. A window still inside its attribution period is a partial count. Count it and you are back to deciding on drafts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't react to single flips.&lt;/strong&gt; When a task type has a near-zero failure baseline, one late failure doubles the failure rate. That is noise wearing a costume. We run a 30-closed-findings floor before trusting a routing change, and we ask how many flips — not just what percentage — moved before acting. The percentage is the headline; the flip count is the evidence.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How long is the window?
&lt;/h2&gt;

&lt;p&gt;There is no universal answer, and anyone selling you one is guessing. Ours is 7 days per task type — and I will be explicit that this is a starting guess, not a measured constant. The way to calibrate it is a replay: re-score the same tasks at 3, 7, and 14 days and see where the numbers stop moving. If the 7-day and 14-day scores agree, 7 days is long enough for that task type. If they don't, lengthen it. The window is a property of the task type — an invoice extraction that gets corrected same-day needs a shorter window than a customer ticket that reopens a week later. Fix it per task type, write it down, and don't let the dashboard quietly use "until now" as the window.&lt;/p&gt;

&lt;p&gt;And timestamp every success event. "Succeeded" without a time is unscoreable: you cannot tell whether it succeeded before or after the window closed, and you cannot replay it. A success event with no timestamp is a rumor.&lt;/p&gt;

&lt;h2&gt;
  
  
  We measure our own funnel the same way
&lt;/h2&gt;

&lt;p&gt;This is not theory we exempt ourselves from. Our own numbers have the same delayed-truth shape:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Google Search Console reports with a lag of a couple of days, so the dashboard we read this morning is already a draft. In the latest 28-day window (October 13 to November 9, 2026), our whole site recorded 6 clicks and 248 impressions — and our GPU cost calculator page recorded zero impressions, meaning it still isn't ranking at all. That zero is a fact about the window, not a verdict on the page; the window hasn't finished telling us yet.&lt;/li&gt;
&lt;li&gt;The last post itself kept changing after it was "done." Its comment count went from 27 on the morning of November 9 to 42 by the afternoon of November 10 — a 55% increase in engagement, days after publishing, driven by readers arriving late with real questions. The provisional engagement number understated the final one by more than half. If we had judged the post on day one, we would have judged it wrong.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Measure first, spend second — and measure again after the window closes.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bottom line
&lt;/h2&gt;

&lt;p&gt;Do not ask "what is our success rate?" Ask "what is our success rate &lt;em&gt;after the attribution window closes&lt;/em&gt;, and how far did it move from the provisional number?" Volume tells you when to buy more of what works. Success rate tells you when what you have stopped working. The attribution window tells you whether either number was real yet.&lt;/p&gt;

&lt;p&gt;If you want to skip the spreadsheet, our free GPU cost calculator prices the same trade-off — workload, GPU tier, and hourly rates — so you can see what a routing change costs before you make it: &lt;a href="https://qisuanai.com/gpu-cost-calculator/" rel="noopener noreferrer"&gt;https://qisuanai.com/gpu-cost-calculator/&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Prices: RunPod Community Cloud list — RTX 4090 $0.34, H100 SXM $2.69 per GPU-hour; RTX 4090 Secure Cloud $0.89 (the rate used in the worked example, carried over from the previous post for comparability). Lambda H100 $3.99 now, $4.59 from November 16, 2026. All re-checked November 10, 2026; community-cloud availability varies by the hour. The worked example is arithmetic on GPU-hours, not a production benchmark — your redo times and correction rates will differ, and they are the numbers that decide your actual bill. The 7-day window is our starting guess, stated as such; calibrate yours by replay. Thread data: discussion under "Don't Scale Your LLM Routing on Request Volume" (November 6, 2026), including Alex Shev's delayed-outcome question and the 30-findings floor, two-window rule, and flip-count discussion with Agentic Affairs.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>gpu</category>
      <category>routing</category>
    </item>
    <item>
      <title>Don't Scale Your LLM Routing on Request Volume — Scale It on Task Success Rate</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Fri, 06 Nov 2026 07:05:48 +0000</pubDate>
      <link>https://dev.to/qisuancloud/dont-scale-your-llm-routing-on-request-volume-scale-it-on-task-success-rate-32l3</link>
      <guid>https://dev.to/qisuancloud/dont-scale-your-llm-routing-on-request-volume-scale-it-on-task-success-rate-32l3</guid>
      <description>&lt;p&gt;Your request volume doubled this month. So you doubled your GPU spend. That is the most common scaling mistake in LLM inference, and it is also the most expensive one — because request volume tells you almost nothing about whether your routing is working.&lt;/p&gt;

&lt;p&gt;This week the same question kept surfacing on X in different words: token prices have collapsed, so why are AI bills still painful? One widely shared thread argued the metric that matters is cost per successful outcome, not cost per token. Another pointed out the same model weights can cost 9x more at one provider than another. The underlying question in all of them is the same: &lt;strong&gt;when is it actually time to send more work to the big model — and when is that just burning money?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The answer is not "when requests go up." It is "when the task success rate says so."&lt;/p&gt;

&lt;h2&gt;
  
  
  Request volume is the wrong trigger
&lt;/h2&gt;

&lt;p&gt;Request volume measures demand, not quality. It goes up for reasons that have nothing to do with your routing: a marketing push, a new customer, a retry storm from a flaky client, a prompt change that made outputs longer. None of those mean the small model is failing more often.&lt;/p&gt;

&lt;p&gt;Scaling on volume has a second cost that is easy to miss: it buys capacity before you know what the capacity is for. If your small model is already completing 96% of chores correctly, doubling your large-model hours buys you almost nothing — you are paying frontier prices to re-do work that was already fine. And the price of that mistake is going up, not down: Lambda's published instance rates rise on November 16, 2026, with the H100 moving from $3.99 to $4.59 per GPU-hour and the A100 (80 GB) from $2.79 to $3.19. Every GPU-hour you buy on the wrong trigger costs more than it did last week.&lt;/p&gt;

&lt;p&gt;Volume is a capacity signal. It answers "do I need more total compute?" It does not answer "does this task type need a bigger model?" Those are different questions, and answering the second one with the first one's data is how teams end up with a large-model bill and a small-model quality problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  The right trigger: task success rate and cost per completed task
&lt;/h2&gt;

&lt;p&gt;Track two numbers per task type — not per request, and not per token:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Task success rate:&lt;/strong&gt; the share of tasks that finish correctly without a redo. A task is a unit of real work — an extracted invoice, a classified ticket, a drafted summary — not one API call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost per completed task:&lt;/strong&gt; total GPU cost divided by tasks that finished successfully, redos included.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cost per request hides redos. Cost per completed task exposes them. The formula is simple:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;effective cost = small-model cost + (redo rate × large-model cost) + final-answer cost&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If a third of your extracted fields come back wrong and need a large-model redo, your "cheap" chores are no longer cheap — and no amount of request-volume data will tell you that. Only the success rate will.&lt;/p&gt;

&lt;p&gt;The decision rule follows from the numbers, not from the traffic graph: &lt;strong&gt;scale a task type up to the large model when its success rate falls below the bar you set, and stays there across two measurement windows — not when request count crosses a round number.&lt;/strong&gt; Conversely, if success rate is stable and cost per completed task is flat, a volume spike means you need more of the same capacity, not a bigger model.&lt;/p&gt;

&lt;h2&gt;
  
  
  The math, with prices checked today
&lt;/h2&gt;

&lt;p&gt;These are public GPU listing prices I re-checked on November 6, 2026 — not a production benchmark, and not a promise about your throughput:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;RunPod Community Cloud: H100 SXM from $2.69 per GPU-hour, A100 SXM from $1.39; RTX 4090 listed at $0.89.&lt;/li&gt;
&lt;li&gt;Lambda (rates effective November 16, 2026): H100 at $4.59 per GPU-hour, A100 (80 GB) at $3.19.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now the arithmetic on GPU-hours, because that is the unit you actually pay for when you self-host. Suppose a day's pipeline needs 10 GPU-hours: 8 hours of chores on a 4090, 2 hours of final answers on an H100:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Chores: 8 × $0.89 = &lt;strong&gt;$7.12&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Final answers: 2 × $2.69 = &lt;strong&gt;$5.38&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Total: &lt;strong&gt;$12.50 per day&lt;/strong&gt; for the work attempted.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now apply the success rate. If 1,000 tasks are attempted and 970 complete successfully, cost per completed task is $12.50 / 970 = &lt;strong&gt;$0.0129&lt;/strong&gt;. If the small model's success rate slips to 85% — 850 complete, 150 need a redo — the same $12.50 buys fewer completions ($0.0147 each), and the redos add large-model hours on top. A 12-point drop in success rate costs you more than a 12% rise in request volume ever would, and it is invisible in every dashboard that only counts requests.&lt;/p&gt;

&lt;p&gt;That is the number to alert on. Set the bar per task type: a classification chore might need 98%, a first-pass summary might be fine at 90%. The bar is a product decision, not a universal constant. What matters is that it is written down before the volume spike arrives — because in the middle of a spike, every team "discovers" that the answer is a bigger model.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to measure it without drowning in instrumentation
&lt;/h2&gt;

&lt;p&gt;You do not need a full observability platform to start. You need three logs per task type:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Outcome:&lt;/strong&gt; completed, redone, or escalated — one row per task, not per request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Redo reason:&lt;/strong&gt; wrong fields, missing citation, format failure, low confidence. The reason tells you whether the fix is a better prompt, a narrower task definition, or a bigger model. Most redo reasons are not "model too small."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost:&lt;/strong&gt; GPU-hours per task type, so cost per completed task is a division, not a guess.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Audit a random sample every week — this is the part readers of my last post kept asking about, and they were right to. One reader asked how big the sample needs to be before an audit hit rate means anything; the honest answer is that it depends on the category. Sample per task type, not across the whole pipeline, and audit 100% of the task types where an error is expensive. A 2% error rate on invoice extraction is a different business from a 2% error rate on first-pass summaries, and one blended number hides that.&lt;/p&gt;

&lt;p&gt;Two practical guardrails:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Two windows, not one.&lt;/strong&gt; A single bad hour is noise — a deploy, a bad prompt, a weird input batch. Require the success rate to stay below the bar for two consecutive measurement windows (a day, a week, whatever fits your volume) before you move a task type up a tier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fix the chore before you upgrade the model.&lt;/strong&gt; If redos cluster on one input shape — long documents, a missing field, an ambiguous category — narrow the task or fix the prompt first. Upgrading the model to paper over a broken task definition is the most expensive way to learn the definition was broken.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  We measure our own funnel the same way
&lt;/h2&gt;

&lt;p&gt;This is not theory we exempt ourselves from. Our own GPU cost calculator page recorded 0 clicks and 0 impressions in Google Search Console over the last 28 days (October 9 to November 5, 2026) — it is not ranking yet, and we know that because we measured instead of assuming. We did not respond by buying promotion for it; we responded by checking what is actually being found (our whole site: 9 clicks, 350 impressions in the same window) and fixing the work in front of us first. The same discipline applies to routing: measure first, spend second.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bottom line
&lt;/h2&gt;

&lt;p&gt;Do not ask "how many requests did we get?" Ask "what share of tasks completed successfully, and what did each completed task cost?" Volume tells you when to buy more of what already works. Success rate tells you when what you have stopped working. Scale on the second number.&lt;/p&gt;

&lt;p&gt;If you want to skip the spreadsheet, our free GPU cost calculator prices the same trade-off — workload, GPU tier, and hourly rates — so you can see what a routing change costs before you make it: &lt;a href="https://qisuanai.com/gpu-cost-calculator/" rel="noopener noreferrer"&gt;https://qisuanai.com/gpu-cost-calculator/&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Prices: RunPod and Lambda public pricing pages, checked November 6, 2026. X discussion referenced: cost-per-outcome framing in AI billing threads, week of November 3–6, 2026.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>gpu</category>
      <category>inference</category>
    </item>
    <item>
      <title>Small Models Do the Chores, Big Models Give the Answer: Cost by Task Split, Not by Model Size</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Tue, 03 Nov 2026 11:58:32 +0000</pubDate>
      <link>https://dev.to/qisuancloud/small-models-do-the-chores-big-models-give-the-answer-cost-by-task-split-not-by-model-size-521i</link>
      <guid>https://dev.to/qisuancloud/small-models-do-the-chores-big-models-give-the-answer-cost-by-task-split-not-by-model-size-521i</guid>
      <description>&lt;p&gt;Most teams pick one model and send it everything. That is the expensive mistake. The question is not "which model should we use?" It is "which parts of this job actually need the big model?"&lt;/p&gt;

&lt;p&gt;In most real pipelines, only a small share of the work needs frontier reasoning. The rest is chores: classify a ticket, extract fields from a document, draft a first summary, triage a request, check a format. A small model can do those chores well. Reserve the large model for the final answer, where judgment, synthesis, and edge cases live.&lt;/p&gt;

&lt;p&gt;This is a cost decision, not a quality slogan. Route by task, and measure cost per completed task — not per request, and not per token in isolation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Split the job before you pick the model
&lt;/h2&gt;

&lt;p&gt;Take a typical support or document workflow and split it into three buckets:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Chores (small model):&lt;/strong&gt; classification, field extraction, retrieval, routine summarization, formatting, first-pass drafting.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checks (rules plus a small model):&lt;/strong&gt; schema validation, format checks, citation presence, obvious contradictions. Deterministic checks cost almost nothing and catch the errors that are cheapest to catch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Final answer (large model):&lt;/strong&gt; synthesis across sources, ambiguous cases, decisions that are hard to reverse, and anything where a wrong answer is expensive.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The point of the split is that the large model never sees the raw, unfiltered workload. It sees a cleaned-up task with the relevant context already extracted. That also makes its output easier to audit: you can see what the small model handed over, and what the large model decided.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost math, with prices checked today
&lt;/h2&gt;

&lt;p&gt;These are public GPU listing prices I re-checked on November 3, 2026 — not a production benchmark, and not a promise about your throughput:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;RunPod Community Cloud: RTX 4090 at $0.34 per GPU-hour, A100 SXM (80 GB) at $1.39, H100 SXM at $2.69.&lt;/li&gt;
&lt;li&gt;Lambda on-demand instances: H100 (80 GB) at $3.99 per GPU-hour, A100 (80 GB) at $2.79.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now do the arithmetic on GPU-hours, because that is the unit you actually pay for when you self-host. Suppose a day's pipeline needs 9 GPU-hours in total, and the split is 8 hours of chores plus 1 hour of final answers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;All-large on a RunPod H100: 9 × $2.69 = &lt;strong&gt;$24.21&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Split — chores on a 4090, final answers on an H100: (8 × $0.34) + (1 × $2.69) = $2.72 + $2.69 = &lt;strong&gt;$5.41&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is the 8:1 case, and I am labeling it as such. If your final-answer share is larger, the saving shrinks, and you should know by how much. At a 50/50 split (4.5 hours each):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Split: (4.5 × $0.34) + (4.5 × $2.69) = $1.53 + $12.11 = &lt;strong&gt;$13.64&lt;/strong&gt;, against $24.21 all-large.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The lesson is not "you will save 78%." The lesson is that your saving is driven by one number: what fraction of GPU-hours genuinely needs the large model. Measure that fraction on your own workload before you commit to a routing design. If you cannot measure it, you are guessing, and GPU bills punish guesses.&lt;/p&gt;

&lt;p&gt;A note on the 4090 side of that math, from my own earlier sizing work: a 7B-class model fits on a 24 GB 4090 for inference, but a long context window changes the story because the KV cache grows with context. In an earlier post I showed how quantizing the KV cache lets a 128K context fit on a 4090. The same logic applies here: a "small model" chore is only cheap if you have also sized its context. A chore with a huge context can quietly become a large-model-sized job.&lt;/p&gt;

&lt;h2&gt;
  
  
  The redo rate is where split designs die
&lt;/h2&gt;

&lt;p&gt;The split only saves money if the small model's work usually stands. Every chore that has to be redone by the large model costs you twice: the small-model hour plus the large-model hour.&lt;/p&gt;

&lt;p&gt;Use this formula before you celebrate a saving:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;effective cost = small-model cost + (redo rate × large-model cost) + final-answer cost&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If a third of your extracted fields come back wrong and need a large-model redo, your "cheap" chores are no longer cheap. Track three numbers per task type, not per request:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Success rate:&lt;/strong&gt; the share of tasks completed without a redo.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Redo rate:&lt;/strong&gt; the share that fall back to the large model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost per completed task:&lt;/strong&gt; total GPU cost divided by tasks that finished successfully, redos included.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cost per request hides redos. Cost per completed task exposes them. If a routing change lowers cost per request but raises cost per completed task, it made things worse.&lt;/p&gt;

&lt;h2&gt;
  
  
  Guardrails that keep the small model honest
&lt;/h2&gt;

&lt;p&gt;A small model fails differently from a large one: it is more likely to be confidently wrong on an edge case, and less likely to flag its own uncertainty. Do not fix that by sending everything to the large model for verification — that just re-creates the all-large bill. Instead:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Route on task type, not on vibes.&lt;/strong&gt; Write the routing rule down: "invoices under X pages and with a matching schema go to the small model; everything else goes to the large model." A rule you cannot write down cannot be measured.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use deterministic checks first.&lt;/strong&gt; Schema, format, required fields, and citation checks catch a large share of chore errors for free.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sample, don't verify everything.&lt;/strong&gt; Audit a random sample of small-model outputs each week, and audit 100% of the task types where an error is expensive.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Define the fallback trigger.&lt;/strong&gt; Low confidence, missing fields, contradictory sources, or an input longer than the small model's comfortable context should route to the large model automatically — before a user sees a bad answer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If a task type keeps tripping the fallback, that is data: either the chore was never a chore, or the small model needs better instructions, better retrieval, or a narrower scope. Fix the task definition before you buy a bigger model.&lt;/p&gt;

&lt;h2&gt;
  
  
  A practical way to start
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Log one week of tasks by type: what came in, what the job actually was, and where the large model was truly needed.&lt;/li&gt;
&lt;li&gt;Pick exactly one chore type to move to the small model — the highest-volume, lowest-risk one.&lt;/li&gt;
&lt;li&gt;Run it with the guardrails above and track cost per completed task and redo rate for two weeks.&lt;/li&gt;
&lt;li&gt;Only then move the next chore type.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One chore type at a time is slower to write about and faster to get right. A wholesale "small model everywhere" migration is how teams discover, a month later, that their redo rate ate the saving.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bottom line
&lt;/h2&gt;

&lt;p&gt;Do not ask "small or large?" Ask "which part of this task needs the large model, and how many GPU-hours is that part?" Price that split with today's listing prices, track cost per completed task, and let the redo rate tell you whether the split is real.&lt;/p&gt;

&lt;p&gt;If you want to skip the spreadsheet, our free GPU Advisor walks through the same trade-off — workload, budget, reliability, and GPU tier — and recommends a cloud for it: &lt;a href="https://qisuanai.com/advisor" rel="noopener noreferrer"&gt;https://qisuanai.com/advisor&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Prices in this post are public listing prices re-checked November 3, 2026, from the Lambda pricing page and the ComputePrices RunPod page. Community Cloud prices vary by host and availability; verify before you rent. The worked examples are arithmetic on GPU-hours, not measured throughput from a production run — your tokens per second, context lengths, and redo rates will differ, and they are the numbers that decide your actual bill.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>gpu</category>
      <category>inference</category>
    </item>
    <item>
      <title>SGLang vs vLLM: Your Agent Is Slow Because You Picked the Wrong Cache</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Fri, 23 Oct 2026 01:58:02 +0000</pubDate>
      <link>https://dev.to/qisuancloud/sglang-vs-vllm-your-agent-is-slow-because-you-picked-the-wrong-cache-12hf</link>
      <guid>https://dev.to/qisuancloud/sglang-vs-vllm-your-agent-is-slow-because-you-picked-the-wrong-cache-12hf</guid>
      <description>&lt;p&gt;If you serve LLM agents in production, you have probably seen this week's numbers: in a multi-turn agent workload with an 8,000-token system context, SGLang's RadixAttention cut time-to-first-token (TTFT) from 440ms to 38ms — an 11.57x speedup — while vLLM v0.6 with prefix caching managed 480ms down to 92ms, a 5.21x improvement (Qwen2.5-Coder-32B-Instruct-AWQ), as benchmarked by NextByteTech in their Dev.to post on ollama vs vLLM vs SGLang in 2026.&lt;/p&gt;

&lt;p&gt;The data is real, and the conclusion is mostly right: for long-context, multi-turn agent traffic, SGLang currently wins. But "mostly" hides the interesting part. Those speedups only materialize when your workload has the right shape. Run the same two engines against different traffic and the gap collapses — or flips. This post is the decision framework the benchmark post didn't give you: when RadixAttention genuinely pulls ahead, and when both engines tie.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why RadixAttention Wins in Agent Workloads
&lt;/h2&gt;

&lt;p&gt;Both vLLM's prefix caching and SGLang's RadixAttention reuse previously computed KV cache for repeated prompt prefixes. The difference is the data structure.&lt;/p&gt;

&lt;p&gt;vLLM's automatic prefix caching (APC) historically hashes &lt;strong&gt;whole blocks&lt;/strong&gt; of the prompt and matches them against cached blocks. It's effective when requests share long, identical prefixes — the classic system-prompt-plus-few-shot-template shape — but it degrades when prefixes diverge mid-sequence or when the overlap doesn't align to block boundaries.&lt;/p&gt;

&lt;p&gt;SGLang's RadixAttention stores the KV cache in a &lt;strong&gt;radix tree&lt;/strong&gt; (a compressed trie) over token sequences. Repeated prefixes become shared tree paths, and the cache can be reused at the granularity of any common prefix, not just aligned blocks. Multi-turn agent conversations are exactly this: the same 8,000-token system context plus growing conversation history, turn after turn. Each new turn shares a massive common prefix with the previous one. The tree reuses nearly all of it, which is why the reported TTFT dropped to 38ms — almost the entire prefill was skipped.&lt;/p&gt;

&lt;p&gt;The second Dev.to comparison (ifnodoraemon's SGLang vs vLLM architecture showdown) makes the mechanism visible in the metrics: at a 4,096-token prompt with 75% prefix overlap and concurrency 64, SGLang hit a &lt;strong&gt;78.6% KV cache hit rate vs vLLM's 41.2%&lt;/strong&gt;, with median TTFT of 85ms vs 380ms (4.47x), P99 TTFT of 280ms vs 1,250ms, and throughput of 5,430 vs 3,120 tok/s. The hit-rate gap &lt;em&gt;is&lt;/em&gt; the performance gap. When your workload is prefix-overlap-heavy, the radix tree's finer-grained reuse shows up directly in every latency percentile.&lt;/p&gt;

&lt;h2&gt;
  
  
  But Sometimes They Tie
&lt;/h2&gt;

&lt;p&gt;Here is where the story gets useful. Three published counter-cases show where the advantage evaporates.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Unique-prompt traffic: the gap nearly disappears
&lt;/h3&gt;

&lt;p&gt;bex.co's September 2026 report (&lt;em&gt;SGLang Crossed 400,000 Production GPUs&lt;/em&gt;) benchmarked Llama 3.3 70B FP8 on Spheron H100s at 50 concurrent requests with &lt;strong&gt;unique prompts&lt;/strong&gt; — traffic with no shared prefix to reuse. The result: SGLang at 1,920 tok/s vs vLLM at 1,850 tok/s, with TTFT within 20ms of each other. No shared prefix, no cache to share; the radix tree has nothing to work with, and the two engines are functionally tied.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Raw throughput on short prompts: vLLM can win
&lt;/h3&gt;

&lt;p&gt;The same bex.co report includes a May 2026 result on a single H100 with Qwen2.5-7B: vLLM delivered &lt;strong&gt;23,500 tok/s of raw throughput vs SGLang's 16,800 tok/s&lt;/strong&gt; — a clear vLLM win on aggregate tokens per second. SGLang still held a 4x+ TTFT advantage (1.8s vs 7.8s), so it wasn't a loss on every axis, but for a throughput-first, latency-tolerant batch workload, the "SGLang is faster" headline would have been wrong.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. High concurrency: the crossover is real
&lt;/h3&gt;

&lt;p&gt;bex.co's DGX Spark concurrency sweep found SGLang winning at 1–4 concurrent requests, with &lt;strong&gt;vLLM winning at 6+ concurrent requests&lt;/strong&gt;. And RunPod's blog on the SGLang vs vLLM KV cache comparison (2x H100, DeepSeek-R1-Distill-Llama-70B, 7k context) concluded that a cache hit yields roughly +20% speed — and that on &lt;strong&gt;fresh context the two engines are comparable&lt;/strong&gt;. If your traffic is many short, independent prompts at high concurrency — the classic chatbot-serving shape — the radix tree overhead buys you nothing and vLLM's throughput engine takes the round.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workload shape&lt;/th&gt;
&lt;th&gt;Winner (per published results)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Multi-turn agent, long shared system prompt (8k tokens), moderate concurrency&lt;/td&gt;
&lt;td&gt;SGLang, large margin (11.57x vs 5.21x TTFT speedup)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4k prompts, 75% prefix overlap, concurrency 64&lt;/td&gt;
&lt;td&gt;SGLang (4.47x median TTFT, 2x KV hit rate)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unique prompts, no shared prefix, 50 concurrent&lt;/td&gt;
&lt;td&gt;Tie (1,920 vs 1,850 tok/s, TTFT within 20ms)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Short prompts, throughput-first batch, 1x H100&lt;/td&gt;
&lt;td&gt;vLLM (23,500 vs 16,800 tok/s raw throughput)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Short prompts, 6+ concurrency (DGX Spark)&lt;/td&gt;
&lt;td&gt;vLLM&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fresh context, no cache reuse (RunPod)&lt;/td&gt;
&lt;td&gt;Tie&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The Decision Checklist
&lt;/h2&gt;

&lt;p&gt;Before you switch engines, measure your workload against these four questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;How much prefix overlap do you actually have?&lt;/strong&gt; If most of your requests share a long system prompt or conversation history (agents, RAG with fixed instruction blocks, few-shot templates), RadixAttention's finer-grained reuse pays off. If every prompt is unique, the tree is dead weight.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What's your concurrency?&lt;/strong&gt; Published sweeps show a crossover: SGLang ahead at low concurrency, vLLM ahead as concurrency climbs (around 6+ on DGX Spark in one report — verify against your own hardware). Size for your actual traffic, not the benchmark's.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Are you prefill-heavy or decode-heavy?&lt;/strong&gt; Prefix caching only accelerates the prefill phase. If your outputs are long and your prompts are short, most of your time is in decode, and the cache advantage shrinks to a rounding error. RunPod's +20%-on-cache-hit figure is a good mental anchor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Which latency percentile matters to you?&lt;/strong&gt; SGLang's biggest reported edge is in TTFT and tail latency (P99 280ms vs 1,250ms in the 75%-overlap scenario). If your SLO is on end-to-end tokens-per-second for batch jobs, that edge may not buy you anything — vLLM's raw throughput win shows the other side of the trade.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The benchmark post's headline number — 38ms vs 92ms TTFT in the multi-turn agent scenario — is correct for that workload, and RadixAttention's radix-tree design is genuinely the better mechanism for shared-prefix traffic. But engines don't run benchmarks; they run your traffic. Unique-prompt serving, short-prompt batch throughput, and high-concurrency chatbot shapes all show the two engines tying or vLLM winning, per bex.co, RunPod, and the second Dev.to comparison. Measure your prefix overlap and concurrency first, then pick the cache that fits.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;We write about production LLM infra and agent systems for builders shipping real products. Follow along here:&lt;/em&gt;&lt;br&gt;
&lt;a href="https://t.me/qisuancloud" rel="noopener noreferrer"&gt;https://t.me/qisuancloud&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Or try our free GPU Advisor: &lt;a href="https://qisuanai.com/advisor" rel="noopener noreferrer"&gt;https://qisuanai.com/advisor&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>opensource</category>
      <category>benchmarking</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Your 128K Context Fits on a 4090 — If You Quantize the KV Cache</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Mon, 19 Oct 2026 06:10:55 +0000</pubDate>
      <link>https://dev.to/qisuancloud/your-128k-context-fits-on-a-4090-if-you-quantize-the-kv-cache-51bp</link>
      <guid>https://dev.to/qisuancloud/your-128k-context-fits-on-a-4090-if-you-quantize-the-kv-cache-51bp</guid>
      <description>&lt;p&gt;&lt;em&gt;Researched October 2026. Sizing math from public model cards and the vLLM docs; not my own benchmarks.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In my &lt;a href="https://dev.to/qisuancloud/your-7b-model-fits-on-a-4090-until-you-open-a-128k-context-window"&gt;last post&lt;/a&gt; I showed that a 128K context window can push a 7B model from 17 GB to 33 GB of VRAM — and that you have three moves, in order of cost-effectiveness: quantize the cache, pick a GQA-heavy architecture, or rent the bigger card. This post is about move #1, the cheapest one. It halves the KV cache with a single flag.&lt;/p&gt;

&lt;h2&gt;
  
  
  The One Flag
&lt;/h2&gt;

&lt;p&gt;In vLLM:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve meta-llama/Llama-3.1-8B-Instruct &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--kv-cache-dtype&lt;/span&gt; fp8
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's it. The weights stay in FP16/BF16 — untouched. Only the KV cache is stored as 1 byte per value instead of 2. Per the &lt;a href="https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/" rel="noopener noreferrer"&gt;vLLM docs&lt;/a&gt;, &lt;code&gt;fp8&lt;/code&gt; maps to E4M3, and there are three calibration options: no calibration (scales default to 1.0), random-token warmup, or dataset calibration via llm-compressor (recommended for max accuracy).&lt;/p&gt;

&lt;h2&gt;
  
  
  Why FP8 Works for KV Cache
&lt;/h2&gt;

&lt;p&gt;Weight quantization gets all the attention, but KV values are arguably the easier target: within a single attention head, their values live in a narrow, well-behaved range. E4M3's tighter range and 3-bit mantissa match that distribution well; E5M2's wider range is mostly wasted on values that never appear, and its 2-bit mantissa visibly degrades attention scores that compound across layers. In other words, the format choice matters — &lt;code&gt;fp8&lt;/code&gt; (E4M3) is the right default, not &lt;code&gt;fp8_e5m2&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the Math Flips
&lt;/h2&gt;

&lt;p&gt;Llama 3.1 8B, FP16 weights (~16 GB), 2 GB overhead:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;FP16 cache → total&lt;/th&gt;
&lt;th&gt;FP8 cache → total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;32K&lt;/td&gt;
&lt;td&gt;4 GB → ~22 GB ✅&lt;/td&gt;
&lt;td&gt;2 GB → ~20 GB ✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;64K&lt;/td&gt;
&lt;td&gt;8 GB → ~26 GB ❌&lt;/td&gt;
&lt;td&gt;4 GB → ~22 GB ✅ fits 4090&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128K&lt;/td&gt;
&lt;td&gt;16 GB → ~34 GB ❌&lt;/td&gt;
&lt;td&gt;8 GB → ~26 GB ❌ still over&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Honest note: at 128K, FP8 cache alone doesn't save Llama 3.1 8B on a 24 GB card — 26 GB is still 26 GB. Your options there: pair it with FP8 weights (8 + 8 + 2 ≈ 18 GB ✅), or pick the GQA-heavy architecture from last post — Qwen2.5 7B at 128K needs ~7 GB of FP16 cache, ~3.5 GB in FP8, for a total around 19 GB ✅. The flag is powerful, not magic; the arithmetic still rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three Caveats Before You Ship It
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Hardware matters.&lt;/strong&gt; The biggest wins are on Hopper GPUs with native FP8 support. On older cards, check your vLLM version's kernel support — the memory savings are real, but the compute path may not be as optimized.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Calibrate if quality matters.&lt;/strong&gt; Default scales of 1.0 are fine to start, but for production, dataset calibration (llm-compressor) exists for a reason. Per-head quantization — finer-grained than per-tensor — needs the FlashAttention backend plus that calibration path.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Measure retrieval, not just perplexity.&lt;/strong&gt; Standard benchmarks show small degradation, but long-context tasks with exact-recall demands (needle-in-a-haystack style document QA) are where you'd notice it first. Run your own eval before and after.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Costs in Practice
&lt;/h2&gt;

&lt;p&gt;The whole point: quantization keeps you on the cheap card. A 64K-context 7B workload that needed an 80 GB card at ~$1.50+/hr in FP16 fits on a 24 GB 4090 at ~$0.35/hr with FP8 cache — same model, same context, roughly a quarter of the hourly cost, for one CLI flag.&lt;/p&gt;




&lt;p&gt;I built a &lt;a href="https://qisuanai.com/advisor" rel="noopener noreferrer"&gt;free GPU Advisor&lt;/a&gt; that walks through these tradeoffs (model size, context, quantization tolerance) and recommends the cheapest fitting setup. No signup.&lt;/p&gt;

&lt;p&gt;Have you tried FP8 KV cache in production — and did the quality hold up on your workload? Curious where it breaks in practice.&lt;/p&gt;

</description>
      <category>gpu</category>
      <category>llm</category>
      <category>vllm</category>
      <category>ai</category>
    </item>
    <item>
      <title>Stop Overpaying for GPUs: Spot vs On-Demand for LLM Inference (With Real Numbers)</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Fri, 16 Oct 2026 07:03:05 +0000</pubDate>
      <link>https://dev.to/qisuancloud/stop-overpaying-for-gpus-spot-vs-on-demand-for-llm-inference-with-real-numbers-2eg6</link>
      <guid>https://dev.to/qisuancloud/stop-overpaying-for-gpus-spot-vs-on-demand-for-llm-inference-with-real-numbers-2eg6</guid>
      <description>&lt;h1&gt;
  
  
  Stop Overpaying for GPUs: Spot vs On-Demand for LLM Inference (With Real Numbers)
&lt;/h1&gt;

&lt;p&gt;If you're running LLM inference on rented GPUs, you're probably overpaying. Not by a little — by 2-3x. The culprit isn't which cloud you picked. It's &lt;em&gt;how&lt;/em&gt; you're paying for it.&lt;/p&gt;

&lt;p&gt;I spend my days comparing GPU cloud pricing (it's literally what I do), and the spot vs on-demand decision is where most people leave the most money on the table. Here's the practical breakdown.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Price Gap Is Bigger Than You Think
&lt;/h2&gt;

&lt;p&gt;Take an RTX 4090 (24GB) — the workhorse for 7B-32B model inference:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;On-demand&lt;/strong&gt;: ~$0.34–$0.74/hour depending on the cloud&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spot/preemptible&lt;/strong&gt;: ~$0.15–$0.30/hour for the same card&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's a 50-70% discount for the exact same silicon. For an A100 80GB, the gap is similar: ~$1.20/hr on-demand vs ~$0.50/hr spot.&lt;/p&gt;

&lt;p&gt;But cheap isn't free. Here's what you're actually trading.&lt;/p&gt;

&lt;h2&gt;
  
  
  What "Spot" Really Means in Practice
&lt;/h2&gt;

&lt;p&gt;Spot instances can be reclaimed by the provider with little warning (typically 30 seconds to 2 minutes). For LLM inference, that means:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Model reload time&lt;/strong&gt;: A 70B model takes 3-5 minutes to load into VRAM. If you get preempted twice an hour, you've lost more than you saved.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Request interruption&lt;/strong&gt;: In-flight requests die. You need retry logic or a queue.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Availability variance&lt;/strong&gt;: During peak hours, spot capacity dries up. Your "cheap" GPU might simply not exist when you need it.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The Decision Framework
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use spot when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You're doing batch inference (embeddings, bulk classification, offline evals)&lt;/li&gt;
&lt;li&gt;Your workload is checkpoint-friendly or stateless&lt;/li&gt;
&lt;li&gt;You can tolerate 5-10 minute interruptions&lt;/li&gt;
&lt;li&gt;You're running experiments and cost matters more than latency&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Use on-demand when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You're serving real users with latency SLAs&lt;/li&gt;
&lt;li&gt;Your model takes &amp;gt;2 minutes to load (large models)&lt;/li&gt;
&lt;li&gt;It's peak hours in US/EU and spot markets are thin&lt;/li&gt;
&lt;li&gt;The cost of an interruption exceeds the savings (do the math!)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The hybrid most people should use:&lt;/strong&gt;&lt;br&gt;
Run your baseline traffic on 1-2 on-demand instances, burst with spot. Most inference workloads have a predictable floor and a spiky ceiling. Pay full price for the floor, spot price for the spikes.&lt;/p&gt;
&lt;h2&gt;
  
  
  The Math People Skip
&lt;/h2&gt;

&lt;p&gt;Don't just compare hourly rates. Calculate &lt;strong&gt;effective cost per 1K tokens&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;effective_hourly = (hourly_rate × hours_run + reload_cost) / hours_run
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Where &lt;code&gt;reload_cost = preemptions × reload_time_hours × hourly_rate&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Example: $0.20/hr spot, preempted 3x/hour, 4-min reloads on a 70B model:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reload overhead: 3 × (4/60) × $0.20 = $0.04/hr extra&lt;/li&gt;
&lt;li&gt;Effective rate: $0.24/hr — still way cheaper than $0.74 on-demand&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;But with 10 preemptions/hour? Effective rate hits $0.33/hr plus the latency pain. At that point, on-demand wins.&lt;/p&gt;

&lt;h2&gt;
  
  
  One More Thing: Egress Fees
&lt;/h2&gt;

&lt;p&gt;The sneakiest cost isn't compute — it's data transfer out. Some clouds charge $0.05-0.09/GB for egress. If you're doing RAG with large context windows or serving multimodal models, egress can exceed your GPU bill. Always check the egress pricing before committing.&lt;/p&gt;




&lt;p&gt;I track GPU cloud prices weekly and post updates when prices move significantly. If you want the short version without reading pricing pages, I share them on my Telegram channel: &lt;a href="https://t.me/qisuancloud" rel="noopener noreferrer"&gt;https://t.me/qisuancloud&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;And if you want to skip the spreadsheet entirely, our free GPU Advisor picks the cheapest option for your workload in about 30 seconds: &lt;a href="https://qisuanai.com/advisor" rel="noopener noreferrer"&gt;https://qisuanai.com/advisor&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Prices referenced are from public pricing pages, October 2026. Always verify before you rent — cloud pricing changes fast.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>gpu</category>
      <category>llm</category>
      <category>cloud</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Your 7B Model Fits on a 4090 — Until You Open a 128K Context Window</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Sun, 11 Oct 2026 02:41:11 +0000</pubDate>
      <link>https://dev.to/qisuancloud/your-7b-model-fits-on-a-4090-until-you-open-a-128k-context-window-31ag</link>
      <guid>https://dev.to/qisuancloud/your-7b-model-fits-on-a-4090-until-you-open-a-128k-context-window-31ag</guid>
      <description>&lt;p&gt;&lt;em&gt;Researched October 2026. Architecture specs from public model cards; all numbers below are arithmetic, not my own benchmarks.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In my &lt;a href="https://dev.to/qisuancloud/your-7b-model-doesn-t-need-an-h100-a-practical-gpu-sizing-guide-for-llm-inference"&gt;last post&lt;/a&gt; I showed that a 7B model's weights need ~16 GB in FP16 — a comfortable fit on a 24 GB RTX 4090. Several readers asked the obvious follow-up: &lt;em&gt;then why does my 7B inference server OOM the moment I enable long context?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Because weights are only half the story. The other half is the &lt;strong&gt;KV cache&lt;/strong&gt;, and it scales in a way that surprises almost everyone.&lt;/p&gt;

&lt;h2&gt;
  
  
  The One Formula
&lt;/h2&gt;

&lt;p&gt;For each token in context, the model must store a key and a value vector per layer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_value
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Take &lt;strong&gt;Llama 3.1 8B&lt;/strong&gt; (32 layers, 8 KV heads, 128 head dim, FP16):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KiB per token
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now multiply by context length:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;KV cache (Llama 3.1 8B, FP16)&lt;/th&gt;
&lt;th&gt;+ 16 GB weights → total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;8K&lt;/td&gt;
&lt;td&gt;~1 GB&lt;/td&gt;
&lt;td&gt;~17 GB fits 4090&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32K&lt;/td&gt;
&lt;td&gt;~4 GB&lt;/td&gt;
&lt;td&gt;~21 GB fits 4090&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128K&lt;/td&gt;
&lt;td&gt;~16 GB&lt;/td&gt;
&lt;td&gt;~33 GB needs 48 GB+&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Same model, same weights — the context window alone decides whether you need a $0.35/hr card or a $1.50+/hr one. (Prices from my &lt;a href="https://dev.to/qisuancloud/i-compared-5-gpu-clouds-for-llm-inference-in-2026-heres-what-i-found-4884"&gt;provider comparison&lt;/a&gt;.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Not All 7B Models Are Equal
&lt;/h2&gt;

&lt;p&gt;KV head count varies wildly between architectures:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Layers&lt;/th&gt;
&lt;th&gt;KV heads&lt;/th&gt;
&lt;th&gt;KiB/token (FP16)&lt;/th&gt;
&lt;th&gt;128K context cache&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.1 8B&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;td&gt;~16 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mistral 7B v0.3&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;td&gt;~16 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5 7B&lt;/td&gt;
&lt;td&gt;28&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;56&lt;/td&gt;
&lt;td&gt;~7 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Qwen2.5 7B at 128K needs roughly &lt;strong&gt;half&lt;/strong&gt; the cache of Llama 3.1 8B. Two "7B models," very different GPU bills.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 70B Reality Check
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Llama 3.1 70B&lt;/strong&gt; (80 layers, 8 KV heads): 320 KiB/token → 128K context = &lt;strong&gt;~41 GB of KV cache&lt;/strong&gt;. Add 140 GB of FP16 weights and you're at ~181 GB — that's three 80 GB cards. Even in FP8 (70 GB weights), you're at ~111 GB: still two cards minimum. Long context is where 70B deployments quietly become multi-GPU projects.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three Things Most Guides Skip
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Batch size is a silent multiplier.&lt;/strong&gt; KV cache scales with batch × sequence length. Serving 8 concurrent users at 8K context costs roughly the same cache as 1 user at 64K. Your "fits on a 4090" math breaks the moment traffic grows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. PagedAttention doesn't shrink the cache.&lt;/strong&gt; vLLM's PagedAttention eliminates fragmentation waste, but the bytes are the bytes — it can't make 33 GB fit in 24 GB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Quantize the cache too.&lt;/strong&gt; KV cache in FP8 halves these numbers with minimal quality loss on most workloads. It's the same lever as weight quantization, applied to the forgotten half of VRAM.&lt;/p&gt;

&lt;h2&gt;
  
  
  A 30-Second Sizing Rule
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;context_budget = (VRAM − weights − 2 GB overhead) ÷ KiB_per_token ÷ batch_size
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If your planned context exceeds the budget, you have three moves: quantize the cache, pick a GQA-heavy architecture, or rent the bigger card — in that order of cost-effectiveness.&lt;/p&gt;

&lt;p&gt;I built a &lt;a href="https://qisuanai.com/advisor" rel="noopener noreferrer"&gt;free GPU Advisor&lt;/a&gt; that walks through these tradeoffs with four questions and recommends the cheapest fitting setup. No signup.&lt;/p&gt;

&lt;p&gt;What's the longest context you're actually serving in production — and did the KV cache math surprise you the first time? Curious what caught people off guard.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>gpu</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Your 7B Model Doesn't Need an H100: A Practical GPU Sizing Guide for LLM Inference</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Fri, 09 Oct 2026 10:00:34 +0000</pubDate>
      <link>https://dev.to/qisuancloud/your-7b-model-doesnt-need-an-h100-a-practical-gpu-sizing-guide-for-llm-inference-39ia</link>
      <guid>https://dev.to/qisuancloud/your-7b-model-doesnt-need-an-h100-a-practical-gpu-sizing-guide-for-llm-inference-39ia</guid>
      <description>&lt;h1&gt;
  
  
  Your 7B Model Doesn't Need an H100: A Practical GPU Sizing Guide for LLM Inference
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Researched October 2026. Sizing rules from public docs and community practice, not my own benchmarks.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The most expensive mistake in LLM inference isn't picking the wrong cloud — it's renting too much GPU. I keep seeing people spin up H100s to serve 7B chatbots. Here's a practical guide to right-sizing your GPU.&lt;/p&gt;

&lt;h2&gt;
  
  
  The One Formula That Matters
&lt;/h2&gt;

&lt;p&gt;Weights in FP16 take roughly &lt;strong&gt;2 bytes per parameter&lt;/strong&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model size&lt;/th&gt;
&lt;th&gt;FP16 weights&lt;/th&gt;
&lt;th&gt;Minimum sensible GPU&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;7–8B&lt;/td&gt;
&lt;td&gt;~14–16 GB&lt;/td&gt;
&lt;td&gt;RTX 4090 / L4 (24 GB)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13B&lt;/td&gt;
&lt;td&gt;~26 GB&lt;/td&gt;
&lt;td&gt;A100 40GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;34B&lt;/td&gt;
&lt;td&gt;~68 GB&lt;/td&gt;
&lt;td&gt;2× A100 40GB or 1× H100 80GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;70B&lt;/td&gt;
&lt;td&gt;~140 GB&lt;/td&gt;
&lt;td&gt;2× A100/H100 80GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Then add ~20–30% headroom for KV cache (grows with context length and batch size). Long-context workloads can double your memory needs — that's the part most guides skip.&lt;/p&gt;

&lt;h2&gt;
  
  
  Quantization Changes Everything
&lt;/h2&gt;

&lt;p&gt;FP8 / AWQ / GPTQ roughly halve the memory requirement:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A 70B model in FP8 (~70 GB) can fit on a single H100 80GB&lt;/li&gt;
&lt;li&gt;A 13B AWQ model runs comfortably on a 24 GB card&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your accuracy budget allows it, quantization is the single biggest cost lever you have — bigger than switching cloud providers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Throughput vs. Latency: Pick Your Bottleneck
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Chatbot (low concurrency):&lt;/strong&gt; you need enough VRAM, that's it. A 4090 serving a 7B model at 60+ tok/s is overkill for 5 concurrent users — but cheap overkill.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Batch/API workloads:&lt;/strong&gt; memory bandwidth is king. This is where A100/H100 earn their premium.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Don't pay for bandwidth you won't use.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Decision Checklist
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;What's the largest model you'll serve in the next 6 months?&lt;/li&gt;
&lt;li&gt;FP16 or can you quantize?&lt;/li&gt;
&lt;li&gt;Max context length × max batch size → KV cache estimate&lt;/li&gt;
&lt;li&gt;Latency SLA? (real-time chat vs. background jobs)&lt;/li&gt;
&lt;li&gt;Spot-tolerant? (see my &lt;a href="https://dev.to/qisuancloud/i-compared-5-gpu-clouds-for-llm-inference-in-2026-heres-what-i-found-4884"&gt;previous post&lt;/a&gt; on spot pricing)&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What This Costs in Practice
&lt;/h2&gt;

&lt;p&gt;Combining sizing with cloud prices (researched Oct 2026): a 7B model on a Vast.ai spot 4090 (~$0.35/hr) vs. an H100 on-demand (~$2.50+/hr) is a &lt;strong&gt;7x cost difference&lt;/strong&gt; for a workload that might not even notice.&lt;/p&gt;




&lt;p&gt;I'm building a free tool that asks about your use case and ranks GPU options: &lt;a href="https://qisuanai.com/advisor" rel="noopener noreferrer"&gt;https://qisuanai.com/advisor&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;What are you serving, and on what? Curious where these rules break down in practice.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>gpu</category>
      <category>ai</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>I Compared 5 GPU Clouds for LLM Inference in 2026 — Here's What I Found</title>
      <dc:creator>Alex Chen</dc:creator>
      <pubDate>Tue, 06 Oct 2026 09:41:57 +0000</pubDate>
      <link>https://dev.to/qisuancloud/i-compared-5-gpu-clouds-for-llm-inference-in-2026-heres-what-i-found-4884</link>
      <guid>https://dev.to/qisuancloud/i-compared-5-gpu-clouds-for-llm-inference-in-2026-heres-what-i-found-4884</guid>
      <description>&lt;h1&gt;
  
  
  I Compared 5 GPU Clouds for LLM Inference in 2026 — Here's What I Found
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Researched October 2026. Prices from public pricing pages and third-party trackers.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Running LLMs in production gets expensive fast. I spent a week reviewing public pricing data from GPU cloud providers to find the cheapest way to serve inference workloads. Here are the results.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Contenders
&lt;/h2&gt;

&lt;p&gt;I looked at five providers that offer on-demand GPU instances suitable for LLM inference:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;RunPod&lt;/strong&gt; — community favorite, wide GPU selection&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lambda Labs&lt;/strong&gt; — straightforward pricing, good availability&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Vast.ai&lt;/strong&gt; — marketplace model, often cheapest&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AWS&lt;/strong&gt; — the default choice, premium pricing&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Cloud&lt;/strong&gt; — strong TPUs, GPU pricing varies&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Price Comparison (A100 80GB, per hour)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Provider&lt;/th&gt;
&lt;th&gt;A100 80GB/hr&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Vast.ai&lt;/td&gt;
&lt;td&gt;~$1.20&lt;/td&gt;
&lt;td&gt;Marketplace, prices fluctuate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RunPod&lt;/td&gt;
&lt;td&gt;~$1.89&lt;/td&gt;
&lt;td&gt;Secure Cloud pricing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda Labs&lt;/td&gt;
&lt;td&gt;~$1.50&lt;/td&gt;
&lt;td&gt;On-demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS (p4d)&lt;/td&gt;
&lt;td&gt;~$3.20&lt;/td&gt;
&lt;td&gt;On-demand, us-east-1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GCP (a2-highgpu)&lt;/td&gt;
&lt;td&gt;~$2.90&lt;/td&gt;
&lt;td&gt;On-demand&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Prices are approximate, researched October 2026. Always check current pricing.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What About RTX 4090?
&lt;/h2&gt;

&lt;p&gt;For smaller models (7B-13B), a 4090 is often enough and much cheaper:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Provider&lt;/th&gt;
&lt;th&gt;RTX 4090/hr&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Vast.ai&lt;/td&gt;
&lt;td&gt;~$0.35&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RunPod&lt;/td&gt;
&lt;td&gt;~$0.69&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A 4090 can handle Llama 3 8B inference comfortably. If your model fits, don't overpay for an A100.&lt;/p&gt;

&lt;h2&gt;
  
  
  My Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Marketplace &amp;gt; fixed pricing&lt;/strong&gt; — Vast.ai is consistently 30-50% cheaper, but you trade reliability&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Match GPU to model size&lt;/strong&gt; — Don't rent an H100 for a 7B model&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spot/preemptible instances&lt;/strong&gt; — Can cut costs another 50-70% if your workload tolerates interruptions&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch egress fees&lt;/strong&gt; — Some providers charge heavily for data transfer out&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What's Next
&lt;/h2&gt;

&lt;p&gt;I'm building a free tool that recommends the cheapest GPU for your specific workload. Drop a comment with your use case and I'll share early access.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What's your go-to GPU cloud? Am I missing a cheaper option?&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://qisuanai.com" rel="noopener noreferrer"&gt;QisuanCloud blog&lt;/a&gt; — or skip the reading: our &lt;a href="https://qisuanai.com/advisor" rel="noopener noreferrer"&gt;free GPU Advisor&lt;/a&gt; ranks GPU clouds for your workload in about 30 seconds. No signup.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>gpu</category>
      <category>llm</category>
      <category>cloud</category>
    </item>
  </channel>
</rss>
