Most teams pick one model and send it everything. That is the expensive mistake. The question is not "which model should we use?" It is "which parts of this job actually need the big model?"
In most real pipelines, only a small share of the work needs frontier reasoning. The rest is chores: classify a ticket, extract fields from a document, draft a first summary, triage a request, check a format. A small model can do those chores well. Reserve the large model for the final answer, where judgment, synthesis, and edge cases live.
This is a cost decision, not a quality slogan. Route by task, and measure cost per completed task — not per request, and not per token in isolation.
Split the job before you pick the model
Take a typical support or document workflow and split it into three buckets:
- Chores (small model): classification, field extraction, retrieval, routine summarization, formatting, first-pass drafting.
- Checks (rules plus a small model): schema validation, format checks, citation presence, obvious contradictions. Deterministic checks cost almost nothing and catch the errors that are cheapest to catch.
- Final answer (large model): synthesis across sources, ambiguous cases, decisions that are hard to reverse, and anything where a wrong answer is expensive.
The point of the split is that the large model never sees the raw, unfiltered workload. It sees a cleaned-up task with the relevant context already extracted. That also makes its output easier to audit: you can see what the small model handed over, and what the large model decided.
The cost math, with prices checked today
These are public GPU listing prices I re-checked on November 3, 2026 — not a production benchmark, and not a promise about your throughput:
- RunPod Community Cloud: RTX 4090 at $0.34 per GPU-hour, A100 SXM (80 GB) at $1.39, H100 SXM at $2.69.
- Lambda on-demand instances: H100 (80 GB) at $3.99 per GPU-hour, A100 (80 GB) at $2.79.
Now do the arithmetic on GPU-hours, because that is the unit you actually pay for when you self-host. Suppose a day's pipeline needs 9 GPU-hours in total, and the split is 8 hours of chores plus 1 hour of final answers:
- All-large on a RunPod H100: 9 × $2.69 = $24.21
- Split — chores on a 4090, final answers on an H100: (8 × $0.34) + (1 × $2.69) = $2.72 + $2.69 = $5.41
That is the 8:1 case, and I am labeling it as such. If your final-answer share is larger, the saving shrinks, and you should know by how much. At a 50/50 split (4.5 hours each):
- Split: (4.5 × $0.34) + (4.5 × $2.69) = $1.53 + $12.11 = $13.64, against $24.21 all-large.
The lesson is not "you will save 78%." The lesson is that your saving is driven by one number: what fraction of GPU-hours genuinely needs the large model. Measure that fraction on your own workload before you commit to a routing design. If you cannot measure it, you are guessing, and GPU bills punish guesses.
A note on the 4090 side of that math, from my own earlier sizing work: a 7B-class model fits on a 24 GB 4090 for inference, but a long context window changes the story because the KV cache grows with context. In an earlier post I showed how quantizing the KV cache lets a 128K context fit on a 4090. The same logic applies here: a "small model" chore is only cheap if you have also sized its context. A chore with a huge context can quietly become a large-model-sized job.
The redo rate is where split designs die
The split only saves money if the small model's work usually stands. Every chore that has to be redone by the large model costs you twice: the small-model hour plus the large-model hour.
Use this formula before you celebrate a saving:
effective cost = small-model cost + (redo rate × large-model cost) + final-answer cost
If a third of your extracted fields come back wrong and need a large-model redo, your "cheap" chores are no longer cheap. Track three numbers per task type, not per request:
- Success rate: the share of tasks completed without a redo.
- Redo rate: the share that fall back to the large model.
- Cost per completed task: total GPU cost divided by tasks that finished successfully, redos included.
Cost per request hides redos. Cost per completed task exposes them. If a routing change lowers cost per request but raises cost per completed task, it made things worse.
Guardrails that keep the small model honest
A small model fails differently from a large one: it is more likely to be confidently wrong on an edge case, and less likely to flag its own uncertainty. Do not fix that by sending everything to the large model for verification — that just re-creates the all-large bill. Instead:
- Route on task type, not on vibes. Write the routing rule down: "invoices under X pages and with a matching schema go to the small model; everything else goes to the large model." A rule you cannot write down cannot be measured.
- Use deterministic checks first. Schema, format, required fields, and citation checks catch a large share of chore errors for free.
- Sample, don't verify everything. Audit a random sample of small-model outputs each week, and audit 100% of the task types where an error is expensive.
- Define the fallback trigger. Low confidence, missing fields, contradictory sources, or an input longer than the small model's comfortable context should route to the large model automatically — before a user sees a bad answer.
If a task type keeps tripping the fallback, that is data: either the chore was never a chore, or the small model needs better instructions, better retrieval, or a narrower scope. Fix the task definition before you buy a bigger model.
A practical way to start
- Log one week of tasks by type: what came in, what the job actually was, and where the large model was truly needed.
- Pick exactly one chore type to move to the small model — the highest-volume, lowest-risk one.
- Run it with the guardrails above and track cost per completed task and redo rate for two weeks.
- Only then move the next chore type.
One chore type at a time is slower to write about and faster to get right. A wholesale "small model everywhere" migration is how teams discover, a month later, that their redo rate ate the saving.
The bottom line
Do not ask "small or large?" Ask "which part of this task needs the large model, and how many GPU-hours is that part?" Price that split with today's listing prices, track cost per completed task, and let the redo rate tell you whether the split is real.
If you want to skip the spreadsheet, our free GPU Advisor walks through the same trade-off — workload, budget, reliability, and GPU tier — and recommends a cloud for it: https://qisuanai.com/advisor
Prices in this post are public listing prices re-checked November 3, 2026, from the Lambda pricing page and the ComputePrices RunPod page. Community Cloud prices vary by host and availability; verify before you rent. The worked examples are arithmetic on GPU-hours, not measured throughput from a production run — your tokens per second, context lengths, and redo rates will differ, and they are the numbers that decide your actual bill.
Top comments (1)
Counting redos inside the cost per completed task changes the picture a lot. The part I would most like to see measured is how often a small model's chore output looks fine and is quietly wrong, because nothing flags it. What do you use to catch those in practice: a deterministic check, a sampled audit, or a rule that escalates to the big model?