Originally published at API Cost Lab.
Last updated: October 2026
Every major LLM provider sells the same model at two prices: full price for answers in seconds, roughly half price for answers in hours. The Batch API is the most underused discount in the industry — same model, same quality, ~50% off, in exchange for patience. Here's when it's worth it and when it isn't.
What Batch Mode Actually Is
A batch request is exactly what it sounds like: instead of sending one request and waiting for the response, you submit a batch of requests as a file, the provider processes them when it has spare capacity, and you collect the results later — typically within 24 hours.
Nothing about the model changes. It's the same weights, the same capabilities, the same output quality. The discount exists because you're letting the provider schedule your work during off-peak capacity instead of demanding instant compute. You're selling flexibility; they're buying it back at 50% off.
On GPT-5.6 Sol, the numbers are concrete: batch processing cuts the promo rates in half — $2.00 input / $10.00 output per 1M tokens instead of $4.00/$20.00. (Cached reads in batch presumably follow the same halving, though the headline discount is what providers publish.)
The mechanics are simple across providers: you upload a JSONL file of requests, poll for completion, download the results. Most providers return results well before the 24-hour SLA — but the SLA is what you're signing up for, so plan around the worst case.
The Worked Example: What 50% Off Looks Like
Take a concrete workload: a nightly eval run — 500 test cases, each with 8K input tokens and 2K output tokens, on GPT-5.6 Sol's promo pricing.
Real-time:
- Input: 500 × 8,000 = 4M tokens × $4.00/1M = $16.00
- Output: 500 × 2,000 = 1M tokens × $20.00/1M = $20.00
- Total: $36.00 per nightly run → ~$1,080/month
Batch (50% off):
- Input: 4M × $2.00/1M = $8.00
- Output: 1M × $10.00/1M = $10.00
- Total: $18.00 per run → ~$540/month
Same model, same evals, $540/month saved — for changing when the work runs, not what the work is. And this is a modest example. Teams running large eval suites, backfills, or bulk classification at 10x this volume save thousands monthly for a scheduling change.
Note this stacks with other optimizations. Batch requests with well-structured cached prefixes get both discounts. Batch on the cheaper model tier (say, GPT-6.1 Sol at $2/$10 → $1/$5 batched) compounds further. The teams with the lowest bills layer every applicable discount; batch is usually the easiest layer to add.
The Latency Tradeoff, Honestly
The catch is real: you don't control when results come back. The SLA is typically 24 hours. In practice, providers often return batch results in minutes to a few hours during off-peak times — but "often" isn't a guarantee, and you should architect for the SLA, not the typical.
This kills batch for anything user-facing or interactive. But it also rules out more than people think:
Fits batch: nightly eval suites, dataset labeling and backfills, bulk document processing, embedding generation, offline classification, regression test runs, periodic report generation, training-data preparation. Anything where the consumer of the output is a pipeline, a dashboard, or tomorrow's standup.
Doesn't fit: chat, interactive agents, anything with a human waiting, real-time moderation, request-response APIs, multi-step agent loops (each step would take hours — a 10-step agent becomes a 10-day agent).
The gray area is bigger than it looks. Many "real-time" workloads aren't: that Slack bot summarizing threads? Nobody needs it in seconds. The weekly analytics digest? The support-ticket triage that runs overnight anyway? Audit your async jobs — teams routinely find 30–50% of their API spend is on work that could wait hours with zero user impact.
The Migration Is Smaller Than You Think
The reason teams don't use batch isn't the tradeoff — it's inertia. The migration is genuinely small:
- Collect requests instead of sending them. Where your code currently calls the API in a loop, write the requests to a JSONL file instead.
- Submit and poll. One submit call, one status poll on a schedule (or a webhook where supported).
- Process results the same way. The response format matches the real-time API — your result-handling code barely changes.
Most teams do this in a day for a single pipeline. The pattern generalizes: once one pipeline runs on batch, adding the next is mostly copy-paste.
Two operational notes. First, error handling differs: a malformed request in a batch of 10,000 doesn't fail fast — you find out hours later. Validate requests before submitting. Second, don't batch what you haven't measured: run the workload real-time first, confirm quality, then move it to batch. Debugging quality issues through a 24-hour feedback loop is miserable.
Stacking Discounts: Batch Plus Everything Else
The batch discount composes with every other optimization, and the combination is where the real money is:
- Batch + caching: batch requests with well-structured cached prefixes get both the ~50% batch discount and the ~90% cached-input discount. On GPT-5.6 Sol's promo, a cached input token in a batch request costs a fraction of a cent per million — the input side of the bill effectively disappears.
- Batch + smaller models: batching GPT-6.1 Sol ($2/$10 → $1/$5 batched) for bulk work gets you frontier-adjacent quality at budget-tier prices, halved again. For classification, extraction, and eval workloads, this combination is routinely 70–80% cheaper than the naive real-time flagship setup.
- Batch + off-peak scheduling: submit batches in the evening, collect results by morning standup. The latency is invisible because nobody was waiting.
Work the full stack on the earlier eval example: 500 cases/night on GPT-6.1 Sol batched with decent caching, and the $36/run real-time flagship cost collapses to single digits. That's not a marginal optimization — it's an order-of-magnitude change in what the workload costs to run.
Finding Your Batch Candidates
If you're not sure what qualifies, run this audit over your API usage for a week:
- List every automated caller — cron jobs, pipelines, background workers, scripts. Not user-facing features; machines.
- For each, ask: who reads the output, and when? If the answer is "a dashboard tomorrow," "a data warehouse," or "an engineer next week," it's a batch candidate. If the answer is "a user staring at a spinner," it isn't.
- Estimate the spend per candidate. Even rough numbers work: requests/day × average tokens × rate. Rank by spend, migrate the top two first.
Most teams find their top batch candidates in places they weren't thinking about: the nightly data-quality checks, the weekly report generation, the eval suite someone runs manually every Friday, the backfill script that runs "just this once" every month. The pattern is always the same — work scheduled by habit on the real-time API that never needed to be real-time.
The Bottom Line
The Batch API is a 50% discount for the cost of patience, and most teams are leaving it on the table because their async work is accidentally running on the real-time API. Audit your jobs, find everything that doesn't need an answer in seconds, and move it. It's the easiest large discount in LLM pricing — same model, half the price, one day of engineering.
Top comments (0)