DEV Community

Libme
Libme

Posted on

LLM Batch API vs Real-Time Calls: When the Batch Discount Is Worth the Wait

Use a batch API when no human is waiting on the result and the job can tolerate hours of latency: backfills, nightly classification, evals, embeddings-style bulk enrichment. Stay on real-time calls for anything interactive or anything where a retry loop needs an answer in seconds. The discount is real, but it only pays off if you also build the bookkeeping that batch jobs demand.

As of late 2026, the major providers (Anthropic's Message Batches, OpenAI's Batch API) advertise a meaningful discount on batched requests in exchange for asynchronous completion, typically within a 24-hour window. Check each provider's current pricing page before you do math with a specific percentage; I'm deliberately not quoting one.

What do you actually give up with a batch API?

Three things, and the discount is only the headline for one of them.

Latency. Results arrive "within a window," not in seconds. Many batches finish much faster than the ceiling, but you can't design around that.

Interactivity. There's no streaming and no mid-job feedback. A prompt bug that would have shown up on request three in a real-time loop shows up after the whole batch returns.

Partial failure semantics. A batch is not all-or-nothing. Each request succeeds, errors, or expires independently, so you need a reconciliation step that real-time scripts get for free from an exception.

The takeaway: a batch discount is a trade of money for latency and operational complexity, and you only win if the second two are cheap for you.

When is the batch discount worth it?

Workload Human waiting? Batch fits? Why
Chat / copilot / agent loop Yes No Latency is the product
Nightly ticket classification No Yes Deadline is "by morning"
Re-scoring 200k old records after a prompt change No Yes One-off bulk, easy to resume
LLM-as-judge eval over a golden set No Yes Results needed once per CI/nightly run
Per-request moderation on user input Yes No Blocks the user action
Summarizing a document on upload Usually Mostly no User expects it in seconds

A rule I use: if the job's deadline is measured in hours and its volume is large enough that the bill shows up as a line item, batch it. If either condition fails, the engineering time to build reconciliation can exceed the savings. At a few thousand requests a month on a small model, you are saving a rounding error and adding a state machine.

The takeaway: batch pays off on large, deadline-in-hours workloads, and costs more in engineering than it saves on small ones.

How do you submit and collect a batch in Python?

Here is the shape using Anthropic's Python SDK (as of late 2026; the method names are stable but check the SDK docs if you're on a much newer version). The part that matters is custom_id: results are not guaranteed to come back in submission order, so you key everything by an ID you control.

import time
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
MODEL = "claude-haiku-5-5"  # pick the cheapest model that passes your eval

tickets = {
    "t-1001": "App crashes when I upload a PNG over 10MB.",
    "t-1002": "How do I change the billing email?",
}

requests = [
    {
        "custom_id": tid,  # your stable key, <=64 chars
        "params": {
            "model": MODEL,
            "max_tokens": 50,
            "messages": [{
                "role": "user",
                "content": f"Classify as bug, billing, or other. One word.\n\n{text}",
            }],
        },
    }
    for tid, text in tickets.items()
]

batch = client.messages.batches.create(requests=requests)
print("submitted", batch.id)

# Poll with a generous interval; batches are not latency-sensitive.
while True:
    batch = client.messages.batches.retrieve(batch.id)
    if batch.processing_status == "ended":
        break
    time.sleep(60)

results, failed = {}, []
for item in client.messages.batches.results(batch.id):
    if item.result.type == "succeeded":
        results[item.custom_id] = item.result.message.content[0].text.strip()
    else:  # errored, canceled, or expired
        failed.append((item.custom_id, item.result.type))

print(results)
print("needs retry:", failed)
Enter fullscreen mode Exit fullscreen mode

In production, persist batch.id and your custom_id list to a database before you start polling. If your process dies at hour six, you want to resume by retrieving the existing batch, not by paying for it twice.

The takeaway: key every batched request by your own custom_id and persist the batch ID, or a crash turns a discount into a double bill.

What are the hidden costs of batching?

Retry accounting. Requests that come back errored or expired need a second pass. Re-submit only those IDs as a new, smaller batch (or fall back to real-time for the stragglers if the deadline is close). What tripped me up the first time was resubmitting the full set because my reconciliation only compared counts, not IDs.

Slower prompt iteration. I now run any new prompt against 20 to 50 real-time calls first, and only batch once the output format is stable. Debugging a malformed-output bug across 100k batched requests is an expensive way to find a missing instruction. If structured output is part of the job, validate against a schema on the way out; the same validate-and-retry discipline applies, just delayed.

Caching interactions. Batch discounts and prompt-caching discounts are separate mechanisms, and whether they stack, and how reliably cache hits occur inside a batch, depends on the provider and changes over time. Check the current docs before assuming both apply; don't build a cost model on the assumption that they multiply.

Rate limits and quotas. Batches typically have their own queue and size limits (requests per batch, total size). Large backfills need chunking, which is one more thing to write.

The takeaway: the real cost of a batch pipeline is the reconciliation and chunking code around it, and you should price that before you chase the discount.

Real-time with a queue vs batch API: which is cheaper to run?

A middle option is to keep real-time calls but run them from a background worker with a concurrency cap. You pay full price, but you get fast partial results, easy retries, and no new state machine. For jobs under a few thousand requests, or where the deadline is tight, this is usually the right call. If you want the discount without writing the polling and reconciliation yourself, the batch endpoints in Anthropic's Message Batches API or OpenAI's Batch API are the managed route, and both return per-request results keyed by an ID you supply. The drawback in both cases is the same: you lose streaming and any guarantee on completion time inside the window.

The takeaway: below a few thousand requests or under a tight deadline, a capped background worker on real-time calls beats batching on total cost of ownership.

FAQ

Is the OpenAI or Anthropic batch API cheaper than normal API calls?
Both providers advertise discounted per-token pricing for batch requests compared with real-time requests, in exchange for asynchronous completion. Check each provider's current pricing page for the exact discount, since it can change.

Do batch API results come back in the same order I sent them?
No. Match results to inputs using the custom_id you supplied on each request, never by position.

What happens if some requests in a batch fail?
The batch still completes, and each request is reported individually as succeeded, errored, canceled, or expired. Collect the failed IDs and resubmit only those.

Bottom line

If you have a large, non-interactive workload with a deadline measured in hours, use a batch API and invest in custom_id bookkeeping and a resubmit path for failures. If you're building anything a user is waiting on, or running a few thousand requests a month, stay real-time and cap concurrency in a worker. Test prompts in real-time first either way; batch is for running a prompt you already trust.

Related reading

Top comments (0)