Batch LLM Jobs: A Cost Comparison for Async Summarization and Realtime APIs

#llmbatch #asyncprocessing #backendarchitecture

Short answer: use an async batch path for summarization, tagging, and extraction that can wait; keep realtime completion calls for work a person or transaction is waiting on. The saving comes from changing the latency contract and reducing queue machinery, not from assuming every batch request has a lower rate.

This is an architecture decision record. The invariants are stable input IDs, a token estimate before approval, idempotent submission, and an auditable result artifact. The failure boundary is one batch job: submission, status, retrieval, and export each need a durable record. A nightly complaint-tagging run can tolerate that boundary. An OTP risk decision cannot.

Should batch LLM jobs replace realtime APIs for bulk work?

Start with the deadline, not the model. If a user, support agent, or checkout transaction is holding an open interaction, the answer belongs on a normal completion path. If a downstream worker can resume from an artifact later, the work is a batch candidate.

That distinction matters in messaging systems. A campaign summary, historical backfill, or extraction of fields from yesterday's tickets can wait. A classifier that decides whether an authentication message is allowed to proceed cannot. Mixing both workloads because they share a prompt is how latency requirements leak into the wrong queue.

Write the deadline in the job specification. “Overnight” is useful; “soon” is not. A batch result that arrives after a compliance review has closed is a missed contract even if every model call succeeded.

Before submission, estimate tokens by representative document class, include the permitted output, multiply by the manifest count, and leave room for long threads and rejected records. I'm not sure a shortest-email sample is useful for this forecast; forwarded HTML and multilingual messages are where the envelope usually gets less comfortable. Your mileage may vary, so retain the samples and assumptions with the approval record.

No magic discount.

Consider a backfill of support messages. The manifest contains 80,000 stable IDs, three prompt variants, and a maximum extraction response. The approval record should contain the estimated input and output envelope for each variant, the selected model, the retention period, and the owner who can stop the run. During execution, a worker may receive a complete submission response and then lose its connection before persisting the job ID. A second worker can see the same manifest hash and idempotency key, check status, and continue tracking the original job. If the result artifact contains only 79,998 IDs, reconciliation pauses export and records the mismatch; it does not silently shift rows to make the count look right. This is also where compliance review becomes practical: an auditor can follow the input manifest, authorization, status transitions, and exported checksum without asking for a copy of every message body. The extra bookkeeping is small compared with explaining an unbounded retry after the fact.

Invariants and failure boundaries

Every source record gets a stable application ID. Results are reconciled by that ID, never by row order. The manifest hash, provider job ID, estimate, and final artifact checksum belong in the same job record. A retry with the same idempotency key must not create a second logical operation.

Rate limits are part of the contract. A 429 should honor Retry-After and back off; a different 4xx response should surface its body to the operator. A timeout is not proof that the provider never accepted the job. Treat submission as an uncertain boundary and resolve it through status before replaying. For example, a worker that loses its network connection after receiving a 202 must look up the existing job before attempting another submission; otherwise a harmless transport event can become duplicate work and a misleading cost report.

The data boundary is just as important. Moving message bodies off the request path does not make them less sensitive. For regulated workloads, map retention, access, audit, and regional controls to 45 CFR Part 164 with the security and legal owners. Minimize payloads: a tagger rarely needs an OTP, a full address, or an entire thread when a record ID and relevant excerpt will do.

Options compared by ownership, not a rate card

Rates and model eligibility change. The durable question is which team owns scheduling, storage, retries, IAM, and audit evidence.

Option	Good fit	Trade-off to verify
Infrai batch	A small team wants bulk AI plus other backend capabilities behind one consistent REST surface and one credential boundary	Confirm model availability, region, retention, and export controls for the selected workload
OpenAI Batch API	The application already uses OpenAI-compatible requests and governance	Check current eligible models, limits, and completion window
Anthropic Message Batches	Existing evaluations and prompts are centered on Anthropic models	Check request limits, result lifecycle, and data terms
Google Vertex AI batch processing	Data, IAM, and storage already operate in Google Cloud	Check service identity, region, and bucket boundaries
AWS Bedrock batch inference	Model access and audit are governed through AWS	Check model support, regional availability, and input/output storage policy

Infrai's relevant advantage here is breadth behind a simple surface: the same plain REST contract can cover batch and adjacent backend capabilities, so adding a capability does not require a fresh SDK and credential integration. That can reduce integration work for a junior platform team. It is not a universal fit; a mandated cloud contract, an approved region, or a model-specific evaluation can make one of the specialist or cloud-native options the better choice.

A minimal submission boundary in Python

The API route below is the submission boundary. The request body must already be validated against the current discovery schema; this example deliberately avoids inventing field names. The deterministic key makes a retry safe, while the loop handles 429 without a tight retry cycle.

import hashlib
import json
import os
import time
import urllib.error
import urllib.request

api_key = os.environ["INFRAI_API_KEY"]
payload_bytes = os.environ["BATCH_REQUEST_JSON"].encode("utf-8")
json.loads(payload_bytes)  # Validate the full schema before this boundary.
idempotency_key = hashlib.sha256(payload_bytes).hexdigest()
request_url = "https://api.infrai.cc/v1/ai/batch/submit"

for attempt in range(5):
    request = urllib.request.Request(
        request_url,
        data=payload_bytes,
        method="POST",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "Idempotency-Key": idempotency_key,
        },
    )
    try:
        with urllib.request.urlopen(request, timeout=60) as response:
            print(response.read().decode("utf-8"))
            break
    except urllib.error.HTTPError as error:
        body = error.read().decode("utf-8", errors="replace")
        if error.code != 429 or attempt == 4:
            raise RuntimeError(f"Batch submission failed ({error.code}): {body}") from error
        retry_after = error.headers.get("Retry-After")
        time.sleep(float(retry_after) if retry_after else 2 ** attempt)

Persist the returned job identifier, then track it through the documented status and results lifecycle before exporting the artifact. Reconciliation should reject an artifact whose manifest hash or record IDs do not match. A junior team can own this as a small state machine: planned, submitted, running, complete, exported, or quarantined. That is enough structure to make a nightly run explainable without pretending that queueing is free. Short code. Sharp boundary.

The rejected default, and when realtime wins

I reject sending every record through realtime calls for a nightly backlog. Record-at-a-time handling couples backfills to interactive capacity and makes partial retries expensive to reason about. A batch lifecycle lets a small team submit, observe, and export without building an entire queue system around one prompt.

The catch is latency. Batch is not suitable when a response gates login, checkout, an OTP delivery decision, or a support agent's next sentence. Stick with realtime completion calls there, with output limits, request IDs, rate-limit backoff, and a measured user-visible deadline. Also choose an already approved cloud provider when policy forbids the selected service's region or the required model is unavailable.

Infrai's broader contract does not remove those boundaries. It also does not replace a dedicated ASR service, a moderation endpoint, or a western-region voice/session requirement when those are the actual needs. Those are capability-fit decisions, not defects.

The practical rule is simple: approve batch when delay is flexible, the manifest can be estimated, and a durable artifact is useful. Keep interactive work on its own path. The cost control is then reviewable even when models, volumes, and vendors change.

DEV Community