The answer is conditional: move summarization, tagging, and extraction into a batch lane only when the work can wait, inputs can be replayed safely, and your measured all-in cost is lower than the realtime path. Keep interactive requests realtime. A provider's advertised discount is an input to that decision, not the decision itself.
That split sounds obvious until a notebook becomes a service. In a notebook, a list of prompts and a loop feel like a batch. In production, the useful distinction is architectural: the caller hands durable work to a queue, receives a job identity, and can disappear while workers process it. Results return to durable storage and pass validation before another system consumes them. If the client must hold the connection or keep process memory alive, it's still a long synchronous request wearing a bulk label.
The practical comparison is therefore wider than token price. It includes waiting time, retries, storage, orchestration, evaluation, and the cost of sending work twice.
Should batch LLM jobs replace realtime API calls for async summarization and extraction?
Use two lanes. A realtime lane serves chat turns, inline writing assistance, and any request whose value falls sharply after a few seconds. A deferred lane handles a nightly document summary, catalog tagging, or structured extraction from an import. The same model may sit behind both lanes; the service contract is different.
Batching earns consideration because deferred work gives a runtime scheduling freedom. It can admit a collection, process it away from an interactive request path, and expose results later. That freedom may be reflected in a different price, but there is no trustworthy universal savings percentage. I'm not sure one could exist: model choice, input-to-output ratio, retry behavior, and the surrounding storage system all change the result. Your mileage may vary — quite a lot.
Start with a latency budget, not a discount. Write down the latest useful completion time for each workload. A support reply needed before an agent sends a message belongs in the realtime lane. Summaries used in tomorrow morning's search index probably don't. A backfill of historical records is an even clearer batch candidate because no person is waiting for row 40,001.
Then define correctness. Summarization needs an evaluation set that checks factual coverage and unsupported claims. Tagging needs a stable label taxonomy plus per-label precision and recall. Extraction needs schema validation, field-level accuracy, and an explicit policy for absent values. A cheaper run that fails the acceptance test is wasted spend. This is where an eval-driven workflow changes the cost conversation: compare the least expensive configurations that clear the same quality bar, rather than comparing invoices from outputs of unequal quality.
The catch is latency and operational surface area. Batch is not suitable when a user is actively waiting, when every record requires immediate human feedback, or when the team cannot operate durable job state. Stick with realtime calls for those cases. A small queue also may not justify a second execution path; fewer moving parts can be worth more than a modest unit-price difference.
Put a cost gate in front of submission
Before wiring a provider, make the estimate executable. The following Python program compares two caller-supplied scenarios. It does not encode a vendor price or promise a discount. Feed it the prices and observed token counts that apply to your chosen API, then add the non-model costs your system actually incurs.
from dataclasses import dataclass
from decimal import Decimal
MILLION = Decimal("1000000")
@dataclass(frozen=True)
class RunPlan:
records: int
input_tokens_per_record: int
output_tokens_per_record: int
input_price_per_million: Decimal
output_price_per_million: Decimal
duplicate_fraction: Decimal = Decimal("0")
fixed_pipeline_cost: Decimal = Decimal("0")
def estimated_cost(self) -> Decimal:
multiplier = Decimal("1") + self.duplicate_fraction
input_tokens = Decimal(self.records * self.input_tokens_per_record)
output_tokens = Decimal(self.records * self.output_tokens_per_record)
model_cost = multiplier * (
input_tokens * self.input_price_per_million / MILLION
+ output_tokens * self.output_price_per_million / MILLION
)
return model_cost + self.fixed_pipeline_cost
def choose_lane(realtime: RunPlan, deferred: RunPlan) -> str:
realtime_cost = realtime.estimated_cost()
deferred_cost = deferred.estimated_cost()
if deferred_cost < realtime_cost:
return f"deferred: {deferred_cost:.4f} vs realtime: {realtime_cost:.4f}"
return f"realtime: {realtime_cost:.4f} vs deferred: {deferred_cost:.4f}"
Keep the inputs beside the eval result and the prompt version. Token averages from one notebook sample can drift when production documents are longer, when a prompt adds examples, or when extraction output expands. Record the distribution, not just the mean (p50 and p95 are a useful start), and rerun the estimate on the exact candidate dataset before a large submission.
Duplicate fraction deserves its own field because retries can erase the expected advantage. Suppose a client times out after sending a job but before recording the returned identity. Blindly submitting again can pay for the same work twice. The fix is a stable submission identity derived from the input manifest and operation version, plus a local state transition committed before the collector begins. Don't treat a timeout as proof that no work was accepted.
Fixed pipeline cost should include object storage, queue operations, result ingestion, and engineering overhead where those are material. The calculator deliberately leaves them as a single measured input. Splitting that input into a dozen guessed line items only creates false precision.
No magic here.
Make the job ledger the source of truth
A production batch needs an immutable manifest. Each row should carry a stable record ID, a source-version ID, and the operation version that names the prompt, schema, model configuration, and evaluator. Store a hash of the complete manifest. The submission record then links that hash to a provider job identity and a local state such as prepared, submitted, collecting, validated, or published.
This state machine prevents a common notebook-to-prod failure: equating “the API finished” with “the dataset is ready.” A completed job can still produce a missing row, an unknown ID, malformed structured output, or an answer below the quality threshold. The collector should join every output to the manifest, reject duplicates, quarantine invalid records, and report missing IDs. Only the validated set advances to publication. Keep transport retries separate from semantic retries: a transport retry repeats an operation because delivery is uncertain and should preserve the same idempotency identity, while a semantic retry changes something meaningful, such as the prompt or schema, and therefore needs a new operation version. Mixing the two makes both audit logs and cost reports hard to interpret. For bulk summarization, partition by a stable source boundary rather than whatever happens to fit in memory. For tagging and extraction, partition sizes should also respect downstream transaction limits, because a huge model job followed by a fragile one-shot database import merely moves the failure point. Small replayable partitions cap the cost of recovery — and make eval regressions easier to isolate. Imagine the collector stopping after it has downloaded results but before it has marked them as validated: on restart, the ledger should direct it to validate the existing artifact, not submit the source again. If that choice depends on an engineer remembering what happened, job state is not durable enough.
One record, one identity.
Privacy constraints can rule the architecture. If a workload handles regulated health information, review data access, storage, transmission, retention, and vendor relationships against the HIPAA Security and Privacy Rules in 45 CFR Part 164. A deferred pipeline often creates extra durable copies of prompts and outputs, so retention cannot be an afterthought. This article can't determine compliance for a particular system; that requires the actual data flow, controls, agreements, and legal context.
Compare runtimes with a replay, not a brochure
Run a shadow test on an already-reviewed slice. Send the same frozen inputs through the realtime and deferred designs, validate both against the same evaluator, and compare the full ledger: accepted records, total input and output tokens, duplicate work, invalid outputs, elapsed time, and all-in cost. The API invoice matters. So does the operator time needed to explain a missing result.
One provider page, such as the official Amazon Bedrock page, can establish which platform and model options are currently offered. It cannot establish that a particular application will save money. That claim needs the workload replay above, using current terms and the team's own traffic shape. Recheck provider documentation before implementation because service availability and commercial terms can change.
The Node.js version of this architecture is the same even though the example here is Python: persist a manifest, submit outside the request handler, store the returned identity, poll from a scheduled worker, and make result ingestion idempotent. Don't keep a promise chain open for hours. Language choice does not remove the need for durable ownership.
Rollout should be intentionally dull. Begin with one replayable partition, verify its manifest hash, run the evaluator, and publish only after reconciliation. Restart the collector after output download but before publication; it should resume without another model submission or duplicate writes. Then increase partition size while watching token variance, evaluation failures, duplicate IDs, missing IDs, queue age, and reconciliation lag. Keep urgent work on the realtime lane throughout.
The operational checklist is short in wording but strict in practice: every run has a frozen input manifest, an operation version, a cost estimate, a quality threshold, a stable submission identity, a persisted job identity, reconciled outputs, and a retention decision. An operator should be able to answer which inputs ran, which outputs were accepted, and what a retry will do without reading transient logs. If any answer is unclear, the bulk path isn't ready.
Ship the ledger first.
Top comments (0)