Short answer: run summarization, tagging, and structured extraction through a deferred LLM API lane only when the completion deadline, data-handling rules, and reconciliation process can tolerate it; compare that lane with realtime processing by cost per accepted result, not by a quoted token discount.
The governing constraint is time. A user waiting for an answer has a latency budget, while a nightly corpus has a settlement deadline, and confusing those two service levels produces either an unnecessarily expensive backlog or an interactive path that misses its promise. The cheaper architecture is therefore workload-specific. A deferred lane earns consideration when work can wait, inputs can be durably identified, outputs can be validated before publication, and operators can prove that every obligation reached one terminal state.
There is a hard limit. Deferred execution is not suitable for an answer that blocks a payment decision, an account-recovery flow, or any other request whose value expires in seconds; keep those operations realtime. It is also unsuitable when policy prohibits retaining an input for the processor's documented job window, or when the team cannot own a queue, replay procedure, and audit trail. In those cases, reduce prompt size, output bounds, or repeated work inside the synchronous design rather than pretending that latency is free.
How should teams compare batch LLM API cost with realtime processing?
Start by defining the unit being purchased. Submitted tokens are easy to invoice, but a business does not consume submitted tokens; it consumes summaries that pass review, tags that conform to a controlled vocabulary, and extracted records that satisfy schema and domain invariants. Let P denote processor charges, Q queue and storage charges, E the amortized engineering and operational cost for the measured period, and A the number of outputs accepted under the same rules. Compare (P + Q + E) / A for deferred and realtime lanes over an identical corpus. A retry, malformed object, duplicate, or superseded result contributes cost but does not increase A.
That denominator changes the discussion. A lower processing tariff may still lose after replay traffic, long-lived staging objects, review labor, and missed deadlines are counted, while a deferred run with a high first-pass acceptance rate may win even though its control plane costs more to operate. I'm not sure a forecast built before observing the corpus can price that tail honestly; a time-bounded shadow run, reconciled against an invoice and an operator-hours record, resolves the uncertainty.
The comparison should preserve the same model identifier, prompt version, output limit, sampling settings, schema, and acceptance rules. Record p50 and p99 completion time, oldest-item age, deadline misses, submitted and accepted tokens where reported, schema rejection rate, duplicate-suppression count, and operator minutes. Don't blend validation failures with transport retries. They have different remedies, and combining them into one failure percentage makes both capacity planning and audit review weaker.
Count accepted work.
The backlog is a ledger, not a folder
Each source object creates a processing obligation. Before submission, persist an immutable source identifier, a content digest, the requested operation, prompt and schema versions, the chosen model identifier, the deadline, and an idempotency key derived from business intent. Attempts belong in a separate table because an attempt number is not identity: retrying the same extraction should produce another attempt against the same obligation, not a new obligation that can settle independently.
Exactly once is an accounting invariant rather than a delivery guarantee. Queues commonly deliver at least once, processes can stop between remote acceptance and local acknowledgement, and polling can repeat after an ambiguous network outcome. The acceptance transaction must therefore validate the output and insert it behind a unique obligation key; if two workers race, one write wins and the other becomes an auditable duplicate. The ledger should balance submitted, running, accepted, rejected, expired, and cancelled items, with every transition timestamped and attributed to a worker or operator. No item should disappear merely because a dashboard only shows the latest attempt.
Consider a hypothetical run of 10,000 documents in which worker w-17 receives the same queue message twice after a 30-second visibility lease. Both deliveries may legitimately poll the same remote job. A unique constraint on (obligation_key, operation_version) prevents two accepted records, while the attempt log retains both deliveries and their correlation identifiers. Treating the second delivery as harmless without recording it would make the final count look right but leave the control environment unable to explain why processor usage exceeded the number of source documents. This is the kind of small discrepancy that becomes expensive during reconciliation.
Keep payloads out of ordinary application logs. Logs need stable identifiers, hashes, transition names, timing, and non-secret configuration fingerprints; controlled storage holds prompts and outputs under an explicit retention rule. Under 45 CFR Part 164, regulated entities and their business associates must evaluate the applicable privacy and security requirements for protected health information, including safeguards and restrictions on use and disclosure. A queue, input object, result object, and replay archive all expand the custody map. Counsel and the designated security function must determine the applicable controls; an architectural note cannot make that determination.
Audit first.
A small Go settlement boundary
The useful code boundary is intentionally narrower than a provider integration. Submission may be vendor-specific, but identity, validation, and conditional acceptance belong to the application. The following Go example shows that division and avoids assuming that a successful poll is equivalent to an accepted business result.
package settlement
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
)
var ErrPending = errors.New("result pending")
type Obligation struct {
SourceID string
ContentDigest string
Operation string
Version string
}
type Client interface {
Submit(context.Context, string, Obligation) (string, error)
Result(context.Context, string) (bool, []byte, error)
}
type Ledger interface {
JobFor(context.Context, string) (string, bool, error)
RecordJob(context.Context, string, string) error
AcceptValidOnce(context.Context, string, []byte) (bool, error)
}
func Key(o Obligation) string {
s := o.SourceID + "\x00" + o.ContentDigest + "\x00" + o.Operation + "\x00" + o.Version
sum := sha256.Sum256([]byte(s))
return hex.EncodeToString(sum[:])
}
func Settle(ctx context.Context, c Client, l Ledger, o Obligation) error {
key := Key(o)
jobID, found, err := l.JobFor(ctx, key)
if err != nil {
return fmt.Errorf("read ledger: %w", err)
}
if !found {
jobID, err = c.Submit(ctx, key, o)
if err != nil {
return fmt.Errorf("submit obligation: %w", err)
}
if err := l.RecordJob(ctx, key, jobID); err != nil {
return fmt.Errorf("record job: %w", err)
}
}
done, output, err := c.Result(ctx, jobID)
if err != nil {
return fmt.Errorf("read result: %w", err)
}
if !done {
return ErrPending
}
_, err = l.AcceptValidOnce(ctx, key, output)
if err != nil {
return fmt.Errorf("accept result: %w", err)
}
return nil
}
AcceptValidOnce should parse the declared output schema, enforce domain invariants, and perform the unique-key write in one appropriate transaction boundary. For payment-adjacent extraction, syntactically valid JSON is insufficient: currency, sign conventions, decimal precision, document identity, and allowed status values may all be acceptance conditions. ErrPending is ordinary scheduling state, so the caller should reschedule it with bounded backoff rather than treat it as an incident. Cancellation deserves its own recorded state because stopping local polling does not prove that remotely accepted work was cancelled.
Node.js can implement the same state machine, but runtime syntax does not alter the settlement invariant. The critical interface is the one between an external completion signal and the application's conditional acceptance transaction.
Operational criteria before choosing a lane
Workload classification comes before product comparison. Interactive assistance, request-time authorization, and user-visible generation normally remain realtime because their latency budgets are short. Nightly summaries, taxonomy tagging, retrospective extraction, evaluation suites, and historical backfills may enter a deferred lane when their deadlines exceed the processor's documented completion objective and the data-processing terms are acceptable. Split unusually large inputs from ordinary ones so a long tail cannot hide inside a comforting average.
The scheduler needs both a monetary authorization and a deadline budget. Stop admitting new work before either is exhausted, but continue reconciling work already accepted by the processor. Alert on backlog age and remaining deadline slack, not merely queue depth: one thousand ten-second items and one thousand hour-long items present the same count and radically different risk. Capacity tests should include duplicate delivery, worker termination after submission, delayed polling, malformed output, schema-version changes, cancellation, and replay. For each test, prove the ledger still balances and that replay cannot create a second accepted result.
Pricing deserves one disciplined measurement rather than a headline claim. Current terms can vary by model, region, input type, completion window, and eligibility; capture a dated price sheet with the decision record, then reconcile the trial against the actual bill. The evidence packet should also contain the corpus manifest, model and prompt versions, acceptance definition, latency distribution, retention configuration, operator time, and rollback threshold. Without those artifacts, a claimed saving is a projection, not an auditable result.
The trade-off is real — deferred processing adds a scheduler, durable state, polling, retention surfaces, replay tooling, and on-call ownership. A small workload with tight deadlines may cost less overall as direct realtime calls even if a deferred tariff is lower. A large, tolerant backlog may justify the control plane because aggregation creates scheduling flexibility and because explicit obligations make cost and correctness measurable. Neither result is universal.
Roll out as a reversible accounting change
Begin in shadow mode: create obligations from production-shaped, appropriately de-identified inputs without sending them, and reconcile their counts against the current path. Next, submit a bounded canary whose outputs cannot publish automatically. Validate every result, compare it with the established acceptance baseline, exercise replay, and account for every terminal and nonterminal item. Expand only while cost authorization, deadline slack, quality thresholds, and data-retention controls remain inside their approved bounds.
Keep an urgent realtime lane. Migration should classify work, not force all traffic through one mechanism.
The decision record can remain compact: workload owner, latency or settlement deadline, data classification, processor retention assumptions, model and prompt versions, expected size distribution, acceptance rules, authorization limit, rollback trigger, and review date. Revisit it after a material change in pricing, model behavior, corpus shape, regulatory scope, or service-level objective. Deferred LLM processing is economically sound only where delayed completion is genuinely acceptable and the organization can reconcile each result; elsewhere, realtime execution is the simpler and more defensible design.
References
- AWS Bedrock official service page: https://aws.amazon.com/bedrock/
- 45 CFR Part 164, Security and Privacy: https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164
Top comments (0)