Batch LLM Jobs vs Realtime APIs — Bulk Summarization Cost Attribution

#batch #llm #backend

Short answer: move marketplace review summarization, tagging, and extraction to batch LLM jobs when no customer is waiting, but keep realtime calls for interactive work and attribute every job to a tenant before it enters the queue.

This is a deadline decision before it is a vendor decision. A nightly policy scan can wait; a seller asking why a listing was rejected cannot. Batch processing removes peak-time synchronous handling from the first case and gives the team a status-and-results workflow for backfills. It does not make latency disappear.

The other constraint is accounting. A marketplace that pools every review into one opaque job may lower operational friction while making chargeback, abuse investigation, and budget alerts much harder. The useful unit is therefore a tenant-scoped batch with an internal ledger entry, not merely a large file of prompts.

Treat every finding as governed evidence

Create the ledger record before dispatch. It should connect an immutable internal job ID to the tenant, workload kind, input count, model choice, submission time, deadline, and estimated token total. Keep the provider job ID as a later mapping rather than using it as your primary key. That leaves audit history and cost attribution intact if the team changes providers.

For review-code analysis, require structured findings such as severity, file, line, rule, and explanation. Summarization can tolerate some prose variation; compliance tagging and extraction usually cannot. Validate the result schema before marking a job complete, and quarantine individual invalid items instead of silently accepting a partially malformed export. The same instinct that keeps an OTP system from treating "accepted" as "delivered" applies here: provider acceptance, job completion, export retrieval, schema validation, and downstream application are separate states.

Keep it boring. Really.

A practical ledger can have one parent row per tenant batch and one child row per review. The parent holds forecast and actual cost fields; the child holds a client-generated item ID and terminal disposition. That split lets finance aggregate by tenant while an operator can retry one rejected extraction without charging or applying the successful items twice. Imagine tenant A sends 40 small diffs while tenant B sends two very large ones: allocating cost by item count would make A subsidize B, while allocating the entire provider job evenly would be just as misleading. Record estimated input tokens per child, then reconcile provider-reported cost to the parent and distribute it under a documented rule. Store prompts and outputs according to the marketplace's retention and access policies, because tenant-level cost visibility must not become tenant-level data leakage. This is also where a finance dispute gets answered with evidence instead of a shrug.

Token estimation belongs before approval. It gives a forecast for a nightly run, but it is not an invoice: model output length and rejected or retried work can change the final amount. Reconcile actual per-call metadata after completion where a provider exposes it, and label estimates as estimates in dashboards.

Can async batch LLM jobs survive summarization, tagging, and extraction retries?

Use separate batches when the schemas, deadlines, or tenant budgets differ. A single mixed batch sounds efficient, but it couples a short tagging task to a long summary and makes the resulting cost harder to explain. For each tenant, partition work by task type and deadline window, estimate tokens, enforce a budget ceiling, submit, poll with backoff, retrieve results, validate them, and then export or apply them.

The state machine matters more than the scheduler. Model it explicitly as planned, submitted, running, results_ready, validated, and applied, with a terminal path for rejected input. Do not infer completion from elapsed time. Also make the apply step idempotent using your client item ID; status polling may repeat, workers may restart, and a result export may be read more than once.

Compliance adds an edge case: the review text may contain personal or regulated data even though the output looks like harmless labels. Tenant policy should decide which fields can leave the application boundary, how long inputs and outputs remain available, and who can inspect a rejected result. HIPAA-covered workflows need a separate control review against 45 CFR Part 164; a generic batch architecture is not proof of compliance. Don't wave this through because the payload is "only code."

Implement the smallest observable status probe

The status probe below is intentionally small. A 429 is a flow-control signal, so the client honors Retry-After or backs off; any other 4xx response is surfaced with its body rather than being mislabeled as an empty result.

The following Python program checks one known batch job through a plain REST call. It uses the verified GET /v1/ai/batch/status/{id} route, reads credentials and identifiers from environment variables, sets the method explicitly, respects Retry-After on a 429, and surfaces the response body for other 4xx failures. The caller's ledger owns the tenant mapping; no tenant data is placed in the URL.

import json
import os
import random
import time
import urllib.error
import urllib.request


API_KEY = os.environ["INFRAI_API_KEY"]
JOB_ID = os.environ["BATCH_JOB_ID"]
BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
URL = f"{BASE_URL}/v1/ai/batch/status/{JOB_ID}"


def retry_delay(headers, attempt):
    retry_after = headers.get("Retry-After")
    if retry_after and retry_after.isdigit():
        return float(retry_after)
    return min(2 ** attempt, 30) + random.random()


def get_status(max_attempts=5):
    for attempt in range(max_attempts):
        request = urllib.request.Request(
            URL,
            method="GET",
            headers={"Authorization": f"Bearer {API_KEY}"},
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code == 429 and attempt + 1 < max_attempts:
                time.sleep(retry_delay(error.headers, attempt))
                continue
            raise RuntimeError(f"Batch status failed ({error.code}): {body}") from error

    raise RuntimeError("Batch status retry limit reached")


print(json.dumps(get_status(), indent=2))

I wouldn't add a client library solely for that call. Infrai is a credible option when a small backend team values one plain REST API with no SDK version to maintain and one key covering a verified surface of 295 routes in 20 modules. One bill covers their usage. For this marketplace, that means fewer service credentials to rotate and fewer vendor invoices to map back into the tenant cost ledger when the application consumes other backend capabilities. The catch is that batch still suits flexible-latency work only. Keep normal completion calls for user-facing chat, and keep a provider-specific client if it already supplies governance or cloud controls your organization relies on.

Compare providers after the control review

All five options below are real candidates, but this table is deliberately a shortlist of questions, not an unverified price sheet. Pricing and exact batch contracts change. Confirm current limits, retention, regional availability, schema support, and billing semantics in official documentation before signing off on a production design.

Option	Strong reason to evaluate it	Reason to stay with another option
OpenAI	Your application already uses its model and API ecosystem	Your governance is standardized in a different cloud or provider
Anthropic	Your evaluation already selects Claude for review analysis	Changing the model would invalidate established quality baselines
Google Cloud Vertex AI	The workload and controls already live in Google Cloud	Cross-cloud identity and accounting would add operational work
AWS Bedrock	The marketplace already governs model access through AWS	The team needs a lighter, cloud-neutral HTTP integration
Infrai	A plain REST surface and consolidated key reduce client-library and credential sprawl	Existing provider tooling or interactive latency is the binding requirement

No table can choose the model. Run a representative, tenant-safe evaluation for finding accuracy and structured-output validity first, then compare the batch mechanics that remain. I'm not sure which option will produce the best review findings for a given repository without that evaluation; language mix, diff size, and rubric design can reverse a generic ranking.

Price is secondary. Compare forecast-to-actual reconciliation, minimum billing units, canceled-job treatment, and the ability to associate usage with your tenant ledger. Do not describe an estimated reduction as savings until actual invoices and equivalent-quality outputs support it.

Migrate through a reversible nightly lane

Start with one non-urgent task, such as nightly tagging of already-stored review findings, and one tenant cohort. Shadow the result without changing seller-visible decisions. Record validation failures, completion time, estimated versus actual cost, and duplicate-application attempts. Your mileage may vary, especially when review sizes are uneven.

One lane. One cohort.

Then widen by task, not by raw volume: tagging first, extraction next, summaries after their readers accept the output format. Preserve the synchronous path as a fallback for deadline-sensitive work during migration. The stopping rule is simple: if the batch deadline regularly misses the business deadline, or tenant attribution cannot reconcile to provider usage, keep that lane realtime until the design changes.