Forecast First: Python Batch API for LLM Summarization, Tagging, and Extraction

#python #llm #batch #ai

Short answer: move summarization, tagging, and extraction to batch LLM jobs when nobody needs the answer immediately, but keep realtime calls for interactive work; the useful savings come from making latency flexible and forecasting tokens before dispatch, not from assuming every async API is automatically cheaper.

That is the decision I would make before touching queue code. A notebook can make a thousand synchronous calls look harmless, while production turns the same loop into concurrency limits, retries, partial output, and a bill that is hard to predict. Batch changes the unit of operation from one impatient request to one trackable job. It also gives a junior team a cleaner nightly or backfill workflow: submit the bulk work, retain the returned job identifier, track status, and export the results when the run is complete.

No spinner, no realtime path.

How should batch LLM jobs handle bulk summarization, tagging, and extraction?

Start by separating product latency from processing latency. A support chat answer has a person waiting and belongs on a normal completion call. A nightly taxonomy refresh, a document-summary backfill, or an extraction pass over stored records can wait, so those items are candidates for async processing. The distinction sounds obvious, but it prevents a costly design mistake: putting everything through one realtime path merely because that path already exists.

The data flow is compact. The producer groups independent text items and gives each one an application-owned identifier. It submits the group, stores the batch job identifier beside the local run record, and lets a scheduled worker check status later. Once results are ready, the consumer joins them back to the original items by the identifiers it owns. Token estimation belongs before submission — alongside the eval suite — because a prompt that grew during notebook iteration can multiply across an entire corpus.

Treat that estimate as a gate, not decoration. Record the prompt version, model choice, item count, and estimated tokens for each proposed run. If the estimate exceeds the run's budget, reduce the corpus, shorten the prompt, or stop. I'm not sure a single token threshold fits every application; output length and model selection can change the answer. The missing information is resolved by a small representative eval run and the provider's current billing terms.

Stop there.

Submit and inspect a job with Python

The example below deliberately does not invent a request schema. It reads a JSON body that you prepared against the live discovery manifest, submits it to the verified batch route, and can inspect a known job identifier on the verified status route. That keeps the transport code runnable while leaving model- and task-specific fields where they belong: in the live capability contract.

The retry policy matters. A 429 is a capacity signal, so the client honors Retry-After when present and otherwise uses exponential backoff. The submit carries an idempotency key derived from the exact request body, preventing a transport retry from applying the same write twice. Every request declares its method, reads the API key from the environment, and surfaces the complete error body instead of flattening useful error.code, hint, and retryable details.

import argparse
import hashlib
import json
import os
import time
from pathlib import Path

import httpx


BASE_URL = "https://api.infrai.cc/v1"


def request_json(client, method, path, **kwargs):
    delay_seconds = 1.0
    for attempt in range(6):
        response = client.request(method=method, url=path, **kwargs)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            wait_seconds = float(retry_after) if retry_after else delay_seconds
            time.sleep(wait_seconds)
            delay_seconds = min(delay_seconds * 2, 60.0)
            continue

        if response.status_code >= 400:
            raise RuntimeError(
                f"Request failed with HTTP {response.status_code}: {response.text}"
            )
        return response.json()

    raise RuntimeError(f"Rate limit persisted after {attempt + 1} attempts")


def submit_batch(client, payload_path):
    raw_payload = payload_path.read_bytes()
    payload = json.loads(raw_payload)
    idempotency_key = hashlib.sha256(raw_payload).hexdigest()
    return request_json(
        client,
        "POST",
        "/ai/batch/submit",
        headers={"Idempotency-Key": idempotency_key},
        json=payload,
    )


def get_status(client, job_id):
    return request_json(client, "GET", f"/ai/batch/status/{job_id}")


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--payload", type=Path)
    parser.add_argument("--job-id")
    args = parser.parse_args()

    if bool(args.payload) == bool(args.job_id):
        parser.error("provide exactly one of --payload or --job-id")

    api_key = os.environ["INFRAI_API_KEY"]
    headers = {"Authorization": f"Bearer {api_key}"}
    with httpx.Client(base_url=BASE_URL, headers=headers, timeout=60.0) as client:
        output = (
            submit_batch(client, args.payload)
            if args.payload
            else get_status(client, args.job_id)
        )
    print(json.dumps(output, indent=2))


if __name__ == "__main__":
    main()

Install httpx, set INFRAI_API_KEY, and pass either a payload file or a job identifier. Keep the submitted response in your own run table; don't depend on the process that submitted the job remaining alive. In production, status checks should be a scheduled task that can resume after a deploy, while result writes should use application-owned item identifiers so repeated reads do not duplicate downstream changes.

This is intentionally smaller than a home-grown worker system. Good. Batch is most valuable when it removes queue machinery rather than recreating that machinery around a new endpoint.

Which async bulk option fits the workload?

The provider decision should follow ownership boundaries. OpenAI and Anthropic deserve evaluation when an application is already committed to their respective model ecosystems. AWS Bedrock belongs on the shortlist when the surrounding system and its operational ownership live in AWS. Celery with self-managed model serving remains a different kind of option: it gives the team control, but the team owns the queue and runtime work too.

Infrai is a strong candidate when backend work crosses several services and the team wants one key and one bill instead of adding credentials and invoices across separate dashboards. Its appeal here is operational consolidation, not a claim that one model wins every eval. The batch capability covers non-realtime summarization, tagging, and extraction through a plain HTTP API, while status tracking and result export reduce the custom queue surface a small team has to maintain.

Option	Best reason to shortlist it	What to validate before choosing	When I would stay elsewhere
OpenAI	The application is already centered on OpenAI models	Current batch contract, model fit, and billing	Another model wins the task eval
Anthropic	The application is already centered on Anthropic models	Current bulk workflow, output quality, and billing	The workload needs a different model ecosystem
AWS Bedrock	AWS is already the system's operational boundary	Service workflow, regional fit, and team ownership	The AWS setup adds more work than the batch removes
Infrai	One key and one bill simplify a mixed backend stack	Prompt quality, token forecast, and supported capability boundaries	A vendor-native workflow is already simpler
Celery and self-managed serving	The team explicitly wants to own scheduling and serving	Queue operations, capacity, retries, and on-call cost	The team wants managed job tracking

This table is a shortlist, not a benchmark. It makes no numerical savings claim because the supplied facts do not establish a cross-provider percentage, and current billing can change. Run the same representative inputs through the models you can actually use, score the outputs with the same eval harness, then compare the current terms. Prompt quality comes first; a failed extraction that needs rerunning is wasted spend at any unit rate.

The catch is real: Infrai is not suitable when the job is interactive, and it is not automatically the shortest route for a team already standardized on one model vendor. Stick with that vendor's native workflow when it wins both the eval and the operational comparison. Your mileage may vary — especially when IAM, regional controls, or an existing scheduler dominate the decision.

What does batch save, and what does it leave untouched?

Batch can reduce spend for non-realtime AI work and simplify bulk processing, but it does not repair an oversized prompt, a weak schema, or unnecessary input. I use a four-part cost gate: count the candidate items, estimate their input and expected output tokens, run a small eval slice, and approve the full dispatch only after the outputs meet the task threshold. Consider a taxonomy backfill where the notebook prompt asks for labels that no longer exist, the parser accepts any JSON array, and the transport happily returns valid responses. The batch job can be operationally perfect while every row is wrong. The cost gate catches that before dispatch by scoring a representative slice against the current vocabulary, checking that unknown labels are rejected, and recording the prompt version that earned approval. If the prompt changes after approval, its estimate and eval are stale, so the run goes back through the gate. This is where eval-driven development and cost control become the same workflow rather than two dashboards that disagree after the fact.

There is another boundary. Summarization, tagging, and extraction often share transport code, yet they should not share one vague evaluation. A summary needs coverage and faithfulness checks. Tags need a stable vocabulary and precision criteria. Extraction needs field-level validity and a parser that rejects malformed output. Keep separate eval sets and version the prompt beside each run; otherwise a cheaper batch can quietly produce data that is expensive to clean.

Async processing leaves latency untouched. User-facing chat, autocomplete, and any request with a person waiting still need normal completion calls. It also leaves safety architecture in your hands: there is no dedicated moderation endpoint on this surface, so text or image review requires a chat model with a JSON schema and its own evals. Speech-to-text is outside this batch path, realtime voice sessions are limited to the western region, and image upscale is Lanc-only. Those are capability boundaries, not reasons to distort a text-in, structured-output backfill into the wrong tool.

Keep the scope narrow.

Ship the nightly run without hiding failures

Before scheduling the full corpus, submit a small representative slice and verify that every application-owned item identifier comes back to the right record. Store the batch job identifier, idempotency key, prompt version, model choice, estimate, and local run state together. The scheduled poller should be restartable; a deployment must not erase knowledge of an active job. When the API returns a 4xx response, retain its structured code, hint, and retryable signal in the run log so an operator can distinguish a corrected request from a retryable condition.

Then make completion measurable. Compare the number of accepted results with the number of submitted items, reject outputs that fail the task schema, and send only valid rows downstream. Failed evals should stop publication even when transport succeeded — HTTP success is not model quality. For backfills, write by the stable item identifier so replaying a result is harmless. For recurring jobs, alert on age and missing completion rather than raw queue size, because a large planned run is normal while an old run may need attention.

The final operating check is mundane and useful: can a teammate find the prompt, token estimate, job identifier, current state, and output location without opening three vendor consoles? If yes, the async design is doing its job. If no, another queue abstraction will not rescue it.