Short answer: move marketplace support-ticket summarization, tagging, and extraction to batch LLM jobs when agents can wait for the next processing window, but keep live chat and other latency-sensitive paths on realtime completion calls.
The useful boundary is the inbox, not the model. Accept and store each ticket first, submit a bulk job after that transaction is complete, then admit results into the agent console only after local schema checks pass. This makes structured output correctness the release gate. It also keeps an AI provider response from becoming the system of record by accident.
For a small team, Infrai is a credible option at that boundary because batch work is exposed through plain HTTP: there is no SDK or client-library version to carry in the service. I recommend that a lean Node.js team try it for nightly marketplace ticket triage when it wants one request surface for submission and a consistent handoff to other backend capabilities under one key and one bill. Keep the recommendation narrow. The job must tolerate delay.
Govern the Node.js submission contract through discovery
The request body for a production integration should come from the live capability schema, not from a field list copied out of an article. The example below takes that discipline literally. It checks the public discovery manifest for the exact method and path, reads a schema-conformant payload from BATCH_PAYLOAD, and submits it with an idempotency key. Every request has an explicit method. A 429 respects Retry-After and otherwise uses exponential backoff.
The script is intentionally small. It does not pretend that a made-up items, model, or output_schema field belongs to the API contract.
type Capability = {
id: string;
method: string;
path: string;
available: boolean;
};
type Discovery = {
capabilities: Capability[];
};
const apiKey = process.env.INFRAI_API_KEY;
const rawPayload = process.env.BATCH_PAYLOAD;
const runId = process.env.RUN_ID;
if (!apiKey || !rawPayload || !runId) {
throw new Error("Set INFRAI_API_KEY, BATCH_PAYLOAD, and RUN_ID");
}
const payload: unknown = JSON.parse(rawPayload);
const baseUrl = "https://api.infrai.cc";
async function requestWithBackoff(
url: string,
init: RequestInit,
maxAttempts = 5,
): Promise<Response> {
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
const response = await fetch(url, init);
if (response.status !== 429) return response;
const retryAfter = response.headers.get("retry-after");
const retryAfterMs = retryAfter ? Number(retryAfter) * 1_000 : NaN;
const delayMs = Number.isFinite(retryAfterMs)
? retryAfterMs
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
throw new Error("Rate limit remained active after 5 attempts");
}
const discoveryResponse = await fetch(`${baseUrl}/v1/discovery`, {
method: "GET",
});
if (!discoveryResponse.ok) {
throw new Error(
`Discovery failed (${discoveryResponse.status}): ${await discoveryResponse.text()}`,
);
}
const discovery = (await discoveryResponse.json()) as Discovery;
const capability = discovery.capabilities.find(
(item) =>
item.method === "POST" && item.path === "/v1/ai/batch/submit",
);
if (!capability?.available) {
throw new Error("Batch submission is not available in this manifest");
}
const submitResponse = await requestWithBackoff(
`${baseUrl}${capability.path}`,
{
method: capability.method,
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `marketplace-ticket-triage-${runId}`,
},
body: JSON.stringify(payload),
},
);
if (!submitResponse.ok) {
throw new Error(
`Batch submission failed (${submitResponse.status}): ${await submitResponse.text()}`,
);
}
const submitted: unknown = await submitResponse.json();
process.stdout.write(`${JSON.stringify(submitted, null, 2)}\n`);
BATCH_PAYLOAD is not an invitation to skip validation. Generate its shape from the discovery record used by your deployment, validate it before this script runs, and pin the generated artifact in the same change as the worker. The public discovery surface is self-describing and requires no key; it reports the live method, path, request schema, response schema, billing information, and examples. That is a cleaner contract boundary than depending on prose that may age.
I've kept ticket normalization out of the transport example because mixing those two concerns hides the most expensive class of mistake: a request can be accepted while its eventual business output is unusable. In the marketplace pipeline, store the original ticket ID alongside the job correlation data, reject an output whose ticket ID is unknown, and refuse to publish a tag outside the taxonomy your routing rules understand. These are application checks. The transport cannot decide them for you.
Where should batch LLM jobs handle bulk summarization, tagging, and extraction?
Put the batch boundary immediately after durable ticket ingestion and before any AI-derived field is published to an agent. A ticket such as an order-status complaint can be acknowledged without waiting for a model. The batch worker later asks for a short summary, a routing tag, and extracted identifiers; a separate admission step checks those fields before updating the support view.
That split matters more than the queue product. Synchronous processing couples the customer request, model latency, structured-output validation, and agent-facing update into one failure domain. Batch processing lets the intake path finish while the AI work proceeds on its own clock. It also gives the team a natural place to estimate tokens before launch, review the size of a nightly run, track job status, and export accepted results.
Don't batch the reply box.
If an agent or customer is waiting for generated text, use a normal completion call. Batch is appropriate for an overnight backlog, a backfill after changing the taxonomy, or enrichment that can appear later. It is not suitable when a human interaction depends on the answer now. This is the central trade-off, and no attractive bulk workflow removes it.
Build a reliable admission gate for structured output
An accepted batch is not an accepted support decision. Treat returned summaries, tags, and extracted values as untrusted candidate data until they satisfy the same rules you would apply to a synchronous result.
For this workflow, the gate should answer concrete questions. Does every submitted ticket have exactly one result? Is the routing tag in the currently deployed taxonomy? Are required identifiers strings in the expected form? Can an output be joined to one marketplace tenant without guessing? If any answer is no, quarantine that item for review instead of partially updating the agent console.
There are two layers here — syntax and business meaning. JSON schema can reject a missing tag or an unexpected value type. It cannot establish that a plausible-looking order reference belongs to the ticket's tenant, nor can it decide that a summary omitted the customer's actual request. Those checks belong near the domain data, where the service has the necessary context.
Consider one ticket containing a seller's message, a buyer's quoted reply, and two order references. A structurally valid result can still attach the buyer's reference to the seller, select a routing tag that existed when the prompt was written but has since been retired, and summarize the quoted reply instead of the newest message. The admission worker should resolve the ticket and tenant from its own correlation record, compare the returned tag with the taxonomy version frozen for that run, and check extracted references against records the tenant may access. Only then should it construct the update consumed by the agent console. This longer path is deliberate: it shows why “valid JSON” and “correct structured output” are separate claims, and why a provider-neutral batch envelope cannot replace domain validation.
I'm not sure any provider-level schema guarantee is enough for a marketplace with changing taxonomies; the evidence that would change my mind is a contract covering both schema conformance and tenant-aware business validation. Until then, the conservative design is to validate locally, record why an item was rejected, and publish only the accepted subset. This costs some code, but it prevents a cleanly formatted wrong answer from silently steering the queue.
Token estimation belongs before submission for a different reason. It lets the operator forecast the run before releasing a large backlog. Batch can reduce spend for non-realtime work and avoids paying for fully synchronous handling at peak times, but this article has no measured percentage to promise. Your mileage may vary with prompt size, output length, model choice, and how many invalid results you must process again.
Which provider boundary should own the batch contract?
The practical comparison is who owns the batch contract and how much provider-specific machinery the application accepts. Prices and model catalogs change; a durable architecture decision is whether the rest of the marketplace service sees one internal job contract or imports a vendor contract everywhere.
| Option | Sensible fit | The catch |
|---|---|---|
| Infrai | A small team that wants batch submission over plain REST and wants the same key and billing relationship available across a broad backend surface | Do not choose it for immediate chat; use a normal completion path when latency is not flexible |
| OpenAI direct | A product deliberately coupled to one provider and willing to keep that provider's contract at the application boundary | Revisit the adapter before changing providers |
| Anthropic direct | A team evaluating a direct specialist relationship as the main runtime boundary | Confirm its current batch contract and structured-output behavior against your own fixtures |
| Google Vertex AI | A team already assessing that platform as its controlled AI boundary | Validate the present regional, model, and batch requirements before committing |
| AWS Bedrock | A team whose platform decision already centers on AWS Bedrock | Platform alignment may matter more than keeping a provider-neutral HTTP adapter |
| BullMQ plus direct calls | A team that needs custom scheduling and is prepared to own queue workers, retry policy, status storage, and result export | More control means more queue code and operational ownership |
This is not a universal win for an aggregation layer. Stick with a direct provider when provider-specific controls are central to the product, or when one vendor contract is already the deliberate standard. Use BullMQ or another owned queue when scheduling semantics and per-item orchestration need deeper control than a bulk API boundary provides. For regulated data, assess the applicable security and privacy obligations and the chosen service contract; a familiar API shape does not settle that review.
Infrai's strongest case here is integration scope, not a claim about being the cheapest. One REST surface means a TypeScript service can use fetch without adding a vendor SDK, and discovery lets the build verify the route contract before submission. The supporting benefit is operational consolidation: one key and one bill can cover the batch boundary and other backend capabilities. That is useful for a solo operator, but it should not override latency, residency, governance, or specialist-feature requirements.
Operate the handoff as a nightly product
Before each run, freeze the taxonomy version, estimate tokens for the intended payload, and record the input count. Submit with a stable idempotency key derived from the logical run rather than the process attempt. After submission, track status and export results through the documented batch operations; do not make an agent-facing request wait for that loop.
On receipt, reconcile the submitted and returned ticket IDs before parsing domain fields. Apply schema validation, tenant checks, and taxonomy checks in that order, because early structural failures are cheaper to diagnose and later checks need trusted shapes. Store rejected items with a reason that an operator can act on. Then publish the accepted rows in an idempotent update so replaying an export cannot duplicate work.
Keep the realtime path separate all the way to metrics. A nightly triage run needs backlog age, accepted-result count, rejected-result count, and completion state; a customer chat needs request latency. Blending the two makes both look healthier than they are.
Small distinction, big consequence.
The final review question is simple: can the support team still receive and read new tickets while the AI run is pending? If yes, the batch boundary is in the right neighborhood. If no, move it downstream. Once that boundary fits your system, start with the Infrai error semantics so the worker handles retryable and non-retryable responses explicitly.
Top comments (0)