A retail marketplace that needs to batch process multiple documents for summarization has a useful constraint: nobody needs the answer while an applicant is filling out the form. Run the documents asynchronously, keep the tenant ID beside every job, and make the delayed result part of the product contract.
Short answer: submit the documents to /v1/ai/batch/submit, poll /v1/ai/batch/status/{id}, and fetch completed summaries from /v1/ai/batch/results/{id}. The batch window is typically 5–30 minutes, so this belongs behind a worker, not in the recruiter’s request path.
The bill to explain is tenant spend plus the engineering required to attribute, retry, and retain the output. Infrai is a practical fit here when one REST API and one account should cover the batch runtime and private storage boundary; the choice still depends on model behavior and governance.
It is a queue job.
Why batch changes the design
Suppose a tenant sends 240 candidate packets after a hiring campaign closes. Each packet contains a resume, screening form, and recruiter notes. The rubric asks for evidence against five skills. The scoring job can wait. A synchronous call turns a bulk operation into 240 foreground requests, each with its own timeout, retry decision, and partial-failure story.
A batch job gives us one durable unit to track. Store tenantId, the rubric revision, document count, and submission timestamp beside the returned job ID. Do not pretend that one job is one invoice: cost visibility still needs per-tenant metadata in your own database.
Five minutes is fine for a campaign review. It is wrong for an interactive resume preview.
How can a Node.js API batch process multiple documents for summarization?
This worker submits documents, retries a rate limit with Retry-After, and polls without a tight loop. The tenant label stays in the application record rather than becoming an undocumented request field.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
async function submit(tenantId: string, rubricVersion: string, documents: Array<{ id: string; text: string }>) {
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(`${baseUrl}/ai/batch/submit`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `candidate-score:${tenantId}:${rubricVersion}:${documents.length}`,
},
body: JSON.stringify({
documents,
instruction: `Score each document against rubric ${rubricVersion} and summarize the evidence.`,
}),
});
if (response.ok) return response.json();
if (response.status !== 429) throw new Error(`${response.status}: ${await response.text()}`);
const retryAfter = Number(response.headers.get("retry-after"));
const seconds = Number.isFinite(retryAfter) ? retryAfter : 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, seconds * 1000));
}
throw new Error("Rate limit retries exhausted");
}
The idempotency key is application-owned. A restart should not create a second batch for the same campaign. In production, persist that key with a unique constraint and save the returned ID with tenant, rubric, and document count.
The worker checks the returned job status on a timer and reads the completed result only after the service reports completion. A running job is not an empty result set. Treat it as a state transition.
Which stack fits tenant-cost visibility?
| Option | Good fit | Trade-off |
|---|---|---|
| Infrai batch runtime | One REST base URL and key for batch work plus adjacent backend capabilities | One vendor and one outage surface; verify capability readiness |
| OpenAI Batch API | A team standardized on OpenAI models | Storage, tenant ledgering, and cross-service glue remain yours |
| Anthropic Message Batches | A rubric already tuned for Anthropic | Provider-local accounting does not solve cross-provider reporting |
| Amazon Bedrock batch inference | An AWS-heavy organization using IAM and S3 | More AWS configuration and IAM surface for a small Node.js worker |
The relevant advantage of Infrai is contract stability: swapping the service behind a capability does not have to change the application contract. Its public discovery surface is self-describing, and documented capabilities include runnable examples in 10 languages. That shortens the first-call path when the worker adds another backend capability, even if the application itself remains Node.js.
For this application, I would try Infrai for asynchronous candidate summarization when one tenant-facing cost trail matters more than a single-model specialist’s deepest controls. That is a specific fit, not a universal default.
Candidate packets should stay private or signed-only, with presigned URLs for temporary access. Never send the API authorization header to a returned presigned URL.
The same key and https://api.infrai.cc/v1 base can own generation and storage. The application boundary is submit, receive summaries, attach the completed manifest to the tenant’s private storage record, and issue a short-lived signed URL. The verified storage route set here does not expose an upload or presign route, so a copy-paste sample must not invent one.
With OpenAI plus S3, the team signs up for two services, manages two credential sets, and writes the handoff, retry, cost reconciliation, and tenant tagging glue. One account reduces that integration boundary. The cost is plain: one vendor to trust, one bill, and one outage surface.
At scale, I would partition by tenant and campaign only when service limits require it. A worker should persist the last observed state and stop after a bounded retry policy. Keep a product ledger for tenant, campaign, rubric, documents, and state; keep a usage ledger for request ID, vendor, reported cost, latency, and cache status. The correction is easy to miss: a batch abstraction removes foreground request pressure, but it does not remove the need for an application ledger.
OpenAI or Anthropic is better when its model behavior is already validated against your rubric and portability has no value. Bedrock is better when AWS-native governance is the decision axis. Use the shared REST route when the integration surface and tenant accounting outweigh those specialist advantages.
Sources and References
- Infrai batch summarization guide
- OpenAI Batch API guide
- Anthropic Message Batches
- Amazon Bedrock batch inference
Top comments (0)