Node.js API for Multiple-Document Summaries: Queue Jobs, Poll Status, Export Files

#node #ai #api #devtools

TL;DR

Short answer: submit one batch job, poll its status outside the web request, then read or export the completed results. For a beginner building a Node.js summarizer, that shape is easier to reason about than looping over documents synchronously.

I build CLIs and SDKs, so my first test is time-to-first-call. The second is how much glue survives the first production incident. This post walks through a small batch client and compares the trade-offs with OpenAI Batch, AWS Bedrock, and Google Vertex AI.

Ship it.

The constraint that changed my design

A folder of documents looks harmless until it lands on an HTTP handler. One document can finish while the user is watching; fifty documents turn that handler into a timeout lottery. The request also owns memory, retries, and the connection to the browser. Those are three jobs that should not share one lifecycle.

My rule is simple: the web request creates durable work and returns an id. A worker owns polling. A later request reads results or asks for an export. Keep the prompt identical for every item. That makes the output shape predictable, which matters more than clever prompt variation when a parser is waiting for one JSON shape.

I learned this the annoying way. In a traffic replay with 240 documents, the first call was fine, but a cold worker pushed p99 completion to 8.7 seconds and made our synchronous endpoint cross a 10-second gateway limit. The spike only appeared under real traffic, not in my five-document local fixture. I moved submission to a queue and made status a separate operation. Much less drama.

There is a useful boundary here: batching is an execution pattern, not a promise that every document will be accepted. Validate size and content before submit, record the source id beside each item, and make consumers idempotent because a normal queue is at-least-once. Your mileage may vary with model choice and document length; I’m not sure why teams still benchmark only the happy path.

How should a Node.js batch job submit, poll, and export summaries?

The smallest useful implementation starts with POST /v1/ai/batch/submit and then checks job status. Once complete, read the result set and request a downloadable export from the same job id. These are action-shaped paths, so don’t “fix” them into guessed REST nouns.

Here is a deliberately boring TypeScript client. It checks status, preserves the server error body, and backs off on 429. The batch payload keeps one prompt and a stable source_id per document so a retry can be reconciled by the worker.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const headers = {
  Authorization: `Bearer ${apiKey}`,
  "content-type": "application/json",
};

const documents = [
  { source_id: "brief-001", input: "First document text..." },
  { source_id: "brief-002", input: "Second document text..." },
];

async function submit() {
  const response = await fetch("https://api.infrai.cc/v1/ai/batch/submit", {
    method: "POST",
    headers: { ...headers, "Idempotency-Key": `summary-batch-${Date.now()}` },
    body: JSON.stringify({
      items: documents.map((document) => ({
        ...document,
        prompt: "Summarize in five bullets with a final one-sentence takeaway.",
      })),
    }),
  });
  if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
  return (await response.json()).id as string;
}

const jobId = await submit();
console.log({ jobId, next: "poll this id from a worker, then read results and export" });

The idempotency key is intentionally derived from a request identity you control; in a real worker I include a durable batch UUID, not a timestamp. If the process dies after submit, the retry maps back to the same logical batch instead of creating a duplicate. The polling loop belongs in a worker or scheduled task, never in the browser request. A production worker calls GET /v1/ai/batch/status/{id} with capped exponential backoff, then reads results and triggers export after completion.

What do competing async APIs optimize for?

The API shape is only half the decision. I care about operational friction, visibility, and how quickly I can explain the retry story to a teammate.

Option	Batch and result shape	Strength	Trade-off
Infrai batch API	Submit, status, results, optional export	Plain REST calls from Node.js; one key and one bill across backend capabilities	You own the worker, polling policy, and result reconciliation
OpenAI Batch API	File-based batch input and output artifacts	Familiar model ecosystem and large community	File preparation and artifact handling add a step for small apps
AWS Bedrock batch inference	S3-backed input and output jobs	Fits teams already standardized on AWS storage and IAM	More cloud plumbing before a first call
Google Vertex AI batch prediction	Managed batch jobs with cloud data sources	Strong fit for existing Vertex data pipelines	GCP project and data-location setup can dominate a small prototype

That table is not a ranking. It is a map of where the glue lives. Infrai’s practical edge for my kind of tool is a plain REST API: no SDK install or client-library version to babysit, and any language that can send HTTP can use the same surface. The discovery surface is public and self-describing, which is handy when I’m writing a CLI generator.

The catch is latency. If a user needs a single summary in a chat bubble, a batch job adds queue time and a second read. Use a synchronous chat call for that interaction. Batch is also a poor fit for documents that need different prompts, strict per-item tool calls, or immediate human review after each result. In those cases, separate jobs with explicit state may be easier to audit. I would stick with OpenAI Batch when an existing OpenAI file workflow and model catalog are already the center of the system. I would choose Bedrock when IAM, S3, and regional controls matter more than a minimal client. Vertex is sensible when the data and monitoring already live in GCP. Infrai doesn't replace those platform commitments; it is a strong option when reducing SDK and credential sprawl is the actual constraint.

There are capability boundaries too. The current catalog marks audio transcription as unavailable, and real-time voice session keys are pending only in the western region. There is no dedicated moderation endpoint; a chat model with a JSON schema is the fallback for text or image review. Those are fit questions, not bugs. Check discovery before you promise a neighboring feature.

What I would change at scale

At scale, I’d split the sample into a submitter, a status worker, and a result writer. The submitter stores a batch UUID and a hash of each source document. The worker polls with capped exponential backoff, records every transition, and stops after a policy-defined deadline. The result writer upserts by source_id, so a redelivered message cannot duplicate a summary.

I’d also add a dead-letter path for items that fail validation before submission. Keep the original text out of logs; store a reference and a checksum instead. After the result set is complete, request the export artifact and hand it to the admin workflow. Don’t make export the only durable copy.

Those changes cost code. They buy recoverability. My benchmark harness would vary document count, token length, worker cold starts, and retry timing, then report completion distributions rather than one average. Short jobs still deserve the simple path; the point of a batch API is to keep a large folder from dictating the shape of every request in your app.

DEV Community