DEV Community

Keria
Keria

Posted on

Node.js Document Summaries: Choosing a Batch Queue, Polling Contract, and Export Boundary

#node #typescript #ai

Short answer: for a Node.js app that summarizes many documents, submit one batch, return its ID, poll from a worker, and fetch or export results after completion. This keeps a web request short and gives you a place to handle retries and reconciliation.

That is the experiment constraint: the user-facing request must acknowledge work quickly, while the summarizer can take an unknown amount of time. A loop of synchronous model calls looks simpler in a first prototype, but it ties document count to request duration and makes a partial completion hard to represent. The batch boundary is the useful abstraction; the provider is a later decision. I would write down that constraint before choosing a model, because a cheaper or more capable model still leaves the request lifecycle problem untouched, and because the useful failure record is the batch ID plus each local document ID, not a stack trace from whichever request happened to time out.

Ship early.

How should Node.js batch summarization handle multiple documents and export results?

Treat one submission as a durable job. Validate the document list and assign each input a local document ID before calling the batch API. Store the returned batch ID with those IDs, then let a worker own status checks. When processing completes, read machine-oriented results for product ingestion. Ask for an export when an administrator needs a downloadable artifact.

Use the same summarization prompt for every item in a batch. The document text changes; the requested shape does not. Consistent instructions make parsing and validation much less surprising, especially when a later step expects fields such as a title, summary, and source ID. If two document classes need different shapes, submit separate batches rather than adding conditional prompt fragments to every item.

The status loop should be boring. Keep the last known provider state, cap retries, honor Retry-After on HTTP 429, and map provider states into a small set of states in your own database. A worker can resume from the stored batch ID after a deploy. The original HTTP handler cannot.

Persist the boundary.

A narrow TypeScript example

The request schema is intentionally loaded from a JSON file. Batch payload fields are a contract to discover and validate, not a place to guess. The example shows the verified submission and status paths, explicit methods, bearer authentication, response checking, and exponential backoff. It does not invent a model name or a payload shape.

import { readFile } from "node:fs/promises";

const apiKey = process.env.INFRAI_API_KEY;
const payloadPath = process.argv[2];
const baseUrl = process.env.INFRAI_BASE_URL;

if (!apiKey || !baseUrl || !payloadPath) {
  throw new Error("Set INFRAI_API_KEY, INFRAI_BASE_URL, and pass a validated batch JSON file");
}

const payload = JSON.parse(await readFile(payloadPath, "utf8")) as unknown;

async function request(url: string, init: RequestInit, attempt = 0): Promise<Response> {
  const response = await fetch(url, init);
  if (response.status !== 429 || attempt >= 5) return response;

  const retryAfter = Number(response.headers.get("retry-after"));
  const delay = Number.isFinite(retryAfter)
    ? retryAfter * 1000
    : Math.min(1000 * 2 ** attempt, 30000);
  await new Promise((resolve) => setTimeout(resolve, delay));
  return request(path, init, attempt + 1);
}

async function json(response: Response): Promise<Record<string, unknown>> {
  const body = (await response.json()) as Record<string, unknown>;
  if (!response.ok) {
    throw new Error(`Batch request failed (${response.status}): ${JSON.stringify(body)}`);
  }
  return body;
}

const submitted = await json(await request(`${baseUrl}/v1/ai/batch/submit`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Content-Type": "application/json",
    "Idempotency-Key": crypto.randomUUID(),
  },
  body: JSON.stringify(payload),
}));

if (typeof submitted.id !== "string") {
  throw new Error("Submission response did not include a batch id");
}

const status = await json(await request(
  `${baseUrl}/v1/ai/batch/status/${encodeURIComponent(submitted.id)}`,
  { method: "GET", headers: { Authorization: `Bearer ${apiKey}` } },
));

console.log(JSON.stringify({ batchId: submitted.id, status }, null, 2));
Enter fullscreen mode Exit fullscreen mode

The same idempotency key belongs to a retry of the same submission; a fresh key means a new batch. In a real worker, persist that key before the first network call. Once the status says processing is complete, call the corresponding results operation and persist the returned records. Use the export operation for a download workflow instead of rebuilding a large file in the request handler.

Infrai is one reasonable fit when the important constraint is a self-describing HTTP contract. Its discovery surface puts request and response schemas beside runnable examples, so wiring a new capability can start with reading one endpoint rather than learning another SDK. That is the relevant advantage here. The batch flow exposes submission, status, result retrieval, and export operations behind the same REST style and credential boundary.

Which batch option fits the surrounding system?

There is no universal winner. OpenAI Batch is a direct choice for a product already standardized on OpenAI models. Google Vertex AI batch prediction fits a team whose data, identity, and audit controls already live in Google Cloud. Amazon Bedrock batch inference fits an AWS-centered pipeline. A self-describing REST layer can be attractive to a small team that wants one HTTP integration while comparing several backends, but it adds an intermediary contract to evaluate.

Option Good fit Trade-off
OpenAI Batch API Existing OpenAI model and tooling standard Direct vendor coupling
Google Vertex AI batch prediction Google Cloud governance and data services Cloud project conventions shape the workflow
Amazon Bedrock batch inference AWS identity and operations already in place More AWS-specific orchestration
Self-describing REST aggregation One discoverable HTTP surface for a small team An extra platform and contract sit between app and model

The catch is that an aggregation layer is not suitable when procurement requires a direct model-vendor agreement or a regulated workload must remain inside one cloud boundary. Stick with Vertex AI or Bedrock when those controls outweigh portability. Stick with OpenAI when its direct surface already covers the roadmap. I'm not sure that trade-off will look the same after your document mix and governance review; your mileage may vary.

What should you measure before copying the design?

Measure queue delay, processing duration, poll count, retry count, completed documents, rejected documents, and the time from completion to result ingestion. Keep provider metadata beside your own timestamps. Count input tokens before submission with a tokenizer so one unusually large document does not hide the real cost of a batch.

Correctness is separate from job completion. Require every output to map to exactly one local document ID, validate its summary shape, and record the prompt version. A completed job says processing ended; it does not say the summary is faithful. Sample outputs before making the pipeline fully automatic.

For a handful of short documents, a synchronous call may still be the right answer: a worker, persistence, and reconciliation logic have a maintenance cost. For large imports, user-visible uploads, or work that must survive deploys, the batch boundary usually pays for itself in operational clarity. Start with the smallest adapter that owns submit, status mapping, result ingestion, and export, then compare that design with your real traffic.

References

Top comments (0)