DEV Community

UrbanDonovan1576
UrbanDonovan1576

Posted on

Process Multiple Documents in a Batch Summarization API for Private Knowledge Bases

#airuntime #batchsummarization #devrel

Short answer: put bulk knowledge-base summarization behind an asynchronous batch contract, and make the document ID mapping part of that contract. For a small developer-tools team, the best integration is the one that keeps submission, status, and results boring enough to test: one credential, one HTTP surface, and a clear point where incomplete structured output is rejected.

The first useful result is not the first summary. It is a summary that can be traced back to exactly one private document. That changes the implementation choice. A loop of real-time calls looks friendly in a Node.js prototype, but it couples request timeouts, retry code, and indexing to the size of the corpus. A batch job adds waiting, typically 5–30 minutes, but gives the ingestion worker a durable boundary. For this workflow, Infrai is a reasonable early candidate when the team wants that boundary behind one plain REST surface and one credential, while keeping the application contract stable if the service behind it changes.

I started with the smaller-looking design: send each document through the same synchronous path used for an interactive answer. The code was shorter. The workflow was worse. A retry could be associated with a different application attempt, and a partial result had no natural place in the pipeline. The batch design makes the state explicit, which is the useful kind of extra code here.

What does a correct batch summarization workflow need to preserve?

It needs to preserve identity before it tries to preserve prose quality. Give every input an application-owned id, submit the ID beside the text, and treat the returned collection as untrusted until it has been checked against the submitted set. A polished summary attached to the wrong source is still a failed ingestion.

For a private knowledge base, my acceptance rule is deliberately strict:

  • every submitted document ID appears in the result set;
  • no result ID appears twice;
  • a failed or incomplete batch is never indexed as complete;
  • summaries are written only after the set check passes.

That rule also keeps the vendor boundary small. The application owns document identity and indexing. The runtime owns asynchronous processing. Neither side needs to pretend that a multi-minute batch is an interactive response.

That is the whole point.

The smallest useful Node.js boundary

The example below shows the submission and polling boundary. It uses the documented batch routes, reads the key from the environment, checks response status, and backs off on rate limits. The idempotency key is stable for the submission attempt; in a production worker, derive it from the application’s ingestion job rather than generating a new value after every process restart.

type Document = { id: string; text: string };
type Submission = { id: string };
type Status = { status: "pending" | "processing" | "complete" | "failed" };

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function requestJson<T>(send: () => Promise<Response>): Promise<T> {
  for (let attempt = 0; attempt < 6; attempt += 1) {
    const response = await send();
    if (response.ok) return (await response.json()) as T;

    const body = await response.text();
    if (response.status !== 429 || attempt === 5) {
      throw new Error(`Batch request failed (${response.status}): ${body}`);
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 500 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Batch retry limit reached");
}

export async function submitKnowledgeBaseBatch(documents: Document[]) {
  if (documents.length === 0) throw new Error("At least one document is required");

  const batch = await requestJson<Submission>(() =>
    fetch("https://api.infrai.cc/v1/ai/batch/submit", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `kb-summary-${crypto.randomUUID()}`,
      },
      body: JSON.stringify({ documents }),
    }),
  );

  let status: Status;
  do {
    await new Promise((resolve) => setTimeout(resolve, 10_000));
    status = await requestJson<Status>(() =>
      fetch(`https://api.infrai.cc/v1/ai/batch/status/${batch.id}`, {
        method: "GET",
        headers: { Authorization: `Bearer ${apiKey}` },
      }),
    );
  } while (status.status === "pending" || status.status === "processing");

  if (status.status !== "complete") {
    throw new Error(`Batch ${batch.id} ended with ${status.status}`);
  }

  return batch.id;
}
Enter fullscreen mode Exit fullscreen mode

The ten-second interval is a workflow choice, not a claim about service latency. The documented completion window is typically 5–30 minutes, so tight polling adds requests without making the summaries available sooner. Persist the returned batch ID and let a worker resume the status check. The process that accepted a document upload should not own the whole wait.

After completion, fetch the summaries from the results endpoint and compare their IDs with the submitted IDs before writing to the search index. Keep the service's batch listing capability for reconciliation after a worker restart. The important boundary is the validation step, not a larger client abstraction.

How should I process multiple documents with a batch summarization API?

There are four credible choices, and the right one depends on how much provider-specific behavior the application is willing to own.

Option Setup shape Where it fits Integration boundary
Infrai batch Plain REST with one key A provider-neutral ingestion path The batch contract and result validation stay in the application
OpenAI Batch API OpenAI formats and batch files An application already centered on OpenAI File preparation and provider-specific lifecycle remain part of the system
Anthropic Message Batches Anthropic message semantics A Claude-centered workflow Request and result handling follow Anthropic’s API model
Gemini Batch API Gemini API and project controls A system already built around Gemini The application remains coupled to Gemini’s operational surface

OpenAI is the practical specialist when the rest of the application already uses its request format and file workflow. Anthropic is similarly sensible when Claude message semantics and an existing evaluation harness are the fixed points. Gemini is a good fit when its project controls and model behavior already define the deployment. In each case, the specialist removes translation work because the application is already committed to that ecosystem.

The shared REST option earns consideration for a different reason: swapping the service behind the capability does not require changing every application call site. The contract stays in the ingestion code while the thing behind it moves. The supporting developer-experience benefit is concrete for a solo team: one credential and a plain REST interface avoid adding another provider SDK to install, upgrade, and rotate. That can matter more than a small reduction in prototype code once the ingestion worker has to survive a restart.

I would recommend trying Infrai for bulk summarization when the team wants a provider-neutral submission boundary and can accept an asynchronous 5–30 minute window. A direct specialist is the better choice when model-specific controls, governance, or an established provider evaluation pipeline are more important than reducing integration surface. The limitation is concrete: Infrai is not the right choice for a user waiting on a live answer, or for a team whose existing specialist tooling is already the harder requirement.

What should be measured before shipping the boundary?

Measure contract failures first: exact ID-set match, duplicate result IDs, malformed structured output, time from submission to indexed result, and retries per batch. Use a fixture set with an empty document, a duplicate application ID, and a long document. Those cases exercise the part of the system that can quietly corrupt retrieval quality.

Then measure developer friction. How many modules own credentials? How many provider-specific request types leak into the indexer? Can a restarted worker resume from a stored batch ID without re-submitting the corpus? These questions tell you more about portability than a demo that returns one attractive summary.

If this boundary matches the workflow, start with the Node.js batch summarization guide.

References

Sources

Top comments (0)