TL;DR: To process multiple private fintech documents with a batch summarization API, use an asynchronous job when summaries can wait several minutes. Pick the provider whose request format, model access, and result handling you can replace without rewriting your ingestion pipeline. A plain REST option such as Infrai is a reasonable fit when portability means keeping the client small: there is no SDK or client-library version to maintain, and any language that can send HTTP can submit the work.
| Option | Best fit | Portability trade-off |
|---|---|---|
| OpenAI Batch API | A stack already centered on OpenAI models | The batch contract and model routing are OpenAI-specific |
| Anthropic Message Batches | Workloads built around Claude | The message schema and model catalog are Anthropic-specific |
| Google Gemini Batch API | Applications already using Gemini | The request and response shape follows Gemini's API |
| A REST batch gateway | A small service that may change model vendors | One more abstraction to evaluate, but the client can stay HTTP-based |
My decision rule is simple: use the least complex option that preserves a real exit. For a weekly-shipping one-person SaaS, that usually means storing documents and job state behind an internal interface, then choosing a batch provider beneath it. Keep the provider-specific code in one adapter. That is a few hours of work now, and it prevents a model-provider migration from becoming a product rewrite later.
How should I process multiple documents with a batch summarization API?
The important distinction is not “which model writes the nicest paragraph?” It is “what remains stable when the model changes?” For private fintech material, the stable parts should be document identity, access policy, prompt version, and summary status. The provider should own the temporary execution details.
That is the whole point.
Batch processing also changes the product contract. A user who needs an answer while looking at a transaction cannot wait for a bulk job. A nightly refresh of several hundred internal documents can. Put those paths behind different queues and make the asynchronous one visible in the UI: submitted, processing, complete, or failed.
The batch workflow in this comparison has three phases: submit an array of documents, poll the job, and retrieve the completed summaries. The observed completion window in the supplied API contract is typically 5–30 minutes. That is an operational fact worth exposing to users, not hiding behind a spinner.
The two criteria that matter most
First, preserve a provider-neutral document envelope. A useful record has an internal documentId, a content hash, a source revision, and a prompt version. Send only the text needed for summarization. Keep customer identifiers and account numbers out of prompts when the summary does not require them. The batch API should return a job identifier, while your database remains the source of truth for which revision was submitted.
Second, make retries boring. A network timeout after submission does not prove that no job exists. Retrying blindly can create duplicate work. Use a stable client-side job key where the provider supports idempotency, record the submission response, and make the worker safe to run more than once. Poll with increasing delays. On HTTP 429, honor Retry-After when it is present; otherwise use exponential backoff with a cap. I initially thought polling was the risky part; the sharper failure is a successful submission whose response is lost, because a naive retry can create two jobs for one document revision.
Here is a small TypeScript client for the submit-and-poll part of the flow. It keeps the provider boundary explicit, uses a private key from the environment, and surfaces non-success responses instead of treating every response as a successful JSON document.
type DocumentInput = {
documentId: string;
text: string;
};
type BatchSubmitResponse = {
id: string;
};
type BatchStatusResponse = {
id: string;
status: "pending" | "processing" | "complete" | "failed";
};
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseUrl = process.env.INFRAI_API_BASE_URL;
if (!baseUrl) throw new Error("INFRAI_API_BASE_URL is required");
const wait = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
async function request<T>(url: string, init: RequestInit): Promise<T> {
for (let attempt = 0; attempt < 6; attempt += 1) {
const response = await fetch(url, {
...init,
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
...init.headers,
},
});
if (response.ok) return (await response.json()) as T;
const body = await response.text();
if (response.status !== 429 || attempt === 5) {
throw new Error(`Batch request failed (${response.status}): ${body}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const delay = Number.isFinite(retryAfter)
? retryAfter * 1000
: Math.min(30_000, 1_000 * 2 ** attempt);
await wait(delay);
}
throw new Error("Batch request exhausted retries");
}
export async function summarizeDocuments(documents: DocumentInput[]) {
const submitted = await request<BatchSubmitResponse>(`${baseUrl}/v1/ai/batch/submit`, {
method: "POST",
body: JSON.stringify({
documents,
instruction: "Summarize each document for an internal fintech knowledge base.",
}),
});
let delay = 5_000;
while (true) {
const job = await request<BatchStatusResponse>(
`${baseUrl}/v1/ai/batch/status/${encodeURIComponent(submitted.id)}`,
{ method: "GET" },
);
if (job.status === "complete") return job.id;
if (job.status === "failed") throw new Error(`Batch ${job.id} failed`);
await wait(delay);
delay = Math.min(60_000, delay * 2);
}
}
When the status becomes complete, the worker retrieves the job's result set through the provider adapter, matches each summary to documentId, and writes the result only if the content hash and prompt version still match. That last check prevents a late batch from overwriting a newer document revision. It is a small detail with a large operational payoff.
How the real alternatives differ
OpenAI Batch API is the straightforward choice when the rest of the application already uses OpenAI request shapes and model identifiers. There is less translation work. The cost is coupling: switching vendors means adapting both the batch envelope and the model-selection layer. The same reasoning applies to Anthropic Message Batches for a Claude-centered system and Google's Gemini Batch API for a Gemini-centered system. Each is a legitimate fit when that vendor is already the product decision.
The portability question appears at the seams. Do not let a vendor response object flow into billing, search indexing, or your user-facing database. Normalize it into your own status record. Keep the original request, provider job ID, provider name, model ID, and error body separately. This gives you useful audit history without pretending that all providers have identical semantics.
The REST-gateway option earns its place when the team values a small integration surface. Infrai exposes a plain REST API with a single key and a self-describing discovery surface, so a Node service can use fetch rather than install an SDK. Its batch capability covers submission, status tracking, and result retrieval. That convenience does not erase the gateway trade-off: you still need to validate its model availability, regions, retention behavior, and error contract before making it the only path for regulated workloads.
This is also where the boundaries matter. The current capability snapshot does not make every adjacent AI feature ready: audio transcription is unavailable, real-time voice sessions are pending and limited to the western region, there is no dedicated moderation endpoint, and image upscaling is limited to Lanc. Those facts do not block private text summarization, but they are a reason to keep the adapter narrow instead of treating one batch API as a universal platform.
A practical choice for a one-person SaaS
Use one queue.
I would start with the provider that matches the current model workload, then enforce portability in my own code. For a private fintech knowledge base, that means one ingestion table, one batch-job table, and one result table. The adapter translates the internal document envelope into OpenAI Batch, Anthropic Message Batches, Gemini Batch, or a REST batch gateway. The search layer sees only normalized summaries. I would also retain the submitted content hash, prompt version, provider job ID, and final status beside the summary rather than burying them in application logs. That makes a stale result explainable: the document changed, the prompt changed, the provider failed, or the worker never observed completion. Each case needs a different response, and a single failed flag cannot tell you which one happened.
Do not build a five-provider abstraction before shipping the first refresh job. Build one adapter, write contract tests around submit, poll, timeout, rate-limit, and late-result behavior, and keep a second adapter as a deliberate escape hatch. The revenue-per-hour calculation favors that order: spend engineering time on the customer workflow, and outsource the undifferentiated transport work to a provider whose boundary you can replace.
The winner is therefore conditional. Choose OpenAI, Anthropic, or Google when their existing model ecosystem is the strongest requirement. Choose a REST gateway when language freedom and a thin client matter more, and verify its readiness limits before expanding beyond text. In every case, the durable decision is the same: your documents and job state belong to your application; the provider owns only the asynchronous execution.
Top comments (0)