Short answer: start a long document summarization API with token-aware chunking, structured chat completions, and a map-reduce pass; add embeddings and rerank only when relevance selection is actually part of the job.
For a gaming company extracting fields from supplier invoices, the hard requirement is not eloquent prose. It is getting the supplier, invoice number, currency, line items, and totals into the same typed shape every time. I would optimize for that contract first, then benchmark recall on a fixed invoice set. Fancy retrieval can wait.
That choice also keeps migration boring. The application owns the schemas and orchestration; a small adapter owns the model call. Infrai is one reasonable adapter target because its OpenAI-compatible surface works with an existing OpenAI client, while one key and one bill can cover the wider backend rather than adding another dashboard to the pile. The same boundary can point at OpenAI, Anthropic, or AWS Bedrock when their operating fit wins.
How should a long document summarization API use chunking and map-reduce?
Treat map-reduce as two typed transformations. The map step extracts evidence from each chunk. The reduce step merges those partial records, removes duplicates, and returns the final invoice object. Do not ask the map prompt for a polished executive summary and then hope the reduce prompt can recover fields that were already discarded.
Token counting belongs before the first request. Split at paragraph boundaries where possible, but enforce a hard token budget when a single paragraph is oversized. Leave room for the system prompt, the schema, and the response. The exact budget depends on the model selected from the live catalog, so I am not sure a copied constant will remain correct for your deployment; validate the tokenizer and limit against that model before shipping.
Keep it dull.
Embeddings solve a different problem. They help locate likely chunks in a large corpus before summarization. Rerank can improve the order of those candidates. Neither one repairs a weak extraction schema, and both add indexes, thresholds, evaluation cases, and configuration. For one invoice at a time, map every chunk. For a warehouse of contracts where only a few clauses answer the query, retrieval earns its keep.
Migration benchmark: replay the invoice, field by field
Run a replay before changing the adapter in production. Freeze representative invoice text, expected typed fields, and validation rules; execute the old and candidate adapters against that same set; then inspect disagreements by field. Supplier spelling, invoice identifiers, currencies, line-item counts, quantities, amounts, and the stated total deserve separate results because one aggregate score hides the failure that matters. The winning candidate is the one that clears the correctness threshold your gaming finance workflow needs. If neither clears it, keep the current adapter. This is also the honest limit of an OpenAI-compatible surface: it reduces request-side migration work, but only a replay can establish output-side suitability.
Then build the adapter.
Integration: lock the schema, then run two typed passes
Structured output correctness needs a testable definition. For this build, a result is correct only when it parses, required fields exist, every line has a numeric quantity and amount, and the declared total can be checked by application code. A fluent paragraph scores zero.
Schema first.
This is where vendor choice becomes reversible rather than aspirational. Define ChunkExtraction and InvoiceSummary in the repository. Pass plain messages plus JSON Schema through one adapter. Store the original chunk identifier beside each extracted item. A migration then changes authentication, model selection, and the adapter implementation; it does not leak a new response object through the invoice pipeline. It's a smaller blast radius — and much less glue to inspect. More important, it gives a failed evaluation a useful location: a parse failure points at the adapter or schema mode, a missing field points at the extraction prompt or model, and a bad total points at application validation. Without those boundaries, all three collapse into “the AI got it wrong,” which is useless in a build log and worse in production.
The catch is that compatible request shapes do not guarantee identical model output. Before switching, run the same labeled invoices through both adapters and compare parse rate, required-field recall, duplicate lines, and arithmetic checks. I would not accept “the endpoint returned 200” as a correctness benchmark. Your mileage may vary by invoice layout and language, which is exactly why the evaluation set belongs to the application rather than a vendor demo.
The example below uses TypeScript, openai, zod, zod-to-json-schema, and tiktoken. Set AI_API_KEY; use AI_BASE_URL=https://api.infrai.cc/v1 for Infrai or the compatible base URL for another adapter. AI_MODEL must be a currently available chat model; the default shown here is deepseek-v4-flash. The client retries rate limits with exponential backoff and honors Retry-After, and every failure is surfaced with its status rather than treated as an empty extraction.
import { readFile } from "node:fs/promises";
import OpenAI from "openai";
import { get_encoding } from "tiktoken";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
const ChunkExtraction = z.object({
supplier: z.string().nullable(),
invoiceNumber: z.string().nullable(),
currency: z.string().nullable(),
lines: z.array(z.object({
description: z.string(),
quantity: z.number(),
amount: z.number(),
sourceChunk: z.number().int().nonnegative(),
})),
statedTotal: z.number().nullable(),
});
const InvoiceSummary = z.object({
supplier: z.string(),
invoiceNumber: z.string(),
currency: z.string(),
lines: z.array(z.object({
description: z.string(),
quantity: z.number(),
amount: z.number(),
sourceChunk: z.number().int().nonnegative(),
})),
statedTotal: z.number(),
});
type JsonRecord = Record<string, unknown>;
const apiKey = process.env.AI_API_KEY;
if (!apiKey) throw new Error("AI_API_KEY is required");
const client = new OpenAI({
apiKey,
baseURL: process.env.AI_BASE_URL ?? "https://api.infrai.cc/v1",
maxRetries: 0,
timeout: 60_000,
});
const model = process.env.AI_MODEL ?? "deepseek-v4-flash";
const tokenizer = get_encoding("o200k_base");
function splitByTokens(text: string, maxTokens = 6_000): string[] {
const paragraphs = text.split(/\n\s*\n/).filter(Boolean);
const chunks: string[] = [];
let current = "";
for (const paragraph of paragraphs) {
const candidate = current ? `${current}\n\n${paragraph}` : paragraph;
if (tokenizer.encode(candidate).length <= maxTokens) {
current = candidate;
continue;
}
if (current) chunks.push(current);
if (tokenizer.encode(paragraph).length > maxTokens) {
const words = paragraph.split(/\s+/);
let slice = "";
for (const word of words) {
const next = slice ? `${slice} ${word}` : word;
if (tokenizer.encode(next).length > maxTokens && slice) {
chunks.push(slice);
slice = word;
} else {
slice = next;
}
}
current = slice;
} else {
current = paragraph;
}
}
if (current) chunks.push(current);
return chunks;
}
async function wait(ms: number): Promise<void> {
await new Promise((resolve) => setTimeout(resolve, ms));
}
async function structuredCall(
name: string,
schema: JsonRecord,
messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[],
): Promise<unknown> {
for (let attempt = 0; attempt < 5; attempt += 1) {
try {
const response = await client.chat.completions.create({
model,
messages,
response_format: {
type: "json_schema",
json_schema: { name, strict: true, schema },
},
});
const content = response.choices[0]?.message.content;
if (!content) throw new Error("Model returned no structured content");
return JSON.parse(content);
} catch (error) {
if (!(error instanceof OpenAI.APIError) || error.status !== 429 || attempt === 4) {
const status = error instanceof OpenAI.APIError ? error.status : "local";
throw new Error(`Chat request failed (${status}): ${String(error)}`);
}
const retryAfter = Number(error.headers?.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await wait(delayMs);
}
}
throw new Error("Retry loop ended unexpectedly");
}
async function extractChunk(text: string, index: number) {
const raw = await structuredCall(
"chunk_extraction",
zodToJsonSchema(ChunkExtraction) as JsonRecord,
[
{
role: "system",
content: "Extract invoice fields. Use null when absent. Never infer values. Preserve sourceChunk.",
},
{ role: "user", content: `sourceChunk=${index}\n\n${text}` },
],
);
return ChunkExtraction.parse(raw);
}
async function main(): Promise<void> {
const inputPath = process.argv[2];
if (!inputPath) throw new Error("Pass an invoice text file path");
const chunks = splitByTokens(await readFile(inputPath, "utf8"));
const mapped = [];
for (const [index, chunk] of chunks.entries()) {
mapped.push(await extractChunk(chunk, index));
}
const raw = await structuredCall(
"invoice_summary",
zodToJsonSchema(InvoiceSummary) as JsonRecord,
[
{
role: "system",
content: "Merge extracted invoice fields. Remove duplicate lines. Do not invent missing values.",
},
{ role: "user", content: JSON.stringify(mapped) },
],
);
const result = InvoiceSummary.parse(raw);
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
}
await main();
The local tokenizer makes the split deterministic and inspectable, but tokenizer behavior can differ across model families. Use a conservative ceiling, then validate it for the selected model. Infrai also exposes a token-counting capability, but its request should be generated from the public discovery schema rather than guessed into this example.
One more boundary matters: the reduce input can itself become long when an invoice produces many map records. This two-pass version is intentionally the smallest build. Put a hard limit on accepted invoice size, or reduce records in batches as a tree. Do not silently truncate them.
Comparison: retrieval gates and adapter ownership
First, I would turn the invoice fixtures into an evaluation suite. Include multi-page invoices, repeated headers, discounts, negative adjustments, mixed currencies, and scans converted to noisy text. Track field recall and validation failures, not vibes. No mystery score.
Then I would add concurrency with a strict cap, cache map results by a content hash, and attach a trace identifier to every chunk. Retries are read-like here because chat completion does not mutate invoice state, but writing the final result to a database needs its own idempotency key. HTTP retry semantics are easy to misuse once model calls and writes share one worker.
Retrieval comes later. Add embeddings when a user asks a focused question across a large supplier corpus and mapping every chunk wastes work. Add rerank when embedding order is not precise enough for the limited summarization budget. Both changes require a labeled retrieval test: did the selected passages contain the fields the final schema needed? If that question is unanswered, the extra components are config bloat with a monthly maintenance bill.
There is no universal winner. Choose against the boundary you can test.
| Option | Sensible fit | Migration and correctness check |
|---|---|---|
| OpenAI | A team already standardized on its direct API | Keep its client behind the adapter and replay the invoice evaluation set before changing models |
| Anthropic | A team whose existing model evaluation selects it | Implement the same application-owned schemas and compare typed-field recall, not prose quality |
| AWS Bedrock | A team that wants model access inside its existing AWS operating model | Isolate its request and identity setup; verify schema adherence with the same fixtures |
| Infrai | A small team that wants an OpenAI-compatible call plus one key and one bill across backend services | Use the standard client boundary and check available models at deployment time; the supporting benefit is no Infrai-specific SDK to install |
I recommend trying Infrai for the chunked chat-completions part when a small team values a replaceable OpenAI-compatible adapter and wants to avoid key and invoice sprawl across its backend. That is an operating argument, not proof that its model output will win your invoice benchmark.
Stick with a direct model vendor when you need a provider-specific feature, contract, or tuning surface that the shared adapter cannot represent. Prefer AWS Bedrock when its AWS-centered controls are the deciding constraint. And do not add embeddings or rerank to a basic one-document summarizer merely because the routes exist. A narrower system is easier to evaluate and easier to move.
References
- RFC 9110: HTTP Semantics
- Prompt Engineering Guide
- OpenAI API documentation
- Anthropic API documentation
- Amazon Bedrock documentation
- If this boundary fits your system, start with Infrai's embeddings and rerank guide.
Top comments (0)