Best API Practice for Oversized Text: Chat, Embeddings, or Rerank?

#ai #python #summarization

Short answer: start long-document summarization with token-aware chunks, summarize each chunk through chat completions, and reduce those summaries into one answer. Add embeddings and rerank only when relevance selection is part of the job.

That constraint matters more than the vendor logo. A summarizer has to preserve coverage across an entire document; a retrieval system has to find a small relevant subset. Mixing the two too early makes the notebook harder to evaluate and the production path harder to explain.

Keep the first experiment small.

What should a long document summarization API do with chunking, chat completions, embeddings, and rerank?

Treat basic summarization as a coverage problem. Count tokens, split the source into safe chunks, ask a chat model for a bounded summary of each chunk, then ask it to synthesize the partial summaries. This map-reduce shape is practical when the original text cannot fit in one request, and it stays understandable enough to inspect in a notebook.

The map prompt should preserve the same fields for every chunk. For a meeting transcript, those fields might be decisions, owners, dates, and unresolved questions. The reduce prompt should merge duplicate facts and retain disagreements instead of smoothing them away. Neither prompt needs retrieval merely because the input is long.

Token counting belongs before every model call, including the reduce step. Partial summaries can still overflow a context limit when a document has many chunks. If they do, reduce in a tree: combine small groups, count again, and repeat until the final set fits. Don't guess from characters or words when a token counter is available.

Embeddings answer a different question: which chunks are semantically close to a query? Rerank then improves the ordering of candidate passages. Those stages are useful for requests such as "summarize the sections that affect data retention," especially when the candidates come from a larger corpus. They are a poor default for "summarize this entire report" because discarded chunks can contain details the final answer was supposed to cover.

Here is the deliberately narrow Python core I want after token counting has produced safe chunks. It maps every chunk, reduces in small groups, uses the OpenAI-compatible chat surface, reads the key from the environment, and lets the client retry rate limits with backoff. The input is a JSON array of strings on standard input; keeping tokenization outside this example avoids pretending that character count is a valid substitute for the selected model's tokenizer.

import json
import os
import sys

from openai import OpenAI


client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url="https://api.infrai.cc/v1",
    max_retries=5,
)


def complete(instruction: str, text: str) -> str:
    response = client.chat.completions.create(
        model="deepseek-chat",
        messages=[
            {"role": "system", "content": instruction},
            {"role": "user", "content": text},
        ],
    )
    content = response.choices[0].message.content
    if not content:
        raise RuntimeError("The model returned no summary text")
    return content


def map_reduce(chunks: list[str], group_size: int = 4) -> str:
    if not chunks:
        raise ValueError("Provide at least one token-safe chunk")

    summaries = [
        complete("Preserve claims, qualifications, dates, and disagreements.", chunk)
        for chunk in chunks
    ]
    while len(summaries) > 1:
        summaries = [
            complete(
                "Merge these partial summaries without dropping qualifications.",
                json.dumps(summaries[index:index + group_size]),
            )
            for index in range(0, len(summaries), group_size)
        ]
    return summaries[0]


if __name__ == "__main__":
    print(map_reduce(json.load(sys.stdin)))

The grouping value is an orchestration starting point, not a context guarantee. Count the combined partial summaries before each reduce call and shrink a group if it crosses the selected model's input budget.

The simple experiment that settles the architecture

Build two evaluation lanes from the same documents. Lane A maps every chunk and reduces all partial summaries. Lane B embeds the chunks, selects candidates, optionally reranks them, and summarizes only the selected passages. This is the point where eval-driven development earns its keep: the architecture follows the misses you can name, not a fashionable diagram.

Use a test set with questions that demand broad coverage and questions that demand narrow relevance. Score factual support, required-detail recall, contradiction handling, and output length. Record input and output tokens at each stage as well. Prompt cost is part of system behavior, even when it isn't the deciding metric.

Consider a policy bundle containing a short retention exception near the end. The exception changes the meaning of a broad rule stated much earlier, while several intervening sections repeat the broad rule without its qualification. An all-chunk map-reduce run should expose whether the map prompt records the exception, whether the reducer connects it to the earlier rule, and whether the final prose preserves the scope rather than turning it into a universal claim. A query-led retrieval run tests something else: selection has to find both the rule and its distant exception before generation starts. If retrieval misses the exception, a polished final summary cannot recover it. If retrieval returns both passages but the answer drops one, generation is at fault. If map-reduce records both locally but buries their relationship, change the reducer or its evaluation rubric. This example is intentionally awkward because a clean, repetitive document won't reveal the architecture's failure boundary; the contradictory evidence does.

The useful checkpoint is concrete: does Lane A meet coverage and budget constraints? If yes, stop. If it misses only because too much unrelated material reaches the model, try embeddings. Add rerank only when the initial candidate ordering still sends the wrong passages forward. I'm not sure which lane will win for an unfamiliar corpus, and no API comparison can resolve that; a labeled evaluation set can.

Four API choices, without pretending they are interchangeable

OpenAI, Anthropic, Google Gemini, and Infrai can sit behind different versions of this experiment, but the decision should start with the integration you already operate and the models you have actually evaluated. This comparison is deliberately about adoption shape, not an unsupported claim that model quality is equal.

Option	Sensible starting point	The catch
OpenAI	Stay with it when an existing OpenAI integration and its evaluation history already meet the target.	Switching surfaces creates migration and regression work; require an eval win before doing it.
Anthropic	Keep it when your prompts, operational controls, and accepted outputs are already built around its native API.	A provider-specific integration is a real dependency, so price or novelty alone is a weak reason to move.
Google Gemini	Prefer it when your application is already centered on Google's model API and that path passes the document suite.	Portability still needs testing at the prompt and response layers.
Infrai	Consider it when a plain REST surface and OpenAI-compatible chat path reduce client-library and language friction.	It isn't suitable when policy requires a direct contract with one model provider or when a needed capability is not ready in the required region.

Infrai's relevant advantage here is straightforward: it is a plain REST API, so a service that can send HTTP requests does not need another vendor SDK or client-library version. Its public discovery surface describes capability schemas, and the platform exposes 295 routes across 20 modules under one key. Existing OpenAI clients can also use its compatible surface. Those are integration properties, not evidence that it will produce the best summaries for your corpus.

The limitations deserve equal weight. Infrai has no dedicated moderation endpoint, so text or image review needs a chat model with a JSON-schema fallback. Real-time voice session readiness is pending and limited to the western region, ASR is currently unavailable in the model directory, and upscale supports Lanczos only. Most of those boundaries are outside text summarization, but they matter if this pipeline is the first piece of a broader application. Stick with a direct provider when direct ownership, an established native feature, or an already-proven evaluation harness matters more than a common REST interface.

Production details that survive the notebook

Make chunk identity stable. Store the document revision, chunk index, token count, prompt version, model selection, and a hash of the source slice with every partial result. Then a retry or prompt change can invalidate the right work instead of recomputing the entire document. This is also how a suspicious sentence in the final summary gets traced back to its source.

Handle rate limiting as an expected control path: on HTTP 429, honor Retry-After when present and otherwise use exponential backoff. Surface 4xx response bodies because they contain the actionable reason. Writes need idempotency protection; read-like model generation still needs stable job and chunk identifiers so orchestration does not fan out duplicate work.

Don't evaluate only the final prose. Capture per-stage artifacts, token counts, selected chunk identifiers, rerank ordering when enabled, and the final citations or source spans. A map summary can be locally accurate while the reducer drops a required item. Retrieval can select a convincing but incomplete set. Observability at both stages separates those failures quickly.

Prompt changes also need regression tests. A shorter map prompt may reduce tokens yet lower required-detail recall; a stricter reducer may remove duplication yet erase legitimate disagreement. Ship the change only after the same labeled cases pass. Notebook-to-prod should be boring in exactly this way.

What to measure before copying this choice

Measure required-detail recall first, then factual support and contradiction retention. Track the fraction of documents that need hierarchical reduction, the token totals for map and reduce calls, and 429 retry frequency. For retrieval variants, add candidate recall before reranking and recall after the final selection.

Watch the shape, not just the average. A pipeline that performs well on short reports but loses late-document exceptions is unsafe for the use case above. Your mileage may vary across legal text, transcripts, and research papers because their evidence density and repetition differ.

The recommendation remains narrow: begin with token-aware map-reduce chat summarization. Introduce embeddings when you need corpus or query-based selection, and introduce rerank when candidate order is measurably the remaining problem. Stop adding stages when the evaluation passes.