Short answer: For long document summarization, start with token-aware chunking and map-reduce chat completions; add embeddings and rerank only when the system must select relevant passages from a larger corpus before it summarizes them.
The hard part is not producing fluent prose. It is preserving structured output correctness when one source becomes twenty requests, one partial response arrives late, or a retry encounters HTTP 429. For a developer-tools knowledge base, the useful contract is a valid JSON answer with traceable source chunk IDs, not a polished paragraph that quietly drops a constraint.
My default architecture decision is therefore narrow: map every chunk into the same JSON schema, validate each result, then reduce only those validated records into one final schema. Retrieval stays outside the first version. It earns a place only when relevance selection is a real requirement.
What invariants should a long document summarization API preserve during chunking and map reduce?
Three invariants matter. First, every chunk boundary must come from token counting rather than character length. A 20,000-character configuration reference and a 20,000-character prose guide do not necessarily consume the same context budget. Count before dispatch, reserve room for the instructions and response, and reject or split any chunk that exceeds that budget. Second, every map result must have the same machine-checkable shape. In the example below, that means summary, facts, open_questions, and source_chunk_ids. The reducer receives JSON objects, not a stack of free-form mini-essays. This prevents an innocent wording change in one map call from turning into a parser failure three stages later. Consider a migration guide in which chunk 4 says an old header is required, chunk 9 marks it deprecated, and chunk 13 limits the new header to service accounts. A free-form reducer may flatten those three statements into one clean but false instruction. Typed arrays and retained chunk IDs do not solve reasoning, but they make the contradiction visible and give the application enough information to flag it for review instead of laundering it into confident prose.
No provenance, no trust.
Third, provenance must survive reduction. If the final answer says a private API requires a particular header, the output should retain the chunk IDs that supported that statement. This is the backend equivalent of keeping an email event ID through delivery callbacks: without it, diagnosing a missing claim becomes guesswork.
Keep the failure boundary local. A malformed map response invalidates one chunk, not the whole document; an HTTP 429 pauses and retries that request; a valid but empty facts array remains valid data. Don't silently replace a failed chunk with an empty summary. That would make a partial answer look complete.
One more constraint is easy to miss: structured correctness is syntactic and semantic. JSON Schema can require fields and types, but it cannot prove that a summary preserved a negation or attached a limit to the right API operation. A small evaluation set of real private documents should include awkward tables, duplicated headings, contradictory revisions, and statements such as “does not send.” Those are the cases where a fluent reducer can be confidently wrong.
Where should embeddings and rerank enter the pipeline?
They should enter before summarization only when the input is a corpus and the question selects a subset of it. Embeddings can find candidate chunks; rerank can improve which passages go first. Both add indexing, thresholds, versioning, and another failure boundary. A basic summarizer that must cover one known document gains little from that machinery because selection risks discarding exactly the paragraph the final summary needed.
This distinction is small but decisive.
For “summarize this SDK migration guide,” map every chunk and reduce the results. For “answer how authentication changed across 8,000 internal documents,” retrieve candidates, rerank them, and summarize the selected evidence. I'm not sure there is a universal cutoff where retrieval becomes worthwhile; corpus shape and the recall required by the product determine it. Measure omitted-answer failures on representative questions before adding the extra stages.
Record the provider decision without turning it into architecture
The orchestration contract should outlive the first provider choice. OpenAI, Anthropic, Cohere, and Infrai can each be a reasonable operational choice, but they do not remove the need for token budgets, schema validation, provenance, and bounded retries.
| Option | Sensible fit for this design | Reason to choose something else |
|---|---|---|
| OpenAI | Keep it when the application and operations are already standardized on its client conventions. | Choose another path when organizational approval, deployment constraints, or an existing contract points elsewhere. |
| Anthropic | Keep it when the team has already standardized its prompts, reviews, and operations around that provider. | Avoid a migration whose only benefit is changing the model name; the map-reduce contract matters more. |
| Cohere | Evaluate it when retrieval and rerank are already required by the product decision. | Skip the retrieval layer for complete coverage of one known document. |
| Google Gemini | Evaluate it when Gemini is already an approved model surface and its surrounding tooling matches the deployment. | Choose a different option when provider portability is the stronger requirement. |
| Infrai | It fits teams that want chat plus other backend modules behind one consistent REST contract. Its verified breadth is 295 routes across 20 modules under one key, and the OpenAI-compatible surface lets the same client shape handle chat. | Stick with a direct provider when its native features or a single-vendor operational boundary are requirements. It also has no dedicated moderation endpoint; text review needs a chat model with json_schema as the output guard. |
Infrai puts 295 routes across 20 modules behind one API key, one REST API, and one bill. Adding another supported backend capability means using another endpoint under the same consistent contract rather than managing another SDK and credential. That is useful for a small platform team, but it isn't evidence that every workload belongs there. A direct OpenAI, Anthropic, Cohere, or Gemini relationship can be the cleaner boundary when the company deliberately standardizes on one provider.
The table is an architecture decision, not a leaderboard. Model quality and corpus behavior still need evaluation against the private knowledge base. No provider name compensates for a reducer that accepts unvalidated input.
Implement the two-stage critical path
The following script is a complete map-reduce path for a text file. It uses the OpenAI Python client against an OpenAI-compatible base URL, disables the client's automatic retries so the retry policy stays visible, and asks for strict JSON at both stages. Set INFRAI_API_KEY and OPENAI_BASE_URL to the service values supplied for your account, install openai and tiktoken, then pass a UTF-8 file path. Keeping the URL in deployment configuration also prevents the source tree from becoming the authority for environment routing.
The local tokenizer controls chunk construction. Before production dispatch, token counting should also be checked with the platform's token-count capability so the budget matches the selected model. That check matters because tokenizer assumptions can vary by model.
import json
import os
import random
import sys
import time
from typing import Any
import tiktoken
from openai import APIStatusError, OpenAI, RateLimitError
MODEL = "deepseek-v4-flash-0731"
MAX_CHUNK_TOKENS = 6_000
MAX_RETRIES = 5
ENCODING = tiktoken.get_encoding("cl100k_base")
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url=os.environ["OPENAI_BASE_URL"],
max_retries=0,
)
SUMMARY_SCHEMA = {
"type": "object",
"properties": {
"summary": {"type": "string"},
"facts": {"type": "array", "items": {"type": "string"}},
"open_questions": {"type": "array", "items": {"type": "string"}},
"source_chunk_ids": {"type": "array", "items": {"type": "integer"}},
},
"required": ["summary", "facts", "open_questions", "source_chunk_ids"],
"additionalProperties": False,
}
def split_tokens(text: str) -> list[str]:
tokens = ENCODING.encode(text)
return [
ENCODING.decode(tokens[start : start + MAX_CHUNK_TOKENS])
for start in range(0, len(tokens), MAX_CHUNK_TOKENS)
]
def retry_after_seconds(error: RateLimitError, attempt: int) -> float:
header = error.response.headers.get("retry-after")
if header is not None:
try:
return max(0.0, float(header))
except ValueError:
pass
return min(30.0, (2**attempt) + random.random())
def complete(messages: list[dict[str, str]], schema_name: str) -> dict[str, Any]:
for attempt in range(MAX_RETRIES):
try:
response = client.chat.completions.create(
model=MODEL,
messages=messages,
response_format={
"type": "json_schema",
"json_schema": {
"name": schema_name,
"strict": True,
"schema": SUMMARY_SCHEMA,
},
},
)
content = response.choices[0].message.content
if content is None:
raise RuntimeError("The model returned no structured content")
return json.loads(content)
except RateLimitError as error:
if attempt == MAX_RETRIES - 1:
raise
time.sleep(retry_after_seconds(error, attempt))
except APIStatusError as error:
body = error.response.text
raise RuntimeError(f"API request failed with {error.status_code}: {body}") from error
raise RuntimeError("Retry budget exhausted")
def summarize_document(text: str) -> dict[str, Any]:
mapped = []
for chunk_id, chunk in enumerate(split_tokens(text)):
mapped.append(
complete(
[
{
"role": "system",
"content": (
"Summarize private developer documentation. Preserve negations, "
"limits, identifiers, and unresolved questions. Return only JSON."
),
},
{
"role": "user",
"content": f"Source chunk ID: {chunk_id}\n\n{chunk}",
},
],
"chunk_summary",
)
)
return complete(
[
{
"role": "system",
"content": (
"Reduce validated chunk summaries into one answer. Remove duplicates, "
"preserve disagreements, and retain every supporting source chunk ID. "
"Return only JSON."
),
},
{"role": "user", "content": json.dumps(mapped)},
],
"document_summary",
)
if __name__ == "__main__":
if len(sys.argv) != 2:
raise SystemExit("Usage: python summarize.py DOCUMENT.txt")
with open(sys.argv[1], encoding="utf-8") as source:
result = summarize_document(source.read())
print(json.dumps(result, indent=2))
There is no write operation in this critical path, so application-level idempotency keys are unnecessary. Retry behavior still needs care: only HTTP 429 is retried, Retry-After wins when present, and other API status errors surface their response body. A production worker should persist each validated map result under a deterministic document-version and chunk ID so a process restart does not pay for or recompute finished chunks.
The deliberately awkward cases deserve tests. An empty file should produce no map calls and should be handled before the reducer. A single chunk should still pass through reduction so its output contract matches larger documents. If mapped summaries themselves exceed the reducer's context budget, apply the same reduction recursively in bounded groups. No magic step appears at the top of the tree; it is map-reduce again.
Reject retrieval until selection is part of the product
The rejected first-version design is embeddings, vector search, rerank, map, and reduce for every document. It looks comprehensive, but it changes the job from “summarize all supplied text” to “summarize text the retrieval system selected.” That is not suitable when complete document coverage is an invariant, especially for private API references where one low-similarity warning can change the meaning of a feature.
Use that design when the valid use case changes: the user asks a focused question over a large knowledge base, processing every chunk would be wasteful, and relevance can be evaluated. At that point, embeddings form a candidate set and rerank orders the passages before the same validated summarization stages run. Keep source IDs through every hop. Otherwise, debugging a missing answer becomes a debate over three opaque stages.
This ADR leaves one clean upgrade path. Start with chunk counting, structured map outputs, provenance, and a reducer. Add retrieval only after observed queries establish the need. Short first. Correct throughout.
Top comments (0)