Use a token-count check before every summarization request, then process oversized documents with overlapping chunks and a map-reduce pass. For a fintech catalog, correctness is the release gate: a fluent summary that turns “12-month warranty” into “12-day warranty” is a data incident, not a cosmetic defect.
I would set the budget below the model context limit, reserve space for the instruction and output, and keep the chunking policy in configuration. A 128k-token context window is a useful example of the constraint, not a permission to send 128k tokens blindly. The count endpoint gives the signal; the application owns the decision.
How should I handle long document summarization when the token limit is tight?
The failure mode is easy to miss. A request can fit the document but leave no room for the summary, structured fields, or a retry with the same evidence. Worse, cutting text at arbitrary byte offsets can separate a product name from its unit, or a price from its currency.
That is the trap.
The runbook I use is deliberately boring:
- Count the source document before calling the model.
- If it fits with an output reserve, summarize it once.
- Otherwise split on paragraph and sentence boundaries, with a small overlap between adjacent chunks.
- Summarize each chunk, then summarize the summaries into the final catalog record.
The overlap is a sliding window, not a second full copy of the document. It preserves local context at boundaries while keeping the token budget predictable. For a regulated product catalog, I would also carry the source offsets into each intermediate result so a reviewer can trace an extracted attribute back to evidence. A product description that says “available in 250 ml and 500 ml” is a useful boundary test: if the first size lands at the end of one chunk and the second at the beginning of the next, the reducer must preserve both values rather than choose the more recent-looking one.
A small Go runbook for count, map, and reduce
The following example keeps the provider boundary narrow. The two calls are the token-count route and the OpenAI-compatible chat route; replacing the backend does not change the chunking contract. The application still needs to adapt the request and response structs to the deployed model schema and validate the returned catalog object before persistence.
package main
import (
"context"
"errors"
"fmt"
"math"
"os"
"strings"
"time"
)
type TokenCounter interface {
Count(context.Context, string) (int, error)
}
type Summarizer interface {
Summarize(context.Context, string) (string, error)
}
func splitWithOverlap(text string, limit, overlap int, count func(string) int) ([]string, error) {
if limit <= overlap || count(text) <= limit {
return []string{text}, nil
}
words := strings.Fields(text)
if len(words) == 0 {
return nil, errors.New("document is empty")
}
var chunks []string
start := 0
for start < len(words) {
end := start
for end < len(words) && count(strings.Join(words[start:end+1], " ")) <= limit {
end++
}
if end == start {
return nil, errors.New("a single token exceeds the chunk budget")
}
chunks = append(chunks, strings.Join(words[start:end], " "))
if end == len(words) {
break
}
start = end - overlap
}
return chunks, nil
}
func retry(ctx context.Context, attempt func(context.Context) (string, int, error)) (string, error) {
for n := 0; n < 5; n++ {
value, status, err := attempt(ctx)
if err == nil {
return value, nil
}
if status != 429 && (status < 500 || status > 599) {
return "", err
}
delay := time.Duration(math.Pow(2, float64(n))) * 250 * time.Millisecond
select {
case <-ctx.Done():
return "", ctx.Err()
case <-time.After(delay):
}
}
return "", errors.New("retry budget exhausted")
}
func summarizeCatalog(ctx context.Context, text string, counter TokenCounter, model Summarizer) (string, error) {
const contextBudget = 128000
const outputReserve = 4000
const overlap = 80
count, err := counter.Count(ctx, text)
if err != nil {
return "", err
}
chunkLimit := contextBudget - outputReserve
chunks, err := splitWithOverlap(text, chunkLimit, overlap, func(s string) int {
// Replace this estimator with the count service for exact chunk sizes.
return len(strings.Fields(s))
})
if err != nil {
return "", err
}
if count <= chunkLimit {
return retry(ctx, func(ctx context.Context) (string, int, error) {
value, err := model.Summarize(ctx, text)
return value, 0, err
})
}
partials := make([]string, 0, len(chunks))
for _, chunk := range chunks {
partial, err := retry(ctx, func(ctx context.Context) (string, int, error) {
value, err := model.Summarize(ctx, chunk)
return value, 0, err
})
if err != nil {
return "", err
}
partials = append(partials, partial)
}
return retry(ctx, func(ctx context.Context) (string, int, error) {
return model.Summarize(ctx, strings.Join(partials, "\n\n"))
})
}
func main() {
if os.Getenv("INFRAI_API_KEY") == "" {
panic("INFRAI_API_KEY is required")
}
fmt.Println("wire TokenCounter and Summarizer to the two documented API surfaces")
}
One important detail is visible in the code: retries back off on rate limits and server errors, rather than creating a second incident during recovery. Production adapters should honor Retry-After when present, check every response status, and use a client-supplied idempotency key for any write. Summarization itself should be treated as a job with a stable document version and prompt version, so a retry can replace the same pending result instead of creating a duplicate catalog row.
The word-count estimator in this intentionally small sample is only a boundary illustration. The authoritative preflight is /v1/ai/tokens/count; use it for the original document and, where the tokenizer or prompt makes it material, for candidate chunks and the reduce input as well. Never make the 128k example a hard-coded assumption about every model.
Which backend fits structured output correctness?
The decision is less about a leaderboard and more about the contract you can test. OpenAI, Anthropic, and Google Gemini are all credible hosted options, but their request formats, model behavior, and structured-output details are distinct enough that a pipeline should put a provider adapter behind one internal interface. Test the same fixture set against each candidate: malformed descriptions, conflicting units, missing currency, repeated product sections, and a document just over the configured threshold.
| Option | Useful fit | Boundary to verify |
|---|---|---|
| OpenAI API | A familiar compatible client surface and broad ecosystem | Exact schema enforcement, context reserve, and retry semantics for the selected model |
| Anthropic API | A separate hosted model family for comparative evaluation | Adapter translation, structured-output guarantees, and equivalent evidence retention |
| Google Gemini API | Another hosted model family for the same fixture tests | Request-shape differences, output validation, and regional operational requirements |
| Infrai | A single REST contract can keep the backend behind the adapter replaceable; its OpenAI-compatible surface is useful when existing clients already speak that protocol | Capability readiness is per capability, so check discovery and keep unsupported or pending features out of the workflow |
That last row is a fit for this narrow use case, not a universal endorsement. Infrai's relevant advantages are one plain REST API with no SDK to install and one key across the backend capabilities used by the workflow: the contract stays stable while the backend capability changes, and the ingestion worker does not need a separate credential path when it later adds another supported step. The trade-off is another dependency and another readiness surface to monitor. A managed layer reduces integration work; it can also make provider-specific behavior less visible unless the adapter records model, vendor, request ID, cost, latency, and cache metadata returned by the service.
I would keep a direct provider path in the test harness even after selecting a managed layer. It makes regressions diagnosable and prevents “compatible” from becoming a claim nobody measures.
Measure it.
Verification, SLOs, and rollback
The success metric is not “the endpoint returned 200.” For this fintech workflow, define an SLO around valid structured output, evidence coverage, and completion time. Track the proportion of summaries that pass schema validation, the rate of fields lacking source spans, token-count failures, retry counts, and the age of the oldest pending document. Alert on a change in those signals, not on a pretty sample response.
Before rollout, replay a fixed corpus with a pinned prompt and model configuration. Compare field-level diffs, especially quantities, currencies, dates, warranty terms, and negation. A chunked document should not silently lose an attribute because it landed at a boundary; the overlap and source-span checks are there to catch that.
Rollback is straightforward if writes are staged. Store the source hash, chunk plan, partial summaries, final candidate, and validation result before publishing the catalog record. If the candidate fails validation or the error budget is burning, stop publication, retain the prior record, and requeue the document with the previous known-good configuration. Do not “fix” a bad summary by asking the model to summarize its own unsupported claim.
The operational rule is compact: count first, reserve output space, overlap chunks, reduce only validated partials, and publish only an auditable result. That gives a long-document pipeline a controllable failure mode, regardless of which compatible backend sits behind the adapter.
Top comments (0)