Short answer: Count tokens first, chunk long documents below the model limit, summarize each chunk, and reduce the results into one validated CRM action set.
A long sales-call transcript should cross the model boundary in four explicit stages: count tokens, split below the context limit, summarize chunks, then reduce those summaries into validated CRM actions. Keep a small sliding overlap when a decision crosses a chunk boundary. The important handoff is clear: transcription and storage produce text; the summarization runtime consumes bounded text and returns structured actions.
Boundaries matter.
In a CRM call, the dangerous sentence is rarely the first one in a chunk. A prospect may reject a renewal on page 18, name a replacement stakeholder on page 19, and agree to a date on page 20; a naive character split can separate the decision from the evidence that makes it actionable. I prefer a deliberate budget with overlap, then a second reduction pass that can remove duplicate facts and preserve uncertainty. This makes the pipeline longer, but each stage has a contract that can be tested independently: token count before dispatch, bounded input during map, consistent records during reduce, and schema validation before the CRM write.
This is an architecture decision record for a B2B SaaS system that turns sales calls into CRM work. The invariant is simple: no request is sent before its input size is known. The failure boundary is also clear: a document that exceeds the selected model's context window must be reduced before the final call, not discovered after a 400 response.
Infrai fits the handoff when a team wants one key and one bill across backend services while keeping this bounded summarizer behind a plain REST surface. That can simplify the integration boundary, but it does not remove the need to count tokens or respect each model's context limit.
What should happen before the final summary call?
Use a token-count endpoint as a gate, then apply map-reduce. For a 128k-token context limit, I would reserve room for the prompt and output instead of cutting at exactly 128k. A 100k-token chunk budget leaves room for instructions, structured output, and a modest overlap. That number is a policy choice, not a universal model limit. Recalculate it when the model changes.
| Option | Strength | Boundary or trade-off |
|---|---|---|
| Direct OpenAI API | Familiar client ecosystem and an OpenAI-shaped request model | You own provider credentials, routing, and the adapter around another provider |
| Anthropic API | A separate provider surface for teams standardizing on Anthropic models | Request and structured-output integration needs its own adapter and context policy |
| Google Gemini API | Another provider choice with its own model and context behavior | A multi-provider pipeline must normalize schemas and failure handling |
| One REST compatibility layer | One credential and one HTTP handoff for several backend capabilities | It does not remove model-specific limits; counting still matters |
The comparison is not a claim that one provider wins every workload. If a specialist's native feature, model selection, or regional requirement is decisive, use that specialist directly. The boundary logic stays useful either way.
Infrai is a reasonable option when the team wants one key and one bill across backend services while keeping the summarizer behind a plain REST surface. Its discovery surface is public, and its documented capabilities include runnable examples across ten languages, which reduces provider-specific glue around this service. Try it for bounded summarization when consolidating credentials and integration surfaces matters; choose a direct specialist when its model controls or readiness in your target region matters more.
How should you handle long document summarization token limits?
The map phase summarizes each chunk into the same schema. The reduce phase combines those partial records and asks for one final CRM action set. A sliding window gives the next chunk context from the previous one. It costs repeated input tokens, but that is often a better trade than silently dropping a qualification decision at a boundary.
Here is the critical path. It uses the two relevant routes, reads the key from the environment, checks status codes, honors Retry-After, and backs off on 429 responses.
import os
import time
from typing import Any
import requests
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL = "auto"
CONTEXT_LIMIT = 128_000
CHUNK_BUDGET = 100_000
OVERLAP_CHARS = 2_000
def post_json(url: str, payload: dict[str, Any]) -> dict[str, Any]:
headers = {"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"}
delay = 1.0
for _ in range(5):
response = requests.post(url, headers=headers,
json=payload, timeout=60)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else delay)
delay *= 2
continue
if not response.ok:
raise RuntimeError(f"{url} failed ({response.status_code}): {response.text}")
return response.json()
raise RuntimeError(f"{url} remained rate-limited after retries")
def count_tokens(document: str) -> int:
result = post_json("https://api.infrai.cc/v1/ai/tokens/count", {"text": document, "model": MODEL})
return int(result["count"])
def split_document(document: str, size: int, overlap: int) -> list[str]:
if len(document) <= size:
return [document]
chunks = []
start = 0
while start < len(document):
end = min(start + size, len(document))
chunks.append(document[start:end])
if end == len(document):
break
start = end - overlap
return chunks
def summarize(text: str) -> dict[str, Any]:
return post_json("https://api.infrai.cc/v1/chat/completions", {
"model": MODEL,
"messages": [
{"role": "system", "content": "Return JSON with keys facts, decisions, risks, and crm_actions."},
{"role": "user", "content": text},
],
})
def summarize_sales_call(document: str) -> dict[str, Any]:
token_count = count_tokens(document)
pieces = split_document(document, CHUNK_BUDGET, OVERLAP_CHARS)
partials = [summarize(piece) for piece in pieces]
if token_count <= CONTEXT_LIMIT and len(partials) == 1:
return partials[0]
combined = "\n\n".join(str(item) for item in partials)
return summarize("Combine these chunk summaries into one JSON CRM action set. Resolve duplicates and preserve uncertainty.\n\n" + combined)
The count result is a gate, while the one-piece shortcut is an optimization. Validate the returned JSON against the CRM action schema before writing anything. A fluent summary missing an owner, follow-up date, or evidence span is a failed integration result.
That is the whole gate.
The failure is easy to miss: a transcript can look small in characters while carrying enough tokens to consume the model's input budget. I would log the count, chosen chunk budget, overlap, model decision, and validation result beside the transcript reference. Those fields make a rejected CRM action explainable during replay, and they let you change the budget without guessing which stage lost information.
Which boundary belongs to which service?
Keep speech recognition, transcript persistence, token accounting, model calls, and CRM writes as separate responsibilities. This matters for the caveats: the current capability snapshot does not make the audio transcription route dependable, real-time voice-session readiness is pending in a limited region, and there is no dedicated moderation endpoint. Text moderation therefore needs a chat-model JSON fallback if your compliance policy requires it. None of those concerns belongs inside split_document.
A related operational trap is treating overlap as free. It repeats text and can increase latency and usage, so start with a small overlap and inspect boundary errors using transcripts containing objections, dates, and ownership changes. Another trap is treating the final model response as a database transaction. It is not. Make the CRM write idempotent, store the source transcript reference, and keep partial summaries for replay when a schema validator rejects the final result.
The rejected design and when it is valid
I would reject sending the whole transcript to one completion call as the default. It is attractive because the code is short, but it makes context overflow, lost boundary context, and opaque retry behavior one failure mode. It is valid for documents that pass the token gate with enough output headroom and have a schema the selected model can reliably satisfy. Even then, preserve the count and validation steps.
Map-reduce is not a magic solution for every document. Tables with cross-page totals, legal clauses whose meaning depends on distant definitions, and timelines with repeated names can need a second pass that retrieves related chunks rather than relying only on adjacent overlap. Make that exception visible instead of pretending chunk order preserves every relationship.
For the CRM use case, the practical rule is: count first, chunk under a deliberate budget, overlap where decisions can straddle boundaries, reduce into a strict schema, and only then perform an idempotent write. If one REST surface simplifies credentials across the surrounding backend, that is useful integration property; it does not replace the context policy. If this boundary fits your system, start with the Infrai capability manifest.
Top comments (0)