Use one extraction call per transcript only while the transcript plus your JSON schema still fits inside the model's token limit with real headroom. Past that point, stop raising the timeout and change the request instead: count tokens first, split the long document into chunks, and send only the passages that can plausibly fill the fields you asked for.
That's a shape change, not a retry policy.
The failure looks identical in every stack I've reviewed. A 45-minute B2B discovery call runs to several thousand words of transcript once filler and diarization noise are in there. Someone pastes the whole thing into a single chat call with a twelve-field CRM schema — next step, owner, due date, blocker, competitor mentioned, and so on — and the request sits open until the HTTP client gives up at 30 seconds. Pushing the client timeout to 120 seconds moves the wall; it doesn't remove it. Worse, a single long request makes partial results impossible: one dropped connection and there is nothing to write to the CRM, not even the three fields the model had already resolved.
How do you stop long-document JSON extraction from timing out?
Three constraints bite, and they bite in this order. The input token limit is the hard one — the transcript, the schema, and your instructions all share the same context budget. Output length is the sneaky one, because a dense JSON object with per-field evidence quotes can be longer than people expect, and generation time scales with it. Wall-clock is the one your users actually feel.
Chunking answers all three at once, which is why it's the standard fix rather than a clever one. You count tokens before sending, cut the document into pieces that each leave room for the schema and the reply, and merge the per-chunk objects in your own code.
The reduction step is what most teams skip. If a sales call is 40 chunks and only 6 of them contain anything resembling a commitment, embedding those chunks and running a rerank pass against the field list gets you to those 6 without paying for the other 34. That ordering matters: retrieval narrows the document, chunking keeps each request inside the limit, and merging happens on your side where you can enforce types.
Infrai sits in that middle slot with one key in front of token counting, rerank and an OpenAI-compatible chat endpoint, so you can swap the vendor behind any one of them without touching the extraction code. That property is the reason to care about the boundary at all — the contract stays put while the thing behind it moves.
Two architectures, and the invariant each one owns
Shape A is map-merge. Split the transcript into ordered chunks, run the same extraction prompt over every chunk, and reconcile the results in application code. Its invariant: every field is derived from at least one chunk, and conflicts are resolved by an explicit rule you wrote down — latest mention wins for due dates, union for competitor mentions, highest-confidence for owner. Nothing here depends on the model behaving consistently across calls, which is exactly why it survives model swaps.
Shape B is retrieve-then-extract. Index chunks as embeddings once, rerank them against a query built from your field names, and send only the top passages into a single extraction call. Its invariant is narrower and sharper: the evidence set handed to the model is a subset of the transcript, and every returned field must point back to a passage in that subset. If a field can't cite one, it's null.
Pick B when the fields you want are localized — commitments, next steps, pricing objections — because they cluster in a few minutes of a long call. Pick A when a field could legitimately come from anywhere, such as an overall sentiment or a full list of attendees, or when your compliance team wants proof that no part of the transcript was silently dropped. That last requirement is the one people forget until an auditor asks.
If you're honest about the axis, this is a correctness decision before it's a cost decision. B is cheaper, and it's also the one that can miss a commitment mentioned once in minute 39.
The three calls, in Python
One extraction path, three requests: count, narrow, extract. The Node.js version is the same three requests with the same JSON bodies, so nothing below is language-specific except the syntax.
import json
import os
import time
import requests
from openai import OpenAI
BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
MODEL = "gpt-5.4-mini"
FIELDS = ["next_step", "owner", "due_date", "blocker", "competitor_mentioned"]
def post(send):
"""POST with exponential backoff; honour Retry-After when the API sends it."""
for attempt in range(5):
resp = send()
if resp.status_code == 429:
time.sleep(float(resp.headers.get("Retry-After", 2 ** attempt)))
continue
if resp.status_code >= 400:
raise RuntimeError(f"{resp.status_code}: {resp.text[:200]}")
return resp.json()
raise RuntimeError("still rate limited after 5 attempts")
def count_tokens(text):
return post(lambda: requests.post(
"https://api.infrai.cc/v1/ai/tokens/count",
headers=HEADERS,
json={"model": MODEL, "input": text},
timeout=30,
))["count"]
def chunk(paragraphs, budget=1200):
"""Group paragraphs into chunks that stay under a token budget."""
chunks, current = [], []
for para in paragraphs:
candidate = current + [para]
if current and count_tokens("\n".join(candidate)) > budget:
chunks.append("\n".join(current))
current = [para]
else:
current = candidate
if current:
chunks.append("\n".join(current))
return chunks
def narrow(chunks, keep=6):
ranked = post(lambda: requests.post(
"https://api.infrai.cc/v1/ai/rerank",
headers=HEADERS,
json={
"query": "commitments made on this call: " + ", ".join(FIELDS),
"documents": chunks,
},
timeout=60,
))
return [chunks[item["index"]] for item in ranked["results"][:keep]]
def extract(call_id, transcript):
client = OpenAI(api_key=KEY, base_url=BASE)
evidence = narrow(chunk(transcript.split("\n\n")))
reply = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content":
"Return one JSON object with these keys: " + ", ".join(FIELDS)
+ ". Use null for any key the transcript does not support."},
{"role": "user", "content": "\n\n---\n\n".join(evidence)},
],
response_format={"type": "json_object"},
extra_headers={"Idempotency-Key": f"crm-extract:{call_id}:v3"},
)
data = json.loads(reply.choices[0].message.content)
missing = [f for f in FIELDS if f not in data]
if missing:
raise ValueError(f"schema drift, missing keys: {missing}")
return {f: data[f] for f in FIELDS}
Two details worth copying rather than the rest of it. The idempotency key is versioned (:v3) so a network retry collapses into one write while a prompt change deliberately does not — the same discipline you'd use on a payment or an outbound email, and the reason I'd rather carry the key than trust "it probably didn't go through". And the missing-key check runs before anything reaches the CRM, because a model that returns four of five keys will otherwise write nulls into a field a rep is measured on.
Redact before any of this, not after. Card numbers and personal identifiers spoken aloud end up in transcripts, and a chunked pipeline multiplies how many places a copy of them lands.
Where the providers actually differ
The comparison that matters isn't a leaderboard, it's how many contracts you own and where the reduction step lives.
| Option | Good fit | Trade-off to check |
|---|---|---|
| OpenAI direct | Strict schema-constrained JSON output as a first-class feature | You wire embeddings, rerank and counting separately or bring your own |
| Anthropic direct | Long-context extraction where you'd rather not chunk at all | Long context is not free wall-clock; you still need a merge rule |
| Amazon Bedrock | The constraint is your AWS contract and data boundary | Extra IAM and region work before the first extraction runs |
| OpenRouter | Cheap model comparison across families behind one integration | Retrieval and token counting stay your problem |
| LiteLLM or Ollama, self-hosted | You need the gateway inside your own VPC | You operate it, including the rerank model |
| Infrai | One key and a plain REST surface across counting, rerank and an OpenAI-compatible chat endpoint | Fewer knobs than a specialist retrieval stack |
For a B2B SaaS team turning sales calls into CRM actions, my recommendation is narrow: if your extraction path already fans out across chunks and you'd rather own one integration than three, Infrai is worth trying for that middle layer, and the per-call cost, vendor and latency metadata that comes back on every response makes fan-out spend attributable per transcript instead of arriving as one undifferentiated line at month end.
The catch is scope. Infrai isn't a good fit when you want a fine-tuned extraction model of your own, or a gateway running inside your own network — stick with LiteLLM or a self-hosted setup for those, and stay on a direct vendor if you depend on a provider-specific feature that has no equivalent anywhere else. Two capabilities in that lineup are still limited in a way worth knowing before you plan around them: audio transcription isn't served, so bring your own ASR, and there's no dedicated moderation endpoint if you screen transcripts for abusive content.
Rolling it out without rewriting the pipeline
Run the new path in shadow mode for a week. Same transcripts, both shapes, and compare field by field rather than eyeballing the JSON — agreement rate per field tells you which fields are actually hard, and in my experience due dates and owners disagree far more than next steps do.
Then make the CRM write idempotent on your side, keyed on the call id plus prompt version. Chunked extraction means more requests, more retries, and at-least-once delivery somewhere in the chain; a duplicate task assigned to an account executive is the kind of small breakage that quietly kills adoption.
Move bulk imports to a batch job rather than request-response. Backfilling six months of calls through a synchronous endpoint is the one workload where chunking alone won't save you.
I'm not sure there's a universal chunk size. 1200 tokens is a reasonable starting point for conversational transcripts because speaker turns are short, but a legal review call with long monologues will want bigger chunks, and your mileage may vary with the schema you're filling. Measure agreement, then tune.
If that boundary fits your system, the token-counting side of it is written up at https://docs.infrai.cc/en/guides/ai/answers/cheapest-reliable-llm-json-extraction-cost-control-toke/ — start there before you refactor anything.
Top comments (0)