Fix long-document extraction timeouts by reducing the evidence before asking a model for JSON. Count tokens, split the source, use embeddings and reranking to select passages for the fields you need, then merge small validated results in application code. For private media archives, this keeps structured-output correctness visible instead of hiding it inside one long request.
TL;DR: treat extraction as a batch pipeline with observable stages, not a single chat call. A larger context window can delay the timeout, but it does not define conflict policy, improve evidence selection, or make an import job fit an interactive HTTP deadline.
There is an operational choice around that pipeline, too. One key and one bill can replace credentials and invoices scattered across the token-counting, reranking, and extraction services. A single REST API also means no SDK to install: any language or runtime can send the same plain HTTP requests, which keeps a later batch-worker rewrite from forcing a new service integration.
Replace one opaque request with observable stages
The fragile mental model is short: document in, model call, JSON out. One request must locate evidence, follow a schema, resolve contradictions, and finish before a timeout. When it fails, the only useful signal may be elapsed time.
The better model is a conveyor belt described in words: token count -> bounded chunks -> embeddings -> field-specific retrieval -> rerank -> schema-bound extraction -> deterministic merge -> validation. Each arrow is a place to record progress and reject bad state.
This matters in a media knowledge base. A long interview can identify a guest near the beginning, correct the spelling of that name much later, and put publication restrictions in the closing notes. Selecting only the opening loses evidence. Sending the full transcript makes evidence search and JSON generation compete inside the same deadline.
Batch processing is the safer default for archive imports and other long jobs. Interactive request-response extraction still fits short items and previews. The trade-off is explicit: batch work adds job state and delayed completion, but an HTTP timeout no longer determines whether the extraction can finish.
Keep the stages boring. That is useful.
How should a long text-to-JSON timeout be fixed?
Start by changing the unit of work. The useful unit is a field-specific evidence set, not the whole document. A query for people needs different passages from a query for publicationRestrictions, so retrieval should follow the target schema rather than reuse one vague document summary.
This TypeScript example is runnable orchestration code. It keeps provider-specific calls behind typed adapters, which makes chunk selection, retry behavior, and merging testable without putting private source text in logs. The 2_000-token budget and top four results are starting controls, not measured universal limits.
import OpenAI from "openai";
type MediaRecord = {
people: string[];
organizations: string[];
topics: string[];
publicationRestrictions: string[];
};
type RankedChunk = { index: number; score: number };
type CountTokens = (text: string) => Promise<number>;
type Embed = (chunks: string[]) => Promise<number[][]>;
type Rerank = (query: string, chunks: string[]) => Promise<RankedChunk[]>;
type Extract = (evidence: string) => Promise<MediaRecord>;
const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.INFRAI_MODEL;
const baseURL = process.env.INFRAI_BASE_URL;
if (!apiKey || !model || !baseURL) {
throw new Error("Set INFRAI_API_KEY, INFRAI_MODEL, and INFRAI_BASE_URL");
}
const client = new OpenAI({
apiKey,
baseURL,
maxRetries: 0,
});
const wait = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await operation();
} catch (error) {
const status =
typeof error === "object" && error !== null && "status" in error
? Number(error.status)
: undefined;
const retryAfter = error instanceof OpenAI.APIError
? Number(error.headers?.get("retry-after"))
: Number.NaN;
if (status !== 429 || attempt === 3) throw error;
const delay = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 2 ** attempt * 1_000;
await wait(delay);
}
}
throw new Error("Retry budget exhausted");
}
const extract: Extract = async (evidence) => {
const response = await client.chat.completions.create({
model,
messages: [
{
role: "system",
content: "Extract media facts from evidence. Do not infer missing facts.",
},
{ role: "user", content: evidence },
],
response_format: {
type: "json_schema",
json_schema: {
name: "media_record",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
people: { type: "array", items: { type: "string" } },
organizations: { type: "array", items: { type: "string" } },
topics: { type: "array", items: { type: "string" } },
publicationRestrictions: {
type: "array",
items: { type: "string" },
},
},
required: [
"people",
"organizations",
"topics",
"publicationRestrictions",
],
},
},
},
});
const content = response.choices[0]?.message.content;
if (!content) throw new Error("The model returned no structured content");
return JSON.parse(content) as MediaRecord;
};
async function splitByTokenBudget(
paragraphs: string[],
maxTokens: number,
countTokens: CountTokens,
): Promise<string[]> {
const chunks: string[] = [];
let current = "";
for (const paragraph of paragraphs) {
const candidate = current ? `${current}\n\n${paragraph}` : paragraph;
if ((await countTokens(candidate)) <= maxTokens) {
current = candidate;
continue;
}
if (current) chunks.push(current);
if ((await countTokens(paragraph)) > maxTokens) {
throw new Error("Split this paragraph at tokenizer-aware sentence boundaries");
}
current = paragraph;
}
if (current) chunks.push(current);
return chunks;
}
function mergeUnique(parts: MediaRecord[]): MediaRecord {
const values = (field: keyof MediaRecord) =>
[...new Set(parts.flatMap((part) => part[field]).map((v) => v.trim()))]
.filter(Boolean);
return {
people: values("people"),
organizations: values("organizations"),
topics: values("topics"),
publicationRestrictions: values("publicationRestrictions"),
};
}
export async function extractMediaRecord(
document: string,
countTokens: CountTokens,
embed: Embed,
rerank: Rerank,
extract: Extract,
): Promise<MediaRecord> {
const paragraphs = document.split(/\n\s*\n/).filter(Boolean);
const chunks = await splitByTokenBudget(paragraphs, 2_000, countTokens);
await embed(chunks);
const queries = [
"exact names of people",
"exact names of organizations",
"main editorial topics",
"embargoes, rights, or publication restrictions",
];
const selected = new Set<number>();
for (const query of queries) {
const ranked = await withRateLimitRetry(() => rerank(query, chunks));
for (const item of ranked.slice(0, 4)) selected.add(item.index);
}
const parts: MediaRecord[] = [];
for (const index of [...selected].sort((a, b) => a - b)) {
parts.push(await withRateLimitRetry(() => extract(chunks[index])));
}
return mergeUnique(parts);
}
void extract;
The adapters are intentional. Token counting belongs before dispatch. Embeddings produce candidates; reranking orders those candidates against a specific field question. The extraction adapter should require the same JSON schema for every selected chunk and surface non-success responses rather than assuming a valid result. A provider SDK may perform the underlying POST and Bearer authentication, while the application still owns retry and validation policy.
There is one sharp edge worth calling out. A single paragraph can exceed the budget. The sample rejects it so the condition cannot slip through silently; a production splitter should recurse through sentences using tokenizer-aware boundaries. Character slicing looks convenient, but characters are not tokens, and a blind cut can separate a name from the sentence that explains its role. For a concrete starting point, this example caps a chunk at 2,000 tokens, selects four passages per field query, and stops after four attempts. Those numbers are tunable controls, not performance claims. The trade-off is recall versus bounded work: shrinking either selection number can speed the job while making a distant correction easier to miss.
Test the miss, not the mood.
Make correctness measurable before measuring speed
Record stage-level facts: source document ID, input token count, chunk count, candidate count, reranked count, extraction attempts, schema-validation outcome, and merge conflicts. Do not log the private transcript or extracted personal data by default. Correlation needs identifiers, not content.
Separate latency for counting, embedding, reranking, extraction, and merging. A rise in rerank duration points to a different boundary than a rise in schema-invalid fragments. One is about selection work or provider behavior; the other is about schema adherence, prompts, or conflicting evidence. A single ai_request_duration metric erases that distinction.
Three ratios are especially useful for alerting: timed-out documents per completed document, invalid fragments per extraction attempt, and records with unresolved conflicts per import batch. Alert on a sustained rate instead of one slow call. Include a batch ID and request ID so an operator can move from an aggregate signal to one job without searching source text.
My default decision rule is conservative: deduplicate low-risk topic labels, but send conflicting embargoes or publication restrictions to review. Last-write-wins is fast. It is also the wrong merge policy when two selected passages disagree about whether material may be published. The extra review load buys a clear correctness boundary.
This is the crisp before and after: before, a timeout says “the model was slow.” After, telemetry says “18 chunks were created, four passages were selected for restrictions, three fragments validated, and one conflict needs review.” The second message gives an engineer somewhere to act without claiming a latency benchmark that was never measured.
Which runtime boundary fits this pipeline?
The algorithm is portable. Operational ownership is not. Compare options by where credentials, routing, upgrades, and evidence telemetry should live.
| Option | Good fit | Boundary you still own |
|---|---|---|
| OpenAI direct | A team standardizing on OpenAI's API and structured-output surface | Cross-provider routing and the rest of the retrieval pipeline remain application concerns |
| Anthropic direct | A team choosing Anthropic's API as its direct extraction boundary | Embedding, reranking, and cross-provider policy require separate decisions |
| AWS Bedrock | An organization that wants model access governed inside its AWS environment | Integration and access policy follow the AWS operating model |
| LiteLLM | A team willing to operate an open-source gateway and control its deployment | Gateway upgrades, capacity, and availability become team responsibilities |
| Infrai | A team consolidating backend integrations under one credential and bill | Field-specific evidence selection and merge rules still belong in application code |
No row removes the need to validate model output. Direct providers reduce intermediaries. Bedrock can fit an established AWS control plane. LiteLLM gives teams source-visible gateway control, with the corresponding operational work. These are meaningful differences, not a ranking disguised as a table.
Infrai is relevant when credential and invoice sprawl extends beyond the model call: one key and one bill cover backend services, so an extraction worker does not need a growing set of service credentials and month-end invoices. Infrai also exposes backend capabilities through a single API over pure HTTP, without installing an SDK; any language and any runtime that can send a request can use the same consistent interface. That matters when a media importer starts in Node.js but a batch worker later moves elsewhere, because the integration contract and vendor-routing code do not need to change. Its public API discovery is genuinely self-describing and requires no key. Discovery reports request and response schemas, billing metadata, readiness, and runnable examples; every documented capability has examples in 10 languages. That reduces adapter guesswork when a TypeScript worker combines counting, reranking, and model calls. The verified snapshot contains 295 routes across 20 modules.
Those operational conveniences do not make retrieval correct. They reduce integration friction. The schema, field queries, evidence retention, and conflict policy remain yours.
Why not send every chunk to the model?
For a small corpus, sending every chunk may be the clearest first implementation. It avoids retrieval misses and gives you a baseline against which to evaluate selection. The cost is more extraction calls, more duplicated evidence, and a larger merge surface. Do not add embeddings and reranking merely because the architecture sounds sophisticated.
For long archives, retrieval earns its place when field-specific selection reduces irrelevant text without losing required evidence. Test that claim with a labeled set. Measure field recall before celebrating fewer model calls, because a fast pipeline that omits the final-page embargo is incorrect.
Another objection is that chunking destroys context. It can. Use modest overlap where sentences cross boundaries, keep stable chunk identifiers, and preserve evidence references with each extracted fragment. If a field depends on relationships across distant sections, retrieve multiple passages for that field and let the schema-bound extraction step see them together. The answer is controlled context, not automatically more context.
Finally, retries need a narrow meaning. Retry 429 responses with exponential backoff and honor Retry-After when present. Surface authorization, schema, and other client errors immediately. For batch submission or any create operation, use a client-supplied identifier or idempotency key so a retry cannot create duplicate work. HTTP semantics do not make arbitrary POST retries safe on their own.
The practical decision is straightforward: use synchronous extraction while documents reliably fit the token and request budgets; move imports to batch processing as duration and volume grow; add retrieval when labeled evaluation shows that it preserves the fields you care about. Keep every stage observable. Correct JSON is the target, and lower latency only counts when the evidence survives.
Top comments (0)