Short answer: make invoice extraction a bounded pipeline, not one heroic prompt. Measure tokens before the model call, chunk by invoice structure, retrieve only relevant passages, validate the JSON, and retry only operations that are safe to retry. This keeps a long supplier invoice from turning into a timeout while preserving a path to another model or gateway later.
I build for an edtech product that turns supplier invoices into fields such as invoice number, billing period, tax, currency, and line items. The awkward documents are not always the largest PDFs. A 12-page invoice with repeated terms and scanned tables can be harder than a clean 40-page statement because the useful evidence is sparse and badly ordered.
Timeout triage starts with the evidence boundary
Start with a provider-neutral contract. The application should pass text and a schema to an adapter and receive either typed JSON or a classified failure. Keep provider SDK objects out of the rest of the codebase. A gateway such as LiteLLM can sit behind that adapter, but the same boundary can also point directly at a self-hosted model.
The first pass is accounting. Estimate tokens, preserve page and table-row metadata, and reject or split input that cannot fit the selected context budget. A timeout is an operational symptom; an overlong prompt is a predictable input condition. Treat those differently in logs and dashboards. When the worker sees an HTTP 408 or its own deadline, it should retain the source hash and chunk IDs before returning the job to the queue; otherwise the next attempt has no way to tell “same work, later attempt” from “new invoice.” That distinction affects deduplication, review screens, and the eventual provider migration, because evidence remains comparable even when the model response format changes.
type InvoiceField = "invoiceNumber" | "billingPeriod" | "currency" | "total";
type Passage = {
id: string;
text: string;
page: number;
tokens: number;
};
type ExtractRequest = {
passages: Passage[];
fields: InvoiceField[];
};
type ModelAdapter = {
extractJson(request: ExtractRequest, signal: AbortSignal): Promise<unknown>;
};
function budgetPassages(passages: Passage[], maxTokens: number): Passage[] {
const selected: Passage[] = [];
let used = 0;
for (const passage of passages) {
if (used + passage.tokens > maxTokens) break;
selected.push(passage);
used += passage.tokens;
}
return selected;
}
The estimate doesn't need to be perfect. It needs to be conservative and consistent. Keep a safety margin for the instruction, schema, and response. Your mileage will vary by tokenizer, so record the estimate and the actual usage returned by your runtime instead of pretending one character ratio is universal.
Ship the boundary first.
The evidence ledger is the product.
How do chunking, embeddings, and rerank fit a long-document JSON extraction flow?
Chunk on document meaning first: invoice header, billing terms, tax summary, then line-item groups. A fixed character window is a useful fallback, but it should not cut a table row in half. Give every chunk an ID and source coordinates so a final value can cite the page and row that supported it.
Embeddings are a recall tool, not an extraction result. Embed chunks and a field-specific query such as “tax total and tax rate,” then take a wider candidate set than you plan to send to the model. Rerank those candidates with lexical and structural signals: exact field labels, currency symbols, proximity to a total, and page position. This two-stage retrieval keeps the prompt small without trusting semantic similarity alone.
For an invoice, I usually retrieve separately per field group. The line-item query should not crowd out the header query just because the line-item section contains more text. If retrieval returns no passage with a plausible label or value, return an explicit missing_evidence status. Guessing a total is worse than asking an operator to inspect page 8.
A minimal TypeScript implementation with bounded retries
The example below shows the orchestration, not a vendor-specific request. It applies a deadline, retries only a classified transient failure, and validates the response before accepting it. HTTP retry semantics matter here: RFC 9110 distinguishes methods and idempotency, so do not blindly replay an operation that may create a side effect.
class TransientModelError extends Error {}
function isRetryable(error: unknown): boolean {
return error instanceof TransientModelError;
}
function validateInvoice(value: unknown): Record<string, unknown> {
if (!value || typeof value !== "object" || Array.isArray(value)) {
throw new Error("invalid_json_shape");
}
const result = value as Record<string, unknown>;
for (const key of ["invoiceNumber", "billingPeriod", "currency", "total"]) {
if (!(key in result)) throw new Error(`missing_field:${key}`);
}
return result;
}
export async function extractInvoice(
adapter: ModelAdapter,
passages: Passage[],
signal: AbortSignal,
): Promise<Record<string, unknown>> {
const selected = budgetPassages(passages, 6_000);
const request: ExtractRequest = {
passages: selected,
fields: ["invoiceNumber", "billingPeriod", "currency", "total"],
};
let lastError: unknown;
for (let attempt = 0; attempt < 3; attempt += 1) {
try {
return validateInvoice(await adapter.extractJson(request, signal));
} catch (error) {
lastError = error;
if (!isRetryable(error) || signal.aborted) throw error;
await new Promise((resolve) => setTimeout(resolve, 250 * 2 ** attempt));
}
}
throw lastError instanceof Error ? lastError : new Error("extraction_failed");
}
The retry loop is intentionally boring. A model completion is a read-like operation in this adapter; a separate “save extracted invoice” request must carry an application idempotency key or be guarded by a unique invoice identifier. Otherwise a timeout after persistence can produce duplicate accounting records even when the JSON itself was correct.
One concrete failure mode is a document that fits the token limit but exceeds the wall-clock limit because retrieval and generation happen serially for every field. The trace then looks deceptively healthy: parsing completes, embeddings complete, and each model call is individually acceptable, yet the sum crosses the worker deadline while a database lease remains open. Parallelize independent field queries, cap concurrency, and set one overall deadline. Don't increase the deadline forever: a queue worker that holds a database lease while waiting can block the next invoice, and a retry can amplify that backlog if the original request is still running.
Which trade-offs decide whether this design is right?
| Decision | Useful default | Cost or limitation |
|---|---|---|
| Chunk size | Structure-aware chunks with token caps | More chunks mean more embedding and storage work |
| Retrieval | Embeddings followed by rerank | Recall errors can hide evidence; keep an audit trail |
| Output | Strict schema validation plus field-level nulls | Validation rejects ambiguous documents instead of guessing |
| Timeout | One deadline across retrieval and generation | A short deadline may return deadline_exceeded on unusually complex invoices |
| Provider boundary | Small HTTP/TypeScript adapter | The adapter becomes your responsibility to test and maintain |
This approach is not suitable when every invoice must be reproduced verbatim with legally complete layout fidelity; a document parser and human review path may be the better primary system. It is also a poor fit for tiny, uniform invoices where a single short request is cheaper to operate. Stick with a simpler call when measurements show the input is consistently below budget and the error rate is already acceptable.
The portability advantage is practical rather than ideological: field schemas, chunk IDs, evidence coordinates, and validation stay in your code while the model endpoint remains replaceable. LiteLLM is one option for presenting a common gateway across providers, but it does not remove the need to test tokenization, latency, structured-output behavior, and failure classes for each backend.
I am not sure any static timeout number will age well as supplier formats change. That is why I alert on p95 latency, token counts, missing-evidence rates, and validation failures by document type, then sample the stored evidence rather than only the final JSON.
The operational checklist is short: persist the source hash, chunk metadata, retrieval scores, model request ID, token usage, deadline outcome, and schema errors; redact invoice content from general logs; replay a fixed corpus in CI; and put ambiguous totals in a review queue. Those records make a provider migration a comparison exercise instead of a rewrite.
References
- RFC 9110: HTTP Semantics (idempotency and retry semantics): https://www.rfc-editor.org/rfc/rfc9110
- LiteLLM, self-hosted LLM gateway: https://github.com/BerriAI/litellm
Top comments (0)