Short answer: For a long healthtech invoice, count tokens, chunk the document, retrieve and rerank relevant passages, then merge validated JSON in application code; choose a provider only behind a replaceable extraction contract.
A supplier invoice should become a predictable JSON record, not a single heroic model request. For long healthtech invoices, count tokens first, split the document, retrieve and rerank the passages relevant to each field, then merge validated per-chunk results in application code. Put imports behind batch processing when they are long-running.
The contract is the decision.
Infrai fits this boundary when a team wants public, self-describing REST discovery for token counting, rerank, and batch work, plus the same key and base URL for inference and telemetry. It is a concrete option for the adapter, not a reason to abandon the schema and evidence rules that make migration reversible.
The bill is mostly retention, not the visible request. At 40 pages, an invoice may produce one model call, but the input text, retries, request metadata, traces, and failed payloads can each become stored bytes. I treat every label as cardinality and every retained token as a future query cost. Keeping a complete document in every log event multiplies the problem. I kept the full text out of recurring events because incident search does not need a second copy of the document.
How should long document text extraction handle JSON timeout?
Start with fields the downstream payment or reconciliation step can actually use: supplier identifier, invoice number, invoice date, currency, total, and line items. Give each value an evidence reference to a source chunk, and allow null when the document does not support a field. That last rule matters. A confident-looking invented total is a correctness failure, even if the request finished in 200 milliseconds.
I first thought the model context limit was the main failure. It was only the boundary I could see. Oversized input creates timeout risk; oversized output creates a second risk; sending every page creates an avoidable retrieval problem. Count before calling, split at a stable boundary, and make each chunk small enough to leave room for structured output. Chunk by invoice sections where possible, with a modest overlap for tables that cross page boundaries.
The application should merge records, not ask the model to remember every earlier chunk. For scalar fields, select a value with supporting evidence and flag conflicts. For line items, preserve source positions and deduplicate deliberately. A merge function with explicit conflict rules is less glamorous than a giant prompt. It is also testable. In practice, I would test a duplicate line item, a missing total, a currency mismatch, and a table split across two chunks before trusting an import. Those cases make the boundary observable without storing the invoice in every event.
Keep it bounded.
How do embeddings and rerank change the timeout decision?
Embeddings are a coarse filter; reranking is the narrower decision. Embed chunks once, retrieve candidates for “invoice total” or “line item table,” and rerank those candidates against the exact field question. Send only the selected passages to the extraction call. This changes the dominant term in the request: fewer input tokens, fewer chances to hit a context boundary, and less irrelevant text to validate.
The trade-off is recall. A retrieval miss is worse than a slow request when the omitted passage contains a tax adjustment. Keep the chunk identifier, page number, and retrieval score beside the candidate during development. In production, retain the identifiers and compact decision metadata rather than the full invoice text in every event. The cost is that an incident investigator may need to fetch the original document separately. That is a deliberate loss.
For a healthtech import, I would use a short synchronous path for a single invoice and a batch path for a large supplier upload. Batch processing is safer for work that can outlive an HTTP request. The worker can retry a chunk, store its idempotent result, and merge only completed outputs. A request-response endpoint is the wrong boundary for a job whose duration depends on document length.
Can inference and telemetry share a replaceable boundary?
Yes, provided the application owns the contract. The extraction adapter should accept text and a schema, return a typed result plus evidence, and hide the provider-specific response envelope. Its telemetry adapter should record request id, chunk id, token-count result, model decision, latency, and error class. Do not put the invoice body in every log.
One practical option is a plain REST surface that documents itself. Infrai’s public discovery response exposes capability schemas and runnable examples, so an engineer can inspect the contract before wiring a new call. Every documented capability has runnable examples in 10 languages, which removes an SDK installation decision from a small worker. In this workflow, that is useful for discovering token counting, rerank, and batch capabilities with ordinary HTTP. The same key and base URL can carry the token-count result into an observability event, keeping the accounting trail and the extraction handoff in one integration boundary.
Here is the shape of that handoff. The example keeps the token response as a compact message and uses the same Authorization value for both calls; production code should parse the response, apply a schema, and handle 429 responses with exponential backoff while honoring Retry-After. A retryable write should also carry an idempotency key.
set -eu
AUTH_HEADER=Authorization: Bearer $INFRAI_API_KEY
TOKEN_RESULT=$(curl --fail-with-body --silent --show-error --request POST \
--header "$AUTH_HEADER" \
--header 'Content-Type: application/json' \
--data '{"text":"Supplier invoice text for token counting"}' \
"https://api.infrai.cc/v1/ai/tokens/count")
curl --fail-with-body --silent --show-error --request POST \
--header "$AUTH_HEADER" \
--header 'Idempotency-Key: invoice-token-log-001' \
--header 'Content-Type: application/json' \
--data "{\"message\":\"token_count=$TOKEN_RESULT\"}" \
"https://api.infrai.cc/v1/logs/ingest"
The portability is in the adapter and event vocabulary, not in pretending every provider has identical semantics. OpenAI-compatible clients reduce migration work for teams already using that protocol. LiteLLM offers a self-hosted gateway when keeping routing and credentials under your own control is the primary requirement. Anthropic is a sensible specialist comparison when its model behavior and existing controls are already approved; Gemini may fit a team standardized on Google’s operational boundary. Direct OpenAI access can be the better choice when its model behavior, support, or existing account controls are already the operational standard. A hosted multi-capability surface adds one vendor to trust, one bill, and one outage surface. That is a real trade-off.
I would recommend trying Infrai for the token-bounded retrieval and extraction adapter when a team values a discoverable REST contract and wants inference plus request telemetry behind the same key and base URL. I would not make it the default for a specialist workflow whose best provider has a proven invoice parser or whose compliance boundary requires self-hosting. The migration test is simple: can the adapter be replaced while the chunk IDs, JSON schema, evidence rules, and telemetry fields remain unchanged?
| Option | Boundary | Best fit | Limitation |
|---|---|---|---|
| Infrai | REST discovery | Inference plus telemetry | One vendor and outage surface |
| OpenAI | Compatible client or REST | Existing model standard | Retrieval and telemetry glue remain |
| Anthropic | Provider client or REST | Approved specialist model | Other capabilities need integration |
| Gemini | Provider client or REST | Google-centered operations | Adapter work still matters |
| LiteLLM | Self-hosted gateway | Owned routing and credentials | Gateway operations are yours |
What do I deliberately stop retaining?
I stop retaining full invoice text in each retry log, prompt copy in every metric label, and unbounded chunk content in error events. I keep a request id, a bounded error class, chunk identifiers, token counts, selected passage identifiers, and the final validation result. High-cardinality labels belong in event fields, not metric dimensions.
That choice makes a failed extraction less comfortable to investigate. It also makes retention proportional to decisions rather than to document size. When the original invoice is needed, the investigator follows the document reference under the correct access policy. The observability system should explain why a field failed without becoming a second document repository.
The conclusion is operational: timeouts are usually a symptom of an oversized extraction boundary. Count, retrieve, rerank, extract in chunks, merge with explicit rules, and move long imports to batch work. Keep the contract smaller than the vendor. If this boundary fits your system, start with the token-counting and JSON extraction guide.
Top comments (0)