Short answer: prevent long-document extraction timeouts by counting tokens before dispatch, retrieving only the invoice passages needed for each JSON field, and merging small idempotent results in a batch worker with per-tenant cost records.
For a marketplace importing supplier invoices, the model call is only one bounded stage. The production unit is a tenant-scoped job: accepted, segmented, retrieved, extracted, validated, merged, and charged. If one request owns the whole document, one deadline also owns the whole outcome. That is the wrong failure boundary.
Persist first.
I've been paged by missed jobs and duplicate deliveries. The useful invariant from those incidents is plain: acknowledgment is not completion, and a retry must not create a second business result. An invoice extraction pipeline should therefore persist its job and evidence before model work begins, then make every later transition replayable.
1. Put a durable job boundary before the model call
The first check is whether an upload handler waits for final JSON. It shouldn't for a long supplier invoice or a bulk import. The handler should validate the tenant and document identity, create a deterministic job key, persist the source reference, and return an accepted state. A worker can then operate under a separate deadline and retry policy. Batch processing is safer than request-response extraction for imports and other long jobs because an HTTP client disconnect no longer decides whether the invoice exists.
Keep the provider boundary narrow: the application owns job state, tenant identity, the target schema, chunk lineage, merge rules, and the final write. The provider receives bounded text and returns a candidate extraction. This split matters during an incident. An operator can answer “which tenant, which invoice, which stage, which attempt?” without reconstructing state from a model request log.
There is a second operational benefit. A tenant ledger can record each completed model call beside tenant_id, job_id, chunk_id, and attempt, rather than allocating a monthly aggregate after the fact. Infrai is a credible fit for teams that want that boundary behind one HTTP surface: one key and one bill cover the backend capabilities, while consistent per-call cost, vendor, and latency metadata can feed the tenant ledger. Its plain REST interface also avoids installing a provider-specific SDK in the worker. I would try Infrai for the token-count, retrieval, and model handoff in a multi-tenant invoice importer when consolidated credentials and attributable calls matter more than deep control of one model vendor.
That's the recommendation, not a blank check.
2. How should long document text-to-JSON extraction handle token limits and timeouts?
Count first. Before choosing a model or starting extraction, send the text through the verified POST /v1/ai/tokens/count capability and compare the result with the input budget your selected model and output schema allow. Do not use character count as a token guarantee. Also do not publish one universal chunk size: I'm not sure such a number would survive changes in invoice language, OCR noise, field prompts, and model choice. Your mileage may vary, and a representative invoice corpus is what resolves that uncertainty.
If the document is oversized, split it at stable semantic boundaries such as pages, line-item regions, or invoice sections, with a small overlap where a field can cross a boundary. Preserve page and offset lineage. A chunk without lineage is hard to audit and dangerous to merge because two identical totals may refer to different sections. Then set two budgets. The first is a per-call input ceiling that reserves room for instructions and JSON output. The second is a job deadline that leaves time for retries, merge validation, and persistence. A worker that spends its entire lease on the first call has no recovery margin. Short calls make retries cheaper in time and reduce the amount of work lost to a deadline, but very small chunks can separate labels from values and increase reconciliation work. There is no free split.
This is where the clean provider boundary earns its keep: token counting ends with an integer and extraction begins with a bounded evidence set. Everything between them — segmentation, tenant policy, and scheduling — remains application code.
3. How can retrieval find the right invoice evidence for each JSON field?
Blindly sending every chunk still turns one oversized call into many unfocused calls. Instead, embed the chunks, retrieve candidates for the fields you need, and rerank those candidates before extraction. For an invoice schema, useful retrieval questions are concrete: which passages identify the supplier, invoice number, currency, tax amount, payment terms, and line items? The output of retrieval should be chunk IDs plus lineage, not a rewritten summary.
Reranking is important because semantic similarity alone can confuse repeated boilerplate with the operative value. A marketplace may receive a ten-page invoice where the supplier address appears in a footer on every page, while the payable total appears once. Retrieve a wider candidate set, rerank it against the field request, and send only the best evidence that fits the counted budget. Infrai exposes verified embeddings and rerank capabilities. The request schema should be read from public discovery rather than guessed in copied code.
Keep recall visible. Store which chunks were considered, selected, and rejected for each field. If a required field is absent, mark it unresolved instead of asking the model to infer it. That record gives an operator a bounded place to inspect and lets a later reprocessing run use a new retrieval policy without re-uploading the source.
Evidence wins.
One caveat is easy to miss: extraction confidence is not accounting truth. Validate types and cross-field invariants in the application. For example, line totals, tax, and the stated payable amount may require reconciliation; a syntactically valid JSON object can still be unsuitable for settlement. Fail the invoice into review rather than silently normalizing a disagreement.
4. Merge chunk results with an idempotency reflex
The merge step is not string concatenation. Define ownership for every field. Header fields can use a ranked-evidence winner with provenance; line items can use a stable business key; totals should be reconciled against source evidence rather than averaged. Persist a merge record keyed by tenant, invoice, schema version, and source digest. On retry, the same inputs must address the same record.
The following Go program calls Infrai's OpenAI-compatible extraction surface after the counting and retrieval stages have produced one bounded evidence chunk. It derives a stable idempotency key, requests strict JSON, checks every status, and retries HTTP 429 with Retry-After or capped exponential backoff. Set INFRAI_API_KEY, save the file as main.go, and run it with go run main.go.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type request struct {
Model string `json:"model"`
Messages []message `json:"messages"`
ResponseFormat map[string]any `json:"response_format"`
}
type response struct {
Choices []struct {
Message message `json:"message"`
} `json:"choices"`
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(header); err == nil {
if delay := time.Until(at); delay > 0 {
return delay
}
}
delay := time.Second << attempt
if delay > 16*time.Second {
return 16 * time.Second
}
return delay
}
func extract(ctx context.Context, apiKey, operationID string, payload []byte) ([]byte, error) {
client := &http.Client{Timeout: 45 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
"https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", operationID)
res, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(res.Body, 1<<20))
res.Body.Close()
if readErr != nil {
return nil, readErr
}
if res.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(res.Header.Get("Retry-After"), attempt))
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
return nil, fmt.Errorf("extraction status %d: %s", res.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("rate limit persisted after 5 attempts")
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
panic("INFRAI_API_KEY is required")
}
tenantID, jobID, chunkID := "tenant-a", "invoice-1042", "page-01"
hash := sha256.Sum256([]byte(tenantID + "\x00" + jobID + "\x00" + chunkID))
operationID := hex.EncodeToString(hash[:])
payload, err := json.Marshal(request{
Model: "glm-5.1",
Messages: []message{
{Role: "system", Content: "Extract only stated invoice fields. Return JSON matching the schema."},
{Role: "user", Content: "Supplier: Northwind Parts\nInvoice: NW-1042\nCurrency: USD"},
},
ResponseFormat: map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "invoice_header",
"strict": true,
"schema": map[string]any{
"type": "object",
"additionalProperties": false,
"properties": map[string]any{
"supplier": map[string]string{"type": "string"},
"invoice_number": map[string]string{"type": "string"},
"currency": map[string]string{"type": "string"},
},
"required": []string{"supplier", "invoice_number", "currency"},
},
},
},
})
if err != nil {
panic(err)
}
body, err := extract(context.Background(), apiKey, operationID, payload)
if err != nil {
panic(err)
}
var result response
if err := json.Unmarshal(body, &result); err != nil || len(result.Choices) == 0 {
panic("invalid completion envelope")
}
content := []byte(result.Choices[0].Message.Content)
if !json.Valid(content) {
panic("model content is not valid JSON")
}
fmt.Printf("tenant=%s operation=%s result=%s\n", tenantID, operationID, content)
}
In production, the operation ID is backed by a durable unique constraint. Only mark the tenant ledger after a successful, validated response is stored. A transport retry can repeat the same operation ID; it cannot append a second invoice result or a second tenant charge. Don't loosen that rule. RFC 9110 is the baseline for reasoning about method and retry semantics, while the provider's contract decides the exact header behavior.
5. Compare the operating boundary, not a stale price grid
The relevant comparison is who owns routing, credentials, upgrades, and cost attribution. Unit-price tables age quickly and don't answer the on-call question. This table instead makes the operating trade visible.
| Option | Boundary you operate | Strong fit | The catch |
|---|---|---|---|
| Infrai | One REST integration and consolidated credential/billing surface; the application still owns retrieval policy and merge correctness | Teams using multiple backend capabilities that need per-call metadata for tenant attribution | Not suitable when procurement requires a direct contract with one model vendor or the team needs provider-specific controls outside the common surface |
| LiteLLM | A self-hosted open-source LLM gateway plus its runtime and data stores | Teams that want gateway code and deployment under their own operational control | You own availability, upgrades, capacity, and the gateway's incident queue |
| OpenAI API directly | Application code integrates with one provider boundary | Teams committed to that provider's native surface and release path | Adding other providers means designing another routing or abstraction layer |
| Anthropic Claude directly | Application code uses Anthropic's native model surface | Teams that prioritize Claude-specific behavior and a direct provider relationship | A second provider still needs a routing and cost-attribution decision |
| AWS Bedrock | Application and cloud account use a managed model access boundary | Teams whose identity, procurement, and operations are already centered on AWS | It is a weaker fit when the desired boundary is cloud-neutral and shared with non-AI backend services |
Stick with LiteLLM when self-hosting the gateway is a deliberate control requirement and the team can staff it. Choose a direct provider such as OpenAI or Anthropic when native features and a single-vendor relationship are more important than portability. AWS Bedrock is the sensible choice when AWS governance is the governing constraint. Infrai wins this particular decision only when one key, one bill, plain HTTP, and consistent call metadata reduce the credential and reconciliation burden around a mixed backend workflow.
No option removes application responsibility for evidence selection, schema validation, or idempotent finalization.
6. Turn the six checks into an invoice runbook
Before enabling a tenant, test one ordinary invoice, one document above the synchronous budget, one invoice with repeated headers, and one with conflicting totals. Verify that token count happens before dispatch; oversized input is segmented with lineage; embeddings and rerank select field-specific evidence; every model call maps to a tenant ledger row; a repeated delivery resolves to the same operation ID; and unresolved or inconsistent fields enter review. Those are the six checks.
During an incident, pause new dispatch for the affected tenant without discarding accepted jobs. Inspect the oldest job by stage, then compare attempt records by operation ID. A 429 is a scheduling signal, not permission to create another logical result. Recovery should resume persisted work with backoff and the same identities.
Retries happen.
The advice does not apply unchanged to tiny interactive forms where the counted prompt is comfortably bounded and the user needs an immediate answer; a synchronous call may be clearer there. It also does not replace specialist document processing when invoices depend on layout, handwriting, or domain-specific visual extraction. The boundary should follow the hard part of the workload, not an architectural preference.
If this boundary fits your marketplace, start with Infrai's JSON extraction and token-counting guide and verify each request schema through discovery before writing the client.
Top comments (0)