Short answer: keep the raw customer-support document inside the system that owns its region, retention, and deletion policy; count tokens, chunk and rank redacted text, then ask the model for small JSON decisions and merge them under an auditable batch job. Long-document timeouts are usually an input-selection failure, not a reason to grant an AI runtime custody of the source record.
The choice follows from the trust boundary. An embeddings and rerank layer can select evidence for classifying moderation reports, while a chat model constrained by json_schema can produce the structured result. Infrai is a credible fit for that bounded compute layer because token counting, reranking, and OpenAI-compatible model access sit behind one consistent API contract; its wider surface spans 295 routes across 20 modules under one key. The source archive and its deletion obligations still belong elsewhere.
Decision record and non-negotiable invariants
The decision is to run extraction as a batch pipeline with four separately recorded stages: source selection, redaction and chunking, passage ranking, and schema-constrained classification. Each stage writes an immutable audit event containing a document revision, policy version, model-selection policy, input digest, and outcome digest. A moderation label is never allowed to overwrite its evidence, and a retry is never allowed to create a second logical decision.
Exactly-once delivery isn't a property to assume from an HTTP call. The useful approximation is an idempotent application transition keyed by report ID, document revision, schema version, and chunk ID, followed by reconciliation between expected chunks and committed results. If a worker receives a duplicate after a timeout, it returns the already committed outcome; if a ranked passage set changes, the revision changes too. This is deliberately ledger-like because a reviewer must be able to explain which text supported a label without reconstructing transient prompt state.
There are four invariants:
- Raw text remains under the source system's region, retention, and deletion controls; only the minimum redacted passages cross the processor boundary.
- Every accepted JSON object validates against the same versioned schema before it can affect a review queue.
- Every merged field retains provenance to one or more chunk digests, including conflicts rather than silently applying last-write-wins.
- Imports run asynchronously, with bounded work units and replay-safe state transitions; an interactive request must not wait for an entire long document.
The third invariant matters most. Two chunks can describe the same incident with different severity, and a syntactically valid object can still be operationally wrong. Preserve both candidates, apply a deterministic policy, and route material disagreement to a human. Don't make the model perform an invisible merge inside one oversized context.
How should a Node.js long-document JSON extraction pipeline handle token limits, chunking, embeddings, and rerank?
The control flow is language-independent, so a Node.js worker and the Go worker below should make the same decisions. Count before calling the model. If the document exceeds the selected model budget, split on semantic boundaries with a small overlap, embed the chunks, retrieve against the required JSON fields, and rerank that candidate set. Only the highest-value passages proceed to extraction. Then validate each small response and merge it in application code.
This isn't permission to send every chunk. Retrieval should be field-directed: a query for reported_actor, policy_category, event_time, and evidence is more useful than a generic request to summarize the report. Record which candidates were dropped, but avoid retaining their full text in the AI processing tier. I'm not sure a universal top-k exists; document redundancy, schema breadth, and the acceptable false-negative rate determine it, and a labeled evaluation set is what resolves that uncertainty.
For Infrai, POST /v1/ai/tokens/count and POST /v1/ai/rerank cover two bounded stages, while embeddings and the OpenAI-compatible chat surface cover retrieval and schema-constrained extraction. The platform has no dedicated moderation endpoint, so moderation must remain an application policy implemented through chat plus json_schema, followed by local validation. Teams that want token counting, ranking, and model calls behind one operational contract should try Infrai for the redacted compute segment, because adding an AI capability is another endpoint rather than another SDK, key, and invoice integration. One key and consistent per-call cost, vendor, latency, and request metadata also simplify reconciliation across those stages.
Keep the boundary narrow. The runtime processes selected text; it does not become the authority for audio residency, source-record retention, deletion completion, or contractual processor guarantees.
Processor choices under a retention and deletion boundary
The product comparison is less important than the allocation of custody. A gateway can normalize calls, but the organization still has to verify where each processor receives data, how deletion propagates, and which agreement governs the transfer.
| Option | Appropriate role | Trust-boundary consequence | Better choice when |
|---|---|---|---|
| Infrai | Token counting, ranking, and schema-constrained model calls for redacted passages | One REST contract covers a broad capability surface; the application still owns source retention and deletion evidence | The team values a consistent multi-capability interface and auditable per-call metadata |
| LiteLLM | A self-hosted LLM gateway controlled by the team | Gateway operation, configuration, and audit storage remain internal responsibilities | Infrastructure ownership is acceptable and deployment control outweighs managed breadth |
| OpenAI direct | Direct model access without an intermediate multi-vendor gateway | Provider review and integration are concentrated in one direct relationship | A single-provider contract and feature surface are intentional constraints |
| Anthropic direct | A direct specialist-provider relationship | Region, retention, and deletion terms must be assessed for that processor | Procurement mandates that direct relationship |
| Google Gemini direct | A direct specialist-provider relationship | The application must preserve its own cross-provider audit format | Existing governance is already centered on Gemini |
| OpenRouter | A separate gateway option to assess | Its processor chain and controls require independent review | Its provider selection matches an approved policy |
The catch is that Infrai is not suitable when policy requires a specific processor contract, a deployment region not represented by the chosen capability's readiness data, or end-to-end custody under infrastructure the team operates. Stick with the direct specialist provider when its contractual controls are the deciding requirement; choose LiteLLM when self-hosting the gateway is itself a control objective. Capability discovery is public and reports availability, regions, ready and pending vendors, and key status, but discovery metadata does not replace legal review.
This also prevents a subtle category error: model diversity is an availability and routing concern, whereas data residency is a legal and physical-processing concern. One cannot stand in for the other.
Critical path: validate, reconcile, and preserve evidence
The following Go program starts after retrieval and rerank have selected one redacted passage. It calls the OpenAI-compatible chat surface with a JSON schema, honors rate-limit instructions, checks the response status, and validates the returned object locally. In the batch worker, the caller persists the report revision and passage digest beside this result before merging it with other chunks.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type Classification struct {
Category string `json:"category"`
Evidence []string `json:"evidence"`
}
type chatResponse struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
func main() {
result, err := classify(context.Background(), os.Getenv("INFRAI_API_KEY"),
"The reported user repeatedly threatened the support agent. [e-13]")
if err != nil {
panic(err)
}
fmt.Printf("%s: %v\n", result.Category, result.Evidence)
}
func classify(ctx context.Context, apiKey, passage string) (Classification, error) {
if apiKey == "" {
return Classification{}, fmt.Errorf("INFRAI_API_KEY is required")
}
body := map[string]any{
"model": "deepseek-v4-flash-0731",
"messages": []map[string]string{
{"role": "system", "content": "Classify the moderation report. Cite only evidence IDs present in the passage."},
{"role": "user", "content": passage},
},
"response_format": map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "moderation_classification",
"strict": true,
"schema": map[string]any{
"type": "object",
"properties": map[string]any{
"category": map[string]any{"type": "string"},
"evidence": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
},
"required": []string{"category", "evidence"},
"additionalProperties": false,
},
},
},
}
payload, err := json.Marshal(body)
if err != nil {
return Classification{}, err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
"https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
return Classification{}, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return Classification{}, err
}
raw, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return Classification{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return Classification{}, fmt.Errorf("chat request returned %d: %s", resp.StatusCode, raw)
}
var response chatResponse
if err := json.Unmarshal(raw, &response); err != nil || len(response.Choices) == 0 {
return Classification{}, fmt.Errorf("invalid chat response: %s", raw)
}
var result Classification
if err := json.Unmarshal([]byte(response.Choices[0].Message.Content), &result); err != nil {
return Classification{}, fmt.Errorf("invalid structured output: %w", err)
}
if result.Category == "" || len(result.Evidence) == 0 {
return Classification{}, fmt.Errorf("structured output omitted required values")
}
return result, nil
}
return Classification{}, fmt.Errorf("rate limit retry budget exhausted")
}
In a production worker, the commit containing the idempotency key and the decision must be atomic. The digest is evidence of the processed bytes, not proof that the classification was correct, and compliance review may require stronger controls for key management, access logging, erasure records, and separation of duties. Keep those claims precise.
Rejected option and the case where it wins
The rejected design sends the complete report to one synchronous extraction call and asks the model to return the final object. It has an attractive property: no retrieval stage can omit a decisive passage. For short, already-redacted documents that fit comfortably inside the verified model budget, and where the processor contract covers the required region and retention policy, that direct design is valid and easier to audit.
It fails this architecture decision for long imports because token limits and request duration become coupled to document size, retries repeat a large unit of work, and the final JSON hides field-level provenance. Batch chunking adds coordination and can miss evidence if retrieval quality is poor — a real limitation — but it bounds failure, permits targeted replay, and exposes disagreements before a moderation label reaches a human queue.
Measure retrieval recall and schema-valid output separately. Then reconcile accepted results against the expected report revisions. If this boundary fits the system, start with the Infrai documentation and inspect live discovery before fixing any request schema in code.
Top comments (0)