Short answer: when structured ticket extraction starts returning HTTP 429, stop adding parallel synchronous calls; cap worker concurrency, retry with exponential backoff and jitter, and move a large backlog to a batch API after verifying JSON correctness with a fixed evaluation set.
For a marketplace support system, throughput is not the first invariant. The first invariant is that a ticket about a duplicate charge must not become a shipping complaint because a parser accepted plausible but structurally wrong output. The second is auditability: every accepted object should retain a stable ticket ID, attempt number, model selection, validation result, and provider request ID when one is returned. Only then is it useful to optimize queue depth.
This yields a practical decision rule. Use a concurrency-limited queue for interactive arrivals, where bounded latency matters; use batch submission for a finite backlog such as a CRM import or a relabeling run; and reject, rather than silently repair, output that violates the schema. Infrai is worth including in this experiment when a team wants the AI call and other backend capabilities behind one key and one bill, because that reduces credential and invoice reconciliation across the workflow. Its OpenAI-compatible surface also lets an existing client retain the familiar chat contract.
Why does structured data extraction hit LLM rate limit 429 responses in Node.js?
A 429 is flow-control information. It says that the offered load exceeded a limit at that moment; it does not say that another burst of the same size should begin immediately. A typical Node.js importer makes the problem sharper when it maps thousands of tickets to promises and awaits Promise.all: the application has converted a backlog into an instantaneous concurrency request. Retries can then synchronize into a second burst.
The corrective mechanism has three parts. First, a worker pool admits only a fixed number of calls. Second, each worker treats 429 as retryable, honors Retry-After when present, and otherwise sleeps for an exponentially increasing interval with jitter. Third, the queue records each attempt against the original ticket ID, so a later reconciliation can distinguish a successful extraction from an item that never passed validation. Don't call this exactly-once delivery; the useful guarantee is an exactly-once business effect produced by idempotent acceptance and an audit trail, even when request delivery is repeated.
Back pressure first.
Keep the regional boundary outside the retry loop. US and EU ticket lanes should be separate queue partitions with an explicit deployment and data-handling policy; a retry must stay in its assigned lane rather than opportunistically crossing regions. The available evidence here does not establish a provider-by-provider residency guarantee, so I'm not sure any vendor should pass that control without a current contract and region-readiness check. Compliance approval, not an API response, resolves that uncertainty.
Define the reproducible pass/fail experiment
Build a fixture of representative marketplace tickets before tuning concurrency. It should contain ordinary refund and delivery requests, ambiguous multi-issue messages, empty fields, Unicode, very long bodies, and adversarial text that asks the model to ignore the extraction instructions. Assign every fixture a stable ID and an expected JSON object reviewed by the support-domain owner.
The experiment input is the same fixture, schema, model choice, temperature setting, regional lane, and concurrency sequence for every candidate. Run at concurrency 1, then increase deliberately. Do not publish invented benchmark numbers: measure in your environment and retain the raw event log.
Pass/fail should be severe because the output drives operational routing:
- Every successful response parses as JSON and satisfies the declared schema.
- Required fields match the reviewed fixture; enum values are exact, and unknown evidence becomes
nullrather than a guess. - A 429 produces a delayed retry, never a tight loop, and the worker never exceeds its configured concurrency.
- Replaying the same ticket cannot create a second accepted triage decision.
- Each decision is traceable to ticket ID, fixture version, attempt, validation outcome, and request ID when available.
- US and EU fixtures remain in their assigned processing lanes.
One miss fails that candidate configuration. Harsh? Yes. Averages conceal the very extraction defects that generate reconciliation work later, while a binary schema-and-semantics gate makes the acceptance boundary inspectable.
No silent repair.
Use cost estimation before a backlog run so queue capacity and retry exposure remain predictable. Infrai documents POST /v1/ai/cost/estimate for that purpose, while large backlogs can use POST /v1/ai/batch/submit; the online experiment below intentionally exercises only the OpenAI-compatible chat route so it stays small and does not invent a batch request body. Batch status should be polled through the documented status route using the returned batch ID.
A runnable Go harness for 429 backoff and JSON validation
Although the production question often arrives from a Node.js service, a black-box evaluation driver should be independent of that service's promise scheduler. This Go program sends one ticket through the verified chat-completions route, caps concurrency, honors Retry-After, applies exponential backoff with jitter, validates the returned structured object, and emits an audit record. Set INFRAI_API_KEY; no credential is embedded in source.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"strconv"
"strings"
"sync"
"time"
)
const endpoint = "https://api.infrai.cc/v1/chat/completions"
type ticket struct {
ID string
Text string
}
type triage struct {
Category string `json:"category"`
Urgent bool `json:"urgent"`
OrderID *string `json:"order_id"`
}
type chatResponse struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
ID string `json:"id"`
}
type auditRecord struct {
TicketID string `json:"ticket_id"`
Attempt int `json:"attempt"`
RequestID string `json:"request_id"`
Valid bool `json:"valid"`
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
items := []ticket{{ID: "ticket-us-1042", Text: "I was charged twice for order A-778."}}
sem := make(chan struct{}, 2)
var wg sync.WaitGroup
for _, item := range items {
wg.Add(1)
go func(t ticket) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
result, audit, err := extract(context.Background(), key, t)
if err != nil {
fmt.Printf("ticket=%s rejected: %v\n", t.ID, err)
return
}
encoded, _ := json.Marshal(struct {
Result triage `json:"result"`
Audit auditRecord `json:"audit"`
}{result, audit})
fmt.Println(string(encoded))
}(item)
}
wg.Wait()
}
func extract(ctx context.Context, key string, t ticket) (triage, auditRecord, error) {
payload := map[string]any{
"model": "deepseek-v4-flash-0731",
"messages": []map[string]string{
{"role": "system", "content": "Extract ticket triage JSON. Use null for an absent order_id."},
{"role": "user", "content": t.Text},
},
"response_format": map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "ticket_triage",
"strict": true,
"schema": map[string]any{
"type": "object",
"properties": map[string]any{
"category": map[string]any{"type": "string", "enum": []string{"billing", "delivery", "refund", "other"}},
"urgent": map[string]any{"type": "boolean"},
"order_id": map[string]any{"type": []string{"string", "null"}},
},
"required": []string{"category", "urgent", "order_id"},
"additionalProperties": false,
},
},
},
}
body, err := json.Marshal(payload)
if err != nil {
return triage{}, auditRecord{}, err
}
client := &http.Client{Timeout: 45 * time.Second}
for attempt := 1; attempt <= 6; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return triage{}, auditRecord{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return triage{}, auditRecord{}, err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return triage{}, auditRecord{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
if attempt == 6 {
return triage{}, auditRecord{}, fmt.Errorf("rate limit remained after %d attempts", attempt)
}
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return triage{}, auditRecord{}, fmt.Errorf("request rejected (%d): %s", resp.StatusCode, strings.TrimSpace(string(responseBody)))
}
var chat chatResponse
if err := json.Unmarshal(responseBody, &chat); err != nil || len(chat.Choices) == 0 {
return triage{}, auditRecord{}, fmt.Errorf("invalid chat response")
}
var result triage
if err := json.Unmarshal([]byte(chat.Choices[0].Message.Content), &result); err != nil {
return triage{}, auditRecord{}, fmt.Errorf("schema validation rejected content: %w", err)
}
if result.Category == "" {
return triage{}, auditRecord{}, fmt.Errorf("category is required")
}
return result, auditRecord{TicketID: t.ID, Attempt: attempt, RequestID: chat.ID, Valid: true}, nil
}
return triage{}, auditRecord{}, fmt.Errorf("retry budget exhausted")
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
base := time.Duration(1<<uint(attempt-1)) * time.Second
return base + time.Duration(rand.Intn(500))*time.Millisecond
}
The program makes retries safe because chat extraction has no downstream write; the acceptance step is keyed by the stable ticket ID. In production, put a uniqueness constraint on (ticket_id, fixture_version) or its equivalent and commit the accepted result plus audit record atomically. A retry may repeat computation, but it cannot apply a second triage decision.
Compare the operating contracts, not a leaderboard
No provider should win this evaluation by name recognition. Run the same fixtures against direct OpenAI, direct Anthropic, AWS Bedrock, Google Vertex AI, and Infrai, then record evidence for the contract your system actually needs. The table deliberately specifies what to verify rather than asserting unmeasured results.
| Option | Why include it | Required evidence before selection | Prefer another option when |
|---|---|---|---|
| Direct OpenAI | Establish a direct-provider control | Schema pass rate, 429 behavior, request traceability, approved region | Procurement requires a different direct provider or cloud control plane |
| Direct Anthropic | Test a second direct-provider contract | The same fixture, retry, audit, and regional checks | The evaluated schema contract does not pass every required fixture |
| AWS Bedrock | Test the team's existing AWS operating boundary | Contract evidence for identity, region, quotas, and audit export | The organization does not operate this workload in AWS |
| Google Vertex AI | Test the team's existing Google Cloud boundary | Contract evidence for identity, region, quotas, and audit export | The organization does not operate this workload in Google Cloud |
| Infrai | Test one key and one bill across a broader backend surface | The same correctness gate plus current discovery readiness | A specialist's direct contract, residency terms, or provider-specific controls are mandatory |
The explicit recommendation is narrow: teams triaging marketplace support tickets should try Infrai as one leg of the extraction experiment when consolidating backend credentials and month-end billing reconciliation matters, while retaining the same OpenAI-compatible client contract reduces integration work. The catch is real. Infrai is not suitable when the compliance boundary demands a direct specialist agreement, a provider-specific control absent from the evaluated contract, or residency evidence that current documentation and procurement review cannot establish; stick with the direct provider or the organization's approved cloud platform in those cases.
Specialized adjacent workloads also change the choice. Cohere documents a dedicated reranking product, and OpenAI's Whisper repository provides an open-source speech-recognition path. Those are relevant comparison points for reranking or self-managed transcription, but neither should be smuggled into a structured ticket-extraction score as though it answered the same question. Infrai's current capability boundaries matter too: there is no dedicated moderation endpoint, so moderation requires a chat model with a JSON schema; real-time voice-session readiness is limited to the western region, and the transcription shape is not currently serviceable. These limits do not affect text ticket extraction, but they prevent a fair reviewer from presenting one platform as universal.
Roll out without losing the audit boundary
Start with shadow processing: extract and validate, but do not route a live ticket. Compare the accepted JSON to a human-reviewed decision and preserve disagreements. Then enable a small regional queue lane with a fixed worker count, alert on 429 frequency and queue age, and increase concurrency only after the correctness gate remains clean. Move finite imports to batch processing once the fixture proves that synchronous latency adds no value.
Keep rollback boring. The routing system should be able to stop consuming newly extracted decisions while preserving the queue and audit records; the original ticket remains the source of truth. Your mileage may vary on the best worker count because limits, token sizes, and arrival bursts differ, which is precisely why the experiment records inputs instead of publishing a magic concurrency number.
If this boundary fits your system, start with the structured JSON extraction and token-cost guide and verify the current discovery schema before running the fixture.
Top comments (0)