A 429 rate limit changes the scheduling problem in LLM structured data extraction, not the extraction prompt. If an interactive request, a queue consumer, and a backfill all compete through one concurrency setting, retries can amplify the burst that caused the throttle in the first place.
Short answer: reserve a small, bounded lane for interactive structured extraction; put deferred work in a durable queue; reschedule 429 responses with server-directed or capped jittered delay; use a batch API only for work that can tolerate reconciliation; and bind every job to its approved US or EU processing lane before it reaches a worker.
That split matters more than the client language. A Node.js service can own admission and queue state while a Python worker owns validation, or the entire flow can run in either runtime. The useful boundary is the job contract — input reference, schema version, prompt version, region, attempt, eligibility time, and idempotency key — because that contract survives a model swap and a notebook-to-prod rewrite.
How should Node.js LLM structured data extraction recover from 429 rate limits?
Start by refusing to treat every failure as retryable. An HTTP 429 is an admission signal: the attempt should release its active slot and become eligible later. A value that fails JSON parsing or schema validation is a different class of event. Retrying both through the same loop hides prompt regressions inside traffic metrics and can spend more tokens without improving the object.
For interactive work, define a total deadline first. The retry budget has to fit inside it, including network time and validation. Once the next delay cannot fit, return a pending state or move the job to the asynchronous path; don't keep a request open merely because its attempt counter has room. For queued work, persist next_eligible_at and let another ready item run. Sleeping inside a worker keeps capacity occupied while doing no extraction.
Keep it boring.
The queue consumer also needs a concurrency cap independent of retry count. Concurrency answers how many calls may be active now; rate answers how many may start over an interval; backoff answers when one rejected item may try again. Those controls interact, but collapsing them into one number makes diagnosis muddy. If 429 frequency rises while active concurrency stays below its cap, inspect start rate and upstream limits. If queue age rises without 429s, inspect worker capacity, input size, and validation time instead.
I would make the 429 path observable with fields such as lane, attempt, selected delay, queue age, and completion state, while keeping source text out of routine logs. The eval harness should inject a 429 through the model adapter and assert four things: no result was committed, the item received a future eligibility time, the active slot was released, and an unrelated ready item could proceed. That is a much sharper test than asserting that a retry function was called.
Make retries a state transition, not a recursive request
The smallest useful implementation is a state machine around a narrow model adapter. It doesn't need a vendor endpoint, and it shouldn't let transport code write directly to the final table. The example below uses full jitter under a configurable cap. A separately parsed server delay, when available through the adapter, takes precedence; policy still decides whether that delay fits the job's deadline.
from dataclasses import dataclass, replace
from datetime import datetime, timedelta, timezone
from random import uniform
from typing import Any, Callable
class RateLimited(Exception):
def __init__(self, retry_after_seconds: float | None = None) -> None:
self.retry_after_seconds = retry_after_seconds
@dataclass(frozen=True)
class ExtractionJob:
job_id: str
idempotency_key: str
region: str
schema_version: str
prompt_version: str
attempt: int
text: str
def retry_delay(attempt: int, retry_after: float | None) -> float:
if retry_after is not None:
return retry_after
ceiling = min(60.0, 2.0 ** attempt)
return uniform(0.0, ceiling)
def process_job(
job: ExtractionJob,
call_model: Callable[[str, str], Any],
validate: Callable[[Any, str], dict[str, Any]],
commit_once: Callable[[str, dict[str, Any]], None],
reschedule: Callable[[ExtractionJob, datetime], None],
) -> None:
try:
candidate = call_model(job.text, job.region)
except RateLimited as error:
delay = retry_delay(job.attempt, error.retry_after_seconds)
eligible_at = datetime.now(timezone.utc) + timedelta(seconds=delay)
reschedule(replace(job, attempt=job.attempt + 1), eligible_at)
return
result = validate(candidate, job.schema_version)
commit_once(job.idempotency_key, result)
commit_once is the load-bearing detail. Queue delivery can happen more than once, and a completed network call does not prove that the acknowledgement or database write happened exactly once. Build the key from stable source identity plus the prompt and schema versions, enforce uniqueness at the storage boundary, and acknowledge only after the accepted object is durable. Process-local memory is not an idempotency mechanism when two workers can see the same job.
Validation deserves its own ledger. Record parsing status, required-field failures, domain-rule failures, and the versions that produced them. Then compare extraction changes on a reviewed corpus before rollout. I care about field-level accuracy and manual-review demand alongside input and output tokens; a prompt that produces prettier prose is irrelevant here, while a prompt that improves one difficult field may justify its extra token cost if the fixed eval set demonstrates the gain.
One caveat: the numeric delay cap in this sample is a policy example, not a universal setting. I'm not sure any static cap can be chosen responsibly without the provider's current limit information and the application's latency budget. Resolve that uncertainty with documented account limits, observed throttling under controlled load, and a test that includes cold workers rather than copying 60.0 into production.
Queue, live call, or batch API?
Choose the execution lane from the deadline and recovery model, not from document count alone.
| Lane | Best fit | Failure handling | Main trade-off |
|---|---|---|---|
| Bounded live call | A user is waiting and measured tail latency fits the product deadline | A small retry budget inside one total deadline | Fast feedback, but burst capacity must be reserved |
| Durable queue | Work may finish later and needs per-item control | Delayed eligibility, redelivery, and idempotent commit | Strong control, but queue age becomes a user-facing concern |
| Batch submission | A large immutable set can wait for asynchronous reconciliation | Manifest-based result matching and explicit unresolved items | Efficient scheduling, but cancellation and partial results add state |
A batch API does not remove the queue contract. The submission needs stable client-side item identifiers, and the returned items must pass through the same schema validator and idempotent commit path as live results. Keep a manifest that maps each source item to its submitted identifier. A result absent from the returned set remains unresolved; it must not silently turn into an empty object or a successful extraction.
The catch is that batching is not suitable for a request whose value disappears after a short user deadline, and a live retry loop is not suitable for a backfill that can swamp interactive capacity. Stick with a durable queue when individual cancellation, priority, or region-aware admission matters. Use an internally operated serving stack when approved data cannot be sent to a managed endpoint, accepting that the team then owns capacity planning, scheduling, model evaluation, and upgrades.
Batch size should come from measurement. Track payload constraints and token estimates before submission, but also track the distribution after tokenization and validation; averages conceal the few long inputs that dominate completion time or exceed a limit. Your mileage may vary across document types, so bucket the scorecard by input length, language, schema, and region rather than publishing one comforting pass rate.
Keep US and EU routing attached to the job
Region is data, not worker configuration inferred at the last moment. Assign the allowed processing lane when the source enters the system, include it in the immutable job and batch manifest, and require the dispatcher to match it to an approved endpoint, queue, worker pool, storage location, and review path. Retries must preserve the same lane. So must deletion.
This is also where generic labels can mislead. “EU worker” says nothing about where model processing, logs, queue payloads, batch artifacts, or human review occur. Draw the actual data flow and verify it against current provider terms, account configuration, and organizational policy. Legal and security reviewers need that concrete map; application code cannot settle residency or transfer requirements by naming a queue.
Don't duplicate raw text into every operational system. A source reference, content hash, byte and token estimates, timestamps, versions, validation summary, and opaque job identifier are often enough to diagnose scheduling behavior. Access to the original text can stay on the narrow extraction path. This reduces the number of stores that must follow the source's retention and deletion rules without making queue health invisible.
Regional lanes can have different load shapes, so measure them separately. A global concurrency graph may look healthy while one lane accumulates age. Alert on oldest eligible job, time in each state, 429 share by lane, attempts per completion, and regional mismatch rejection. The mismatch counter should stay at zero, but testing it matters — inject a job whose lane cannot match the worker.
No model call.
Measure this before copying the architecture
Begin with a fixed extraction set containing the real document shapes the application accepts: clean text, missing fields, long inputs, ambiguous values, and every supported language. Review the expected objects. Version the prompt, schema, normalizer, and scorer together so a result can be reproduced, then report per-field outcomes instead of hiding a weak field inside whole-object accuracy. Next, replay traffic through each execution lane. Vary arrival bursts and worker starts; inject 429 outcomes; force queue redelivery; and reconcile a batch with one unresolved identifier. The release scorecard should put extraction quality beside queue age, end-to-end latency, attempts per completion, duplicate suppressions, unresolved batch items, regional mismatch rejections, and input/output token use. Those measurements connect prompt-cost choices to operational behavior instead of treating cost, correctness, and throttling as three unrelated dashboards. Finally, promote one lane at a time. A notebook proves that a model can return a plausible object. Production evidence has to show that the object is valid, committed once, processed in the approved region, and still delivered within its deadline when admission tightens. Pick the least complex lane that passes those tests; add another only when a measured workload needs its recovery model.
Top comments (0)