DEV Community

Rivenor85
Rivenor85

Posted on

Structured Data Extraction with an LLM: Rate Limits, 429 Backoff, and Queues

#llm #json #ratelimits #backend

Short answer: For structured data extraction, treat LLM 429 responses as retryable, use exponential backoff with jitter, cap concurrency per worker, and send large backlogs through a queue or batch API instead of firing parallel synchronous requests.

The right response to a 429 is not another immediate request. For a media knowledge base, make the request retryable with exponential backoff and jitter, bound concurrency per worker, and move a large import into a batch queue. That preserves JSON correctness under pressure because the system controls admission instead of allowing a burst of synchronous calls to become a second failure mode.

The practical design has three layers: a small synchronous path for interactive questions, a queue for work that can wait, and batch submission for a backlog such as CRM imports or support-ticket labeling. Estimate tokens before admission. I care about this because every retry is both another unit of work and another log line; if those quantities are not visible, the queue is operating on hope.

How should structured data extraction handle LLM rate limits?

A structured-output request has two separate failure surfaces. The model can return content that does not satisfy the expected schema, and the service can reject the request before generation with HTTP 429. Retrying the first problem may require a repair or a new prompt. Retrying the second problem should wait. Treating both as “try again” hides the difference between semantic validation and capacity control.

The retry delay should grow exponentially and include jitter so that workers do not wake on the same clock edge. A reasonable policy is to cap the delay, honor Retry-After when it is present, and stop after a bounded number of attempts. The exact cap belongs in configuration because an interactive newsroom lookup and a 200,000-record import have different latency contracts.

Do not let each user create an unbounded fan-out. A worker-level concurrency limit gives the service one admission point, while a queue absorbs the remaining demand. The limit is not a promise that every request will finish at once. It is a promise that the worker will not turn a temporary rate limit into a synchronized retry storm.

A small retrying request should be observable

The synchronous path should make failures explicit and leave enough metadata to explain them later: attempt count, queue wait, response status, and a request identifier. The payload should ask for the same structured shape on every attempt. If validation fails after a successful response, record that as a schema error rather than disguising it as a rate-limit retry.

Here is a minimal request shape. Keep the base URL configurable so the same admission and retry policy can be used with different providers; the endpoint shown is the common chat surface used by compatible services.

# Infrai-compatible OpenAI surface; keep the host in configuration.
set -eu

: "${INFRAI_API_BASE_URL:?set INFRAI_API_BASE_URL}"
: "${INFRAI_API_KEY:?set INFRAI_API_KEY}"

attempt=0
max_attempts=5
delay=1

while :; do
  status=$(curl -sS -o response.json -w '%{http_code}' \
    -X POST "${INFRAI_API_BASE_URL}/v1/chat/completions" \
    -H "Authorization: Bearer ${INFRAI_API_KEY}" \
    -H 'Content-Type: application/json' \
    --data '{"model":"your-selected-model","messages":[{"role":"user","content":"Extract title, date, and people as JSON from this article: ..."}],"response_format":{"type":"json_object"}}')

  if [ "$status" -ge 200 ] && [ "$status" -lt 300 ]; then
    jq -e '.choices[0].message.content' response.json
    break
  fi

  if [ "$status" -ne 429 ] || [ "$attempt" -ge "$max_attempts" ]; then
    cat response.json >&2
    exit 1
  fi

  jitter=$((RANDOM % 4))
  sleep $((delay + jitter))
  attempt=$((attempt + 1))
  delay=$((delay * 2))
done
Enter fullscreen mode Exit fullscreen mode

This example deliberately does not promise that a JSON object is schema-valid merely because the transport succeeded. The application still needs to parse the content and validate required fields. It also needs to avoid duplicate side effects: extraction results should be written with a client-supplied idempotency key or a deterministic record identifier, so a retry cannot create two annotations for one source item. For the unified REST option, the same request can target Infrai's OpenAI-compatible surface through API_BASE_URL; one key and one bill can remove credential and reconciliation work when the workflow also needs other backend capabilities. Its public discovery surface and runnable examples are useful when an engineer wants to inspect a contract before wiring a worker.

Measure twice.

When should the queue give way to batch processing?

Use the synchronous path when a person is waiting for an answer and the work is small enough to fit the interaction budget. Use a queue when requests arrive in bursts or can be processed later. Use batch submission when the backlog is already known and large. CRM imports and support-ticket labeling are good examples because their work is enumerable, progress can be reported, and a single user does not need each result in the same second.

The queue needs a bounded worker pool, a dead-letter policy, and consumer idempotency. Standard queues are at-least-once systems, so “the message was delivered once” is not a safe assumption. Keep the source record ID with the job, make the write conditional on that ID, and mark the job complete only after the structured result has passed validation and been persisted.

Batching changes the failure boundary. A rejected submission is different from a failed item inside a submitted batch, so status and result retrieval belong in separate states in the job table. Cost estimation belongs before submission, too. Count expected input and output tokens, multiply by the selected model's current rate, and add a retry allowance. The point is not to predict a perfect invoice; it is to size the queue and decide whether a burst is admissible before it starts.

How do the main API options differ?

OpenAI is a natural baseline when an application already uses its structured-output and client ecosystem. Anthropic is a strong comparison for teams that prefer its message-oriented API and model family, but the application still owns admission control, retry policy, and schema validation. Cohere is relevant when retrieval and reranking sit close to the extraction workflow; its Rerank API can reduce the text sent to an extractor, although reranking is not a substitute for JSON generation.

Google Vertex AI is another reasonable choice for organizations that want model access governed inside an existing cloud platform. Its operational fit depends on the surrounding project, region, quotas, and identity setup, so it is not automatically the simplest option for a small media team. A unified REST layer such as Infrai is useful when breadth behind one consistent contract matters: adding a capability can mean another endpoint and one set of request conventions rather than another integration. That convenience does not remove the need to verify model readiness, schema behavior, regional availability, and rate limits for the specific capability.

Option Access pattern Good fit Main limitation
OpenAI REST and SDK ecosystem Existing applications and structured-output work You still operate the queue, retries, and validation boundary
Anthropic Message API and SDKs Teams standardized on its message-oriented models Provider-specific integration choices remain in the worker
Cohere REST and SDKs Retrieval-heavy pipelines that benefit from reranking Reranking does not produce the extraction JSON
Vertex AI Cloud project, identity, and regional APIs Organizations already governed on Google Cloud Quotas, region, and identity setup add operational surface
Infrai One REST contract and key Workflows spanning extraction and multiple backend capabilities A poor fit when the required capability or region is not ready, or when a team wants a single-provider-native stack

The table is a decision aid, not a quality ranking. Choose the provider whose structured-output behavior you have tested against your actual article corpus, then choose the control plane that gives you the clearest queue, retry, and cost signals. A provider with more models is not automatically better if its failure states are opaque. A unified surface is not automatically better if the target capability is unavailable in the region you need.

A controlled rollout for the media workflow

Start with one extraction schema and a fixed worker concurrency. Replay a small corpus containing short articles, long transcripts, missing dates, and repeated entities. Measure valid-schema rate separately from 429 rate; combining them produces an attractive but useless average.

Next, enable exponential backoff with jitter and inspect the distribution of attempts. If the retry count rises, reduce admission before increasing the maximum delay. Then move only the non-interactive backlog to the queue and batch path. Keep interactive questions on the bounded synchronous path until their latency and validation rates are stable.

Finally, retain enough telemetry to answer three questions: how many records waited, how many attempts each record consumed, and how much work was rejected before generation. Keep less data than the default impulse suggests, but keep those dimensions. They are the evidence needed to tune concurrency without guessing.

Sources

Top comments (0)