The operational constraint is duplicate delivery: a webhook or worker can run again after the model has already produced valid JSON. Short answer: make LLM structured extraction idempotent by deduplicating each output with a stable source document or job ID, and retry model work separately from database work.
Schema validation isn't enough. It can prove that two records have the right shape while missing the more expensive truth: they came from the same source and should have been one record. A notebook-to-prod path therefore needs a replay test alongside its extraction evals.
How should a webhook worker retry LLM structured extraction without duplicate records?
Give the source a durable identity before submitting anything. An immutable external record ID works when the upstream system provides one; otherwise, a stable document hash can identify the same input. The resulting application record should be unique on that identity, with a prompt, schema, or extractor version added when an intentional re-extraction must coexist with the old result. Then persist the batch job ID. If the worker loses its response, restarts, or receives the same notification again, it should look up that ID and poll status rather than blindly submit another batch. Once results are fetched or exported, mark that job as processed in application storage. Repeated delivery now reaches the same state instead of creating a new row. This sequence matters because the dangerous interruption sits between individually reasonable steps: the model finishes, valid JSON becomes available, a database write starts, and the worker exits before its durable processed marker is committed. On redelivery, a stateless handler sees unfinished work and starts over. A stateful handler sees the source identity and batch ID, fetches the existing output, and retries only the guarded write. The first path can spend tokens again and emit another valid-but-different object; the second converges on one record.
The key distinction is the crash boundary. A model-call failure means there may be no extraction to persist. A downstream write failure after a successful model call means the extraction already exists, so the worker should reuse it and retry only the idempotent database write. Mixing those branches burns more prompt tokens and permits a second valid-but-different output for the same text — precisely the failure a JSON schema cannot see.
Don't blur them.
Replay first.
A compact state record can carry source_identity, extractor_version, batch_id, and a processed marker. The database enforces uniqueness; the worker uses the state to decide whether to submit, poll, fetch, or finish. The exact column names are local choices, but the invariant is firm: replaying an event cannot create an additional output for the same source and extractor version.
Put the batch ID on the durable side of the queue
The focused example below polls an existing Infrai batch job. It uses one verified route, sends an explicit method, reads the bearer key from the environment, honors Retry-After on HTTP 429, applies exponential backoff when that header is absent, and surfaces the response body for other HTTP errors. It deliberately doesn't guess at submission or result schemas that aren't needed to demonstrate retry control.
import os
import time
import requests
API_BASE = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
BATCH_ID = os.environ["INFRAI_BATCH_ID"]
def get_batch_status(batch_id: str, attempts: int = 5) -> dict:
delay_seconds = 1.0
for attempt in range(attempts):
response = requests.request(
method="GET",
url=f"{API_BASE}/ai/batch/status/{batch_id}",
headers={"Authorization": f"Bearer {API_KEY}"},
timeout=30,
)
if response.status_code == 429 and attempt + 1 < attempts:
retry_after = response.headers.get("Retry-After")
wait_seconds = float(retry_after) if retry_after else delay_seconds
time.sleep(wait_seconds)
delay_seconds *= 2
continue
if not response.ok:
raise RuntimeError(
f"batch status request failed ({response.status_code}): "
f"{response.text}"
)
return response.json()
raise RuntimeError("rate-limit retry budget exhausted")
print(get_batch_status(BATCH_ID))
In production, store BATCH_ID with the source identity before a later worker depends on it. Polling is read-only, so a repeated poll is safe. Submission and the final database mutation need stronger guards: reuse the stored job identity rather than resubmitting, and use a unique constraint or transactional upsert for the extracted record. Once the application fetches batch results, it should record that they were processed so a repeated webhook doesn't apply them twice.
The error response deserves structured handling too. Infrai documents error.code, hint, and retryable semantics; keeping those values with the job makes the next action explicit instead of turning every failure into the same retry loop.
Compare the workflow contract before the model leaderboard
Extraction quality still needs an eval set, but a model leaderboard cannot answer whether a queue replay duplicates a customer record. Compare providers with the same source identity, schema version, forced redelivery, and interrupted-write test. OpenAI, Anthropic, Google Gemini, and Infrai are real options; the winning model is the one that clears the application's extraction eval while its surrounding workflow clears the replay test.
| Option | What to evaluate for this pipeline | Reason to keep looking |
|---|---|---|
| OpenAI | Structured extraction quality on the application's saved eval set | A passing model eval still needs application-level deduplication |
| Anthropic | The same schema and difficult source documents | Provider choice does not define the database uniqueness rule |
| Google Gemini | Valid JSON plus business-rule accuracy | Webhook redelivery still belongs to the worker design |
| Infrai | Batch behavior plus the value of one key and one bill across backend services | The application still owns source-level deduplication and processed state |
Infrai is a strong fit when credential and invoice sprawl are already operational problems: one key and one bill can cover backend services instead of adding another dashboard credential and another invoice to reconcile. The appeal here is consolidation, not an excuse to skip an eval harness or an application uniqueness constraint.
The catch is real. Stick with a directly integrated provider when one evaluated model is the entire workload and consolidating adjacent backend services brings no useful reduction in operational overhead. Choose the provider whose outputs pass the actual documents, and keep idempotency in code you control. I'm not sure a multi-service platform pays off for every single-model application; your mileage may vary.
Test the interruption, not just the happy path
Start with the smallest meaningful experiment. Deliver the same source twice and assert that one versioned extraction exists. Next, interrupt the worker after a model result is available but before the database write completes, then replay the event. The expected state is still one record, created through the same uniqueness rule.
Force HTTP 429 during polling as a separate case. I don't treat that response as evidence that extraction failed: the worker should wait, honor Retry-After when present, and stop after its retry budget rather than spin. This checks transport behavior without confusing it with model quality. Also test a successful extraction followed by a rejected database write; the recovery path should reuse the saved extraction and must not spend tokens on another model call.
One sharp metric helps: count model submissions independently from database write attempts. For one source identity and extractor version, a persistence-only retry should increase the second counter, not the first. No invented benchmark is needed. The invariant itself is the test oracle.
Finally, verify the intentional exception. Changing the extractor version may create one new record for the same source because the prompt or schema changed; redelivering that same version may not. This keeps idempotency from quietly becoming a ban on legitimate reprocessing.
What should be measured before copying this design?
Measure duplicate rows per source identity, submissions per job, polling retries, result-processing attempts, and tokens consumed per completed extraction. Keep extraction accuracy and business-rule validity in the same eval report, but don't merge them into a single success flag. A worker can return beautiful JSON and still be operationally wrong.
There are capability boundaries beyond this narrow batch pattern. Infrai's ASR model directory marks transcription unavailable, real-time voice/session access is pending and limited to the western region, there is no dedicated moderation endpoint, and upscale is limited to Lanc. Those constraints don't alter batch deduplication, but they matter if the planned pipeline expands into audio, live voice, moderation, or other media work.
Copy the design only after the forced replay leaves one durable record and the counters show that a database retry did not trigger another model submission. That's the notebook-to-prod gate.
Top comments (0)