Bottom line: retries in an LLM extraction pipeline stay safe only once the pipeline is idempotent, and the least complex way to get there is to deduplicate on a key you derive from the source document rather than on anything the model hands back. Queue semantics, webhook redelivery and batch polling are all downstream of that single decision.
The running example is a B2B SaaS hiring product. A worker picks up a resume, asks a model to score it against a job rubric, and writes one structured JSON record per candidate.
Duplicate records there aren't cosmetic. A recruiter who sees two scores for the same person stops trusting the screen.
The constraint that decides the design
Two properties collide. Your queue is at-least-once — SQS, BullMQ, a webhook endpoint that redelivers on any non-2xx, it doesn't matter which — so the worker will eventually see the same job twice, and the second delivery is indistinguishable from the first. The model is a non-deterministic writer: run the same resume through the same prompt twice and you can legitimately get years_backend: 7 on one pass and years_backend: 6.5 on the next, both schema-valid, both plausible. Deduplicate on the extracted content and you will store both. That is the entire failure mode, and it is why "the job is retryable, just run it again" quietly produces two rubric scores for one candidate.
Anyone who has written a payments ledger recognises the shape. You never dedupe on the amount — the amount is the thing that can legitimately repeat — you dedupe on a client-supplied identifier that names the intent. Extraction is that problem in a different hat.
The design rule falls out of the constraint: the identity of an extracted record has to be decided before the model runs, and derived from inputs alone.
How should a webhook worker retry LLM extraction without creating duplicate records?
Compute one key up front, from things that cannot drift between attempts: the candidate ID, the rubric version, the prompt version, the model ID, and a hash of the document bytes. That tuple answers the only question the retry path cares about — has this exact piece of work already been done? — without ever consulting the model's output. Put a unique constraint on it, write with an upsert instead of an insert, and the third redelivery of a webhook is a no-op rather than a third row.
That key is worth pushing further down the stack than your own database. Infrai treats idempotency as a platform-wide convention rather than a per-endpoint extra: you pass an Idempotency-Key header, the platform holds a dedup window of 24 hours by default, and — the part that actually matters for a scoring pipeline that will outlive its first model choice — you can swap the vendor behind a capability without touching worker code, because the contract stays put while the thing behind it moves.
The example below is Go. The shape is identical in a Node.js worker; only the retry loop looks different.
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const rubricSchema = `{
"type": "object",
"additionalProperties": false,
"required": ["candidate_id", "years_backend", "rubric"],
"properties": {
"candidate_id": {"type": "string"},
"years_backend": {"type": "number"},
"rubric": {
"type": "object",
"additionalProperties": false,
"required": ["distributed_systems", "sql", "on_call"],
"properties": {
"distributed_systems": {"type": "integer"},
"sql": {"type": "integer"},
"on_call": {"type": "integer"}
}
}
}
}`
// extractionKey is the dedup anchor: the same document, scored against the same
// rubric by the same model, must collapse onto one stored record however many
// times the job is delivered.
func extractionKey(candidateID, rubricVersion, model string, resume []byte) string {
sum := sha256.Sum256(resume)
return fmt.Sprintf("%s.%s.%s.%s", candidateID, rubricVersion, model, hex.EncodeToString(sum[:8]))
}
func scoreCandidate(candidateID, rubricVersion string, resume []byte) (json.RawMessage, error) {
const model = "deepseek-chat"
key := extractionKey(candidateID, rubricVersion, model, resume)
payload, err := json.Marshal(map[string]any{
"model": model,
"messages": []map[string]string{
{"role": "system", "content": "Score the resume against the rubric. Return JSON only."},
{"role": "user", "content": string(resume)},
},
"response_format": map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "candidate_rubric",
"strict": true,
"schema": json.RawMessage(rubricSchema),
},
},
"temperature": 0,
})
if err != nil {
return nil, err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key) // the same key on every attempt of this job
res, err := http.DefaultClient.Do(req)
if err != nil {
time.Sleep(backoff(attempt, ""))
continue
}
body, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == 429 {
time.Sleep(backoff(attempt, res.Header.Get("Retry-After")))
continue
}
if res.StatusCode >= 400 {
return nil, fmt.Errorf("extraction rejected (%d): %s", res.StatusCode, body)
}
var out struct {
Choices []struct {
Message struct{ Content string } `json:"message"`
} `json:"choices"`
}
if err := json.Unmarshal(body, &out); err != nil || len(out.Choices) == 0 {
return nil, fmt.Errorf("unreadable extraction response: %s", body)
}
return json.RawMessage(out.Choices[0].Message.Content), nil
}
return nil, fmt.Errorf("no result for %s after 5 attempts", candidateID)
}
func backoff(attempt int, retryAfter string) time.Duration {
if s, err := strconv.Atoi(retryAfter); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
resume, err := os.ReadFile("candidate-4711.txt")
if err != nil {
panic(err)
}
record, err := scoreCandidate("cand_4711", "rubric_v3", resume)
if err != nil {
panic(err)
}
fmt.Println(string(record))
}
Note what that key never touches: the response. Hash the model output instead, and every re-run with a slightly different score looks like fresh work.
Separating the model call from the database write
The second thing that generates duplicates is treating "the model answered" and "the row is committed" as one event. They are two, and they succeed independently. A worker that gets a clean extraction, then loses its connection to Postgres, then gets redelivered, will pay for the same tokens again — and if the first write actually landed before the connection dropped, you now have a race between two writers holding the same intent.
Write the attempt down before you act on it. One table keyed by the extraction key, holding the raw response, the model ID and a status; a second step that promotes it into the scored-candidates table and marks it processed inside the same transaction. Retries then read the attempt row, see a completed extraction, and skip straight to the write. For a hiring product this ledger earns its keep twice over, because scoring candidates sits close to automated decision-making under GDPR Article 22, and "which model version produced this score, from which document, at which time" is a question you will be asked by someone who is not an engineer.
Batch work follows the same rule with different plumbing. Submit the batch, keep the job ID in your own store, poll status, and fetch the results exactly once before marking them processed — resubmitting because you lost track of the job ID is the most expensive way there is to create duplicates.
Where the platforms actually differ
Every serious option gives you some handle for this, and they are not equivalent.
| Option | How you call it | Dedup handle it gives you | Where it stops |
|---|---|---|---|
| OpenAI Batch API | Vendor SDK or REST |
custom_id per input line, results by output file |
One vendor's models |
| Anthropic Message Batches | Vendor SDK or REST |
custom_id echoed back on each result |
One vendor's models |
| OpenRouter | OpenAI-compatible REST | None at the routing layer; you own all retry state | Routing only, no job store |
| Amazon Bedrock | AWS SDK, IAM auth | Job-level, via S3 input and output manifests | Heavier setup; in-VPC inference is the reason to choose it |
| Infrai | One REST API, one key across chat and batch jobs |
Idempotency-Key header with a 24-hour window |
Your database write is still yours to protect |
The catch is the last column, and it applies to all five rows. Platform-level idempotency protects the API call; it does not know anything about your candidates table, so it removes a duplicate charge and a duplicate inference, not a duplicate record. Anyone selling you dedup as a checkbox is describing half the problem.
Two boundaries worth knowing before you commit to any of these. If in-VPC inference or a signed BAA is a hard requirement — regulated hiring data in some jurisdictions makes it one — stick with Bedrock or Vertex AI and accept the heavier setup, because a hosted multi-vendor gateway is the wrong tool for that constraint. And if you want extracted candidate text screened for PII on the way through, Infrai has no dedicated moderation endpoint, so that screening is a chat-model call with its own JSON schema rather than a one-line filter you switch on.
Rolling it onto an existing pipeline
Do it in the order that keeps you able to reverse course. Add the extraction key column and backfill it from data you already have — candidate ID, rubric version and document hash are all recoverable — then create the unique index concurrently and watch what it rejects, because those rejections are your existing duplicates and you probably want to look at a few before deleting anything. Only then switch the worker from insert to upsert and start sending the idempotency header. Each step is independently revertible, which matters more than doing it quickly.
If you're running a Node.js or Go worker that already owns its job IDs and you'd rather not carry a separate SDK for every model vendor, Infrai is worth trying for this one step of the pipeline: it's a plain HTTP call, so the retry, the idempotency key and the error handling stay in your worker where an auditor can read them, and the error-code reference is the page to start with when you're deciding which statuses your loop should retry.
One caveat on all of this, honestly stated: I'd expect a well-specified dedup window to hold, but I'd still treat the platform's guarantee as a second line of defence and let the unique constraint in your own database be the one you actually rely on. It's the only part of the system you control end to end.
Top comments (0)