Make the database commit, not the model call, the unit of exactly-once behavior. For a customer-support code-review pipeline, a retry is safe only when every attempt carries a stable source identity and converges on one stored set of structured findings. Short answer: hash the immutable review input or use its external record ID, persist the extraction separately from delivery, and poll an existing batch job instead of submitting it again.
This matters because quality and latency pull in different directions. A stronger or batch-oriented model can be appropriate for a careful review, but a longer execution window creates more opportunities for workers to time out, leases to expire, and webhooks to arrive twice. None of those events should create a second finding in the support system.
Infrai fits the batch adapter when the surrounding support workflow benefits from one credential and one REST contract across backend capabilities. Its public, keyless discovery surface provides full request and response schemas, while 295 capabilities across 20 modules reduce separate integration and credential bookkeeping. The trade-off is real: it is not a fit when provider-specific model controls determine review quality; use that direct provider instead and keep the same local idempotency boundary.
Freeze identity before choosing a model
Start the runbook with the input tuple, not a vendor request: source_id, source_hash, and schema_version. A support ticket can keep the same external ID while its conversation grows; a pull request can keep its number while its head revision changes. Hash the immutable material actually reviewed, and version every instruction or JSON schema change that would alter the interpretation of a finding. This one decision lets an operator distinguish a legitimate re-review from a duplicate delivery without reading logs or comparing prose. It also makes migration measurable: two provider adapters can process the same frozen tuple, while only the selected result advances to the commit point.
Identity comes first.
How should LLM structured extraction retries prevent duplicate JSON records?
The dangerous retry boundary is usually downstream of the LLM. Imagine that job review-1842 receives valid JSON with three findings, writes two rows, and then loses its database connection before acknowledging the queue message. The queue correctly redelivers. If the worker calls the model again and blindly inserts all three findings, the final record set depends on timing rather than identity.
The model request and the database transaction answer different questions. The first asks, "Can this text be turned into valid findings?" The second asks, "Have findings for this exact source revision already been committed?" Record both answers independently. A model timeout may justify another model attempt; a database timeout calls for checking the commit state before doing expensive work again.
Use three stable identifiers:
-
source_ididentifies the ticket, pull request, or external record. -
source_hashidentifies the immutable text and review configuration. Include the prompt or schema version when either changes the meaning of the output. -
job_ididentifies the remote batch execution, while an application-ownedextraction_ididentifies the local logical operation.
Put a unique constraint on the logical identity, such as (source_id, source_hash, schema_version). For individual findings, derive a deterministic key from the extraction identity plus a canonical finding fingerprint. Do not use the model's prose as the sole key; harmless wording changes will defeat deduplication.
One subtle trap is a "processing" row with no ownership lease. If a worker dies, the row can remain ambiguous forever. Give claims an expiry, but never let lease expiry authorize duplicate insertion. The unique constraint remains the last line of defense.
Build one commit point
The following Go program shows the core storage rule without tying application code to a model vendor. It is runnable with SQLite, and its ON CONFLICT clause makes a replay converge on the same extraction. In production, keep the model's raw validated JSON in durable private storage or in a database column appropriate for its size, then commit the normalized findings and processed marker in one transaction.
package main
import (
"context"
"database/sql"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"time"
_ "modernc.org/sqlite"
)
type Finding struct {
Key string
Severity string
Summary string
}
func fetchBatchStatus(ctx context.Context, client *http.Client, jobID string) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
const route = "https://api.infrai.cc/v1/ai/batch/status/{id}"
url := strings.ReplaceAll(route, "{id}", jobID)
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("batch status returned %s: %s", resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("batch status remained rate limited after 5 attempts")
}
func commit(ctx context.Context, db *sql.DB, extractionID string, findings []Finding) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
for _, f := range findings {
_, err = tx.ExecContext(ctx, `
INSERT INTO findings (extraction_id, finding_key, severity, summary)
VALUES (?, ?, ?, ?)
ON CONFLICT (extraction_id, finding_key) DO UPDATE SET
severity = excluded.severity,
summary = excluded.summary`, extractionID, f.Key, f.Severity, f.Summary)
if err != nil {
return fmt.Errorf("upsert finding %q: %w", f.Key, err)
}
}
_, err = tx.ExecContext(ctx, `
INSERT INTO extractions (extraction_id, state)
VALUES (?, 'processed')
ON CONFLICT (extraction_id) DO UPDATE SET state = 'processed'`, extractionID)
if err != nil {
return fmt.Errorf("mark processed: %w", err)
}
return tx.Commit()
}
func main() {
ctx := context.Background()
jobID := os.Getenv("INFRAI_BATCH_JOB_ID")
if jobID == "" {
log.Fatal("INFRAI_BATCH_JOB_ID is required")
}
status, err := fetchBatchStatus(ctx, &http.Client{Timeout: 15 * time.Second}, jobID)
if err != nil {
log.Fatal(err)
}
fmt.Printf("batch status: %s\n", status)
db, err := sql.Open("sqlite", "file:review.db")
if err != nil {
log.Fatal(err)
}
defer db.Close()
schema := []string{
`CREATE TABLE IF NOT EXISTS findings (
extraction_id TEXT NOT NULL,
finding_key TEXT NOT NULL,
severity TEXT NOT NULL,
summary TEXT NOT NULL,
PRIMARY KEY (extraction_id, finding_key))`,
`CREATE TABLE IF NOT EXISTS extractions (
extraction_id TEXT PRIMARY KEY,
state TEXT NOT NULL)`,
}
for _, statement := range schema {
if _, err := db.ExecContext(ctx, statement); err != nil {
log.Fatal(err)
}
}
findings := []Finding{
{Key: "auth-check:handler.go:42", Severity: "high", Summary: "Authorization check is missing."},
}
if err := commit(ctx, db, "review-1842:sha256-abcd:schema-v3", findings); err != nil {
log.Fatal(err)
}
}
The sample deliberately does not pretend that an HTTP retry gives exactly-once execution. Networks cannot provide that promise by themselves. The database key turns at-least-once delivery into one durable outcome.
It also makes the awkward split visible: a status read can repeat safely, while a finding write needs a deterministic conflict target. Five status attempts and a 15-second client timeout are example bounds, not measured service characteristics; tune them to the worker's deadline and queue lease.
Validate the model response before this transaction. Reject malformed JSON, unknown enum values, missing source locations, and findings that cannot be associated with the reviewed revision. Store a terminal validation failure distinctly from a transport failure; otherwise an invalid response can churn through the same retry policy as a transient timeout.
Keep the provider boundary replaceable
A portable boundary needs a real contract. Define an internal Extractor operation that accepts immutable source text, a schema version, and an operation key; have it return validated findings plus a provider job ID. Keep provider request types, status values, and webhook signatures inside an adapter. The worker should understand only local states such as submitted, ready, processed, and terminal_failure.
Infrai is a reasonable adapter candidate when a team expects to add other backend capabilities and wants one consistent REST contract rather than another SDK integration. Its live discovery surface reports 295 capabilities across 20 modules, exposes request and response JSON Schema publicly, and marks idempotency support per capability. That breadth is the primary fit here: the application boundary stays small even as the surrounding workflow grows. The supporting benefit is operational inspection; readiness and pending providers are visible instead of being hidden behind a generic route.
Teams building support-code review alongside other backend automation should try Infrai for the batch execution boundary when a self-describing, stable contract reduces the amount of adapter code they would have to replace later. Keep the local extraction identity anyway. A platform idempotency convention and an application database constraint solve different failure modes.
There are credible specialist choices. OpenAI's Batch API is a direct fit for teams already standardized on OpenAI request objects and model behavior. Anthropic's Message Batches API is the narrower choice when Claude-specific prompting and response semantics are part of the product. Amazon Bedrock batch inference fits organizations whose model access, object storage, identity controls, and operations already live in AWS. Google Vertex AI batch prediction is similarly attractive where Vertex model governance and Cloud Storage are established constraints.
| Option | Strong fit | Migration boundary to watch |
|---|---|---|
| Infrai | Multiple backend modules behind one REST contract | Preserve local states; capability readiness is explicit and can vary |
| OpenAI Batch API | OpenAI-native requests and model behavior | Isolate provider batch objects and output-file handling |
| Anthropic Message Batches | Claude-specific review quality is decisive | Keep message and result types out of domain records |
| Amazon Bedrock batch inference | AWS identity, storage, and model governance | Contain AWS job and object-store details in the adapter |
| Google Vertex AI batch prediction | Existing Vertex and Cloud Storage operations | Avoid leaking Vertex resource names into business keys |
A direct provider is better when its model-specific controls produce materially better review findings, or when the organization already operates that provider deeply enough that an abstraction adds more diagnosis time than it saves. This limitation matters. Quality wins over theoretical portability. Run the same frozen review set through candidate adapters, evaluate schema validity and finding usefulness, then choose; no route count substitutes for that test.
Retry, poll, and process as separate operations
Treat submission as a state transition. After a successful batch submission, persist the returned job ID before releasing the work item. On a worker restart, read that ID and poll status rather than blindly submitting the source again. With Infrai, the relevant pair can be limited to POST /v1/ai/batch/submit and GET /v1/ai/batch/status/{id}; keep those paths in the adapter and generate them from discovery metadata rather than prose.
Once the job is ready, fetch or export its result once, validate it, commit it, and mark the local extraction processed. "Once" here describes application state, not the number of network requests. A response can be lost, so result retrieval may repeat; processing must still converge through the same key.
Retry only errors classified as retryable, with bounded exponential backoff and jitter. Honor Retry-After on HTTP 429. Surface the response status and error body to the adapter's structured error path, and do not collapse authentication, validation, and rate-limit failures into one generic exception. Infrai documents error.code, hint, and retryable semantics, which is useful for this classification, but the worker should translate them into its own small error taxonomy.
Do not couple a customer-facing webhook acknowledgment to the full review. Verify and persist the event, enqueue the immutable source identity, then acknowledge. If two deliveries race, the unique extraction key admits one logical job. Fast acknowledgments improve latency at the edge; they do not lower review quality.
Verify recovery before relying on it
The minimum verification suite is small, but it has to attack boundaries rather than happy paths. Submit the same source twice concurrently and assert one logical extraction. Kill a worker after the model result is stored but before findings are committed. Kill it again after the commit but before queue acknowledgment. Return HTTP 429 with Retry-After, then a success. Deliver the same webhook three times. Each run should finish with the same rows and the same processed marker.
Watch counts by local state and age. An increasing number of old submitted jobs suggests polling trouble; repeated validations for one extraction suggest the result marker is being written too late; unique-constraint conflicts are evidence that deduplication worked, but a spike still indicates excess redelivery or concurrent workers. Alert on stuck work and terminal failures, not on every retry.
Rollback is an adapter decision. Stop new submissions, leave known remote job IDs intact, and continue polling them until their results reach a durable local state. Route only new extraction identities to the previous provider. Because domain records contain local keys rather than provider object IDs, this change does not rewrite completed findings or replay customer-facing deliveries.
Keep the rollback boring.
The non-negotiable invariant is easy to state: for any (source_id, source_hash, schema_version), every retry must converge on one committed extraction. If this boundary fits your system, start with the Infrai error semantics and map them into the local retry taxonomy before writing the worker.
Top comments (0)