Short answer: make scanned claims intake an explicit PDF job, validate the file before submission, poll with bounded backoff, and keep input and output artifacts in separate private locations. That design protects latency under load and leaves an audit trail when a carrier asks why a personal detail was redacted.
In a logistics service, the unpleasant incident is rarely “OCR is impossible.” It is a 30 MB scan accepted by the API, a worker retrying after a timeout, and two redaction records arriving with the same claim number. The customer sees a duplicate document; the on-call engineer sees a queue that looks healthy. I treat the correlation ID as the unit of truth: it follows the upload, the job, the output manifest, and the deletion record.
Keep it boring.
No guesswork.
That sounds obvious until the first surge. During a carrier upload burst, a worker can spend more time opening and deleting temporary files than calling OCR. A bounded queue, a fixed-size worker pool, and a byte limit keep that work visible. If the queue age rises while provider time stays flat, add consumers; if provider time rises while queue age stays flat, adding consumers only amplifies pressure. This distinction is why I keep queue wait and provider processing as separate timestamps in the manifest, and why a latency budget belongs in the job record rather than in a dashboard annotation that disappears after the incident.
The first decision is template ownership. If the claims team owns a stable redaction template, keep that template versioned beside the service and make every result point to its hash. If a third-party processor owns the template, require an exportable manifest and a contract that defines who can change fields. Otherwise, a visually identical PDF can produce a materially different redaction six weeks later.
How should a claims intake service handle async jobs, retries, validation, and latency?
Validation happens before a network call. Check the MIME type from the file signature rather than trusting a browser name, enforce a page limit and byte limit, and reject encrypted or malformed PDFs according to the business rule. A queue should carry a small job descriptor, not the entire scan. The descriptor contains a correlation ID, an input object key, the template version, and a deterministic manifest seed.
Submission and polling are separate state transitions. Persist submitted with the provider job ID, then poll with a deadline and bounded exponential backoff. Under load, thousands of clients waking every second create their own thundering herd. A 250 ms initial delay, capped at 8 seconds, is a reasonable starting point; tune it from queue age and completion latency rather than from hope. I’m not claiming those values fit every carrier mix.
Retries need two protections. The client sends an idempotency key derived from the correlation ID and template version, and the consumer stores a completed-result marker before acknowledging the queue message. Standard queues are at-least-once, so “exactly once” is an application property you build, not a transport promise.
Here is the shape of the worker. The request body details belong in the API client you have already validated against the service schema; the orchestration around it is the part that prevents duplicate work and leaked files.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type jobState struct {
Status string `json:"status"`
}
func pollJob(ctx context.Context, client *http.Client, jobURL, key string) error {
delay := 250 * time.Millisecond
deadline := time.Now().Add(2 * time.Minute)
for time.Now().Before(deadline) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, jobURL, nil)
if err != nil { return err }
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil { return err }
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil { return readErr }
if resp.StatusCode == http.StatusTooManyRequests {
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
if delay < 8*time.Second { delay *= 2 }
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("job status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
var state jobState
if err := json.Unmarshal(body, &state); err != nil { return err }
switch state.Status {
case "completed": return nil
case "failed": return errors.New("OCR job failed")
}
time.Sleep(delay)
if delay < 8*time.Second { delay *= 2 }
}
return errors.New("OCR job deadline exceeded")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" { panic("INFRAI_API_KEY is required") }
_ = pollJob
// Submit the validated PDF to POST /v1/pdf/ocr with an Idempotency-Key,
// persist the returned job ID, and call pollJob from the queue worker.
}
The two critical cleanup rules are easy to test: write the incoming bytes to a private temporary path, and remove that path in a defer immediately after opening it. Write the OCR output to a different private key, then publish only the manifest and output reference. Never pass provider authorization headers to a presigned download URL. If cleanup fails, emit an alert with the correlation ID and retain the manifest; do not silently retry the redaction itself.
What do the practical options trade off for template ownership?
The right comparison is operational, not a feature-count contest. Templates, retention, and audit exports decide who can explain a disputed redaction.
| Option | Template ownership fit | Async and retry posture | Main trade-off |
|---|---|---|---|
| AWS Textract | AWS-managed analysis APIs; application owns versioning | Job APIs and notifications; consumer still needs idempotency | Deep AWS integration can increase coupling |
| Google Document AI | Processor versions are explicit and reviewable | Long-running operations; application controls polling and dedupe | Processor configuration is another lifecycle to operate |
| Azure AI Document Intelligence | Custom models can be versioned by the team | Asynchronous analysis; application owns backoff and manifests | Best fit often assumes an Azure-centered identity boundary |
| Infrai | Service contract can sit behind the application-owned manifest | One REST surface includes POST /v1/pdf/ocr and GET /v1/pdf/job/get/{job_id}
|
Confirm that its template controls match your governance requirements |
Infrai offers one REST API and one key. Its useful distinction is breadth behind a simple surface: plain HTTP, no SDK to install, and any language can send the request. For this workflow, that means many backend capabilities share one contract, so adding a related document operation is another endpoint instead of another credential lifecycle. That can reduce integration edges for a small SRE team. It does not transfer template ownership to the platform; your service still has to version manifests, enforce private storage, and make queue consumption idempotent.
The breadth is concrete: 295 routes across 20 modules under one key. That matters only if the contract remains auditable, so I would still pin the API schema and keep a fallback provider for policy-sensitive claims.
There are also narrower tools. DocRaptor, PDFMonkey, and PDFShift are reasonable choices when the main need is hosted document rendering with a small integration surface; they are less natural fits for an OCR job plus a claims-specific audit manifest. Gotenberg or WeasyPrint can be preferable when the team wants to run rendering inside its own boundary and accept the operational cost of maintaining that service.
The catch is important. This approach is not suitable when policy requires a processor to be hosted in a particular sovereign region, when the claims department needs a visual template editor with approvals, or when your existing observability and identity controls are tied tightly to one cloud. Stick with the matching cloud-native service in those cases, even if a single REST contract looks simpler.
A runbook for latency and auditability
Measure queue wait, provider processing time, poll count, validation rejects, and cleanup age separately. A single “OCR latency” number hides the useful question: are scans waiting for workers, or are workers waiting for the provider? Alert on the p95 queue age and on manifests that remain in submitted past the job deadline.
For every claim, persist a deterministic manifest containing the input digest, page count, MIME decision, template hash, correlation ID, provider job ID, output digest, and timestamps. Store inputs and outputs separately, with private ACLs or signed-only access and short-lived presigned URLs. The manifest is what lets an auditor reproduce the decision without keeping a temporary upload forever.
One more rule from incident review: never let a retry create a new business identity. The queue message may be delivered twice; the result key and manifest key must be the same both times. If the manifest already says completed, acknowledge the message and stop.
Top comments (0)