Short answer: treat each scanned document as an immutable evidence packet, then run asynchronous OCR jobs behind an idempotent scheduler with bounded retries, validation gates, and short-lived encrypted files. Latency under load improves when admission control protects the OCR workers and the audit writer gets its own queue.
The failure signal is a broken chain of custody
A logistics board book is assembled from bills of lading, delivery receipts, customs forms, and signed exception sheets. The useful output is searchable text, but the defensible output is text that can be tied back to the exact bytes a driver or clerk signed. A filename is not a chain of custody. It can be reused, normalized, or silently replaced.
The incident pattern is familiar: a burst of inbound scans fills the worker pool, a retry writes a second OCR result, and an operator cannot tell which image produced the text shown in a review screen. That is a scheduling and evidence problem together. Store a content hash, source identifier, capture time, and signer metadata before dispatching OCR. Make the hash the idempotency key. Duplicate delivery then becomes a no-op rather than a second legal record. I've been paged for missed jobs where the queue metrics looked fine because they measured throughput instead of age; the forensic work started by matching object hashes across three systems, then tracing a delayed acknowledgement that allowed a second delivery. The fix was a state transition with an owner and timestamp, not another retry setting.
Measure first.
How should asynchronous jobs, retries, validation, and secure files work together?
Use a durable job record with explicit states: accepted, running, validated, published, and quarantined. A queue message carries only the job ID and a signed reference to the encrypted object store. Workers fetch the object into a process-private temporary directory, set a deadline, and delete the file in a defer block. The object itself has a retention timer; cleanup must not depend on a worker finishing normally.
Retries need a reason, not just a counter. Retry transport timeouts and rate limits with exponential backoff and jitter. Do not retry a validation rejection or a corrupt signature envelope. Cap attempts, move the job to quarantine, and preserve the original error code in the audit event. In one review, the useful clue was attempt=3 code=OCR_TIMEOUT; without that field, the queue looked healthy while the board book was already late.
Here is a small Go worker sketch. The interfaces are deliberately generic so the same control plane can front a self-hosted engine or a hosted OCR service.
type Job struct {
ID string
Object string
SHA256 string
Attempt int
Deadline time.Time
}
func process(ctx context.Context, j Job, store Store, ocr OCR, audit Audit) error {
if j.Attempt > 4 {
return audit.Quarantine(ctx, j.ID, "RETRY_EXHAUSTED")
}
path, err := store.TempFile(ctx, j.Object, 10*time.Minute)
if err != nil { return retryable(err) }
defer os.Remove(path)
got, err := sha256File(path)
if err != nil || got != j.SHA256 {
return audit.Quarantine(ctx, j.ID, "HASH_MISMATCH")
}
text, sig, err := ocr.Extract(ctx, path)
if err != nil { return retryable(err) }
if err := validate(text, sig); err != nil {
return audit.Quarantine(ctx, j.ID, "VALIDATION_REJECTED")
}
return audit.Publish(ctx, j.ID, got, text, sig)
}
The publish operation must be conditional on the job still being running. A compare-and-swap on the state, or a database uniqueness constraint on (job_id, result_hash), prevents a late retry from replacing an already validated result. Keep the audit append-only. Redaction belongs in a separate view, because rewriting the evidence record makes later verification ambiguous.
Keeping latency predictable during a scan storm
Measure queue age, not only request duration. Alert on the oldest accepted job and on the time between validated and published. Admission control should reject or defer new work before memory pressure causes every request to time out. A token bucket per source prevents one carrier from starving customs documents, while a small reserved lane handles signature-critical exceptions.
Separate CPU-heavy OCR from I/O-heavy hashing and audit writes. Each pool gets a concurrency limit derived from load tests, and the limits are adjustable without redeploying workers. Your mileage may vary: OCR engines differ sharply by page count and image resolution, so a single requests-per-second target is not a capacity plan. Replay a production-shaped corpus, including skewed pages and multi-page PDFs, then set an SLO for queue age with a rollback threshold.
Verification is a replay, not a green dashboard. Sample published jobs, recompute the object hash, check the signature envelope, and compare the OCR text checksum with the audit event. Exercise a worker kill after extraction but before publish; the next attempt should either publish the same result or record a deterministic duplicate, never create a second signed fact.
Rollback means stopping admission, draining running jobs, and restoring the last known-good OCR configuration. Do not delete quarantined objects until an evidence owner signs off. The catch is that this design is not suitable when a workflow needs interactive, sub-second OCR or edits to the original scan; use a synchronous preview path and a separate immutable archive in that case. Stick with a simpler batch process when signatures have no legal or financial weight.
Top comments (0)