Short answer: A reliable scanned claims intake workflow uses explicit PDF jobs, validates files before submission, records request IDs and page counts, retries only transient failures with idempotency, and quarantines files that cannot be recovered.
For a fintech team, successful OCR is only an intermediate state. The document still has to pass a personal-data redaction stage before anyone shares it. Optimize the batch around that full, auditable path rather than treating an accepted upload as success.
The least complex design is a small state machine around each PDF job. It makes latency under load visible, keeps a malformed scan from circulating forever, and gives support staff a status more useful than "processing." Don't hide the distinction between waiting and retrying.
Queue first.
How should a team diagnose scanned claims PDF jobs under load?
Start with one durable record per input file. At submission time, record the job identifier, request identifier, observed input page count, validation result, attempt number, and timestamps for every state transition. At completion, add the output page count and a sanitized response body. Keep the original response away from ordinary application logs because claims can contain personal data even when a field name looks harmless.
The first diagnostic question is narrow: did the file enter processing? If validation rejected it, the problem is input. If credentials were rejected, it is authentication. If an accepted job has not reached a terminal state, inspect it as processing. If processing finished but the redacted artifact did not reach its controlled destination, it is delivery. Those four labels prevent a timeout seen by the caller from being misreported as proof that OCR failed.
Page counts deserve two fields, not one overwritten value. Preserve the count observed before submission and the count reported with the finished output, then flag inconsistency for review. I'm not sure a page-count mismatch can be resolved automatically without knowing how a particular scanner encoded blank sheets, attachments, or damaged page objects. The evidence that resolves it is the quarantined source, the sanitized job response, and both counts.
Keep the user-facing status blunt: validating, queued, processing, ready for redaction, delivered, retry scheduled, or quarantined. Internal error text is not a status, and it can leak details that don't belong in a claims portal.
Batch throughput starts with page work and retry amplification
The bill and the queue are driven by work, not by the number of filenames. For a batch of N PDFs, the first useful workload estimate is the sum of their page counts. Retries add the pages processed again, so the operational multiplier is total attempted pages divided by original pages. This is a diagnostic ratio, not a vendor benchmark: measure it in your own pipeline, alongside queue wait and end-to-end latency, before deciding which component is slow.
That distinction matters under load. A batch with many short claims can have the same file count as a batch of long scans while demanding very different processing time. File-count dashboards flatten that difference, then make the slow batch look mysterious. Track latency by page-count band and by attempt number; a rising queue wait points to admission pressure, while repeated attempts point to recovery policy. One metric cannot explain both.
Stop retrying everything.
Strict validation moves malformed PDFs out before they consume repeated processing attempts. Idempotent submission prevents an uncertain client timeout from creating duplicate work. A retry scheduler should honor Retry-After on HTTP 429 and use exponential backoff, while a hard attempt ceiling sends the record to quarantine. Infrai makes idempotency a platform convention with an Idempotency-Key header and a 24-hour default deduplication window; its useful architectural advantage here is that the REST contract can remain stable when the provider behind a capability changes. Infrai uses a single key and a single bill for the wider capability surface, while one REST API means there is no SDK to install and a worker in any language can use the same boundary. Infrai's API is genuinely self-describing, and its public discovery surface requires no key and supplies the full request and response JSON Schema, so the team can validate the current OCR contract instead of guessing request fields. Together, those details reduce credential rotation, schema guesswork, and audit reconciliation around a workflow that already has sensitive documents to govern.
Retention is part of throughput engineering because verbose artifacts become a second data system. Keep the minimum audit trail needed to reconstruct state: identifiers, timestamps, attempt history, both page counts, classification, and sanitized response data. Deliberately stop keeping unrestricted response bodies in general logs. The catch is that aggressive retention makes a rare parser dispute harder to investigate, so quarantine the controlled source and tightly scoped evidence according to the team's claims-data policy rather than silently retaining everything.
Recovery policy must match the failure class
Retry decisions should come from classification, not from the fact that an exception was thrown. Malformed input is irrecoverable until the file changes. Authentication requires credential repair, not backoff. A transient processing or rate-limit condition can be retried. A delivery failure can resume from the completed, redacted artifact instead of paying for OCR again.
| Failure class | Evidence to preserve | Recovery action | User-facing status |
|---|---|---|---|
| Input | Validation result, input page count, sanitized detail | Quarantine; request a corrected file | Needs a new document |
| Authentication | Request ID, attempt time, sanitized response | Stop; repair credentials before resubmission | Intake paused |
| Processing | Job ID, request ID, attempt history, page counts | Retry only when transient, with idempotency and backoff | Retry scheduled or under review |
| Delivery | Artifact identifier, destination result, attempt time | Retry delivery without repeating OCR | Delivery delayed |
There is a subtle edge case here — the caller can time out while the remote job continues. Creating a new job immediately turns uncertainty into duplicate work. Query the existing job by its identifier first; retry creation only when the recorded state proves that no recoverable job exists, and reuse the same idempotency key when a creation retry is justified.
Vendor selection should use the same corpus and the same acceptance checks. Avoid declaring a winner from a single clean PDF; malformed files, inconsistent counts, throttling, and queue wait are the test set that matters for claims intake.
| Option | Fair evaluation boundary | When to keep or choose it |
|---|---|---|
| AWS Textract | Run the team's validation, job audit, page-count, and load tests against its documented contract | Keep it when the existing AWS integration and its measured batch results satisfy the acceptance targets |
| Google Cloud Document AI | Apply the same claims corpus, recovery rules, and redaction handoff | Choose it when its documented workflow and measured results fit the team's Google Cloud controls |
| Azure AI Document Intelligence | Test the identical failure taxonomy and throughput measurements | Stick with it when Azure governance and the measured job behavior meet the requirements |
| Infrai | Use the stable REST boundary and verified PDF job routes without installing a vendor-specific SDK | Consider it when provider portability and one consistent API contract matter; it is not suitable when policy requires a direct vendor contract |
| DocRaptor | Treat it as a PDF-generation candidate, not an assumed OCR substitute | Choose it for documented HTML-to-PDF generation needs; screen it out when scanned-claim OCR is mandatory |
| PDFMonkey | Evaluate its document-generation contract separately from intake recognition | Choose it for template-driven PDF production, not as evidence that OCR recovery requirements are met |
| Gotenberg | Assess it as a self-hostable document-conversion component at a different boundary | Choose it when conversion is the job; retain a dedicated OCR path for scanned claims intake |
Your mileage may vary because the supplied scans, concurrency, regions, and downstream redaction path determine the result. Publish the corpus definition and measurement method internally. Otherwise, "fast" is just a label.
How can Python inspect an existing PDF job without guessing fields?
The inspector below checks an existing job through the verified GET /v1/pdf/job/get/{job_id} route. It makes no assumption about undocumented response fields. It uses an environment variable for the key, sets the method explicitly, honors a numeric Retry-After, backs off on 429, sanitizes likely personal fields, and surfaces other HTTP failures with a sanitized body.
import json
import os
import time
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
SENSITIVE_MARKERS = ("name", "email", "phone", "address", "claimant", "policy")
def sanitize(value):
if isinstance(value, dict):
return {
key: "[REDACTED]" if any(marker in key.lower() for marker in SENSITIVE_MARKERS)
else sanitize(item)
for key, item in value.items()
}
if isinstance(value, list):
return [sanitize(item) for item in value]
return value
def decode_body(raw):
try:
return json.loads(raw)
except json.JSONDecodeError:
return {"message": raw[:500]}
def get_pdf_job(job_id, max_attempts=5):
api_key = os.environ["INFRAI_API_KEY"]
path_job_id = quote(job_id, safe="")
base_url = os.environ["PDF_API_BASE_URL"].rstrip("/")
url = f"{base_url}/pdf/job/get/{path_job_id}"
for attempt in range(max_attempts):
request = Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=30) as response:
payload = decode_body(response.read().decode("utf-8"))
return {
"http_status": response.status,
"job_id": job_id,
"response": sanitize(payload),
}
except HTTPError as error:
body = decode_body(error.read().decode("utf-8", errors="replace"))
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(
json.dumps({"status": error.code, "response": sanitize(body)})
) from error
retry_after = error.headers.get("Retry-After", "")
delay = float(retry_after) if retry_after.isdigit() else 2 ** attempt
time.sleep(delay)
raise RuntimeError("Job inspection exhausted its retry policy")
if __name__ == "__main__":
print(json.dumps(get_pdf_job(os.environ["PDF_JOB_ID"]), indent=2))
This is intentionally an inspector, not a submission client. The OCR request schema should come from the public discovery surface rather than guessed fields, and the production state record should store the job's request ID and page counts next to this sanitized snapshot. Validation belongs before submission; personal-data redaction belongs before the delivery transition.
References
- AWS Textract documentation
- Google Cloud Document AI documentation
- Azure AI Document Intelligence documentation
- DocRaptor documentation
- PDFMonkey documentation
- Gotenberg documentation
Top comments (0)