A US/EU SaaS should use four PDF endpoint capabilities for digital archiving: asynchronous submission, status and evidence, protected retrieval, and policy-aware erasure. The page that proves why says searchable-document lag is rising while logistics intake is healthy. Operators can see PDFs arriving, but warehouse staff can't find the text from newly scanned bills of lading. Promote a PDF only after OCR fidelity and signature evidence pass validation; keep regional placement and retention in an artifact ledger rather than scattering them across worker settings.
That is the least complex shape that separates four jobs with different failure modes. Intake must be durable and idempotent. OCR is slow and retryable. Archive reads need narrow authorization. Erasure needs policy state, not a blind object deletion. A synchronous "upload and wait for text" call couples all four and gives the caller no safe interpretation when its connection ends before the work does.
I've been paged for missed jobs and duplicate deliveries. The painful cases rarely begin with a worker being obviously down; they begin with an accepted item that has no durable identity, a retry that becomes a second job, or a queue-age graph hidden behind a healthy request-rate graph. For scanned logistics records, duplicates are worse than wasted compute: two searchable artifacts can appear to represent one shipment while carrying different OCR output or audit metadata.
Start with identity.
How should US/EU SaaS PDF endpoints handle private digital archiving?
Treat an endpoint as a capability boundary, not as a URL naming exercise. The submission capability accepts an immutable source reference plus tenant, document revision, requested processing region, OCR profile, retention-policy identifier, and an idempotency key. It returns a stable job identity after durable admission. The observation capability reports state and validation evidence without returning the document body. The retrieval capability releases only the promoted PDF and searchable representation to an authorized caller. The erasure capability requests a policy transition and exposes its eventual disposition.
Those capabilities don't have to map one-to-one to public routes. A team may combine status and evidence in one read model, or keep erasure behind an administrative service. The important separation is semantic: a network timeout during submission must not imply that no job exists, and an OCR worker finishing must not imply that an artifact is ready for archive retrieval.
The lifecycle should move forward: accepted, processing, validating, then archived or rejected. An archived revision is immutable. A corrected scan creates a new revision with a new input digest; it does not overwrite the bytes and evidence that an auditor may already have examined. Repeating submission with the same tenant, revision, region, policy, and source digest resolves to the same job identity. This rule turns client retries into lookup behavior instead of duplicate production.
| Capability | Durable result | Failure it contains |
|---|---|---|
| Submit | Job identity bound to source and policy | Ambiguous retries |
| Observe | State plus validation evidence | Premature retrieval |
| Retrieve | Authorized promoted artifact | Exposure of temporary output |
| Erase | Auditable disposition workflow | Orphaned derivatives |
No endpoint can rescue mutable input.
For browser ingestion, a Blob is a useful boundary because it represents immutable raw data and can be consumed as an ArrayBuffer or a stream. Don't treat a blob URL as an archive identifier. It is a browser access mechanism, and the application should revoke an object URL when it is no longer needed. Compute the source digest at the controlled ingestion boundary, store the source in the approved location, and send workers an opaque reference rather than copying document bytes into queue messages or logs.
Make one identity survive every retry
The idempotency key must describe the requested result, not the attempt. Timestamps, random request IDs, and retry counters therefore don't belong in it. Region and retention policy do belong in it because changing either changes the authorized processing context. The same applies to the OCR profile when profile selection can change the searchable output.
This Go example creates a deterministic key from a typed request. It deliberately contains no commercial client and no endpoint path; it belongs in the application boundary before any queue or PDF processor is selected.
package archive
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
)
type Identity struct {
TenantID string `json:"tenant_id"`
DocumentID string `json:"document_id"`
Revision int `json:"revision"`
Region string `json:"region"`
OCRProfile string `json:"ocr_profile"`
RetentionPolicy string `json:"retention_policy"`
SourceSHA256 string `json:"source_sha256"`
}
func IdempotencyKey(id Identity) (string, error) {
payload, err := json.Marshal(id)
if err != nil {
return "", err
}
sum := sha256.Sum256(payload)
return hex.EncodeToString(sum[:]), nil
}
Persist that identity before acknowledging admission. A worker should lease the job for a bounded period and commit a result only while it still owns the lease and the source digest still matches. Retries preserve the identity. A terminal rejection records a machine-readable reason and the validator version, but it should not copy names, addresses, shipment contents, document text, or temporary download credentials into an error field.
This is where operational complexity earns its keep. A queue, lease, and ledger add moving parts, yet they make burst handling and ambiguous retries observable. The synchronous alternative looks smaller until a large scan batch occupies connections, callers retry, and nobody can tell whether an absent response means rejected, running, or complete. Still, asynchronous processing is not suitable for an interactive one-page preview where the user needs immediate visual feedback and the result is not authoritative. Keep a bounded preview path for that case; don't let it publish into the archive.
Admission control should watch estimated queue age, not just queue length. Ten complex image-heavy PDFs and ten single-page scans aren't equivalent work. Measure queue age, processing duration, validation duration, promotion lag, retry count by reason, and age of the oldest temporary artifact as separate signals. One end-to-end percentile conceals which stage needs action.
Fidelity includes the signature and audit trail
OCR confidence alone cannot decide archival acceptance. The useful definition of fidelity comes from the logistics workflow: expected pages are present, the searchable text covers the required fields, page images remain readable, document order is intact, and signature-related evidence remains linked to the exact source and output digests. If a scanned proof of delivery has a visible signature but the searchable layer puts the tracking number on the wrong page, the PDF may look fine and still fail the retrieval job it was created to serve.
Use two independent checks. A visual check compares rendered pages against an approved baseline or inspects layout invariants such as clipping and rotation. A semantic check extracts text and tests the fields the product actually searches: shipment identifier, date, facility, and document type are plausible examples, but the owning team must define the authoritative set. Don't silently "correct" source text in the archive. Store normalized search terms as derived data and retain the original OCR output as evidence tied to its processor and profile version.
The audit record should connect source digest, output digest, OCR profile, validator version, policy version, timestamps, and state transitions. It should also record which service principal caused a transition. It should not become another copy of the document. Signature verification material is governed by the organization's signature design, so a PDF endpoint should preserve and reference that evidence rather than claiming that successful rendering proves a signature valid.
Build a test corpus around ugly documents: rotated pages, faint thermal scans, handwriting near a crop boundary, mixed page sizes, accented names, duplicated pages, a signature on the last page, and a barcode crossing a fold. I first reach for a happy-path sample because it makes deployment comparison easy; it also misses the failures that create a 02:00 page. A fixed corpus gives each processor and each release the same examination. I'm not sure which OCR threshold is right for a given operation until rejected documents have been reviewed by the people who search them; their false-accept and false-reject costs settle that policy.
Fast isn't faithful.
Latency should cover the whole promotion path, from durable admission through OCR and validation to archive availability. Report queue wait separately from processor time. A fast processor behind an old queue is slow to the warehouse user, while a slower processor with bounded queue age may provide a more predictable service. Select against representative scans and burst shapes, not a single warm request.
Put privacy and retention in an artifact ledger
A region request field is only intent. The ledger is the enforceable record. For each revision, enumerate the source scan, temporary renderings, OCR output, searchable derivative, validation evidence, promoted PDF, and any authorized delivery copy. Each entry needs its actual placement, policy identifier, creation time, expiry or hold state, and disposition status. Then an operator can answer the uncomfortable question: "Which copy is still present?"
Count every copy.
US and EU labels aren't retention policies by themselves. Legal and compliance owners must supply the permitted processing locations, access rules, transfer conditions, retention periods, and hold behavior for each document class. Engineering turns those decisions into routing, authorization, lifecycle jobs, and evidence. Workers should receive the minimum reference and context needed for their stage. Logs and metrics should use opaque job identifiers, never OCR text or signed URLs.
Erasure is a workflow because one archived record can have several derivatives. A request moves an eligible record through erasing to erased only after every governed artifact reports disposition. A hold blocks that transition through an explicit policy state. Quietly excluding held objects from a cleanup query leaves no audit trail and makes an apparently successful deletion impossible to explain.
The catch is operational cost. Separate regional processing planes reduce the chance of an accidental cross-region transfer, but they duplicate deployment, queue monitoring, capacity planning, key management, and incident response. They are not suitable by default for every small SaaS. Use a shared plane only when the approved policy allows every processor and artifact location involved; split planes when tenant commitments or policy boundaries require isolation. Privacy drives the topology, not diagram symmetry.
Retention has a second catch: very short temporary-object lifetimes can erase evidence before a retry or investigation completes, while very long lifetimes expand exposure and leave cleanup debt. Tie expiry to explicit job states and alert on overdue transitions. A daily blind purge may be easy to operate, but it cannot distinguish a forgotten temporary file from the only recoverable input for a valid in-flight job.
Roll out from the alert backward
Return to the page: searchable-document lag is high, but intake still looks healthy. The earlier signal should have been rising age of the oldest admitted job, followed by growing validation or promotion lag. Instrument each transition in the ledger and count only durable state changes. Queue depth remains useful for capacity context, but age tells the on-call that user-visible freshness is being consumed.
Deploy the ledger and idempotency rule around the existing processor first. Run the fixed corpus through the new path on every processor or profile change. Next, shadow only documents that policy permits, compare evidence rather than exposing document bodies to developers, and verify that temporary artifacts reach disposition. Move reads to the governed retrieval boundary before the final write cutover, then perform a synthetic erasure drill in each processing region.
The runbook should map a signal to an action. Rising admission age calls for checking arrival rate, tenant fairness, and available worker capacity. Flat admission age with rising processing duration points toward document complexity or processor behavior. Rising promotion lag sends the operator to validators and the artifact store. Duplicate identity conflicts require inspecting the caller's revision and digest rules before adding workers. More capacity can't repair identity.
Tune the warning on sustained age and burn against the freshness objective, with a separate critical threshold for immediate user impact. Don't page on one old test document or a brief batch spike. The final failure mode is an alert so sensitive that normal logistics bursts wake the on-call; after enough false pages, the real delay receives a slower response. Thresholds need replay data, a minimum duration, and exclusions for explicitly paused test tenants. Your mileage may vary because scan mix and promised freshness vary, so review both missed-delay incidents and no-action pages after rollout.
Choose PDF endpoint boundaries that make admission durable, retries idempotent, fidelity a promotion gate, and privacy policy authoritative over every artifact. The endpoint names can change. Those invariants should not.
Top comments (0)