Short answer: keep one immutable, signed PDF as the archive record, then separate preview, OCR text, and evidence retrieval into four contracts. Give each contract its own latency, privacy, and retention policy. That is the least complex way to preserve fidelity when a logistics customer challenges a delivery signature.
The page usually arrives at 02:13 UTC. A carrier's scanned receipt is searchable in the dashboard, but the PDF in cold storage has a clipped signature. The on-call sees a successful 200 response and a green queue counter. Nothing looks broken until a customer asks who accepted the shipment.
This is where an SRE's definition of success changes. The browser being fast is useful; proving that the bytes presented to a reviewer are the bytes accepted at the dock is the actual requirement. I once assumed a sharper preview would expose this class of defect. It did not. A preview can look perfect while the stored artifact and its digest disagree.
Start from the alert, then work backward
The first useful alert names the missing relationship, not just a slow request. Record a content digest, byte length, page count, renderer version, and signature status beside every archive object. Connect upload, OCR, PDF creation, signing, object storage, and retrieval in one trace. When a worker retries, the same idempotency key must resolve to the same object reference.
A practical alert says that a signed digest cannot be fetched, an object's byte length changed, or an OCR result points at a different PDF version. Preview latency still deserves a dashboard, but it should not page the same person as an evidence-integrity violation. A p95 thumbnail spike is noisy; an audit link that cannot reproduce the signed bytes is urgent.
The runbook should show the document identifier, tenant region, retention start event, and last successful evidence read. It should also show whether a legal hold is active. Those fields let the responder decide whether to retry, quarantine a new rendition, or escalate to the records owner without opening the PDF in an ad-hoc tool.
One rule keeps the investigation short: never overwrite the archive object in place. A corrected OCR layer is a new version linked to the same source bytes. The signature remains attached to the version it covers.
Evidence is the product.
How should PDF endpoints balance fidelity, latency, privacy, and retention?
Use four routes as policy boundaries, even if they share a queue and a storage cluster.
- Archive write accepts the final PDF and metadata, computes a digest, and returns an immutable object identifier. Synchronous work ends at durable acceptance; indexing can happen later.
- Preview read serves a bounded rendition for the web UI. It may trade fidelity for speed, but it never becomes the signed record.
- Text read returns OCR text and confidence metadata. Keep it replaceable so a recognition correction cannot rewrite legal evidence.
- Evidence read returns the original bytes, digest, signature envelope, and audit events to an authorized reviewer.
The split makes failure behavior legible. Archive writes need strict durability and retention controls. Previews need a tighter latency budget and an eviction policy. Text reads can tolerate asynchronous refresh. Evidence reads need stronger authentication, a complete access log, and a response that can be verified independently of the browser.
Do not hide these distinctions behind one timeout. A five-second preview budget may be reasonable for a dashboard, while an evidence export can be queued and reported as a job. The important part is that a caller can tell “accepted for archive” from “rendered for display.” Mixing those states is how a green health check masks a records problem.
Make retries boring with immutable evidence
Duplicate delivery is normal in queue systems. Design for it.
The worker below writes an object only when the supplied digest is new, then records an audit event with the same key. The interface is generic so the idempotency policy can be tested with an in-memory fake before wiring a storage implementation.
package archive
import "context"
type ArchiveStore interface {
PutIfAbsent(ctx context.Context, key string, body []byte, meta map[string]string) (bool, error)
AppendAudit(ctx context.Context, key, event string) error
}
func Archive(ctx context.Context, store ArchiveStore, key string, pdf []byte, digest string) error {
created, err := store.PutIfAbsent(ctx, key, pdf, map[string]string{"sha256": digest})
if err != nil {
return err
}
if !created {
return store.AppendAudit(ctx, key, "duplicate delivery acknowledged")
}
return store.AppendAudit(ctx, key, "archive bytes committed")
}
The retry must not mint a second retention clock or signature. Keep the signing input canonical: digest, document identifier, tenant, and creation timestamp in a documented order. Store the signature envelope next to the bytes, and log key rotation as an audit event. A failed downstream OCR attempt can be retried without touching either the digest or the retention start.
For browser clients, remember that a PDF is binary data. The platform's Blob interface represents immutable, raw data and can be read as bytes or text; that makes it useful for preview and download code, but it does not make a browser-generated Blob an archive authority. The authority is the object whose digest and signature are recorded server-side.
Privacy and retention belong in the route contract
US and EU tenants rarely share one policy. Classify documents before they enter a preview cache. Minimize copied fields, encrypt in transit and at rest, and make cache keys tenant-scoped. Access logs should record the actor and purpose, while the PDF itself stays out of ordinary application logs.
Retention starts at a declared event, such as delivery acceptance, not at whichever worker happened to finish last. A legal hold pauses deletion without changing the original expiry calculation. Test expiry, hold release, tenant export, and deletion proof as workflows. A checkbox in an admin panel is not evidence.
The catch is operational load. Immutable storage, signature verification, regional routing, and long retention all add key-management and support work. This design is not suitable when a product only needs transient previews; a short-lived object and one rendering path may be enough. Stick with the simpler choice when there is no signature, audit, or regulatory review requirement.
Privacy failures often start in a convenience feature. A cache key that omits the tenant can serve the right bytes to the wrong person. A trace attribute containing the full OCR text can leak more than the PDF endpoint itself. In a multi-region SaaS, the request context should carry tenant region and policy version from archive write through evidence read; otherwise a failover can select a technically healthy copy that violates the customer's residency rule. Keep trace fields to identifiers and measurements, make a reviewer request an explicit logged action, and document which team can release a legal hold. This extra complexity feels slow during an incident, especially when the visible symptom is only a missing thumbnail, but it gives the responder a bounded search instead of a guess.
Roll out with failure drills and honest thresholds
Ship the contracts first. In staging, replay the same message three times, alter one byte after signing, delay OCR for ten minutes, and force a preview cache miss. The expected result is boring: one archive object, one digest, visible duplicate events, and an evidence read that refuses a mismatched byte stream.
Then test the policy edges. Export a held document, release the hold, let an object approach expiry, and verify the deletion record without deleting the evidence needed for the test. Run the same cases in each tenant region. Your mileage may vary because retention language differs by contract and jurisdiction; have counsel review the policy event, not just the storage setting.
Track four numbers per tenant: archive-accept latency, preview p95, evidence-read verification failures, and objects approaching expiry. Sample traces across regions so a US request cannot silently read an EU copy that violates the tenant policy.
Short thresholds create pages. Long thresholds create disputes.
Calibrate against replayed incidents, attach the runbook to the alert, and review false positives after every change to the renderer or OCR pipeline. I'm not sure any single dashboard catches every jurisdictional nuance, so keep a periodic records review alongside the automated checks. The decision rule is simple: optimize preview speed only after fidelity and an explainable retention trail are measurable.
Top comments (0)