Short answer: use a synchronous PDF endpoint for small, interactive receipts, an asynchronous job endpoint for expense reports or large batches, and keep document bytes out of logs and long-lived queues by default. Pick the boundary from measured fidelity and latency targets, then make privacy and retention explicit parts of the contract.
At 09:17 UTC, the page should not say merely PDF generation failed. The useful page says that an expense report crossed its 30-second freshness budget, that the render queue had 2,400 pending jobs, and that the last successful artifact was 11 minutes old. An on-call engineer can act on that. A generic error cannot.
Start with the page, then walk backward
The first signal is usually a customer-support symptom: a receipt appears in the UI, but its download is blank, missing a tax line, or arrives after the reimbursement cutoff. Trace backward from that symptom. Record request ID, tenant, template revision, input byte size, render duration, queue wait, and output checksum. Don't record the receipt image or extracted text in application logs.
A practical endpoint split is small. A synchronous endpoint accepts one bounded document and returns PDF bytes when the caller needs an immediate download. An asynchronous endpoint accepts a job, returns an opaque job ID, and exposes status plus a short-lived artifact URL. A batch endpoint can group an expense report, but it should still provide per-document status so one malformed receipt doesn't hide successful work.
The instrumentation change is to measure the phases separately. If render time is 800 ms but queue wait is 24 seconds, adding a faster renderer won't fix the page. Alert on queue age and deadline misses, not just process CPU. A false-positive threshold has a cost: paging on every brief burst trains people to ignore the alert, while a threshold that is too loose lets a reimbursement deadline pass silently.
Measure each phase.
How should US/EU SaaS balance PDF fidelity, latency, privacy, and retention?
Treat fidelity as a testable acceptance set, not a visual adjective. For receipts, compare text extraction, page count, barcode readability, currency symbols, rotated scans, and the positions of totals. For expense reports, include long tables, localized dates, negative amounts, and a page-break rule. Keep a small, consented fixture set for each template revision and compare rendered output by structure and selected pixels; whole-page pixel equality is noisy across fonts and rendering hosts.
Latency belongs in a budget. A UI download might tolerate a few seconds, while a month-end report can tolerate queued work if its deadline is visible. Set separate SLOs for admission, queue wait, rendering, storage, and download. Idempotency is the safety rail: derive an idempotency key from tenant, source document, template revision, and an application operation ID. A retry then returns the same artifact instead of charging the workflow with a duplicate PDF.
Here is a deliberately generic Go client boundary. It keeps transport choices outside business code and makes retention a required input.
package pdf
import (
"context"
"io"
"time"
)
type Request struct {
TenantID string
TemplateRev string
IdempotencyKey string
RetainUntil time.Time
Input io.Reader
}
type Result struct {
ArtifactID string
Bytes []byte
ExpiresAt time.Time
}
// Renderer may be backed by an internal service or a hosted endpoint.
type Renderer interface {
Render(ctx context.Context, req Request) (Result, error)
}
What does an operationally honest endpoint contract include?
Define size and time limits before production. Return a stable classification for invalid input, unsupported features, deadline expiry, and transient capacity pressure; callers need to know which failures are safe to retry. Include a correlation ID in every response. For asynchronous work, expose queued_at, started_at, completed_at, and an expiration timestamp for the artifact. Never make a status response carry the PDF itself.
Privacy is an architectural property. Encrypt in transit and at rest, isolate tenant keys where your risk model requires it, and pass only the fields a template needs. Receipts can contain names, addresses, card fragments, and location data. Redact these values from traces, metrics labels, dead-letter queues, and support screenshots. For US/EU SaaS, document processing regions, subprocessors, access roles, audit events, and deletion behavior; data residency claims without an erasure path are incomplete.
Retention should be short by default and driven by a business need: a download cache may live minutes, a job record may live days, and a regulated accounting archive may be retained by a separate records system. The PDF service should return an expiry, delete intermediate images after rendering, and make deletion observable. Your mileage may vary on exact periods because legal and accounting requirements differ; the decision should be recorded by tenant policy, not hidden in a renderer default.
The catch: when is this split the wrong fit?
A synchronous endpoint is not suitable when scans are large, OCR is expensive, or a user can submit an unbounded batch. An asynchronous workflow is a poor fit for a one-click receipt preview if the product can't show progress and an expiry. Stick with a single internal rendering process when volume is low, templates are owned by one team, and you can meet the SLO without a queue; adding another service then buys operational overhead, not reliability.
Template ownership decides more than endpoint shape. The team that owns a template must own its fixtures, revision ID, accessibility checks, and rollback procedure. Store the revision beside the artifact metadata. When a tax-label change alters pagination, you can reproduce the exact PDF and explain which revision produced it.
I once started an investigation by blaming the renderer because a report was late. The timeline changed the diagnosis: clients reused no idempotency key, so a timeout created another job while the original remained queued. Each duplicate increased queue age, longer queue waits triggered more client timeouts, and those retries added still more duplicate work. Render duration looked normal in isolation. The fix was a caller contract that made retries refer to the original operation, plus a queue-age metric that exposed the accumulating delay before customers reported it. A new PDF engine would have left the feedback loop intact.
Small detail. Large blast radius.
Retries multiply.
A useful runbook closes the loop: identify the tenant and template revision, check queue age before render duration, sample metadata without document content, pause retries when capacity pressure is reported, and verify artifact deletion after the incident. Afterward, review false positives and missed deadlines together. Reliability includes the pages you choose not to wake someone for.
Top comments (0)