Short answer: make template ownership the first decision. Use a hosted PDF API when an external execution boundary is permitted, your team can pin and audit the watermark template revision, and measured latency under representative load meets the sharing deadline; keep local PDF libraries when decrypted customer files, keys, or the authoritative template must stay inside infrastructure you control.
For a fintech backend, watermarking is part of publication control rather than a cosmetic pass. The output may leave the company, so the system must prove which input, policy, template revision, and recipient produced it. A fast renderer with an ambiguous evidence trail is the wrong renderer.
Define the document contract before choosing the execution boundary
Template ownership has two meanings that teams often collapse. Design ownership answers who approves the watermark wording and placement. Runtime ownership answers which system resolves a revision, decrypts the source, renders the mark, and publishes the result. A hosted service may perform the second role without receiving authority over the first, but only if the application sends an immutable revision or a fully resolved overlay rather than an unversioned template name.
That distinction changes the architecture. The ledger-facing application should create a publication intent containing a stable operation key, source digest, recipient scope, template revision, and policy revision. The renderer consumes that intent and returns an artifact whose digest can be attached to the same record. Passwords and decrypted pages don't belong in the audit log; identifiers, hashes, timestamps, and state transitions do. This is an exactly-once publication problem even when transport is at least once: repeating a render can be tolerable, while exposing two independently watermarked artifacts to the recipient is not.
Keep the state machine small. accepted means the intent is durable, rendered means an output has been verified, and published means one artifact became externally visible. A retry may revisit the rendering transition, but a compare-and-swap on the publication record prevents a second winner. This arrangement also separates a compliance question from an implementation preference: an auditor can inspect the authorization and artifact evidence without needing to trust a worker's ephemeral logs.
No renderer fixes unclear ownership.
What should own password-protected customer files under production load?
The application should own authorization and the durable publication record. The execution location can then be hosted or local. A hosted PDF API is preferable when policy allows the necessary document content to cross that boundary, template revisions can be supplied or pinned in an auditable way, and elastic external capacity performs better for the workload shape your team actually sends. A local library is preferable when residency rules, key custody, or template governance require plaintext and rendering assets to remain within your controlled environment.
The catch is that neither choice removes queues. A hosted path includes local admission, upload, remote admission, rendering, and download. A local path includes local admission, worker scheduling, decryption, rendering, encryption, and artifact storage. Comparing one API response time with one library function duration hides most of the customer wait. Compare the full interval from durable publication intent to externally visible artifact, then retain the component intervals so the tail can be explained.
Don't benchmark a single clean page.
Use fixture classes that reflect the production corpus: page-count bands, byte-size bands, image-heavy pages, embedded fonts, and password-protected inputs. Replay the expected burst shape, not merely the average request rate. I wouldn't accept a provider's idle latency or a laptop microbenchmark as capacity evidence because neither establishes how admission queues behave at the release boundary. I'm not sure which path will have the lower tail for an arbitrary corpus; only a controlled run using the same fixtures, concurrency schedule, deadline, and publication rule can resolve that uncertainty.
Latency needs a budget before it needs a percentile. If the external-share workflow has a deadline, allocate that time across admission, processing, transfer, and publication, while reserving room for a bounded retry. Record p50 to understand the ordinary path, but gate on p95 and p99 because batch releases and reconciliation jobs create contention that averages conceal. Those percentiles are measurements to collect, not universal targets; compliance and customer commitments must set the actual limit.
Make retries preserve one publication decision
Timeouts create the dangerous case: the caller lacks a result but cannot infer that processing did not happen. Retrying with a new identity turns uncertainty into duplicate work and potentially duplicate external publication. The operation key therefore has to be stable across attempts, and the application has to reconcile an indeterminate attempt before it lets another artifact become visible.
The following Go sketch keeps vendor details outside the domain record. It deliberately models publication as a separate conditional action; a renderer can be swapped without rewriting the audit rule.
package documents
import (
"context"
"crypto/sha256"
"errors"
)
type Intent struct {
OperationKey string
SourceSHA256 [32]byte
TemplateRev string
PolicyRev string
RecipientScope string
}
type Artifact struct {
Bytes []byte
SHA256 [32]byte
}
type Renderer interface {
Render(ctx context.Context, intent Intent, password []byte) ([]byte, error)
}
type Publisher interface {
PublishOnce(ctx context.Context, operationKey string, artifact Artifact) error
}
func RenderAndPublish(ctx context.Context, r Renderer, p Publisher, in Intent, password []byte) error {
if in.OperationKey == "" || in.TemplateRev == "" || in.PolicyRev == "" {
return errors.New("incomplete publication intent")
}
b, err := r.Render(ctx, in, password)
if err != nil {
return err
}
artifact := Artifact{Bytes: b, SHA256: sha256.Sum256(b)}
return p.PublishOnce(ctx, in.OperationKey, artifact)
}
In production, PublishOnce must be backed by a durable uniqueness or compare-and-swap guarantee, not an in-memory mutex. The audit event should state that an operation key selected an artifact digest under specific template and policy revisions. It should not contain the password, the source bytes, or the rendered output. Keep raw document access governed by the document store's retention and access policy rather than duplicating sensitive material into observability systems.
Backpressure belongs at admission. Bound the queue, bound concurrency, and propagate a deadline through render and publication. For a remote executor, distinguish HTTP 429 admission pressure from an ambiguous client-side deadline, and use the service's documented idempotency semantics if it has them. For a local executor, enforce memory, CPU, page-count, and input-size budgets so one pathological document cannot monopolize a worker. In both cases, reconciliation should compare durable intents with published artifact digests and surface accepted operations that never reached publication.
Use an ownership matrix, not a feature scorecard
A feature checklist makes hosted and local options appear interchangeable until a policy constraint invalidates one column. An ownership matrix asks the more useful question: who can change each consequential input, and who can produce evidence after the change?
| Control | Hosted execution | Local execution | Decision evidence |
|---|---|---|---|
| Template approval | Application retains approval; contract must preserve revision identity | Repository and deployment controls preserve revision identity | Revision, approver, effective time, rendered-input hash |
| Decryption boundary | External processing must be permitted and reviewed | Plaintext remains in the operated worker boundary | Data-flow review and key-access policy |
| Capacity admission | Provider quotas and remote queues join local admission | Worker pool and local queues are operated directly | Burst test with end-to-end tail latency |
| Dependency changes | Contract and behavior need regression tests | Library, fonts, and worker image need patch control | Pinned fixtures and release evidence |
| Publication uniqueness | Remote result is reconciled to the local intent | Local result is reconciled to the local intent | One operation key mapped to one visible artifact |
A hosted API is not suitable when external plaintext processing is prohibited, when the required template cannot be immutably identified, or when network transfer consumes the latency budget. Stick with a local library in those cases, accepting responsibility for native dependencies, fonts, worker isolation, patching, and capacity. Conversely, local rendering is a poor fit when burst capacity repeatedly exceeds a worker pool the team can responsibly operate and the hosted boundary passes security, retention, regional-processing, deletion, and latency review.
There is also a hybrid boundary: resolve and approve the template locally, send only the minimum rendering inputs to a hosted executor, and retain publication authority locally. This can preserve governance while externalizing compute, but it does not bypass data residency or key-custody limits. Legal and compliance reviewers must decide whether the transferred representation is permitted; architecture cannot redefine regulated data by changing its container.
Roll out by moving execution, not authority
Begin with a shadow path over synthetic and approved test fixtures. Both executors receive the same template and policy revisions, but neither shadow result is externally published. Compare page count, visible watermark placement, text and image behavior, metadata policy, and end-to-end latency distributions; raw PDF byte equality alone is insufficient as a visual assertion because serialization details can differ while the rendered pages remain equivalent.
Then move a bounded document class, keeping the durable intent and PublishOnce rule unchanged. Emit queue wait, processing duration, transfer duration where applicable, artifact size, template revision, policy revision, attempt number, and terminal publication state. Never emit passwords or customer content. A deployment is acceptable only when reconciliation finds every eligible intent in one explainable terminal state and the latency tails remain within the predeclared budget under the expected burst.
Rollback should switch the executor selected for new intents; it should not rewrite template history or replay already published documents. That is the payoff of separating authority from execution — the migration changes where bytes are processed while the approval record, idempotency key, audit trail, and single-publication invariant remain stable.
Choose the boundary only after this exercise. Hosted execution wins when its governance contract and measured tail behavior fit; local execution wins when custody and deterministic control dominate. The template owner, however, should remain explicit in either design.
Top comments (0)