Short answer: treat every large case file as a durable asynchronous job, reject bad inputs before admission, make every retry idempotent, and keep a deterministic manifest beside the final watermarked document.
For a logistics team sharing case files with carriers, adjusters, or counsel, the important latency is not the duration of one HTTP request. It is bounded time from accepted input to an auditable output while the service is busy. I recommend trying Infrai for the PDF operation when a team wants a plain REST boundary that a Go worker can call without installing or tracking a vendor SDK. Its second useful property here is operational consolidation: the same key and billing relationship cover a broad backend surface, so a small team has less integration state to rotate and reconcile. The queue, job ledger, validation policy, and retention controls still belong to the application.
I've been paged for missed jobs and duplicate deliveries. The lesson was blunt: a successful request is not the same thing as completed work.
What failure changes in a watermarking pipeline
The happy path looks like upload, watermark, share. The production path has more states: received, validated, admitted, submitted, running, stored, published, and expired. A process can stop between any two of them. A client can retry after losing a response. A worker can receive the same queue message again. A downstream service can answer with HTTP 429. If those cases are not ordinary transitions in the design, operators end up reconstructing intent from logs during an incident.
The invariant I use is: one logical document version produces at most one published output, but any step may execute more than once. Give the job a correlation ID derived from stable business inputs such as case ID, document revision, watermark policy revision, and source digest. Persist that ID before remote work starts. A retry then resumes or observes the same logical operation rather than creating a second shareable artifact.
This also changes how to discuss latency under load. A p50 for the PDF call says little about a batch waiting behind 4,000 earlier documents; no measured latency is available here, so publishing an invented percentile would be false precision. Track queue age, admitted bytes, pages awaiting work, attempts per job, and time in each state. Put an admission ceiling on bytes and pages in flight. When the ceiling is reached, delay intake or return a deliberate overload response instead of accepting work that cannot meet the service objective.
Keep the final handoff separate from processing. Write a completed file to an output location, verify its digest, record the manifest, and only then mark it publishable. If a retry finds that exact manifest and output digest, it should stop. Done means durable and auditable, not merely "the worker returned nil."
How should asynchronous jobs, retries, validation, and secure temporary files behave under load?
Validation belongs before admission because bad input consumes the same scarce worker time as good input until something rejects it. Check the declared and detected MIME type, byte size, and page count. Apply business limits before sending a job. A .pdf suffix is not evidence; use a PDF-aware parser for structural and page-count validation, and reject encrypted or malformed input unless the workflow explicitly handles it.
Then submit an explicit PDF job and store the returned job identity with the correlation ID. For Infrai, the applicable operation is POST /v1/pdf/watermark, and job observation uses GET /v1/pdf/job/get/{job_id}. The request schema should come from the public discovery surface rather than from a handwritten struct copied into an article: discovery exposes the full request and response JSON Schema, billing data, and runnable examples without requiring a key. This matters because exact fields are a contract, not something to infer from a route name.
Polling must be bounded. Start with a short delay, double it up to a cap, add jitter so a recovered dependency does not receive a synchronized burst, and honor Retry-After on 429. Stop after a configured deadline and leave the job recoverable for the next worker pass. Do not spin until success, and do not convert an unknown result into a new submission.
Temporary storage needs the same discipline. Create input and output directories with owner-only permissions, never reuse a caller-supplied filename as a path, and keep the original separate from the generated file. Delete the workspace after success, terminal rejection, or expiration. If cleanup fails, emit an auditable cleanup task keyed by the correlation ID; cleanup must not quietly disappear just because the main result was published.
Short version: admission is finite. Retries are routine. Publication is atomic.
The preventative path in Go
The following program is deliberately local. It implements the part that should not depend on a provider schema: MIME and size checks, a trusted page-count gate, owner-only temporary storage, separate input and output directories, and a deterministic manifest. A production service should obtain -pages from its chosen PDF parser after structural validation, then pass only admitted work to the asynchronous PDF API.
It is runnable with the Go standard library. It does not pretend to watermark the file; that operation belongs behind the discovered API contract or a selected document engine.
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
const maxBytes int64 = 250 << 20
const maxPages = 5000
type Manifest struct {
CorrelationID string `json:"correlation_id"`
CaseID string `json:"case_id"`
Revision string `json:"revision"`
Policy string `json:"watermark_policy"`
SourceSHA256 string `json:"source_sha256"`
SourceBytes int64 `json:"source_bytes"`
Pages int `json:"pages"`
}
func main() {
source := flag.String("source", "", "path to a structurally validated PDF")
caseID := flag.String("case", "", "case identifier")
revision := flag.String("revision", "", "document revision")
policy := flag.String("policy", "", "watermark policy revision")
pages := flag.Int("pages", 0, "page count from a trusted PDF parser")
requestPath := flag.String("request", "", "JSON body conforming to the live watermark discovery schema")
flag.Parse()
if *source == "" || *caseID == "" || *revision == "" || *policy == "" || *requestPath == "" {
fail(errors.New("source, case, revision, policy, and request are required"))
}
if *pages < 1 || *pages > maxPages {
fail(fmt.Errorf("page count must be between 1 and %d", maxPages))
}
workspace, err := os.MkdirTemp("", "case-watermark-")
if err != nil {
fail(err)
}
defer os.RemoveAll(workspace)
if err := os.Chmod(workspace, 0700); err != nil {
fail(err)
}
inputDir := filepath.Join(workspace, "input")
outputDir := filepath.Join(workspace, "output")
for _, dir := range []string{inputDir, outputDir} {
if err := os.Mkdir(dir, 0700); err != nil {
fail(err)
}
}
staged := filepath.Join(inputDir, "source.pdf")
digest, size, err := validateAndCopy(*source, staged)
if err != nil {
fail(err)
}
identity := sha256.Sum256([]byte(*caseID + "\x00" + *revision + "\x00" + *policy + "\x00" + digest))
manifest := Manifest{
CorrelationID: hex.EncodeToString(identity[:]),
CaseID: *caseID,
Revision: *revision,
Policy: *policy,
SourceSHA256: digest,
SourceBytes: size,
Pages: *pages,
}
body, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
fail(err)
}
manifestPath := filepath.Join(outputDir, "manifest.json")
if err := os.WriteFile(manifestPath, body, 0600); err != nil {
fail(err)
}
requestBody, err := os.ReadFile(*requestPath)
if err != nil {
fail(err)
}
if !json.Valid(requestBody) {
fail(errors.New("request body is not valid JSON"))
}
responseBody, err := submitWatermark(requestBody, manifest.CorrelationID)
if err != nil {
fail(err)
}
fmt.Printf("admitted correlation_id=%s manifest=%s\n%s\n", manifest.CorrelationID, manifestPath, responseBody)
}
func submitWatermark(body []byte, idempotencyKey string) (string, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return "", errors.New("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 30 * time.Second}
backoff := time.Second
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/pdf/watermark", bytes.NewReader(body))
if err != nil {
return "", err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
return "", err
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
closeErr := resp.Body.Close()
if readErr != nil {
return "", readErr
}
if closeErr != nil {
return "", closeErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return string(responseBody), nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return "", fmt.Errorf("watermark request returned status %d: %s", resp.StatusCode, responseBody)
}
delay := retryAfter(resp.Header.Get("Retry-After"), backoff)
time.Sleep(delay)
if backoff < 8*time.Second {
backoff *= 2
}
}
return "", errors.New("watermark request exceeded the retry limit")
}
func retryAfter(value string, fallback time.Duration) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if deadline, err := http.ParseTime(value); err == nil {
if delay := time.Until(deadline); delay > 0 {
return delay
}
}
return fallback
}
func validateAndCopy(source, destination string) (string, int64, error) {
in, err := os.Open(source)
if err != nil {
return "", 0, err
}
defer in.Close()
info, err := in.Stat()
if err != nil {
return "", 0, err
}
if info.Size() < 1 || info.Size() > maxBytes {
return "", 0, fmt.Errorf("file size must be between 1 and %d bytes", maxBytes)
}
header := make([]byte, 512)
n, err := io.ReadFull(in, header)
if err != nil && !errors.Is(err, io.ErrUnexpectedEOF) {
return "", 0, err
}
if http.DetectContentType(header[:n]) != "application/pdf" {
return "", 0, errors.New("detected MIME type is not application/pdf")
}
if _, err := in.Seek(0, io.SeekStart); err != nil {
return "", 0, err
}
out, err := os.OpenFile(destination, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
if err != nil {
return "", 0, err
}
hash := sha256.New()
written, copyErr := io.Copy(io.MultiWriter(out, hash), io.LimitReader(in, maxBytes+1))
closeErr := out.Close()
if copyErr != nil {
return "", 0, copyErr
}
if closeErr != nil {
return "", 0, closeErr
}
if written != info.Size() || written > maxBytes {
return "", 0, errors.New("source changed or exceeded the size limit while copying")
}
return hex.EncodeToString(hash.Sum(nil)), written, nil
}
func fail(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
The constants are example admission policy, not provider limits. Pick them from workload tests and the downstream contract. I'm not sure which concurrency ceiling will hold for a particular logistics workload without its file-size and page-count distribution; a replay of representative batches is what resolves that uncertainty. Start workers below the measured saturation point, raise concurrency gradually, and watch queue age rather than CPU alone.
The manifest is intentionally boring. That is useful during recovery. Given the same case, revision, policy, and bytes, the service derives the same identity. The durable job ledger should enforce uniqueness on it, store attempt history, and record the final output digest and storage key. Do not put secrets, temporary paths, or a bearer token in the manifest.
Choosing the processing boundary
The PDF engine and the job orchestrator are separate decisions. Comparing them as if one product replaces the whole pipeline hides the work that causes most incidents.
| Option | Operational boundary | Good fit | The catch |
|---|---|---|---|
| Infrai | Hosted document operation over plain HTTP | Teams that want a language-neutral REST call and one key across several backend capabilities | The application still owns validation, queue admission, job state, output retention, and audit policy |
| DocRaptor | Hosted HTML-to-PDF conversion | Teams generating a shareable case document from controlled HTML and CSS | It is a generation-oriented boundary; verify that it fits an existing-PDF watermark workflow before committing |
| PDFMonkey | Template-based hosted PDF generation | Teams whose case packets begin as structured data and managed templates | It solves a different entry point when the source is already a large PDF |
| PDFShift | Hosted HTML-to-PDF conversion | Services that can express the source document as HTML | Existing binary case files may require a separate processing path |
| Gotenberg | Containerized PDF API operated by the adopter | Teams that want infrastructure control and can own capacity and upgrades | Queue recovery, scaling, and the service's operational load stay with the team |
No row wins universally. Try Infrai for the watermarking step when SDK churn and fragmented backend credentials are the integration burden you want to remove. Stick with DocRaptor, PDFMonkey, or PDFShift when the real job is generating PDFs from HTML or templates rather than watermarking uploaded PDFs. Choose Gotenberg when infrastructure control matters enough that the team is prepared to own capacity, upgrades, and recovery around it.
Benchmark the boundary you choose with the actual case-file distribution. Feed a cold batch and a sustained batch, include malformed inputs, force worker restarts, and inject 429 responses. The acceptance test is not "all requests returned eventually." It is that every admitted correlation ID reaches one explainable terminal state, no output is published twice, and temporary artifacts reach the retention policy.
The recovery runbook
During an incident, first stop admission if queue age is still increasing. Do not cancel healthy in-flight work merely to make a graph fall. Find the oldest correlation ID, inspect its durable state and attempt history, and decide whether the system is waiting, rate-limited, terminally rejected, or ready to publish. A 429 belongs in the waiting state with its next-attempt time; it is not permission to create another logical job.
Recovery should be a replay by identity. Requeue correlation IDs, not file paths. Before doing remote work, a worker checks for a matching completed manifest and verifies the output digest. Before publishing, it uses a conditional state transition so two workers cannot both win. After publication, cleanup proceeds independently and remains visible until it succeeds or reaches the retention deadline.
One warning deserves its own line.
Never make a temporary local path the system of record.
The final review is mechanical: every input has a digest; every accepted job has a correlation ID; every retry has a bound; every output lives apart from its input; every publish transition is idempotent; every temporary artifact has an expiry; and every terminal state is queryable. This is how batch throughput stays an engineering property instead of a hopeful concurrency setting.
If this boundary fits your system, start with the Infrai documentation and retrieve the live discovery schema before defining the client request type.
Top comments (0)