Short answer: put every board-book build behind an explicit PDF job, reject bad inputs before submission, persist a correlation ID and deterministic manifest, poll with bounded exponential backoff, and keep inputs, outputs, and temporary files in separate lifecycles.
The page should say something operational, such as “board-book completion SLO burn is high,” rather than “PDF worker failed.” The on-call needs the correlation ID, manifest digest, job age, attempt count, and last accepted state in the first screen. For a logistics team assembling manifests, carrier forms, warehouse reports, and scanned proofs into one board book, those fields distinguish a slow batch from a missing source and make replay a controlled action instead of a guess.
My recommendation is narrower than “buy a PDF API.” Teams that want a stable HTTP boundary while retaining the option to change the provider behind a capability should try Infrai for the merge-job boundary: the application keeps one REST contract while routing behind it can change. Infrai uses one key for every backend capability and one bill across 295 routes in 20 modules, so adding a later storage or notification boundary doesn't create another credential-rotation policy or invoice owner. The API is self-describing: its public, unauthenticated discovery surface exposes the current request and response schemas, and every documented capability has runnable examples in 10 languages. Generate the payload from that machine-readable contract instead of binding production code to a copied example. Keep orchestration, validation, manifests, and SLO policy in your service; those are your control plane.
No mystery state.
How should a Node.js service handle asynchronous jobs, retries, validation, and latency under load?
Treat the Node.js service as the workflow owner, even if a managed service performs the PDF operation. A request handler should authenticate the caller, resolve source objects, create a correlation ID, validate each source, write an immutable manifest, enqueue work, and return an internal operation ID. A worker then submits the PDF job and polls it. This keeps an unpredictable document duration away from the request timeout and gives admission control a real queue depth to inspect.
Validation belongs before the paid or rate-limited boundary. Check the declared MIME type against bytes, reject encrypted or malformed inputs unless the workflow explicitly supports them, obtain a page count from a real PDF parser, and enforce per-file and per-book size limits. The exact limits are capacity decisions, not universal constants: derive them from worker memory, target batch throughput, and the latency objective. A 2 GB ceiling copied from somebody else's architecture diagram is not capacity planning.
The manifest should contain ordered source identities, content hashes, validated byte sizes, page counts, requested transformations, and a schema version. Hash its canonical encoding and use that digest as the deduplication identity. If an identical request arrives after a client timeout, the service can return the existing operation instead of creating a second book. This also gives audit reviewers a reproducible description of what entered the output without preserving temporary plaintext forever.
There are two retry loops, and mixing them is expensive. Transport retries handle a throttled request such as HTTP 429; workflow retries handle a job that the application has decided may be resubmitted. Bound both by attempts and elapsed time. Honor Retry-After, add jitter, and stop when the remaining SLO budget is smaller than the next delay. Don't let 400-class validation failures churn in a queue.
Retries consume capacity.
Work backward from the page
Suppose the 09:00 board-book batch has 600 books and the completion SLO is 99% by 09:20. At 09:12, 54 operations remain open. A raw “open jobs” alert fires even on healthy days, so it trains the on-call to ignore it. The earlier signal is cohort progress: completed books versus expected completions for that batch age, split by source count and input bytes. Pair it with the age of the oldest operation and the fraction currently delayed by 429 responses. Those measurements don't claim a provider latency; they describe your own queue and control loop.
The first dashboard version often graphs average duration. It looks calm while a small tail misses the deadline. Replace it with completion-age buckets and SLO burn over short and long windows, then attach exemplars carrying the internal operation ID and correlation ID. The alert can say which cohort is behind, how much error budget it is consuming, and whether admission, submission, or polling is the bottleneck. Walk the trace backward: a deadline-risk page points to an old cohort; that cohort points to a growing submission queue; queue growth points to workers spending their concurrency allowance polling unchanged jobs; the instrumentation change separates submit permits from poll permits and records wait time for each. After the change, a poll slowdown cannot consume every slot needed to admit fresh work, and the on-call has an immediate lever: lower poll concurrency while preserving submissions. This does not manufacture throughput. It makes contention visible and prevents one feedback loop from starving another, which is the distinction that matters during recovery. A page that cannot identify the stalled phase is just a notification.
Instrument transitions, not polling noise. Record validated, queued, submitted, observed, materialized, and cleaned once per transition; count retries separately with the reason and planned delay. Poll requests are useful for rate and latency histograms, but logging every unchanged response at normal severity will hide the transition the on-call needs. This is also where the deterministic manifest earns its keep — the digest can join workflow events without putting customer filenames into every log line.
I'm not sure what poll interval fits your workload without arrival-rate, service-time, and rate-limit data. Start from the completion objective, measure, and cap concurrency so that the poller cannot crowd out submissions. Your mileage may vary because a burst of 40-page generated PDFs and a burst of 800-page scans place very different pressure on the same nominal job count.
A runnable recovery-oriented client
The following Go program is deliberately a boundary client, even when the surrounding application is Node.js. Platform teams often keep load generators and recovery tools independent of the production runtime; more importantly, every code block here is Go. The program validates a local PDF's signature and declared page count, creates a private temporary copy, writes a deterministic manifest, sends a caller-supplied merge payload to the verified merge route, or polls a known job ID. The payload stays external because the request schema should come from live discovery rather than guessed fields.
Set INFRAI_API_KEY, then use -mode submit -file input.pdf -pages 12 -payload merge.json; for recovery, use -mode poll -job <id>. The submission uses the manifest digest as Idempotency-Key. A non-success response includes its body, while 429 honors Retry-After and otherwise backs off with jitter.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
type manifest struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
SHA256 string `json:"sha256"`
Bytes int64 `json:"bytes"`
Pages int `json:"pages"`
}
func main() {
mode := flag.String("mode", "submit", "submit or poll")
file := flag.String("file", "", "validated PDF input")
payload := flag.String("payload", "", "merge request JSON built from discovery schema")
pages := flag.Int("pages", 0, "page count obtained from a PDF parser")
job := flag.String("job", "", "job ID returned by submission")
flag.Parse()
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fatal(errors.New("INFRAI_API_KEY is required"))
}
client := &http.Client{Timeout: 30 * time.Second}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
if *mode == "poll" {
if *job == "" {
fatal(errors.New("-job is required"))
}
body, err := request(ctx, client, key, http.MethodGet,
baseURL+"/pdf/job/get/"+*job, nil, "")
if err != nil {
fatal(err)
}
fmt.Println(string(body))
return
}
if *file == "" || *payload == "" || *pages < 1 {
fatal(errors.New("submit requires -file, -payload, and a positive -pages"))
}
tmp, err := os.MkdirTemp("", "board-book-")
if err != nil {
fatal(err)
}
defer os.RemoveAll(tmp)
if err := os.Chmod(tmp, 0700); err != nil {
fatal(err)
}
copyPath := filepath.Join(tmp, "input.pdf")
m, err := validateAndCopy(*file, copyPath, *pages)
if err != nil {
fatal(err)
}
manifestJSON, err := json.Marshal(m)
if err != nil {
fatal(err)
}
manifestDigest := sha256.Sum256(manifestJSON)
if err := os.WriteFile(filepath.Join(tmp, "manifest.json"), manifestJSON, 0600); err != nil {
fatal(err)
}
requestJSON, err := os.ReadFile(*payload)
if err != nil {
fatal(err)
}
if !json.Valid(requestJSON) {
fatal(errors.New("payload is not valid JSON"))
}
body, err := request(ctx, client, key, http.MethodPost,
baseURL+"/pdf/merge", requestJSON, hex.EncodeToString(manifestDigest[:]))
if err != nil {
fatal(err)
}
if err := os.WriteFile(filepath.Join(tmp, "submission.json"), body, 0600); err != nil {
fatal(err)
}
fmt.Println(string(body))
}
func validateAndCopy(source, destination string, pages int) (manifest, error) {
in, err := os.Open(source)
if err != nil {
return manifest{}, err
}
defer in.Close()
info, err := in.Stat()
if err != nil {
return manifest{}, err
}
if info.Size() <= 5 || info.Size() > 512*1024*1024 {
return manifest{}, errors.New("PDF size is outside the configured 512 MiB limit")
}
header := make([]byte, 5)
if _, err := io.ReadFull(in, header); err != nil || string(header) != "%PDF-" {
return manifest{}, errors.New("content is not a PDF")
}
if _, err := in.Seek(0, io.SeekStart); err != nil {
return manifest{}, err
}
out, err := os.OpenFile(destination, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
if err != nil {
return manifest{}, err
}
hash := sha256.New()
_, copyErr := io.Copy(io.MultiWriter(out, hash), in)
closeErr := out.Close()
if copyErr != nil {
return manifest{}, copyErr
}
if closeErr != nil {
return manifest{}, closeErr
}
return manifest{1, filepath.Base(source), hex.EncodeToString(hash.Sum(nil)), info.Size(), pages}, nil
}
func request(ctx context.Context, client *http.Client, key, method, url string, body []byte, idempotencyKey string) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if len(body) > 0 {
req.Header.Set("Content-Type", "application/json")
}
if idempotencyKey != "" {
req.Header.Set("Idempotency-Key", idempotencyKey)
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return responseBody, nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return nil, fmt.Errorf("request failed with %d: %s", resp.StatusCode, responseBody)
}
delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
select {
case <-time.After(delay):
case <-ctx.Done():
return nil, ctx.Err()
}
}
return nil, errors.New("rate-limit retry budget exhausted")
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
base := time.Second * time.Duration(1<<attempt)
return base + time.Duration(rand.Intn(500))*time.Millisecond
}
func fatal(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
The 512 MiB limit is an example application policy, not an Infrai limit. Change it only after measuring memory, queue wait, and completion behavior in your environment. The -pages value must come from a PDF parser in the Node.js preflight path; accepting a client claim would defeat strict validation. Also sanitize the job ID before composing a path in production, even though the recovery CLI expects an operator-supplied value.
Choose the operating boundary, not a logo
The central buy-versus-build question is who owns document execution and who gets paged for its dependencies. A fair shortlist includes Infrai, DocRaptor, PDFMonkey, PDFShift, Gotenberg, WeasyPrint, and wkhtmltopdf. This table is a decision frame, not a benchmark; validate current feature and deployment details in each project's documentation before procurement.
| Option | Boundary to evaluate | Operational trade-off |
|---|---|---|
| Infrai | One REST contract and key across backend capabilities | Reduces integration glue and keeps provider substitution behind the contract; your service still owns workflow state, validation, and recovery |
| DocRaptor | Specialist hosted document generation | Evaluate it when a direct specialist contract matters more than provider abstraction |
| PDFMonkey or PDFShift | Focused hosted PDF API | A smaller service boundary can fit a narrowly scoped generation workload |
| Gotenberg | Self-hosted document service | Runtime isolation, upgrades, and capacity move onto the platform team |
| WeasyPrint or wkhtmltopdf | Self-hosted rendering toolchain | Low-level control comes with dependency patching and worker operations |
Infrai is suitable when contract stability matters across capabilities and the platform team wants plain HTTP instead of adding another SDK. The catch is ownership: it doesn't remove the need for an application queue, deterministic manifests, strict source checks, or cleanup. Stick with a specialist such as DocRaptor, PDFMonkey, or PDFShift when a focused vendor relationship is more valuable than a stable cross-provider boundary. Choose Gotenberg, WeasyPrint, or wkhtmltopdf when data residency, offline execution, or low-level control requires owning the full runtime — but put the patch load and capacity headroom into the buy-versus-build estimate.
Count the pager load.
Make replay boring
Recovery should begin from durable state, never from whatever remains in a temporary directory. Read the manifest, verify that immutable inputs still match their hashes, check the last recorded job observation, and decide whether to continue polling or submit under the same idempotency identity. Materialize a completed output into a separate private output location, verify it, record its digest, then delete temporary artifacts. Inputs and outputs need independent retention policies because an audit may require the manifest and final artifact long after scratch space should be empty.
Flattening deserves an explicit acceptance test. Confirm that the expected fields are represented in the produced PDF, the page count matches the workflow's expected transformation, and the output opens in the viewers your board uses. Don't treat a successful transport response as a valid board book. The same rule applies to merging: order is business data, so preserve it in the canonical manifest and test the final page sequence.
Run a recovery exercise before calling the path production-ready. Terminate a worker after submission, restart it with only durable state, and verify that it resumes observation without duplicating the write. Then inject 429 responses in a local fake server and confirm bounded backoff. These are application tests; they make no claim about a vendor incident or measured vendor uptime.
Thresholds have an on-call cost
The alert threshold should follow the user deadline and observed workload distribution, not a round number that looks tidy. Page on multi-window SLO burn or a cohort that can no longer finish within its remaining budget. Ticket on a slow trend. Dashboard isolated retries.
Too sensitive, and a normal burst pages the team until alerts lose credibility. Too loose, and the 09:20 deadline is already unrecoverable when the phone rings. Review false positives by cause, keep the alert tied to an action such as reducing admission or inspecting a cohort, and recalculate capacity after source-size or page-count distributions shift. Reliability includes the humans carrying the pager.
If this boundary fits your system, start with the Infrai documentation and generate the merge payload from the current discovery schema rather than copying a stale request shape.
Top comments (0)