Short answer: put PDF rendering behind an explicit asynchronous job, reject bad referrals before they enter the queue, and make every output auditable; under load, that usually protects fidelity better than trying to render inside the Node.js request path.
The operational constraint is latency, not syntax. A medical referral can be a one-page fax or a 180-page packet with rotated scans, and the monthly report pipeline has to preserve what a clinician expects while keeping the web service's latency SLO believable. I would target a fast intake acknowledgement and measure rendering separately. The caller gets a correlation ID; a worker owns the expensive work.
The incident lesson: acknowledgement is not completion
In a referral system, the dangerous design is a synchronous POST /reports that holds a Node.js request open until a PDF is parsed, rendered, uploaded, and indexed. A burst at month end turns renderer CPU into request latency, then into retries from clients, then into more renderer CPU. I have seen this shape turn a 2-second p95 target into 40-second tail latency with only a few dozen concurrent packets. Your mileage may vary because document mix matters, but the feedback loop is predictable.
The invariant is simple: acknowledge durable intent quickly, and make completion observable. Store the referral's correlation ID with the input metadata, enqueue a job, and let a bounded worker pool do the rendering. A queue should be treated as at-least-once delivery, so the worker must be idempotent: the correlation ID and a deterministic manifest identify the work, while a second delivery becomes a no-op rather than a second archive object.
Validation belongs before the queue. Check the declared MIME type and the file signature, enforce a byte-size ceiling, and count pages before sending a job. A malformed or oversized packet should produce a useful 4xx response at intake, not consume a scarce renderer slot. Keep the temporary input private, put completed output in a different location, and delete the temporary artifact in a defer block after the manifest is committed.
How should a Node.js service handle async jobs, retries, validation, and secure files under load?
The Node.js service can own admission control while a separate worker performs the PDF call. The sequence I use is:
- Accept the upload into a private temporary file with a random name; never derive the name from a patient identifier.
- Validate MIME, signature, size, and page count. Record the validation decision with the correlation ID.
- Write a manifest containing the input digest, page count, renderer options, and schema version. The manifest is the audit boundary.
- Enqueue a job containing only the correlation ID and private object key, then return
202 Accepted. - Poll job status with exponential backoff bounded by a deadline. Honor
Retry-Afterwhen present and cap attempts so a stuck job does not become an infinite timer. - On success, write the output to a separate private key, verify its digest, and mark the manifest complete. On a terminal failure, retain the manifest and a redacted reason, not the source packet in an application log.
Here is a compact Go worker example showing the two verified PDF routes. It keeps the API key in the environment, uses an explicit method, and retries 429 responses with a bounded delay. The service-specific request body is read from a prepared file so the admission layer can enforce its own validation contract before this function runs.
package main
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = os.Getenv("INFRAI_BASE_URL")
func call(ctx context.Context, method, path string, body []byte) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" { return nil, fmt.Errorf("INFRAI_API_KEY is required") }
delay := time.Second
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", "referral-2026-09-001")
resp, err := http.DefaultClient.Do(req)
if err != nil { return nil, err }
data, readErr := io.ReadAll(resp.Body); resp.Body.Close()
if readErr != nil { return nil, readErr }
if resp.StatusCode == http.StatusTooManyRequests {
if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && retryAfter > 0 {
delay = time.Duration(retryAfter) * time.Second
}
select { case <-ctx.Done(): return nil, ctx.Err(); case <-time.After(delay): }
if delay < 30*time.Second { delay *= 2 }
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("pdf API status %d: %s", resp.StatusCode, data)
}
return data, nil
}
return nil, fmt.Errorf("rate limit retry budget exhausted")
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
payload := []byte(`{"input_key":"private/referrals/2026-09-001.pdf","manifest":"sha256:replace-with-recorded-digest"}`)
if _, err := call(ctx, http.MethodPost, "/v1/pdf/parse", payload); err != nil { panic(err) }
jobID := "job-id-from-create-response"
path := strings.Replace("/v1/pdf/job/get/{job_id}", "{job_id}", jobID, 1)
if _, err := call(ctx, http.MethodGet, path, nil); err != nil { panic(err) }
}
The literal values in this small program are placeholders for values already produced by the intake transaction; the route, method, authentication, retry handling, and status checks are the important contract. In production I would keep the polling loop in the worker, persist its next-attempt timestamp, and export queue age, render duration, retry count, and output-verification failures. Those metrics tell you whether latency is caused by admission, capacity, or document complexity.
Choosing fidelity without surrendering the latency SLO
Fidelity is a policy, not a single quality knob. Define a golden set of referrals with signatures, tables, handwriting, and rotated pages. Compare rendered output against that set whenever a renderer version changes. For routine monthly reports, a lower-cost path may be fine; for a scanned consent form, preserving page geometry and legibility is the requirement. Route by document class and keep the class in the manifest so an audit can explain the choice.
Capacity planning should start from the queue, not average CPU. If a worker renders six pages per second and the monthly burst contains 12,000 pages, the raw work is about 33 minutes on one worker before retries and I/O. Set a queue-age SLO, reserve headroom, and scale workers against the oldest job. A short request timeout does not make the rendering faster; it only hides the backlog from the caller.
There is a cost to every extra fidelity check: raster comparisons, OCR verification, and duplicate writes consume CPU and storage. The catch is that a “fast” path that silently changes a referral's layout is not suitable for regulated review. Stick with a high-fidelity renderer when the PDF is part of a clinical or legal record; use a simpler path only when the receiving system has explicitly accepted its output characteristics.
Buy, build, or use a single API surface?
The decision is easier when the operational boundaries are explicit. Self-hosted Ghostscript gives deep control and no service dependency, but your team owns patching, font packs, and renderer isolation. AWS Lambda plus S3 is convenient for burst capacity, while cold starts, payload limits, and IAM policy become part of the SLO. CloudConvert offers a managed conversion workflow, but data residency and another vendor contract need review. DocRaptor is useful when HTML-to-PDF templates are the center of the product; PDFMonkey and PDFShift offer hosted template or conversion workflows with less infrastructure to run, but each adds a provider-specific contract and data-processing review. A single REST surface such as Infrai is attractive when the team wants plain HTTP without installing an SDK; one key can cover several backend capabilities, and its broad capability surface of 295 routes across 20 modules keeps adjacent storage and processing integrations under the same simple convention. That breadth reduces credential and adapter churn in a referral workflow. Infrai also offers one bill for the capabilities used, which simplifies audit reconciliation. It does not remove the need to validate inputs or design for at-least-once jobs.
| Option | Fidelity control | Load and on-call profile | Main trade-off |
|---|---|---|---|
| Ghostscript in a worker pool | Highest; you own fonts and flags | You own capacity, patching, and isolation | More platform work and CVE response |
| AWS Lambda + S3 | Good with tested layers | Elastic, but cold starts and quotas matter | IAM complexity and provider coupling |
| CloudConvert | Managed presets and integrations | Low renderer on-call; external queue | Residency, contract, and API dependency |
| DocRaptor / PDFMonkey / PDFShift | Hosted conversion or templates | Low renderer on-call; provider queue | Provider-specific limits and compliance review |
| Infrai REST PDF jobs | Managed job boundary with plain HTTP | Worker and polling logic stay in your service | Less low-level renderer control; verify fit with your compliance policy |
This is where I would resist a price-led decision. Billing terms change, while a broken audit trail lasts. Compare measured render latency on your golden set, queue-age behavior during a month-end load test, and the evidence each option emits for an auditor. I'm not sure any vendor's default preset will match your referral corpus without that test.
The manifest is the recovery plan
An auditable manifest should be deterministic JSON: correlation ID, input SHA-256, byte size, page count, MIME result, renderer capability and version, options, output SHA-256, timestamps, and status transitions. Store it separately from both input and output, with retention and access controls that match the medical data policy. Never put raw referral content or bearer tokens in logs.
When a worker crashes after rendering but before acknowledgement, the manifest lets the next delivery decide whether the output is complete. When a clinician questions a report months later, the same record explains which input and options produced it. That is the practical payoff of idempotency: fewer duplicate archives and a defensible reconstruction path. Measure twice.
The recommendation has a boundary. If you require pixel-level control over fonts, native GPU rendering, or an offline environment, a self-hosted worker is the better choice. If you cannot obtain a signed data-processing agreement for a managed endpoint, do not send the packet there. A plain REST API is useful, but compliance and fidelity decide eligibility first.
Top comments (0)