Short answer: accept an invoice request quickly, freeze its inputs into an immutable job, generate the PDF outside the request path, and publish it only after validation, hashing, and signature verification have all succeeded. Retries must reuse the same job identity and evidence manifest; latency under load must be controlled with admission limits and measured as queue time plus execution time, not hidden inside one average.
For a marketplace, the PDF is only half the output. The other half is proof: which order revision was rendered, which renderer version handled it, which bytes were delivered, and whether those facts can still be verified after the temporary workspace is gone. A synchronous endpoint couples all of that work to a client timeout. It also encourages a dangerous ambiguity: the client may retry after losing the response while the server has already produced an invoice.
The operating rule is blunt. No verified manifest, no published invoice.
Start with an immutable job and an explicit state machine
The Node.js API should authenticate the caller, validate the request shape, resolve the exact order revision, assign a stable job ID, and enqueue a small immutable message. It should return 202 Accepted with that job ID rather than hold the connection open for PDF generation. The worker can be a separate process or service; the Go example below makes that boundary visible without tying the design to a particular queue or framework.
Do not enqueue a mutable database pointer and hope it still describes the same order later. Put the order revision, invoice number, currency, line-item totals, template version, and requested locale in a canonical job payload. If privacy policy prevents copying customer fields into the queue, store a content hash plus a versioned snapshot in access-controlled storage and make the worker reject any mismatch. That is a policy choice, not a shortcut.
Use a state machine such as accepted -> running -> validated -> published, with terminal rejected and exhausted states. A retry moves the same job from a retryable failure back to running; it doesn't create a fresh invoice identity. Keep the attempt number as operational metadata, while the job ID and invoice number remain fixed. This separation gives support staff an audit trail without making downstream consumers deduplicate several nominally different invoices.
The queue also needs a lease or visibility timeout longer than the expected render duration, plus renewal for long jobs. If a worker loses its lease, it must stop before publication. The exact timeout can't be chosen from an article: measure the upper tail of valid render time with representative invoices, then leave enough renewal margin for scheduling pauses and storage latency. I'm not sure what that number is for your templates, and a guessed value would be worse than an explicit load test.
| Decision | Managed queue | Self-hosted queue | SRE check |
|---|---|---|---|
| Delivery and leases | Less control-plane work | More tuning and on-call ownership | Can a worker prove exclusive publication rights? |
| Retry scheduling | Usually built in | Must be configured and operated | Are backoff and attempt limits visible? |
| Audit retention | May require a separate evidence store | Can share retention controls, with more upkeep | Can records outlive queue cleanup? |
| Capacity | Service limits still apply | Capacity is yours to forecast | Is queue-age SLO protected during bursts? |
The catch is operational ownership. A managed queue is not suitable when contractual controls require infrastructure custody it cannot provide; a self-hosted queue is a poor choice when the team cannot staff upgrades, recovery drills, and capacity planning. Neither option fixes an underspecified state machine.
How should a Node.js service handle asynchronous jobs retries and latency under load?
Treat queue age as a first-class latency component. End-to-end completion time is time waiting + time rendering + time validating + time publishing; reporting only worker execution makes an overloaded system look healthy right until its backlog breaches the business deadline. Define separate service-level indicators for acceptance latency, oldest runnable job age, successful completion time, and verified publication rate. Percentiles should be segmented by template version and a defensible complexity bucket, such as page count after rendering, because one aggregate can conceal a small class of pathological invoices.
Capacity planning starts with arrival rate and measured service time. If jobs arrive faster than the worker pool can complete them for a sustained interval, the queue grows; retries add more demand, so a dependency slowdown can amplify the backlog. Bound concurrency around the scarce resource, often CPU or renderer memory, and put a retry budget beside normal capacity. Don't let retry traffic consume every worker slot. A simple policy might reserve capacity for first attempts and admit retries only when oldest-job age is below its warning threshold, but the percentages must come from load tests and business priorities rather than folklore.
Backoff should be bounded, include jitter, and distinguish retryable outcomes from permanent rejection. Invalid order totals, an unsupported template version, or a failed evidence signature are not improved by waiting. A transient dependency timeout may be. Record a stable reason code such as INPUT_INVALID, DEPENDENCY_TIMEOUT, or SIGNATURE_INVALID, but keep sensitive payloads and raw customer data out of metric labels and routine logs.
One subtle failure deserves a longer look. Suppose the worker writes the finished PDF, loses its queue lease before recording publication, and receives the job again. If publication uses a random output name, the second attempt can create another document and another audit event; if it overwrites blindly, it can conceal a disagreement between attempts. Instead, derive the final object key from the stable job ID, make publication conditional, and compare the stored content hash with the new validated hash. Equal hashes mean the repeated attempt is idempotent. Different hashes mean stop and investigate the deterministic inputs, renderer version, and canonicalization rules. Do not pick whichever result arrived last.
Walk that sequence all the way through during a review: attempt 1 acquires the lease, renders invoice INV-2026-0042, validates order revision 7, computes digest A, and publishes under job-01JEXAMPLE; its acknowledgement is then lost. Attempt 2 receives the same immutable job and computes digest A again. The conditional publication reports that the stable key already exists, so the worker reads the existing evidence metadata, verifies its signature, compares the digest and immutable identity fields, and records a duplicate delivery without replacing any bytes. Now change only one test input so attempt 2 computes digest B. That is not a harmless retry and it must not become a last-writer-wins update: quarantine the second result, preserve both operational attempt records, alert on the determinism violation, and keep the already verified invoice available according to policy. This one exercise tests the lease, idempotency key, canonical manifest, signature verification, and audit vocabulary together; five isolated happy-path tests won't expose the gaps between those components.
Short queues matter.
Measure the wait.
When oldest runnable job age crosses the warning threshold, shed or defer low-priority batch work before the evidence deadline is threatened. When the critical threshold is crossed, reject new nonessential generation requests with a clear retry contract rather than accepting work the system cannot finish within its objective. This is where buy-versus-build claims become concrete: measure recovery from a burst, worker loss, and dependency slowdown, then count the on-call work required to restore the queue-age SLO.
Generate validate sign and publish in that order
The safe implementation has two commits. The first persists the immutable job before acknowledging acceptance. The second publishes the PDF and its signed evidence record after every validation passes. Everything between those commits is disposable work.
The following worker core is deliberately small. It writes generated bytes to a process-private temporary file, checks the PDF marker and expected order token, computes a SHA-256 digest, signs a canonical JSON manifest with Ed25519, verifies that signature, and atomically renames the file into a publication directory on the same filesystem. In production, the private key belongs behind an approved key-management boundary; passing it into a function keeps this example focused on ordering and verification.
package main
import (
"bytes"
"crypto/ed25519"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
type Job struct {
ID string `json:"job_id"`
OrderID string `json:"order_id"`
OrderRevision int `json:"order_revision"`
InvoiceNumber string `json:"invoice_number"`
Currency string `json:"currency"`
AmountMinor int64 `json:"amount_minor"`
TemplateVersion string `json:"template_version"`
}
type Manifest struct {
JobID string `json:"job_id"`
OrderID string `json:"order_id"`
OrderRevision int `json:"order_revision"`
InvoiceNumber string `json:"invoice_number"`
TemplateVersion string `json:"template_version"`
RendererVersion string `json:"renderer_version"`
PDFSHA256 string `json:"pdf_sha256"`
CompletedAt string `json:"completed_at"`
}
func validateJob(j Job) error {
if j.ID == "" || j.OrderID == "" || j.InvoiceNumber == "" {
return errors.New("missing stable identity")
}
if j.OrderRevision < 1 || j.AmountMinor < 0 || len(j.Currency) != 3 {
return errors.New("invalid order snapshot")
}
if !strings.HasPrefix(j.TemplateVersion, "invoice-") {
return errors.New("unsupported template version")
}
return nil
}
func render(j Job) []byte {
return []byte(fmt.Sprintf(
"%%PDF-1.7\nInvoice %s\nOrder %s revision %d\nAmount %d %s\n%%%%EOF\n",
j.InvoiceNumber, j.OrderID, j.OrderRevision, j.AmountMinor, j.Currency,
))
}
func process(j Job, privateKey ed25519.PrivateKey, workDir, publishDir string) error {
if err := validateJob(j); err != nil {
return fmt.Errorf("INPUT_INVALID: %w", err)
}
file, err := os.CreateTemp(workDir, "invoice-*.pdf")
if err != nil {
return err
}
tempName := file.Name()
defer os.Remove(tempName)
pdf := render(j)
if _, err := file.Write(pdf); err != nil {
file.Close()
return err
}
if err := file.Sync(); err != nil {
file.Close()
return err
}
if err := file.Close(); err != nil {
return err
}
if !bytes.HasPrefix(pdf, []byte("%PDF-")) || !bytes.Contains(pdf, []byte(j.OrderID)) {
return errors.New("OUTPUT_INVALID: invoice validation failed")
}
digest := sha256.Sum256(pdf)
manifest := Manifest{
JobID: j.ID, OrderID: j.OrderID, OrderRevision: j.OrderRevision,
InvoiceNumber: j.InvoiceNumber, TemplateVersion: j.TemplateVersion,
RendererVersion: "renderer-2026.1", PDFSHA256: hex.EncodeToString(digest[:]),
CompletedAt: time.Now().UTC().Format(time.RFC3339Nano),
}
canonical, err := json.Marshal(manifest)
if err != nil {
return err
}
signature := ed25519.Sign(privateKey, canonical)
if !ed25519.Verify(privateKey.Public().(ed25519.PublicKey), canonical, signature) {
return errors.New("SIGNATURE_INVALID: evidence verification failed")
}
if err := os.MkdirAll(publishDir, 0700); err != nil {
return err
}
finalName := filepath.Join(publishDir, j.ID+".pdf")
if err := os.Rename(tempName, finalName); err != nil {
return err
}
fmt.Printf("published=%s manifest=%s signature=%s\n",
finalName, string(canonical), hex.EncodeToString(signature))
return nil
}
func main() {
_, privateKey, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
panic(err)
}
job := Job{
ID: "job-01JEXAMPLE", OrderID: "order-1842", OrderRevision: 7,
InvoiceNumber: "INV-2026-0042", Currency: "USD", AmountMinor: 2599,
TemplateVersion: "invoice-v3",
}
workDir, err := os.MkdirTemp("", "invoice-work-")
if err != nil {
panic(err)
}
defer os.RemoveAll(workDir)
if err := process(job, privateKey, workDir, "published"); err != nil {
panic(err)
}
}
This example is runnable, but its render function is a test double rather than a full document renderer. Replace that function while preserving the contract: bytes enter validation before publication, and the exact published bytes produce the digest in the signed manifest. Browser-side or Node.js code may represent generated output as a Blob; the Blob API provides an immutable raw-data object and methods such as arrayBuffer() for reading its contents. Do not treat a JavaScript object reference as compliance evidence. Hash the actual bytes crossing the publication boundary.
Temporary files need their own threat model. Use an operating-system-created unpredictable name, a directory accessible only to the worker identity, restrictive permissions, a fixed storage quota, and cleanup on both success and failure. Never place customer names, order IDs, or invoice numbers in temporary filenames. Keep the workspace and final directory on the same filesystem if atomic rename is part of the publication design; object storage needs its own conditional-write protocol rather than filesystem assumptions.
The signature covers the manifest, not a vague database row. Canonical serialization matters because verification must reproduce the same bytes. Store the manifest, signature, public-key identifier, and PDF digest under the same retention and access policy as the invoice evidence. Key rotation should add a new key identifier without making older records unverifiable.
Verify load behavior rollback and evidence recovery
Unit tests should reject missing identities, negative amounts, unsupported template versions, malformed output, altered manifests, and signatures checked with the wrong public key. Integration tests should redeliver the same job, kill a worker before publication, expire a lease during rendering, fill the temporary volume, and confirm that cleanup removes abandoned files. A valid test outcome is not merely “the request returned 202”; it is one published PDF, one verifiable evidence chain, and a terminal job state that agrees with both.
Then load test the whole path. Drive a steady baseline, a burst above planned arrival rate, and a dependency slowdown while tracking acceptance latency, queue age, attempt count, render duration, validation failures, temporary disk consumption, and completion latency. Stop increasing concurrency when throughput flattens or queue age worsens; extra workers beyond that point are load, not capacity. Repeat with large and small invoices mixed in the proportion expected from production data. Your mileage may vary because templates, fonts, page counts, and signing boundaries change the service-time distribution.
Rollback has two different meanings. A code rollback returns workers to a known renderer and validator version, while a data rollback must never delete already published evidence merely because an application deployment was reverted. Pause new claims, drain or quarantine jobs from the suspect version, keep immutable manifests, and reprocess only under an explicit policy that preserves the original record. Test public-key recovery and manifest verification without the primary application database; otherwise the audit trail has the same failure domain as the system it is meant to explain.
Ship only when a burst test proves the queue-age objective and a recovery drill proves old signatures remain verifiable. Stick with a synchronous path only when generation is tightly bounded, client disconnects cannot create ambiguity, and the same validation and evidence guarantees still fit inside the request deadline. For marketplace invoice PDFs with retries and audit retention, that is usually a demanding constraint, so the asynchronous boundary earns its operational cost.
Top comments (0)