How to Bound Compliance Evidence Latency Under Load (With Secure Job Admission)

#compliance #queues #node #sre

Short answer: A Node.js service should implement compliance evidence as asynchronous jobs, not long HTTP requests: validate and fingerprint each request before enqueueing it, render from immutable inputs in a bounded worker pool, use finite retries, and publish the PDF only after signature verification. For a marketplace monthly report, that keeps latency under load tied to admission work while secure temporary files and the expensive render stay behind explicit concurrency and cleanup limits.

The signature is the decision point. A PDF that exists but cannot be connected to the accepted request, source snapshot, renderer version, and archive write is not compliance evidence. Return 202 Accepted with a stable job ID for a new request, return the same job for the same idempotency key and fingerprint, and reject a reused key with different input as 409 Conflict. Don't promise a completion time the queue cannot defend.

I've been paged by missed jobs and duplicate deliveries. The useful lesson was plain: retries are normal queue behavior, so correctness cannot depend on a worker running exactly once.

The failure mode is an unbounded latency budget

A synchronous endpoint combines unrelated clocks: request parsing, marketplace data lookup, template rendering, PDF serialization, signing, and object storage. Under light traffic the sum can look harmless. During a monthly boundary, however, a few slow renders retain memory and event-loop attention while new requests arrive. Raising the HTTP timeout only moves the failure outward to a proxy or caller; it does not create capacity.

Split the service into an admission path and a completion path. Admission authenticates the caller, validates the reporting period and marketplace account, resolves an immutable source snapshot identifier, calculates a canonical request digest, records the job, and durably enqueues its ID. Completion loads that record, renders in an isolated worker, validates the artifact, signs its digest, and atomically changes the record from running to succeeded with the archive location. A status read returns the state and timestamps. The HTTP response never contains a temporary pathname.

Keep one clock for each promise. Measure admission latency at the API, queue wait from accepted_at to started_at, execution from started_at to finished_at, and end-to-end age from acceptance to publication. A single job_duration histogram hides whether load is waiting or work. Queue age is the signal that tells an operator to shed optional work, add workers within downstream limits, or stop accepting a batch.

Short queues matter.

How should a service validate asynchronous jobs before retries under load?

Validation has two phases because the trust boundary changes. Before enqueueing, reject malformed periods, unauthorized marketplace IDs, unsupported output options, and requests whose canonical form exceeds the documented limit. Normalize the accepted fields into a deterministic byte sequence and hash it. Store that input digest beside the idempotency key; never hash an object whose property order or implicit defaults can change between deployments.

Validate twice.

After rendering, validate the bytes rather than trusting a filename or a successful child-process exit. Check the configured maximum size while streaming, require the expected PDF media type at the boundary, parse enough structure to establish that the artifact is a readable PDF, and compare the rendered report metadata with the job record. Then compute the artifact digest and sign a versioned evidence statement containing the job ID, input digest, artifact digest, source snapshot ID, renderer version, and completion time. The exact PDF conformance profile and signature format depend on the regulator and verifier; I'm not sure a generic profile is sufficient for your jurisdiction, so settle that contract with the party that will verify the evidence.

Retries operate on states, not exceptions. A worker claims a job with a lease, increments an attempt counter, and renews the lease while it works. A retryable dependency timeout can return the job to the queue with bounded exponential backoff and jitter. Invalid input, an artifact that fails deterministic validation, or a signature policy violation is terminal. Put a ceiling on attempts and elapsed job age; after that, move the record to failed with a stable reason code and retain its audit events. Never ask an operator to infer the final state from logs alone.

The commit must be idempotent. Write the finished PDF to a content-addressed or job-scoped staging key, verify its digest, publish it to the final archive key with conditional semantics, and only then commit the success state. If a lease expires after publication but before the database commit, the next attempt should observe the same digest at the final key and finish the existing job. It must not create a second report or signature.

Publish once.

Implement the admission and audit contract

In Node.js, keep CPU-heavy rendering away from the main event loop. A fixed worker_threads pool is appropriate for JavaScript CPU work; a supervised child process can be a clearer isolation boundary for a native renderer. Both need a hard concurrency limit. Starting one worker or process per job makes queue pressure become memory pressure, which is a poor overload policy. Stream artifact bytes through hashing and size enforcement rather than accumulating a Blob or Buffer for every concurrent report. The Blob API represents immutable raw data, but immutability does not make an in-memory copy free.

Use a private temporary directory created with a cryptographically unpredictable suffix, restrictive permissions, and no caller-controlled path components. Open files with exclusive creation, pass file descriptors where the renderer permits it, and remove the whole job directory in a finally path after the archive commit or failed attempt. On startup, a janitor may remove directories older than the maximum lease age, but it should first prove that no live lease owns them. Avoid shared names such as report.pdf; retry overlap turns that convenience into cross-job data exposure.

The following Go probe is intentionally outside the service. It is runnable against a Node.js deployment and checks the public contract that matters during load: admission remains bounded, duplicate submissions converge on one job ID, and responses never leak local paths. Save it as probe.go, set TARGET, and run go run probe.go.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strings"
    "sync"
    "time"
)

type accepted struct {
    JobID string `json:"job_id"`
}

func main() {
    target := os.Getenv("TARGET")
    if target == "" {
        target = "http://127.0.0.1:3000"
    }
    payload := []byte(`{"marketplace_id":"market-42","period":"2026-08","format":"pdf"}`)

    client := &http.Client{Timeout: 2 * time.Second}
    const requests = 24
    ids := make(chan string, requests)
    errs := make(chan error, requests)
    var wg sync.WaitGroup

    for i := 0; i < requests; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            ctx, cancel := context.WithTimeout(context.Background(), 1500*time.Millisecond)
            defer cancel()
            req, err := http.NewRequestWithContext(ctx, http.MethodPost, target+"/evidence-jobs", bytes.NewReader(payload))
            if err != nil { errs <- err; return }
            req.Header.Set("Content-Type", "application/json")
            req.Header.Set("Idempotency-Key", "market-42-2026-08")

            resp, err := client.Do(req)
            if err != nil { errs <- err; return }
            defer resp.Body.Close()
            body, err := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
            if err != nil { errs <- err; return }
            if resp.StatusCode != http.StatusAccepted && resp.StatusCode != http.StatusOK {
                errs <- fmt.Errorf("unexpected status %d", resp.StatusCode)
                return
            }
            if strings.Contains(string(body), "/tmp/") || strings.Contains(string(body), `\\`) {
                errs <- fmt.Errorf("response exposed a local path")
                return
            }
            var a accepted
            if err := json.Unmarshal(body, &a); err != nil || a.JobID == "" {
                errs <- fmt.Errorf("invalid admission response")
                return
            }
            ids <- a.JobID
        }()
    }

    wg.Wait()
    close(ids)
    close(errs)
    for err := range errs {
        if err != nil { panic(err) }
    }
    var first string
    for id := range ids {
        if first == "" { first = id }
        if id != first { panic("idempotent requests produced different jobs") }
    }
    fmt.Printf("admission contract passed for %d requests; job=%s\n", requests, first)
}

The probe's 1.5s context is a test budget, not a universal service objective. Set it from observed admission work and caller needs, then test with realistic arrival bursts and report sizes. Your mileage may vary when signing uses an external hardware module or archive writes cross regions. The important constraint is that render duration cannot consume the admission budget.

Verify the signature trail, then practice rollback

A deployment check should submit a known marketplace snapshot, wait through the status endpoint, download the archived bytes, recompute the artifact digest, verify the signature against the recorded key ID, and compare every signed field with the job record. Also test a duplicate request during running, a duplicate after succeeded, a conflicting payload with the same idempotency key, worker termination after archive publication, lease expiry, an oversized artifact, and queue saturation. Those are state-machine tests, not just renderer snapshots.

Track p50, p95, and p99 separately for admission, queue wait, render, signing, and archive publication. Alert on oldest-ready-job age and terminal failure rate; raw queue depth has no time dimension and changes meaning when job sizes vary. Correlate logs and traces with the opaque job ID, attempt number, lease owner, input digest, and artifact digest. Do not log report contents, signature private material, temporary paths, or authorization headers. An audit event should be append-only and carry the actor, action, prior state, next state, reason code, and timestamp.

Rollback has two distinct cases. If an application release changes admission or canonicalization, stop that release from accepting work before routing traffic back; otherwise two versions may derive different digests for the same logical request. If a renderer release is suspect, pause new claims for that renderer version, let healthy attempts finish, and requeue only jobs that have not published an artifact. Keep the old verification keys and renderer metadata available for the full evidence retention period. Already published evidence remains immutable; a correction is a new, linked artifact, not an overwrite.

This design has a catch: a queue, lease protocol, signing service, archive, and audit store add operational weight. It is not suitable when the caller truly needs a tiny, non-sensitive document inline and can safely repeat the whole operation. Stick with synchronous generation for that narrow case, with an explicit timeout and idempotent response caching. For compliance evidence, the extra state is justified because it makes duplicate delivery, retry recovery, and signature verification observable.

No queue removes a downstream limit. Size the worker pool against renderer CPU and memory, signing throughput, archive bandwidth, and database connection budgets, then enforce admission control before those limits collapse together. Fail fast with a documented overload response when the system cannot accept another durable job. Quiet overload is how monthly reporting becomes an incident.

DEV Community

How to Bound Compliance Evidence Latency Under Load (With Secure Job Admission)

The failure mode is an unbounded latency budget

How should a service validate asynchronous jobs before retries under load?

Implement the admission and audit contract

Verify the signature trail, then practice rollback

References

Top comments (0)