How to Implement Legal Contract Review Jobs in a Node Service Under Load

#contract #performance #security

Short answer: accept a property contract only after cheap validation, persist one idempotent review job, process it from a bounded queue, and publish the signed artifact only after its audit record commits; scale from measured service time and queue age, not raw request latency.

A Node.js API can own the admission endpoint while isolated workers do document parsing, legal-rule evaluation, signature placement, and audit persistence. That split matters for a property manager uploading hundreds of lease renewals before a deadline: the API should return a job identifier quickly, yet a fast 202 Accepted must never be mistaken for a completed legal review. The operational SLO belongs on terminal job latency and correctness. HTTP explicitly describes 202 as noncommittal, so expose a status resource and make clients poll it or receive a separately authenticated notification.

The capacity rule is blunt: if arrival rate can exceed completion rate for longer than the queue-age objective, reject or defer new batches. Don't hide overload behind an unbounded queue.

How should contract review jobs handle retries and validation under load?

Treat each upload as an immutable input and each processing attempt as disposable. The durable job record should contain a generated job ID, tenant ID, input digest, ruleset version, state, attempt count, timestamps, and a pointer to encrypted object storage. It should not contain the whole contract. A separate append-only audit event records each accepted transition, including the actor and the same digest; this makes the reviewed bytes, signed bytes, and recorded decision distinguishable during a dispute.

Validation has two gates. At admission, enforce authenticated tenant context, declared length, a conservative size ceiling, an allowlist of document types, and an idempotency key before storing anything. In the worker, distrust the declaration: identify the format from content, reject encrypted or malformed documents that the parser cannot safely inspect, limit pages and decompressed size, and verify that required property and signer fields exist. OWASP's file-upload guidance recommends defense in depth because a Content-Type header is supplied by the client and cannot establish the file's type by itself.

Retries belong around transient operations, not around the whole workflow. A retryable storage timeout can use capped exponential backoff with jitter. A validation rejection is terminal. A signature write that may have succeeded before its response was lost must be reconciled by idempotency key and output digest before another write is attempted. Otherwise one ambiguous timeout can produce two signed artifacts, which is far worse than a slow job.

Use explicit states such as accepted, validating, reviewing, signing, committing, completed, and rejected. Permit transitions through one compare-and-swap operation in the durable store. The worker may crash anywhere — the lease must expire and allow redelivery — but a stale worker must not commit after a newer attempt has taken ownership. A monotonically increasing lease token, checked during every state update, closes that race.

Size the queue before choosing concurrency

Start with a service demand distribution, not an average. Suppose a planning test uses 12,000 representative lease packets and observes a 2.4-second p95 worker time at the intended CPU and memory limit. Those figures are an example capacity worksheet, not a production benchmark. At 20 accepted contracts per second, the rough busy-worker requirement is 20 x 2.4 = 48; adding 30% headroom gives 63 workers after rounding up. Then test 63 against parser memory, storage request limits, database connection capacity, and signing concurrency. The smallest downstream limit is the real cap.

Queue age is the early warning signal. End-to-end latency rises after utilization has already become uncomfortable, while oldest-ready-job age shows that work is accumulating. Define separate objectives: for example, 99% of admitted single-contract jobs reach a terminal state inside the agreed window, no completed job lacks its audit event, and batch admission stays below a tenant-specific outstanding-work quota. The exact window and quota require observed document sizes and a legal operations deadline; I'm not sure what values are defensible until those two inputs exist.

Batch throughput also changes the API shape. Do not expand a 5,000-contract manifest inside the request handler and enqueue 5,000 messages in one transaction. Persist the manifest, return its job ID, and let a rate-limited dispatcher create child jobs in pages. That keeps one large property portfolio from consuming every worker and lets weighted fair scheduling preserve capacity for interactive renewals. It also makes cancellation honest: stop undispatched children, allow leased children to finish, and report both counts rather than claiming the entire batch vanished.

Decision	Operate it yourself when	Use a managed queue or document service when	Main SLO risk
Durable queue	Queue semantics are strategic and the team can carry upgrades, backups, and paging	On-call capacity is scarcer than infrastructure flexibility	Redelivery gaps or runaway queue age
Temporary artifact storage	Data residency requires infrastructure under direct control	Lifecycle deletion, encryption, and access logging are available under acceptable terms	Retention drift or inaccessible evidence
Document parser	Formats are narrow and parser behavior must be tightly controlled	Format churn and hostile-file isolation would otherwise dominate maintenance	Memory exhaustion or inconsistent extraction
Signing boundary	Keys must remain in an existing controlled trust boundary	External key custody and audit controls satisfy counsel and security review	Duplicate or unaudited signatures

The catch is lock-in at the job-state and audit boundary. Keep those records in an application-owned schema even when execution is managed. Conversely, self-hosting is not suitable when the platform team cannot staff queue recovery, key rotation, parser patching, and restore tests; in that case, accept a narrower integration surface and test an export path. Price is secondary to missed-deadline exposure and on-call load.

Implement bounded processing and secure temporary files

In a Node.js deployment, the HTTP process should stream uploads to controlled storage rather than buffer a Blob or full file in heap, and CPU-heavy parsing should run outside the event loop. The worker below is Go because the same queue contract should be language-independent: the Node.js admission service can publish the durable job, while this isolated process demonstrates bounded concurrency, validation, retry classification, and temporary-file cleanup without binding the design to an SDK.

The example expects a queue adapter with visibility leases and a processor that performs content inspection. It creates one private directory per attempt, writes with exclusive creation, syncs the file, and removes the directory on every return path. Set the host's temporary root to an encrypted volume, prohibit executable mounts where the platform supports that policy, and never log document text or filenames supplied by a tenant.

package review

import (
    "context"
    "crypto/sha256"
    "errors"
    "fmt"
    "io"
    "math/rand"
    "os"
    "path/filepath"
    "time"
)

var ErrInvalidContract = errors.New("invalid contract")

type Job struct {
    ID       string
    TenantID string
    Attempt  int
    Input    io.ReadCloser
}

type Queue interface {
    Receive(context.Context) (Job, error)
    Complete(context.Context, string, [32]byte) error
    Reject(context.Context, string, string) error
    RetryAt(context.Context, string, time.Time) error
}

type Processor interface {
    ValidateAndSign(context.Context, string, string, string) error
}

type Worker struct {
    Queue       Queue
    Processor   Processor
    Concurrency int
    MaxAttempts int
}

func (w Worker) Run(ctx context.Context) error {
    if w.Concurrency < 1 || w.MaxAttempts < 1 {
        return errors.New("invalid worker limits")
    }
    sem := make(chan struct{}, w.Concurrency)
    for {
        job, err := w.Queue.Receive(ctx)
        if err != nil {
            return err
        }
        sem <- struct{}{}
        go func() {
            defer func() { <-sem }()
            w.handle(ctx, job)
        }()
    }
}

func (w Worker) handle(ctx context.Context, job Job) {
    defer job.Input.Close()
    dir, err := os.MkdirTemp("", "contract-review-")
    if err != nil {
        w.scheduleRetry(ctx, job)
        return
    }
    defer os.RemoveAll(dir)

    path := filepath.Join(dir, "input.bin")
    f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
    if err != nil {
        w.scheduleRetry(ctx, job)
        return
    }
    hash := sha256.New()
    _, copyErr := io.Copy(io.MultiWriter(f, hash), io.LimitReader(job.Input, 25<<20))
    syncErr := f.Sync()
    closeErr := f.Close()
    if err := errors.Join(copyErr, syncErr, closeErr); err != nil {
        w.scheduleRetry(ctx, job)
        return
    }

    digest := fmt.Sprintf("%x", hash.Sum(nil))
    err = w.Processor.ValidateAndSign(ctx, job.TenantID, path, digest)
    if errors.Is(err, ErrInvalidContract) {
        _ = w.Queue.Reject(ctx, job.ID, "validation_failed")
        return
    }
    if err != nil {
        w.scheduleRetry(ctx, job)
        return
    }
    var sum [32]byte
    copy(sum[:], hash.Sum(nil))
    _ = w.Queue.Complete(ctx, job.ID, sum)
}

func (w Worker) scheduleRetry(ctx context.Context, job Job) {
    if job.Attempt >= w.MaxAttempts {
        _ = w.Queue.Reject(ctx, job.ID, "retry_limit_reached")
        return
    }
    shift := min(job.Attempt, 6)
    base := time.Second * time.Duration(1<<shift)
    jitter := time.Duration(rand.Int63n(int64(base / 2)))
    _ = w.Queue.RetryAt(ctx, job.ID, time.Now().Add(base+jitter))
}

Production code needs two refinements around that compact example. First, enforce the 25 MiB ceiling by attempting to read one additional byte; LimitReader alone truncates and therefore cannot prove the input was within the limit. Second, completion, output publication, and the audit event need an atomic database transaction or an outbox relay. The sample leaves those in interfaces because their correct implementation depends on the selected durable store, but the invariant does not: readers must never observe completed before they can retrieve both the signed artifact reference and its audit evidence.

Keep it bounded.

Verify latency safety and audit completeness

A unit test for retry delay is useful, but it cannot answer the capacity question. Build a load corpus that represents actual page counts, file sizes, scanned images, signature-field counts, and malformed inputs without containing production legal data. Drive arrivals as bursts and sustained plateaus. Measure accepted rate, completed rate, oldest queue age, terminal latency by batch size, attempt count, validation rejection rate, parser CPU, peak worker memory, temporary-volume utilization, database lock time, and audit-outbox lag.

Failure injection should stop one dependency at a time and assert invariants rather than merely checking that workers restart. Kill a worker after artifact creation but before completion; redelivery must reconcile the digest and produce one visible result. Expire its lease while it is still processing; the stale lease token must prevent its commit. Fill the temporary volume; admission should shed load before workers thrash. Delay audit publication; completed results must remain hidden until evidence catches up. Rotate signing credentials during a batch and verify that every audit event records the credential version without exposing key material. The pass condition is tied to the service-level objective: at the maximum admitted batch rate, queue age returns to baseline after the defined burst, terminal latency stays inside its objective, and every completed record joins to exactly one current artifact plus a complete transition history. Report percentiles by workload class because a single blended p95 can conceal that 200-page leases starve ordinary renewals. Alert on symptoms an operator can act on: oldest queue age against its budget, no completions despite active leases, retry amplification, dead-letter growth, temporary-volume saturation, and audit lag. Raw queue depth is context, not a page, because ten image-heavy packets and ten text-only packets impose different work.

Test the ugly paths.

Deploy with a rollback that preserves evidence

Roll out workers by ruleset and worker-build version. Shadow evaluation may compare decisions, but it must not create signatures or customer-visible audit events. During a canary, route a small, explicitly selected share of new jobs to the new version, hold concurrency below the downstream caps, and compare terminal latency, rejection reasons, retry rate, and decision differences before increasing the share.

Rollback means stop leasing new work to the suspect version, let safe in-flight steps finish or allow their leases to expire, and route redelivery to the previous compatible worker. Never delete job or audit records to make a rollback look clean. If a schema migration cannot be read by both versions, use expand-and-contract deployment; the rollback window ends only after the old reader is intentionally retired.

For each release, retain the input digest, output digest, ruleset version, worker version, signer identity, timestamps, and state-transition actors for the period counsel approves. Retention is a policy decision, not an engineering guess. Temporary copies have a much shorter life: delete them when an attempt ends, independently sweep abandoned attempt directories, and test that cleanup path under forced termination.

This design is deliberately less convenient than an in-process promise chain. It is also measurable: admission is bounded, work is recoverable, retries are classified, temporary bytes have a lifecycle, and the audit trail shares the completion boundary. That is the minimum shape I would take into a batch-throughput review.

DEV Community