Node.js Service Design for Medical Referral Intake with Async Validation and Retry Control

#node #healthcare #documentprocessing #reliability

Decision note

Short answer: Acknowledge the referral quickly, validate its envelope before expensive parsing, and hand document work to a bounded queue with leased jobs and quarantined temporary files. That shape keeps latency understandable when a medical intake burst arrives, while still allowing merge and split work to run in batches.

Design	Intake response	Batch behavior	Main risk
Parse and merge in the HTTP handler	Tied to document size	Competes with interactive traffic	Event-loop stalls
Unbounded background workers	Fast until the queue grows	Bursts consume every worker	Latency becomes opaque
Bounded queue with staged files	Short, stable acknowledgement	Measurable throughput	Requires queue and cleanup operations

I run a one-person SaaS, so revenue per hour matters. The request handler should spend milliseconds recording intent, not minutes moving PDF bytes. Ship weekly, measure the boring parts, and outsource undifferentiated storage plumbing only after its contract is clear.

The system below is also useful for game-support bundles, where screenshots, receipts, and chat exports are merged for a case and later split into reviewable documents. The domain changes; the pressure from bursty batches does not.

How should a Node.js service implement medical referral intake under load?

Separate acknowledgement latency from completion latency. The HTTP endpoint validates a small JSON envelope, writes an idempotency key and object references, and returns 202 Accepted with a status reference. A worker then claims the job using a lease. If the process disappears, the lease expires and another worker can claim the work.

The queue must be bounded by work, not merely by request count. A packet with two short pages and one with two hundred image-heavy pages are different units of load. Record page count, byte count, and fan-out as soon as those values are known. Set a target queue age from the service-level objective, then cap claims so CPU and storage remain available for normal API traffic.

Keep retries narrow. Storage timeouts and temporary network failures are plausible retry candidates; malformed JSON, an unreadable encrypted file, and a policy-limit violation are terminal outcomes. Exponential backoff with jitter prevents a shared dependency from receiving a synchronized second wave. Persist the error class and a redacted detail rather than a stack trace that might contain protected health information.

I initially treated completion time as the only useful metric. It wasn't. A 100 ms acknowledgement can hide a queue that is already hours old, so I now watch acknowledgement latency, queue age, processing latency, retry count, staged bytes, and cleanup lag together. Your mileage may vary; the right thresholds depend on page and image distributions that only production-shaped load tests reveal.

Validation is a boundary, not a parser feature

Validate in layers. Transport checks cover request size, signatures, and an allowlist of media types. Envelope checks cover referral identifiers, consent fields, schema version, and the declared number of documents. Content checks happen after quarantine: inspect magic bytes, enforce page and byte limits, run the approved malware policy, and confirm that the parser can actually read the file.

Do not trust a .pdf suffix or a client-supplied Content-Type. Store a normalized state such as accepted, rejected, or needs_review, alongside a machine-readable reason. That record makes a retry safe and gives support a useful audit trail when a sender keeps submitting the same packet.

The handler can remain deliberately dull:

type IntakeJob = {
  id: string;
  idempotencyKey: string;
  sourceRefs: string[];
  attempt: number;
  availableAt: number;
};

function validateEnvelope(input: unknown):
  | { ok: true; value: IntakeJob }
  | { ok: false; reason: string } {
  if (!input || typeof input !== "object") return { ok: false, reason: "invalid_json" };
  const value = input as Record<string, unknown>;
  const refs = value.sourceRefs;
  if (!Array.isArray(refs) || refs.length === 0 || refs.length > 200) {
    return { ok: false, reason: "source_count_out_of_range" };
  }
  if (refs.some((ref) => typeof ref !== "string" || ref.length > 256)) {
    return { ok: false, reason: "invalid_source_reference" };
  }
  return {
    ok: true,
    value: {
      id: crypto.randomUUID(),
      idempotencyKey: String(value.idempotencyKey ?? ""),
      sourceRefs: refs,
      attempt: 0,
      availableAt: Date.now(),
    },
  };
}

The 200 reference limit is an example policy boundary, not a promise to callers. Put the active limit in configuration and copy it into the job record; otherwise a later policy change can make an old decision impossible to explain.

What should secure temporary files do during merge and split jobs?

Create a directory per job with restrictive permissions. Generate filenames on the server, stream bytes into those files, and enforce a maximum while streaming. Never accept a caller-provided path and pass it to a filesystem API. Resolve references through an allowlisted storage adapter, and use an encrypted volume when policy requires it. Keep the directory name unrelated to a patient identifier, and keep access logs separate from the file contents. When a merge reads five source objects, stage each one under the same job boundary so cleanup can reason about a single lease. A split operation should publish each finished object under a temporary key first, verify its byte count and digest, then move it into the visible namespace. If the worker is terminated after object three is published, the status record still needs to distinguish three complete outputs from two pending outputs. That distinction is what lets a replay avoid duplicating an outbound referral and what lets a janitor remove only artifacts whose lease and expiry both permit deletion.

Keep it boring.

Every artifact needs an expiry timestamp independent of job status. A janitor removes expired directories after checking that no active lease points at them. The finally block in a worker is useful for the normal path, but it cannot clean up a process killed by an out-of-memory event or SIGTERM. Test those paths, plus out-of-disk behavior and restart recovery.

The Blob interface is a standard byte container at an application edge. It does not supply retention, encryption, access control, or deletion policy. Treat it as a transport type, not a security boundary.

interface BundleEngine {
  merge(inputs: readonly string[], output: string): Promise<{ outputRef: string; pageCount: number }>;
  split(input: string, outputDir: string): Promise<Array<{ outputRef: string; pageCount: number }>>;
}

async function runBundle(job: IntakeJob, engine: BundleEngine): Promise<void> {
  const dir = await createJobDirectory(job.id);
  try {
    const localInputs = await stageReferences(job.sourceRefs, dir);
    const merged = await engine.merge(localInputs, `${dir}/merged.pdf`);
    await publishResult(job.id, merged.outputRef, { pageCount: merged.pageCount });
  } finally {
    await expireJobDirectory(dir);
  }
}

Publish output atomically. A status reader should see either the previous complete object or the new complete object, never a partially written bundle.

Choosing the worker boundary and the runner-up

Keep the merge/split engine behind a small interface. A JavaScript PDF library can avoid a network hop for direct object manipulation; a separate process can isolate renderer memory from the Node.js event loop; a browser-based renderer can be appropriate when the source is HTML and print fidelity is the requirement. Each boundary changes startup time, font handling, observability, and failure recovery.

Benchmark with representative referrals: page count, scanned-image density, font variety, encryption, and merge fan-out all matter. A synthetic single-page PDF says little about a burst of mixed packets. Capture p50 and tail processing time, then size the worker pool from the tail and the queue-age objective rather than from CPU count alone.

The catch is operational weight. This design is not suitable when a caller must receive a fully rendered document in the original HTTP response, or when there is no durable queue and no protected temporary volume. Stick with a synchronous endpoint with strict size limits until the workflow can tolerate eventual completion. The simpler option is sometimes the more honest SLO.

For a solo founder, the first useful release is small: one durable job table, one bounded worker pool, one janitor, and a load test that replays realistic referral and game-bundle batches. Add a dead-letter view and an explicit replay action only when the idempotency behavior is documented. That is enough surface area to ship weekly without turning document intake into a second product.

DEV Community