Medical referral intake is easiest to protect when the request path stays boring. Short answer: validate a small request, put an opaque referral ID on an asynchronous job, make every write idempotent, and give each temporary byte an explicit deletion deadline. That design lets a Node.js service render and archive a monthly batch without putting clinical content in queue messages or logs.
The concrete workload here is a developer-tools service that turns 10,000 monthly referrals into PDF reports. Throughput matters, but privacy and retention decide whether the batch is operable.
Start With the Bill and the Retention Boundary
The largest line item is usually repeated work and retained bytes, not the initial HTTP request. One referral can exist as an upload, a queue payload, a renderer scratch file, a PDF, and a backup copy. Multiply that by 10,000 and an apparently small feature becomes a storage and review obligation.
Draw the byte lifecycle before choosing worker counts. Keep identifiers, checksums, state, and batch membership in a database. Store document bytes in encrypted object storage under keys that contain no patient name. A queue message needs only an opaque referral ID and batch ID; it should never carry the form itself. The worker fetches the source, renders the report, writes the archive object, and records the checksum.
I use three clocks: a temporary-file TTL, a processing deadline, and a legal retention period. They are different controls. A renderer scratch file might live for 15 minutes, a failed job might be retried for two hours, and an approved archive might be retained for seven years only when the records owner requires it. Your mileage may vary; the legal and clinical owner sets that last clock.
The catch is deletion evidence. If you cannot show which job deleted a scratch file and its derivatives, treat the bytes as retained and account for access reviews and incident response.
How Should a Node.js Service Implement Medical Referral Intake?
Split intake into a fast admission transaction and a slower worker contract. The API checks content type, byte limits, required identifiers, and a client-supplied idempotency key. It stores the source and a digest, then returns a batch or referral status. Semantic checks happen in the worker: organization ownership, allowed report period, and whether that digest already has an archive.
Bad syntax is terminal. Retrying it only creates noise.
This Python reference shows the policy independent of the Node.js transport. A production service can expose the same contract while keeping validation easy to unit-test.
from dataclasses import dataclass
from hashlib import sha256
from pathlib import Path
MAX_BYTES = 10 * 1024 * 1024
@dataclass(frozen=True)
class Intake:
referral_id: str
organization_id: str
period: str
source_path: Path
expected_sha256: str
def validate_intake(item: Intake) -> None:
if not item.referral_id or not item.organization_id:
raise ValueError("missing stable identifiers")
if len(item.period) != 7 or item.period[4] != "-":
raise ValueError("period must be YYYY-MM")
size = item.source_path.stat().st_size
if size == 0 or size > MAX_BYTES:
raise ValueError("source size outside policy")
digest = sha256(item.source_path.read_bytes()).hexdigest()
if digest != item.expected_sha256:
raise ValueError("checksum mismatch")
def process(item: Intake, archive, renderer) -> str:
validate_intake(item)
key = f"{item.organization_id}/{item.period}/{item.referral_id}.pdf"
if archive.exists(key):
return "already-archived"
output = renderer.render(item.source_path)
archive.put_if_absent(key, output,
metadata={"sha256": sha256(output).hexdigest()})
return "archived"
The archive path is the idempotency boundary, backed by an atomic put_if_absent. A retry after a worker timeout checks for the result before rendering again. If the renderer itself has side effects, record a render attempt with a unique constraint and let the archive write remain the final gate.
An asynchronous job should carry a small, explicit failure taxonomy. A network timeout is retryable, a dependency's rate-limit response is delayed, and a malformed PDF is terminal. Store the class, attempt count, and next-run time; never copy clinical text into a dead-letter record.
Exponential backoff with jitter keeps a monthly batch from stampeding a dependency. Honor a bounded Retry-After, cap total attempts, and move exhausted work to a review queue. At-least-once delivery is normal, so duplicate messages must produce the same archive status.
import random
def next_delay(attempt: int, retry_after: int | None = None) -> int:
if retry_after is not None:
return min(retry_after, 900)
base = min(2 ** attempt, 900)
return max(1, int(base * random.uniform(0.7, 1.3)))
def classify(error: Exception) -> str:
if isinstance(error, (TimeoutError, ConnectionError)):
return "retryable"
if isinstance(error, ValueError):
return "terminal"
return "review"
Exactly-once claims usually hide a database constraint. Make that constraint visible, and return the same status when a client repeats an idempotency key. This is also where a retry policy belongs: classify the exception before scheduling it, honor a bounded Retry-After, and cap attempts. A malformed document should move directly to review; a transient connection failure can wait with jitter. Those decisions are operational data, so keep them alongside the job rather than burying them in worker logs.
Keep Temporary Bytes Private by Construction
Create scratch files in a private directory with generated names and restrictive permissions. Stream writes and enforce a total byte limit while receiving data. An extension such as .pdf is not proof of content, and the original filename should never become a path component.
The browser's Blob API supplies bytes; server-side policy still decides size, type checks, and retention. The MDN Blob reference documents that client-side primitive.
import os
import tempfile
from pathlib import Path
def write_private_temp(chunks: list[bytes]) -> Path:
fd, name = tempfile.mkstemp(prefix="referral-", suffix=".bin")
os.fchmod(fd, 0o600)
path = Path(name)
try:
with os.fdopen(fd, "wb") as handle:
for chunk in chunks:
handle.write(chunk)
return path
except Exception:
path.unlink(missing_ok=True)
raise
Always unlink in a finally block around rendering, including terminal failures. Container-local storage helps limit exposure, but snapshots, crash dumps, and debug bundles can outlive the process. Disable request-body logging, redact identifiers that could identify a person, and emit byte counts and latency without document content. A useful test is to kill a worker between render completion and archive confirmation, then inspect what the restart can discover. The expected result is either an existing archive, which the idempotency check accepts, or a bounded scratch file that the cleanup process removes; there should be no second clinical payload hidden in a retry message.
Operate a Monthly Batch With Proof of Completion
The scheduler creates one job per referral and records a batch ID. Workers use bounded concurrency chosen from renderer memory, not CPU alone. A canary batch establishes p95 render time and peak scratch usage before concurrency is raised.
Queue depth alone is a trap. I once saw a zero-depth queue while a renderer still held open files; completion had been declared too early. Now a batch closes only when terminal outcomes, archive checksums, and deletion events agree with the input count.
Keep an append-only audit event for accepted, started, retried, archived, deleted, and quarantined states. Record actor, timestamp, reason code, and checksum, but no diagnosis text. When policy allows source deletion after archival, delete the source and its derivatives together, retaining only the audit fields needed to prove it.
This pipeline is not suitable for interactive, sub-second previews or records that must remain editable in place. Use a separately governed preview store for those cases. It also does not replace a HIPAA risk assessment, access-control review, or records schedule.
References
- https://developer.mozilla.org/en-US/docs/Web/API/Blob
- https://www.rfc-editor.org/rfc/rfc9110
- https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final
Top comments (0)