Short answer: make the Node.js intake path durable and boring: accept a bounded upload, validate it before enqueueing, move bytes into a private temporary area, and let workers render or merge bundles asynchronously. Under load, protect the queue and the renderer separately. For marketplace claims, preserve the original scan as the source of truth; spend render cost only when a reviewer or downstream exchange needs a normalized document.
That decision is about fidelity versus render cost. A blurry OCR preview is cheap, but it cannot replace the signed scan when a dispute reaches an auditor. A lossless page image costs more CPU, memory, and storage, yet it keeps the evidence intact. I have seen teams optimize the preview path first and then discover that their “successful” retry had replaced a page with a zero-byte temporary file. The HTTP request returned 202, so the bug hid in the queue.
What must stay true during scanned claims intake?
Write the invariants down before choosing a queue library:
- The original bytes are immutable after acceptance, and their digest is recorded.
- A job can be delivered more than once without creating a second claim bundle.
- Validation failures are terminal and explainable; transient storage or renderer failures are retryable.
- Temporary files are private, bounded in size, and removed after the job reaches a terminal state.
- Queue pressure cannot make upload handlers consume unbounded memory.
The request handler should do only work that protects those invariants. It can stream a multipart body to a file opened with exclusive creation, calculate a SHA-256 digest, inspect the declared media type, and insert an intake record. It should not render 300 pages while holding a client connection open. Return an idempotency key with the accepted job, not a promise that the document is already searchable. That distinction matters during a burst: if 40 uploads arrive together and each request starts a decoder, the event loop may remain responsive while the host runs out of memory, and the queue will report healthy even though every worker is paging. A bounded stream, a separate render concurrency limit, and a measured admission threshold make the pressure visible where it can be acted on; otherwise the first symptom is a timeout in a completely unrelated checkout request.
That boundary matters.
The digest is more than a checksum for logs. It gives a retry a stable identity and lets a worker notice that a path was accidentally reused. A claim record can use (claim_id, digest, operation) as a uniqueness constraint; the exact database syntax is less important than making duplicate delivery harmless.
How should Node.js handle asynchronous jobs, retries, validation, and latency under load?
Separate the queues by failure domain. An intake queue should remain responsive when a renderer is saturated. A render queue can have a smaller concurrency limit because image decoding is memory hungry. A notification queue should never block either one. The queue itself is not a substitute for a state machine: store accepted, validated, rendering, complete, and rejected transitions with an attempt count and timestamps.
Here is a compact worker outline. It is Python-shaped pseudocode so the storage and retry decisions are visible without tying the design to a particular Node.js package.
import hashlib
import os
import tempfile
MAX_BYTES = 25 * 1024 * 1024
RETRYABLE = (TimeoutError, ConnectionError)
def process_claim(job, db, renderer):
row = db.lock_claim(job.claim_id)
if row.status == "complete":
return {"status": "already-complete"}
if row.attempts >= 5:
db.reject(row.id, "retry-limit")
return {"status": "rejected"}
try:
with open(row.path, "rb") as source:
digest = hashlib.sha256()
size = 0
for chunk in iter(lambda: source.read(1024 * 1024), b""):
size += len(chunk)
if size > MAX_BYTES:
raise ValueError("claim exceeds size limit")
digest.update(chunk)
if digest.hexdigest() != row.sha256:
raise ValueError("digest mismatch")
db.mark_validated(row.id)
artifact = renderer.render_bundle(row.path, fidelity="lossless")
db.complete(row.id, artifact_uri=artifact)
return {"status": "complete"}
except ValueError as error:
db.reject(row.id, str(error))
return {"status": "rejected"}
except RETRYABLE as error:
db.record_retry(row.id, str(error))
raise
finally:
if db.is_terminal(row.id):
try:
os.unlink(row.path)
except FileNotFoundError:
pass
In production, the queue acknowledgement must happen after the state transition is committed. A worker crash before acknowledgement causes redelivery; that is expected. The uniqueness constraint and the locked state transition make it safe. A worker crash after acknowledgement but before complete is the dangerous case, so use an outbox or a lease that expires and returns the claim to rendering.
Retries need a budget, not optimism. Exponential backoff with jitter prevents a storage hiccup from producing a synchronized wave of attempts. Do not retry malformed PDFs, unsupported codecs, or a failed digest check; those are validation outcomes. Do retry a bounded number of network timeouts, then move the claim to a review queue with the last error and elapsed time.
Latency is a distribution, not one number. Track upload-to-accepted, accepted-to-validated, and validated-to-complete separately. A p95 of 400 ms can hide a p99 of 40 seconds when a few huge scans monopolize workers. Set a maximum body size and per-job page budget, then load-test with the same mix of one-page receipts and 300-page bundles that production sees.
Which temporary-file and validation boundaries prevent expensive failures?
Use a directory that is not served by the web process. Create a random file with exclusive permissions, write with a size limit, and store only an opaque identifier in the job payload. Never construct a path from a claim number supplied by a client. Keep the original extension out of the security decision; inspect magic bytes and parse the container before handing it to an image or PDF library.
Validation should be layered. First reject an over-limit body and a missing content length only when the protocol requires it. Next verify the media type from bytes, then check page count, decompression ratio, and any embedded object policy. Finally, scan or render in a sandbox with a CPU and memory limit. A file that passes MIME sniffing can still be a decompression bomb.
The catch is that strict validation is not suitable for every document source. If a partner sends an uncommon but contractually valid codec, rejecting it silently is worse than routing it to a quarantined review worker. Stick with a permissive intake only when the quarantine boundary and retention policy are explicit; otherwise, choose a narrower accepted format and tell the sender exactly why it was rejected.
Temporary storage also needs a cleanup contract. A scheduled sweeper can remove files whose lease expired, but it must consult the database first so it does not delete a job that is still rendering. Encrypt the volume when scans contain medical or financial data, and keep access logs for reads as well as writes.
What does a fidelity-versus-render-cost decision record look like?
The choice should be visible to reviewers instead of hidden in a worker flag:
| Option | Fidelity | Render cost | Failure boundary | Use it when |
|---|---|---|---|---|
| Original scan only | Exact | Low | Viewer must support source format | Evidence retention and later reprocessing matter |
| Lossless page render | Exact enough for review | High CPU and storage | Renderer capacity and timeouts | Claims reviewers need predictable pages |
| Lossy thumbnail plus original | Original preserved, preview reduced | Moderate | Thumbnail can be regenerated | Search lists need fast visual cues |
| OCR text plus original | Text may contain recognition errors | Variable | OCR quality and language model | Retrieval is valuable but evidence must remain untouched |
For a marketplace bundle merge, I would retain each source digest, create a deterministic page order, and produce a reviewer rendition only after validation. Splitting is the inverse operation: record the source range and resulting digests so a later retry cannot silently choose a different page boundary. “Fast” is not permission to discard the source.
There is no universal page threshold. Your mileage may vary because scanner resolution, codec, and renderer implementation dominate the cost. Measure a representative corpus and publish the p50, p95, and p99 with concurrency, rather than quoting a single benchmark.
What should be rejected, and when is it still useful?
A tempting design is synchronous rendering inside the upload request. It is easy to explain and works for a tiny internal tool, where a handful of one-page files and a generous timeout are realistic. It becomes the wrong boundary when a marketplace receives bursts: connection slots fill, retries duplicate work, and a slow renderer turns a storage delay into an API outage.
Another rejected option is deleting the original after OCR. That saves storage, but it destroys the ability to prove what was submitted and to improve recognition later. Keep the original under a retention policy; delete derived previews first when space is tight.
The practical rule is narrow: make acceptance fast, make processing idempotent, and spend rendering capacity where a human or an integration actually needs it. The architecture earns trust by naming what it refuses to retry and what it refuses to throw away.
Top comments (0)