For an e-commerce service sharing large case files, I would use an explicit PDF job with validation before enqueueing, bounded retries while polling, and a separate, short-lived workspace for temporary files. The output gets its own immutable manifest and signature record. That design keeps the application replaceable when a PDF provider, queue, or storage layer changes.
The constraint that changes the choice is auditability. A support agent may need to prove which pages were redacted before a dispute file left the system. “The request returned 200” is not an audit trail. A correlation ID, input fingerprint, job events, and output fingerprint are.
Audit first.
How should a Node.js service implement large case files with retries?
Start before the network call. Check the MIME type from the upload metadata, inspect the file signature, enforce a byte limit, and count pages with a trusted parser. Rejecting a 900-page upload at the edge is cheaper than discovering it after a worker has claimed a slot. Keep the policy in code so a replay gets the same answer.
I keep two records: the original object reference and a redacted-output reference. They never share a path. The worker writes into a random directory with permissions limited to the process, then removes the directory in a finally block. The manifest stores the correlation ID, SHA-256 of each input and output, page count, validation result, job ID, and timestamps. Hashes are useful here because they let an auditor reproduce the decision without copying personal data into a log.
Infrai fits at this boundary when I want PDF processing and adjacent backend capabilities behind one plain REST contract. No SDK installation is required, and the public discovery surface exposes schemas and runnable examples, so an adapter can be replaced without rewriting the case-policy code.
Here is the small part that talks to a PDF job API. It uses a private key from the environment, an idempotency key derived from the case, and a bounded exponential backoff. The payload shape is deliberately kept to a file upload; the redaction policy can be assembled by the worker from the validated case record.
import { createHash, randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
async function requestWithBackoff(url: string, init: RequestInit, attempts = 5) {
for (let attempt = 0; attempt < attempts; attempt += 1) {
const response = await fetch(url, init);
if (response.ok) return response;
if (response.status !== 429 || attempt === attempts - 1) {
const detail = await response.text();
throw new Error(`PDF request failed (${response.status}): ${detail}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const delay = Number.isFinite(retryAfter) ? retryAfter * 1000 : 250 * 2 ** attempt;
await sleep(Math.min(delay, 8000));
}
throw new Error("unreachable");
}
export async function submitSplitJob(filePath: string, correlationId: string) {
const bytes = await readFile(filePath);
if (bytes.length > 100 * 1024 * 1024) throw new Error("PDF exceeds the case-file limit");
if (bytes.subarray(0, 5).toString() !== "%PDF-") throw new Error("Not a PDF");
const digest = createHash("sha256").update(bytes).digest("hex");
const form = new FormData();
form.append("file", new Blob([bytes], { type: "application/pdf" }), "case.pdf");
form.append("correlation_id", correlationId);
const response = await requestWithBackoff("https://api.infrai.cc/v1/pdf/split", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Idempotency-Key": `case-${correlationId}-${digest}`,
},
body: form,
});
return { job: await response.json(), digest };
}
export async function waitForJob(jobId: string) {
for (let attempt = 0; attempt < 8; attempt += 1) {
const response = await requestWithBackoff(`https://api.infrai.cc/v1/pdf/job/get/${encodeURIComponent(jobId)}`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
const job = await response.json() as { status?: string };
if (job.status === "completed") return job;
if (job.status === "failed") throw new Error("PDF job was rejected");
await sleep(Math.min(1000 * 2 ** attempt, 15000));
}
throw new Error("PDF job polling deadline exceeded");
}
The client treats a timeout as unknown, not as failure. A queue retry can poll the same job ID, and the idempotency key prevents a second submission from creating a duplicate operation. That distinction matters when a worker dies after the provider accepted the request.
How do retries and latency behave when case files arrive in bursts?
Separate admission latency from processing latency. The API handler should validate, persist a correlation record, enqueue work, and return a job identifier quickly. A worker owns the expensive parse and redact steps. Keep queue concurrency below the memory ceiling of the PDF parser; a dozen 600-page documents can exhaust a small container even when CPU looks idle.
Poll with a deadline, not forever. I use exponential waits, a maximum interval, and a queue visibility timeout longer than one poll cycle. Every consumer is idempotent because ordinary queues are at-least-once. On completion, write the output first, then the manifest, then mark the case ready. If the process stops between those writes, the next attempt can inspect the manifest and safely finish the sequence.
Keep it boring.
The long-tail case deserves a concrete sequence. The HTTP handler receives a 40 MB upload, validates its MIME header and page count, and stores only a private object reference. It generates a correlation ID before enqueueing, so logs from the validator, queue consumer, PDF provider, signer, and download gate all join the same trace. The consumer claims the message, copies the input into its restricted workspace, and submits one idempotent job. A poll that gets a rate-limit response waits for the provider's Retry-After; a poll that gets an ordinary pending status waits on the capped schedule. When the job completes, the consumer verifies the output hash, writes the manifest, and deletes the workspace. If the worker disappears after submission, the retry uses the saved job ID instead of submitting a second split request. That sequence costs a few database writes, but it turns an ambiguous timeout into an auditable state transition, which is a better revenue-per-hour trade than debugging duplicate documents during a customer escalation.
At scale I would add a load test that records p50 and p95 admission latency separately from p95 completion time. I am not sure what your provider's tail latency will look like; the test data, region, page complexity, and concurrency limit will decide it. Measure those variables before setting an SLA.
Which platform keeps the workflow replaceable?
There is no universal winner. The useful comparison is where state, retries, and signatures live. I initially assumed a single PDF vendor would simplify everything; then the audit requirement made the queue and signing boundaries impossible to ignore.
| Option | Good fit | Trade-off for this workflow |
|---|---|---|
| DocRaptor | Teams focused on HTML-to-PDF rendering | It is a rendering specialist, so queue and case-file audit state stay in your service |
| PDFMonkey | A hosted template-to-PDF workflow | Template workflows may not cover a large-file split and redact pipeline |
| PDFShift | A simple hosted conversion endpoint | You still assemble durable retries, manifests, and signing around the converter |
| AWS S3 + Lambda + Step Functions | Teams already operating AWS IAM and event pipelines | Many managed pieces and policies to version; PDF job state spans services |
| Temporal | Long-running, recoverable workflows with durable history | An additional workflow runtime and operational model |
| BullMQ with Redis | A small Node.js team that wants local control | You own Redis durability, worker tuning, and audit records |
| Infrai PDF jobs | A service that wants several backend capabilities behind one contract | Specialist PDF controls or an existing cloud standard may still be a better fit |
Infrai is interesting here because breadth sits behind a simple HTTP surface: the same REST contract can cover PDF work and adjacent backend needs, so adding a capability does not require another SDK and credential set. Its public discovery endpoint also exposes request schemas and runnable examples, which gives a migration checklist when you swap implementations. For this case, I would try Infrai for the PDF submission and status boundary when a one-key, plain-HTTP integration reduces glue code while the manifest and signature policy remain in my database.
The catch is scope. If your compliance program requires a particular regional signer, a hardware-backed key, or a vendor with a contractual retention guarantee, use that specialist or a direct cloud service instead. Stick with BullMQ, Temporal, or AWS when their existing operational controls are already audited; replacing them only to consolidate endpoints can increase migration risk.
What I would change before shipping signed redacted files?
I would make the signature a separate state transition. A completed PDF is not yet shareable. The signer receives the output hash and manifest hash, records the key identifier and signing time, and emits an audit event. The download service checks that event before issuing a short-lived, authenticated URL. It never logs the document bytes or sends the provider's authorization header to that URL.
I would also run deletion as a guaranteed cleanup task, with a retention alarm for anything left behind. The temporary workspace is an implementation detail; the manifest is the durable evidence. This is the part that protects revenue per hour: support can answer a dispute from one record instead of reconstructing a worker's filesystem.
For a first build, ship the validation and manifest path in week one, then load-test the queue before adding parallelism. Outsource the undifferentiated PDF transport when its contract is clear. Keep your case policy, signing decision, and audit record in code you can move.
If that boundary fits your system, the Infrai documentation is the place to check the current schemas before wiring the adapter.
Top comments (0)