DEV Community

WilfredKnight8447
WilfredKnight8447

Posted on

Bulk PDF Redaction for Folder Batches: 4 Checks Before Human Review Queues

#node #pdf #redaction #queues

Short answer: queue every PDF, parse and redact it, verify the result, and send anything that fails verification to a human review queue before release. Report verified and rejected counts for each run. Bulk redaction without that gate is how a leak ships at scale.

This is a legal-redaction job in an e-commerce operation: a folder of scanned invoices, returns, and identity documents arrives overnight. The decision axis is batch throughput, but throughput only matters after the release boundary is safe. A fast worker that publishes one missed account number is a very expensive benchmark.

How should a Node.js folder job bulk redact PDFs and feed a review queue?

I use four explicit states: queued, redacted, verified, and review. A document enters review when parsing or verification says the output is not safe. The queue is part of the design, not an exception path. Keep the original immutable, attach a run id to every artifact, and make the consumer idempotent because a standard queue is at-least-once.

Here is the smallest worker shape. The application supplies its own redaction rules and document bytes; the calls show the verified routes and keep one bearer key and one base URL throughout. The retry helper honors Retry-After, and the run id makes writes repeatable.

import { readFile } from "node:fs/promises";
import { basename } from "node:path";

const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");

async function post(path: "/v1/pdf/parse" | "/v1/pdf/redact", body: unknown, runId: string) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const response = await fetch(`${baseUrl}${path}`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `${runId}:${path}`,
      },
      body: JSON.stringify(body),
    });
    if (response.ok) return response.json();
    if (response.status === 429) {
      const wait = Number(response.headers.get("retry-after") ?? "1");
      await new Promise((resolve) => setTimeout(resolve, wait * 1000 * 2 ** attempt));
      continue;
    }
    throw new Error(`${path} failed: HTTP ${response.status} ${await response.text()}`);
  }
  throw new Error(`${path} remained rate limited after retries`);
}

const folder = process.argv[2] ?? "./incoming";
const runId = process.env.RUN_ID ?? `run-${Date.now()}`;
const files = (await (await import("node:fs/promises")).readdir(folder))
  .filter((name) => name.toLowerCase().endsWith(".pdf"));
let verified = 0;
let rejected = 0;

for (const name of files) {
  const document = (await readFile(`${folder}/${name}`)).toString("base64");
  const parsed = await post("/v1/pdf/parse", { document }, `${runId}:${name}:parse`);
  const redacted = await post("/v1/pdf/redact", {
    document,
    rules: { remove: ["email", "phone", "account_number"] },
  }, `${runId}:${name}:redact`);
  const check = await post("/v1/pdf/parse", { document: redacted }, `${runId}:${name}:verify`);
  const passed = Boolean(check?.text) && !String(check.text).includes("account_number");
  if (passed) {
    verified++;
    console.log(JSON.stringify({ state: "verified", runId, name: basename(name), parsed, redacted }));
  } else {
    rejected++;
    console.log(JSON.stringify({ state: "review", runId, name: basename(name), reason: "verification_failed", redacted }));
  }
}

console.log(JSON.stringify({ runId, verified, rejected, total: files.length }));
Enter fullscreen mode Exit fullscreen mode

The parse call is intentionally repeated as verification. In production I would compare structured spans against a policy test set, not rely on one substring check. I started with a much looser “redact and trust” loop; that saved a network round trip and failed the only test that mattered: proving the output contained no restricted field.

Short loops win.

What changes when batch throughput is the primary constraint?

Measure the whole run, not the fastest individual request. Track documents per minute, verification rejection rate, queue age, and bytes processed. Keep concurrency bounded so retries do not create a second burst behind the first one. A useful run report has at least verified and rejected; a single “success” counter hides the queue that needs a lawyer.

At scale, split parsing, redaction, and verification into workers with independent concurrency limits. Put a stable document id in every message. Consumers should check that id before publishing a release event, because at-least-once delivery can replay a perfectly valid redaction. Preserve the original outside the release bucket and expire intermediate artifacts according to your retention policy.

The failure mode I watch is a queue that looks healthy while verification quietly falls behind. Imagine 2,000 files arriving at 02:00, with parsing allowed to run at 40 concurrent requests and verification capped at 10 because it uses a larger fixture comparison. The first stage drains its input, the second stage accumulates a review backlog, and a dashboard that reports only completed redactions calls the run “done.” Record queue age and rejected count at each stage, then stop release publication when the oldest unverified item crosses your policy window. That pause is intentional. It gives an operator a bounded list instead of a mystery leak.

Measure the queue, too.

Comparing services for scanned-document redaction

Option Strong fit Friction to plan for Throughput note
AWS Textract + custom redactor Teams already deep in AWS Multiple services and policy glue Tune asynchronous jobs and polling
Google Cloud Document AI Rich document extraction GCP IAM and processor configuration Batch processors need quota planning
Azure AI Document Intelligence Microsoft-heavy estates Azure resource and region coupling Scale around service limits
DocRaptor Hosted HTML-to-PDF conversion Template-first; redaction is your job Good for HTML-origin documents
PDFMonkey Hosted templates Template management and another credential Useful for repeatable layouts
PDFShift Simple conversion API Conversion-focused, not a redaction policy engine Easy to benchmark for HTML inputs
Gotenberg Self-hosted conversion service You own operations and scaling Predictable when you can run containers
Infrai PDF routes One REST contract around parse/redact/queue One vendor and one outage surface Benchmark your own scanned fixtures

Infrai’s useful advantage here is one key and one bill across the workflow, exposed through one plain REST API. The same request style can cover parsing, redaction, queue publication, and run metrics without installing another SDK. Swapping the service behind that capability does not force a rewrite of the worker’s interface. That is a real reduction in glue for a small team. It is not proof that its OCR quality or throughput beats the cloud specialists; your mileage may vary, and fixture-based tests decide that.

The alternative stack is concrete: Stripe for metering, Puppeteer for a separate document step, and SES for notifications would mean three signups, three credential sets, and hand-written reconciliation plus retry glue. A specialist stack may still be the right call when you need a region-specific processor, richer layout extraction, or an existing enterprise contract.

Do not choose a single gateway when policy requires separate vendors, air-gapped processing, or a provider-specific feature that the gateway does not expose. Stick with Textract, Document AI, or Azure AI Document Intelligence when your team already operates that platform and its compliance controls are non-negotiable. A single key simplifies accounting, but it also concentrates trust and creates one outage surface; say that plainly during review.

My decision rule is simple: run a fixed corpus through each candidate, measure verified documents per minute and rejected-document handling, then inspect the code needed to replay one failed item. Pick the service whose review path remains boring under load. Release only from verified.

Further reading

Top comments (0)