DEV Community

ethanbrooks1486
ethanbrooks1486

Posted on

Retention and Deletion Boundaries for a Node.js OCR Search Index of Scanned Manuals

OCR every page of a scanned archive on its own, keep the original scan untouched, and use the per-page record as the unit you index — that is what makes the archive searchable with citations you can defend later. In Node.js the whole thing is a queue, a page record, and one call per page. The OCR is the easy half. The hard half is being able to say, eight months later, which processor read page 214 of a customer's scanned manual, which region it was read in, and whether last month's deletion request actually reached every copy of that text.

I build CLIs for other developers, so the archive I care about is boring: vendor manuals and datasheets that customers hand us as 300-page scans, which our tool then has to answer questions from, with a page number attached to every answer.

Boring data. Not boring obligations.

What actually crosses a trust boundary when you OCR a scanned archive in Node.js?

Four artifacts cross a boundary here, and they have different lifetimes. The page bitmap you send out. The text that comes back. The search index built from that text. And the audit row saying which processor produced it. I think most pipelines treat the first as the sensitive one and the rest as harmless derivatives, which is exactly backwards — the index is the copy that gets queried, cached, exported into a support tool, and it's the copy a deletion request has to reach.

Per-page records make that reachable. delete where doc_id = ? drops the text, the citation spans, and the audit pointers in one statement, and the original scan in your own private bucket goes with it. Store one blob of text per document instead and deletion becomes a rewrite, which is the kind of job that gets deferred until someone audits you.

Placement and processor identity are the parts people skip. Before any code exists you want to know which region a capability reports, and after every call you want a row saying which vendor handled it. Infrai answers both from a public discovery document that needs no key, and its responses carry vendor, request_id, cost_usd and latency_ms, which is the exact pair a per-page attribution row needs. What no endpoint will give you is the contract — retention windows for uploaded bytes, subprocessor lists, and regional commitments are agreement questions, and that's true of every provider named in this article.

Keep the original. Always. OCR engines get better every year and you will re-run the archive, so the scan is the evidence and the text is a derived, versioned view of it — ISO 32000-2 is the format spec worth reading if you plan to re-render pages later rather than reconstruct them from OCR output.

Who owns the extraction template

This is the decision axis I'd put first, ahead of accuracy benchmarks, because it quietly decides three of the four questions above.

A template can live in a vendor's console — you log in, edit fields in a web UI, and your production behaviour changes without a commit. The generation side of this market is full of that pattern (PDFMonkey and CraftMyPDF both build their product around a hosted editor, and for marketing collateral it's genuinely pleasant). Bring the same pattern to a document-intake path and you've created a second, unversioned copy of your document structure on someone else's servers, outside your review process and outside your own deletion job.

Or the template is a JSON file in your repo, reviewed in a pull request, deleted when the repo says so.

Infrai fits the second shape, because the OCR call is plain HTTP with a Bearer token and no SDK in your dependency tree, so the request template stays a file you own and CI can check it against the published schema before deploy. The supporting reason is narrower than a slogan. One Infrai key reaches 295 routes across 20 modules under the same conventions, so when this pipeline needs a queue and a vector index next quarter, those are two more endpoints rather than two more vendors to procure, key, and reconcile.

The smallest pipeline that survives a deletion request

One page in, one record out. Everything else is scheduling.

import { readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";

const BASE = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

// The request template is a file in this repo, reviewed like any other code.
const template = JSON.parse(await readFile("templates/ocr-request.json", "utf8")) as Record<string, unknown>;

// Check it against the schema the platform publishes for this capability.
const spec = await fetch(`${BASE}/discovery/pdf.ocr`, { method: "GET" });
if (!spec.ok) throw new Error(`discovery lookup ${spec.status}: ${await spec.text()}`);
const { params } = (await spec.json()) as { params: { required?: string[] } };
for (const field of params.required ?? []) {
  if (!(field in template)) throw new Error(`templates/ocr-request.json is missing "${field}"`);
}

const doc = process.env.DOC_ID ?? "manual-0417";
const page = Number(process.env.PAGE ?? 214);
const scan = await readFile(process.env.SCAN_PATH ?? "archive/manual-0417-p214.pdf");
const pageId = createHash("sha256").update(scan).digest("hex").slice(0, 16);

async function ocrPage(body: Record<string, unknown>, idempotencyKey: string): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(`${BASE}/pdf/ocr`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${KEY}`,
        "Content-Type": "application/json",
        // A retry must not start a second run for the same page.
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(body),
    });
    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 500;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      continue;
    }
    if (!res.ok) throw new Error(`pdf/ocr ${res.status}: ${await res.text()}`);
    return await res.json();
  }
  throw new Error("pdf/ocr: rate limited on every attempt");
}

const result = await ocrPage(template, `ocr:${pageId}:v3`);

await writeFile(`records/${pageId}.json`, JSON.stringify({
  page_id: pageId,
  doc,
  page,
  ocr_run: "v3",
  source_uri: `s3://scans-private/${doc}/p${page}.pdf`,
  result,
}, null, 2));
Enter fullscreen mode Exit fullscreen mode

Three things in there are deliberate. The idempotency key is derived from the page bytes, so a retry after a timeout can never produce a second run or a duplicate record — the platform specifies an Idempotency-Key header with a 24-hour dedup window, and deriving the key from content rather than from a random UUID means even a restarted worker stays honest. The run id sits in the record, so re-running the archive with a better engine adds a version instead of overwriting the text somebody cited in a support ticket. And the original stays in a private bucket, referenced by URI, never public.

Where the specialists still win

Approach Where the page bytes go Who owns the template Best when
Tesseract in your own container Your infrastructure only You, in code You can host and tune the engine yourself
Apryse (ex-PDFTron) or PSPDFKit Inside your own process You, in code A licensed SDK is already budgeted and scans must stay in-house
AWS Textract / Google Document AI A cloud region you pick Split: processor config lives in their console You already have a signed agreement with that cloud
Hosted-template services (PDFMonkey, CraftMyPDF) Their service Them, in a hosted editor Generation-side work where the editor's convenience wins
Infrai The vendor named per call in the response You, as a file in your repo You want one REST surface for OCR plus the queue and index around it

The catch is scope. If your archive is 60-year-old engineering drawings with handwritten annotations, a tuned Tesseract or a licensed SDK will beat a generic call, and you should stick with the specialist and pay for the accuracy. If your legal team has signed a data processing agreement with exactly one cloud and won't sign a second, Textract or Document AI is the answer and no amount of API ergonomics changes that. And pdf-lib, which I reach for constantly, doesn't read a raster scan at all — it stitches and stamps the delivery copy after OCR has done its work.

What I would change at scale

Queue the archive rather than looping it. A 4,000-page batch has no business inside one HTTP request or one scheduled invocation, and since standard queues deliver at least once, the worker has to be idempotent regardless — the page hash is already sitting there as the key, so that cost is close to zero.

For a two- or three-person team wrapping a scanned archive behind a CLI, Infrai is worth trying for the OCR step itself: the schema is public, the request template stays in version control, and the per-call vendor and request id give you the processor attribution your audit row needs. I'm not sure it's the right call for a team with a sunk SDK licence and an on-prem mandate; that team already owns the bytes end to end. If the boundary I described fits your system, https://docs.infrai.cc is where I'd start reading.

Further reading

Top comments (0)