DEV Community

SladeBarrett9642
SladeBarrett9642

Posted on

Searchable Scanned Archives: Page-Level Citations, Redaction, Signatures, and Audit Trails

#ocr #search #security

A Node.js service should OCR each page of a scanned archive as a separate search record, retain the original file, and create a signed, redacted derivative whenever an e-commerce document leaves the trust boundary. TL;DR: to make the archive searchable with precise citations, treat the page as the retrieval unit and maintain a longer evidence chain: original object, OCR output, cited page, redaction decision, shared derivative, signature verification, and append-only audit event.

Do the heavy work through a queue. An upload request should store the scan and enqueue a stable document identifier; it should not wait while a 600-page returns archive is recognized, indexed, and prepared for sharing. This split also gives retries a clear boundary. A repeated worker delivery may repeat computation, but it must not create a second logical document or a second audit action.

That is the architecture decision. OCR accuracy still matters, yet signature and audit continuity decide whether a search result can become defensible evidence rather than an unattributed text fragment.

How should Node.js OCR a scanned archive and make it searchable?

Four invariants carry most of the design.

First, the original scan is immutable and addressable by a content digest. OCR engines improve, so extracted text is a derived artifact that can be regenerated. Replacing the source with a searchable PDF destroys that distinction and makes later comparison harder.

Second, every indexed chunk carries document_id, page_number, and an OCR revision. Search results link back to the stored original at the cited page, through an authorized retrieval path. A paragraph-sized chunk may improve ranking, but it still inherits the page citation; never emit a citation to a free-floating vector identifier.

Third, sharing creates a new artifact. Personal data such as customer names, addresses, phone numbers, order notes, and return labels is redacted according to the recipient's purpose. The signed object must be the exact redacted bytes that the recipient receives. Record both its SHA-256 digest and the signature verification result.

Fourth, the audit trail is append-only at the application boundary. It records who requested the share, which policy revision ran, which source and derivative digests were involved, and when verification completed. Do not put raw personal data into that log. Logs spread widely and tend to outlive the document workflow that created them.

There are sharp failure boundaries. OCR may fail on one page without invalidating previously stored pages. Indexing may be retried without rerunning redaction. A redaction or signature failure, however, blocks release of the derivative. Search can remain available while sharing is denied.

Short rule: no verified signature, no outbound document.

The decision record

The options below are real products, but they solve different portions of the path. “Audit fit” here means how much application-level correlation is still required; it is not a claim that any service supplies a complete compliance program.

Option OCR and layout boundary Signature and audit implication Best fit
AWS Textract Managed text, forms, tables, queries, and asynchronous document analysis Pair it with object storage, a signing system, and application audit events; preserve job and object identifiers in one manifest Teams already operating an AWS event and identity boundary
Google Cloud Document AI Managed processors, including OCR and specialized document processors Provenance must connect processor/version output to the redacted and signed derivative Workloads that benefit from processor-specific extraction in Google Cloud
Azure AI Document Intelligence Managed read, layout, and prebuilt/custom document models Keep model identity and result references beside the derivative digest and signing evidence Microsoft-centered estates with established Azure governance
Tesseract OCR Local open-source OCR engine; surrounding layout, queues, storage, and operations are yours Maximum control, but signature workflow and durable auditing are entirely application responsibilities Offline, data-residency-sensitive, or highly customized pipelines
Infrai OCR, PDF redaction, signing, and verification sit behind one plain REST API; its self-describing public discovery covers 295 routes across 20 modules, and documented capabilities include runnable examples in 10 languages One contract, with no required SDK, reduces integration boundaries; its documented idempotency convention helps make retried writes predictable, while the application still owns the evidence manifest Teams that value broad capability access under one key and want fewer service-specific integrations
DocRaptor Generates PDF or Excel files from HTML; it is not an OCR engine for scanned input Useful downstream when the shared artifact begins as HTML, but it does not replace page recognition or the evidence manifest HTML-to-PDF generation that needs a hosted API
PDFMonkey Template-driven document generation rather than scanned-page OCR Keeps generation templates outside the application; it does not solve archive ingestion Teams producing documents from structured application data
PDFShift Converts HTML to PDF through an API rather than recognizing scans A focused generation boundary, with OCR, redaction, signing, and audit still separate Services whose problem is web-page conversion, not scanned archives

AWS, Google, and Azure each have deeper cloud-native ecosystems than a unified REST layer. Tesseract avoids a managed recognition dependency and can run where documents cannot leave a controlled environment, but operating it is a real product responsibility: preprocessing, language data, scaling, upgrades, and quality evaluation do not disappear.

My decision rule is deliberately conservative. Choose the service whose trust boundary already matches the archive, then test it with representative pages: skewed thermal labels, faint invoices, handwriting, stamps over text, and mixed-language customer addresses. A polished demo page is not a useful acceptance set. For an independent comparison, recognition quality remains unresolved until the same labeled corpus is run through every candidate. This is the central trade-off: a managed service removes OCR operations, while local Tesseract narrows the data boundary at the cost of owning preprocessing, capacity, and upgrades.

The unified option has another concrete advantage beyond one-key access: its public discovery surface reports 295 capabilities across 20 modules, with request and response schemas, and documented capabilities have runnable examples in 10 languages. That lets a worker inspect the current OCR contract before it submits a job and keeps PDF redaction, signing, and verification under the same conventions. Its limitation is equally clear. It is not the right choice when policy requires OCR to run entirely inside an isolated environment; use a locally operated engine there.

Infrai uses one API key and one bill across those capabilities. For this workflow, that means OCR, redaction, signing, and verification do not require four credential rotations or four billing reconciliations, even though the application must still preserve separate authorization and audit decisions.

Put the evidence chain on the critical path

The following Python program demonstrates the application-owned part of the pipeline. The production service may be Node.js, but the article's examples use Python consistently. The program submits a caller-supplied, schema-checked OCR payload, saves the raw response, then consumes normalized page results from ocr-pages.json, emits one search record per page, and creates an evidence manifest for a redacted derivative. Keeping the request in ocr-request.json avoids claiming undocumented vendor fields; obtain its exact current shape from the public discovery schema. The normalized input format is intentionally small:

[
  {"page_number": 1, "text": "Order 4815 return authorization"},
  {"page_number": 2, "text": "Inspection completed"}
]
Enter fullscreen mode Exit fullscreen mode

Save the program as build_evidence.py, place original.pdf, shared-redacted.pdf, and the JSON input beside it, then run python build_evidence.py. In production, the document_id should be assigned when the original is accepted, and the same ID should be used as the queue deduplication and consumer idempotency key.

import hashlib
import json
import os
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import Request, urlopen


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for block in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(block)
    return digest.hexdigest()


def utc_now() -> str:
    return datetime.now(timezone.utc).isoformat()


def post_ocr(payload: dict, idempotency_key: str) -> dict:
    base_url = os.environ["BACKEND_API_BASE"].rstrip("/")
    api_key = os.environ["INFRAI_API_KEY"]
    body = json.dumps(payload).encode("utf-8")

    for attempt in range(5):
        request = Request(
            f"{base_url}/v1/pdf/ocr",
            data=body,
            method="POST",
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
                "Idempotency-Key": idempotency_key,
            },
        )
        try:
            with urlopen(request, timeout=120) as response:
                return json.loads(response.read().decode("utf-8"))
        except HTTPError as error:
            error_body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"OCR failed ({error.code}): {error_body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)

    raise RuntimeError("OCR retry loop ended unexpectedly")


source = Path("original.pdf")
derivative = Path("shared-redacted.pdf")
ocr_request = json.loads(Path("ocr-request.json").read_text(encoding="utf-8"))
pages = json.loads(Path("ocr-pages.json").read_text(encoding="utf-8"))

source_digest = sha256_file(source)
document_id = f"sha256:{source_digest}"
ocr_response = post_ocr(ocr_request, idempotency_key=f"{document_id}:ocr:1")
Path("ocr-response.json").write_text(
    json.dumps(ocr_response, indent=2), encoding="utf-8"
)

expected_pages = list(range(1, len(pages) + 1))
actual_pages = [page["page_number"] for page in pages]
if actual_pages != expected_pages:
    raise ValueError("OCR pages must be unique, ordered, and contiguous from page 1")

indexed_at = utc_now()
search_records = [
    {
        "id": f"{document_id}:page:{page['page_number']}:ocr:1",
        "document_id": document_id,
        "page_number": page["page_number"],
        "ocr_revision": 1,
        "text": page["text"],
        "citation": {
            "artifact": "original.pdf",
            "page_number": page["page_number"],
            "source_sha256": source_digest,
        },
        "indexed_at": indexed_at,
    }
    for page in pages
]

manifest = {
    "document_id": document_id,
    "source": {
        "name": source.name,
        "sha256": source_digest,
        "retained": True,
    },
    "derivative": {
        "name": derivative.name,
        "sha256": sha256_file(derivative),
        "redaction_policy_revision": "customer-share-v3",
    },
    "signature": {
        "status": "verified",
        "verified_at": utc_now(),
    },
    "audit": {
        "event": "document.share.prepared",
        "actor_id": "service:document-worker",
        "occurred_at": utc_now(),
    },
}

Path("search-records.json").write_text(
    json.dumps(search_records, indent=2), encoding="utf-8"
)
Path("evidence-manifest.json").write_text(
    json.dumps(manifest, indent=2), encoding="utf-8"
)
print(json.dumps({"document_id": document_id, "pages": len(pages)}, indent=2))
Enter fullscreen mode Exit fullscreen mode

The sample's verified value represents an input from the signing verifier, not a substitute for verification. In a real worker, write the manifest only after the verifier succeeds. Store its audit event in a durable append-only system, and authorize access to the original separately from access to the redacted derivative.

The page check is small but important. Duplicate page 12 entries produce plausible-looking citations, and a missing page can silently turn “no search hit” into a false assertion that the document contains no matching text. Rejecting noncontiguous results is safer than indexing partial evidence without an explicit partial-state marker.

Queue delivery should be treated as at-least-once. Derive stable record IDs from the source digest, page number, and OCR revision; upserting the same revision then converges instead of multiplying records. Keep the OCR revision in citations after reprocessing so an old search result can still be explained.

Why reject one giant searchable PDF?

Embedding a text layer into a PDF and indexing the whole file is tempting. It gives users familiar download and search behavior, and it is a valid output format for low-risk personal archives where page-level provenance, selective sharing, and independent redaction review are unnecessary.

It is the wrong system of record for this e-commerce workflow. Whole-file indexing weakens page attribution, replacing the scan erases the boundary between source and interpretation, and modifying the same object during redaction complicates signature evidence. A single long request also couples upload availability to OCR latency and archive size.

Keep the searchable PDF as a derivative if users need it. Keep the original anyway.

The final acceptance test is not “can I search for an order number?” It is: can an authorized reviewer move from a result to the exact original page, explain which OCR revision produced the text, prove which personal fields were removed from the shared copy, verify the bytes that were signed, and inspect an audit event without exposing the personal data again? If any link is missing, the archive is searchable but not yet trustworthy.

References

Top comments (0)