Searchable Edtech Reports in Node.js — OCR and Page-Level Indexing

#node #pdf #ocr

Short answer: archive the original monthly report, extract its existing text layer first, and send only pages without useful text through OCR. Index one record per page with stable archive and report identifiers. This is the least complex approach that preserves the PDF students and educators received while avoiding a full-document render on every ingestion.

Report input	Extraction path	Fidelity check	Render cost	Pick this when
Generated PDF with usable text	Parse the text layer	Compare expected fields and page count	Low	Your report renderer emits selectable text
Scan or image-only page	Render that page, then OCR	Inspect rotation, confidence, and key fields	Higher	The page has no useful text layer
Mixed document	Decide page by page	Keep extraction provenance per page	Proportional to scanned pages	Covers or attachments may be scans
Complex chart or table	Extract nearby text and metadata; retain the original	Test representative queries against expected passages	Variable	Search helps discovery, but the PDF remains the visual authority

The key distinction is simple: search text is a derivative. The archived PDF is the record. Do not rebuild the archive copy from OCR output.

How should an API make a PDF archive searchable?

A monthly learning report may contain a generated cover, selectable attendance summaries, charts, and a scanned teacher note. Treating all four as the same input wastes work and can replace good embedded text with weaker recognition output. Treating the whole file as text-native misses the note. The useful unit of routing is the page.

Start with structural signals: did extraction return characters, are they mostly printable, and do expected report tokens appear? A nonempty string alone is weak evidence. A page can contain a header, footer, or hidden text while its main content is still an image. Set the acceptance rule from your own report templates and languages, then keep it observable.

For example, record sourceKind as text or ocr, plus character count and extraction duration. Those few fields make the pipeline explainable. They also expose a template change: if the normal text-path ratio suddenly falls, alert on the ratio rather than waiting for a search complaint.

Do not use OCR confidence as truth. Use it as a diagnostic signal alongside checks for expected fields such as reporting month, course name, or learner identifier.

Pick the text-first path, then selective OCR

Choose direct extraction when the report generator already writes a usable text layer. It skips rasterization and keeps the pipeline short.

There is still a fidelity trap. PDF describes a portable document, but stored text order does not have to match the order a person reads a two-column page. Headers may land in the middle of a paragraph. Chart labels may arrive detached from values. Test reading order, Unicode normalization, and repeated furniture against actual templates.

A good acceptance fixture is small: one report with a long learner name, one with an empty section, one containing a chart, and one covering every supported writing system. Store expected phrases by page.

Fast path first.

Choose OCR for pages whose meaningful content is rasterized or whose extracted layer fails the acceptance rule. Rendering has a real cost: each selected page becomes an image before recognition. Resolution affects both work and legibility, so establish it with representative small type and rotated scans rather than a universal setting. Keep the decision reversible by saving the extractor version, OCR version, language configuration, and source checksum beside the derived page record. When recognition improves or a template changes, reprocess only affected pages while leaving the archive object untouched. Consider a 24-page report with generated attendance tables on pages 1 through 23 and a scanned teacher note on page 24. Running OCR across all 24 pages adds render work and may degrade text that was already good. Never running OCR hides the note. Page routing pays the recognition cost once, for the one page that needs it, while provenance tells an operator why the paths differed. The trade-off is more branching and more metadata than an all-OCR pipeline. That complexity is justified only when the archive contains a meaningful mix of born-digital and scanned pages.

Build the Node.js ingestion boundary

The worker needs three ports: durable object storage, a page extractor, and a search index. Keep concrete engines behind those interfaces. That makes the contract testable and keeps retry behavior in one place.

type SourceKind = "text" | "ocr";

type PageText = {
  page: number;
  text: string;
  sourceKind: SourceKind;
  durationMs: number;
};

interface ArchiveStore {
  putOriginal(key: string, pdf: Uint8Array): Promise<{ checksum: string }>;
}

interface PageExtractor {
  pageCount(pdf: Uint8Array): Promise<number>;
  embeddedText(pdf: Uint8Array, page: number): Promise<string>;
  ocr(pdf: Uint8Array, page: number): Promise<string>;
}

interface SearchIndex {
  replaceReportPages(reportId: string, pages: PageText[]): Promise<void>;
}

const usefulText = (text: string): boolean => {
  const normalized = text.normalize("NFKC").replace(/\s+/g, " ").trim();
  const printable = [...normalized].filter((char) => !/\p{C}/u.test(char));
  return normalized.length >= 80 && printable.length / normalized.length >= 0.95;
};

async function ingestReport(
  reportId: string,
  pdf: Uint8Array,
  store: ArchiveStore,
  extractor: PageExtractor,
  index: SearchIndex,
): Promise<void> {
  const archiveKey = `reports/${reportId}/original.pdf`;
  await store.putOriginal(archiveKey, pdf);

  const count = await extractor.pageCount(pdf);
  const pages: PageText[] = [];

  for (let page = 1; page <= count; page += 1) {
    const startedAt = performance.now();
    const embedded = await extractor.embeddedText(pdf, page);
    const sourceKind: SourceKind = usefulText(embedded) ? "text" : "ocr";
    const text = sourceKind === "text" ? embedded : await extractor.ocr(pdf, page);

    pages.push({
      page,
      text: text.normalize("NFKC"),
      sourceKind,
      durationMs: Math.round(performance.now() - startedAt),
    });
  }

  await index.replaceReportPages(reportId, pages);
}

The 80-character threshold and 0.95 printable ratio are example policy values, not universal facts. Calibrate them with labeled pages from your templates. The decision must be explicit, measured, and replaceable.

Indexing should be idempotent. A retry replaces the report's page set instead of appending duplicates. Use a stable key such as reportId:page, attach the archive key, and filter every query by the caller's authorization scope before returning snippets. Results should link back to the original page.

Store the original first. Build all page records next. Publish the replacement set only when extraction finishes, so readers do not see half of a monthly report. If the index cannot atomically replace a set, write under a generation identifier and switch the active generation after completion.

A diagram in words: report job to immutable archive object; archive object to page router; text pages to direct extraction; image pages to render and OCR; both branches to a normalized page record; page records to a scoped index; search hits back to the archived PDF. One source, two extraction paths, one query surface.

Observability belongs at each arrow. Count reports completed and failed, pages routed to OCR, empty outputs, and replacement failures. Measure end-to-end report latency separately from per-page extraction time. Alert on sustained changes in error rate or OCR-route ratio using a known baseline from your workload.

Test retrieval, not just extraction

A parser test proves that characters came out. It does not prove that an educator can find a report. Build a small evaluation set of queries with expected report and page pairs: a learner name, a course phrase, a date, and a phrase found only in a scanned note. Include a query that must return no result outside the authorized school or account boundary.

Run that set when the template, extractor, OCR configuration, tokenizer, or chunking rule changes. Track page-level recall and false positives, then inspect failures. This connects fidelity to the search job instead of rewarding the pipeline for producing lots of text.

Roll out a new extraction generation beside the current one, compare routing rates and evaluation results, and promote it only after the checks pass. Keep the source checksum in both generations so a comparison never mixes different report files.

Limits

OCR cannot recover detail that was never captured in the page image, and extracted text cannot fully represent visual relationships in every chart or table. Use search to locate the likely page and let the archived PDF carry the visual meaning.

This selective approach is not suitable when every input is a clean scan and no embedded text can be trusted; a uniform OCR path is easier to operate there. It is also a poor fit when users need semantic answers from charts rather than page discovery, because page text alone does not encode every visual relationship. Password protection, digital signatures, retention rules, accessibility requirements, and jurisdiction-specific student-data obligations need separate design work. This field guide covers extraction and indexing; it does not turn derived search text into the authoritative document.

DEV Community