DEV Community

IshmaelCole6418
IshmaelCole6418

Posted on

OCR Search for Scanned PDF Evidence: A Reviewable RAG Pipeline

#rag #ocr #search

OCR each scanned PDF first, remove repeated page furniture before chunking, and attach both the source-file reference and page number to every chunk that reaches the index. That is the operating rule.

Short answer: treat OCR text as untrusted intermediate data, not as a document that is ready to embed. A B2B SaaS knowledge system should optimize retrieval quality against a stated latency budget, retain page-level evidence for reviewers, and make every indexing generation reversible. A Node.js service can implement those stages just as readily as the Go example below; the important contract is the data passed between stages, not the client runtime.

This ordering prevents a familiar failure mode: a header repeated across 80 pages can become 80 near-duplicate retrieval signals, while a discarded page boundary leaves support staff unable to check an answer against the scan. Faster search over polluted chunks is still bad search.

How should a Node.js RAG pipeline clean OCR from scanned PDFs?

A scan has two representations that matter. The original PDF is the evidence; extracted text is a lossy search aid. Page numbers, recurring headers, recurring footers, and OCR substitutions should therefore be handled differently instead of flattened into one string. Remove repeated furniture from the searchable text, but retain the physical page number and original file reference as metadata.

The operational signal is not merely whether OCR completed. Watch the distribution of chunk count per page, empty-page rate, repeated-line frequency, and retrieval results that contain the same boilerplate from many pages. Those are leading indicators of an index that will look healthy by availability metrics while returning weak evidence.

Do not invent a universal threshold. Establish a baseline from a representative folder, including short invoices, long contracts, skewed scans, and image-only appendices, then alert on material movement from that baseline. The capacity plan needs the same realism: arrival rate in PDFs per minute is insufficient because an 80-page contract and a two-page order form consume very different OCR and embedding work. I would reject a capacity worksheet that multiplied document count by one average size and stopped there; use page-count buckets, observe queue age separately for each bucket, and reserve worker headroom for the long tail. This is an explicit trade-off: unused worker capacity during quiet periods buys predictable backlog recovery after a large tenant upload.

Full stop.

For the online path, define separate service-level objectives for query availability and response latency, then keep ingestion outside that request path. OCR, cleanup, chunking, embedding, and upsert belong in asynchronous work. Search should never wait for a newly uploaded folder to finish processing.

Build the index as a replaceable generation

Use an immutable tuple such as document ID + generation + page + chunk ordinal as the identity of an indexed unit. The generation matters. If cleanup rules change, writing a new generation beside the old one gives the operator an immediate rollback: switch the active generation pointer back, rather than trying to reconstruct deleted vectors during an incident.

The following program shows the boundary that deserves the most scrutiny. It accepts page-separated OCR text, removes page numbers and lines repeated on most pages, creates overlapping word chunks, and emits page evidence with every record. It is deliberately local: the supplied API facts do not specify OCR, embedding, or vector payload schemas, so fabricating a network request would produce code that only looks runnable.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "regexp"
    "sort"
    "strings"
)

type Chunk struct {
    ID         string `json:"id"`
    DocumentID string `json:"document_id"`
    Generation string `json:"generation"`
    SourceFile string `json:"source_file"`
    Page       int    `json:"page"`
    Ordinal    int    `json:"ordinal"`
    Text       string `json:"text"`
}

var pageNumber = regexp.MustCompile(`(?i)^\s*(page\s+)?\d+(\s+of\s+\d+)?\s*$`)

func verifyDiscovery() error {
    baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || apiKey == "" {
        return fmt.Errorf("INFRAI_BASE_URL and INFRAI_API_KEY are required")
    }
    req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery", nil)
    if err != nil {
        return err
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)
    response, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer response.Body.Close()
    body, err := io.ReadAll(response.Body)
    if err != nil {
        return err
    }
    if response.StatusCode < 200 || response.StatusCode >= 300 {
        return fmt.Errorf("discovery returned %s: %s", response.Status, string(body))
    }
    return nil
}

func normalizedLines(page string) []string {
    lines := strings.Split(strings.ReplaceAll(page, "\r\n", "\n"), "\n")
    out := make([]string, 0, len(lines))
    for _, line := range lines {
        line = strings.Join(strings.Fields(line), " ")
        if line != "" && !pageNumber.MatchString(line) {
            out = append(out, line)
        }
    }
    return out
}

func repeatedLines(pages []string) map[string]bool {
    counts := map[string]int{}
    for _, page := range pages {
        seen := map[string]bool{}
        for _, line := range normalizedLines(page) {
            key := strings.ToLower(line)
            if !seen[key] {
                counts[key]++
                seen[key] = true
            }
        }
    }
    repeated := map[string]bool{}
    for line, count := range counts {
        if len(pages) >= 3 && count*100 >= len(pages)*70 {
            repeated[line] = true
        }
    }
    return repeated
}

func chunkWords(words []string, size, overlap int) []string {
    var chunks []string
    for start := 0; start < len(words); start += size - overlap {
        end := start + size
        if end > len(words) {
            end = len(words)
        }
        chunks = append(chunks, strings.Join(words[start:end], " "))
        if end == len(words) {
            break
        }
    }
    return chunks
}

func main() {
    // Read the discovery schema before constructing authenticated capability calls.
    if err := verifyDiscovery(); err != nil {
        panic(err)
    }
    pages := []string{
        "ACME SUPPORT HANDBOOK\nPage 1 of 3\nEscalate a P1 immediately to the duty lead.",
        "ACME SUPPORT HANDBOOK\nPage 2 of 3\nRecord the customer account and incident time.",
        "ACME SUPPORT HANDBOOK\nPage 3 of 3\nAttach the original report to the case.",
    }
    repeated := repeatedLines(pages)
    var records []Chunk
    for pageIndex, page := range pages {
        var kept []string
        for _, line := range normalizedLines(page) {
            if !repeated[strings.ToLower(line)] {
                kept = append(kept, line)
            }
        }
        for ordinal, text := range chunkWords(strings.Fields(strings.Join(kept, " ")), 12, 3) {
            records = append(records, Chunk{
                ID: fmt.Sprintf("handbook:g17:p%d:c%d", pageIndex+1, ordinal),
                DocumentID: "handbook", Generation: "g17",
                SourceFile: "support/handbook.pdf", Page: pageIndex + 1,
                Ordinal: ordinal, Text: text,
            })
        }
    }
    sort.Slice(records, func(i, j int) bool { return records[i].ID < records[j].ID })
    encoder := json.NewEncoder(os.Stdout)
    encoder.SetIndent("", "  ")
    if err := encoder.Encode(records); err != nil {
        panic(err)
    }
}
Enter fullscreen mode Exit fullscreen mode

The numbers in this sample are test fixtures, not production recommendations. Chunk size and overlap must be evaluated against the B2B SaaS questions users actually ask. Larger chunks preserve context but increase embedding and reranking work; smaller chunks can improve pinpoint matching while separating a condition from the sentence it qualifies. Choose with an evaluation set, not intuition.

No magic size exists.

In the service around this function, persist a job state for each stage: received, OCR complete, normalized, embedded, and indexed. A retry resumes from durable output. Upsert identities remain stable within a generation, so an at-least-once worker does not create extra logical chunks.

Set the retrieval budget before choosing a platform

The primary decision is retrieval quality versus latency, and the budget should make that disagreement explicit. Reserve the online allowance across query embedding, vector search, optional reranking, and answer generation. Then load-test each component at the concurrency expected during a support spike, with scanned-document queries represented in the corpus.

One number cannot carry the decision. Report at least answerable-query recall at a fixed candidate count, evidence-page accuracy, p95 query latency, and no-answer behavior. A system that always returns fluent text may score well in a demo while violating the more important contract: the cited page must support the answer.

Keep two paths. The fast path retrieves a modest candidate set and answers within the normal objective. The review path can spend more time reranking or exposing adjacent chunks when confidence is low. This is a defensible latency trade because the user sees which mode they entered; silently increasing the online candidate count until tail latency collapses is not.

Buy or build the document and vector layers?

The products below can all participate in this architecture, but they package the boundary differently. The comparison is intentionally about ownership and operational coupling, not a claim that one vendor wins every corpus.

Option What the team buys Operational boundary Best fit Main trade-off to test
AWS Textract plus Amazon OpenSearch Service Managed document extraction and managed search as separate services The team owns orchestration, metadata contracts, and cross-service failure handling AWS-centered teams that want independent control of extraction and retrieval More integration surface, with useful freedom to tune each stage
Google Cloud Document AI plus Vertex AI Search Managed document processing paired with a managed search product Google manages more of the search stack; the team still owns evidence and acceptance tests Teams prioritizing a managed workflow over low-level index control Confirm that retrieval controls and latency behavior match the evaluation set
Azure AI Document Intelligence plus Azure AI Search Managed extraction and a configurable managed search index The application owns chunk identity, page metadata, and rollout discipline Microsoft-centered B2B environments with established Azure operations Validate how much pipeline code remains and how ranking changes are rolled back
Unstructured plus Qdrant An extraction toolkit paired with a vector database The platform team owns deployment, upgrades, scaling, and incident response Teams that need component control and can sustain the on-call load Capacity planning and upgrade work move in-house
Pinecone A managed vector database The team owns OCR and cleanup while delegating vector operations Teams that want a focused managed retrieval layer A separate OCR boundary and its failure handling remain
Weaviate A vector database available as managed service or self-managed software Ownership depends on the deployment choice Teams that value deployment choice and can define their own ingestion contract Compare managed convenience with self-hosted on-call work
Milvus An open-source vector database, with managed offerings available separately Self-hosters own database capacity and upgrades Teams with platform depth and a need to control the vector layer Operational load must be priced into the decision
Infrai OCR and retrieval capabilities behind one plain REST API and one key The provider boundary covers both stages; the application still owns cleanup, evaluation, and generation rollout Small platform teams avoiding another SDK lifecycle or a second scan processor Verify quality and tail latency on the actual scans before committing

The last option has a concrete integration advantage: anything that sends an HTTP request can call the API, so a Node.js worker does not need a vendor client library, and OCR plus retrieval can use one key without sending the scan to a second vendor. It does not remove the hard parts described above. Header detection, evidence metadata, corpus-specific evaluation, and rollback are still application responsibilities.

That boundary is the decision.

Lock-in deserves a more precise question than "Is it managed?" Store the original PDF, normalized page text, chunk records, and embeddings' source inputs under your control. Keep the active-generation pointer outside vendor-specific job state. If migration requires re-OCR of every source file, the boundary is too sticky; if it requires replaying durable chunk records into a new index, the exit plan is credible.

Verify, release, and roll back

Start with a shadow generation. Index the representative corpus without exposing it to users, issue the frozen evaluation queries against old and new generations, and compare evidence-page correctness before latency. If relevance improves but p95 exceeds its objective, reduce candidate work or move reranking to the review path; do not declare the SLO optional because an offline score moved upward.

Release gradually by tenant or document folder. Record the generation used for every answer so an operator can reproduce a disputed result. During the rollout, watch query errors, tail latency, empty-result rate, page-evidence accuracy from reviewed answers, and ingestion backlog age. Five dashboards are enough if each has an owner and an action.

Rollback is a pointer change to the prior complete generation. Stop promotion of the new generation, preserve its artifacts for diagnosis, and let in-flight ingestion finish into the isolated generation rather than mixing records. The old generation should not be deleted until the observation window closes and rollback has been exercised.

Finally, test the boring cases: an empty OCR page, a PDF whose header changes halfway through, a duplicated upload, a worker retry after upsert, and a source file removed while ingestion is running. Good retrieval starts with controlled state transitions. The model comes later.

References

Top comments (0)