3 AM, pager goes off. A batch job that's been converting invoices to spreadsheets for two months straight is now returning empty rows for 40% of a client's upload. No errors. No stack trace. Just... nothing. The pipeline ran, the file "processed," and the output sheet had a header row and silence.
Turns out the client had switched scanners. Same file extension, same MIME type, completely different problem underneath. I'd built an extraction pipeline that assumed every PDF had a text layer, because every PDF I'd tested against did. The new batch was scanned images wrapped in a PDF container, and my parser was faithfully extracting zero characters from a page full of pixels.
That bug taught me something that should've been obvious from the start: "PDF" isn't one file format from a processing standpoint. It's two, wearing the same extension.
The two things a .pdf can actually be
A digitally generated PDF, exported from Word, a web app, an accounting tool, whatever, stores its text as actual text objects. There's a font, a character encoding, positioning data. When you extract from it, you're reading structured data that was always structured.
A scanned PDF is a photograph. Someone ran a document through a scanner or snapped it with a phone, and the "PDF" is just that image dropped into a PDF wrapper, sometimes with an invisible OCR text layer bolted on by the scanning software, often without one at all. Ask a text extractor to read it, and you get exactly what's there: nothing, or garbage if the bolted-on OCR was bad.
Building one code path that assumes text exists is the root bug. You need two paths, and more importantly, you need a way to decide which one a given file needs before you waste a single CPU cycle on the wrong approach.
Detecting which one you've got
The heuristic that's worked best for me isn't complicated. Pull the text layer, count what comes back, and compare that against how much of the page is covered by images.
import fitz # PyMuPDF
def classify_page(page, min_chars=20, image_area_threshold=0.85):
text = page.get_text().strip()
char_count = len(text)
page_area = page.rect.width * page.rect.height
image_area = 0
for img in page.get_images(full=True):
rects = page.get_image_rects(img[0])
for r in rects:
image_area += r.width * r.height
image_coverage = image_area / page_area if page_area else 0
if char_count >= min_chars and image_coverage < image_area_threshold:
return "digital"
return "scanned"
Two signals, checked together, catch more than either one alone. Char count alone fails on a page that has a scanned background image with a thin strip of real text overlaid on it (I've seen this on government forms constantly). Image coverage alone fails on a digital PDF that happens to have a large logo or watermark image taking up most of the page.
Run this per page, not per document. A 12-page contract with a digitally-generated cover page and 11 scanned exhibit pages is common, and treating the whole file as one type means you either miss the cover page's clean text or waste OCR time re-reading it.
Routing to the right path
Once you know what you're dealing with, the pipeline splits.
def extract_page(page):
if classify_page(page) == "digital":
return extract_text_layer(page) # fast, near-instant
else:
return run_ocr(page) # slow, CPU/GPU heavy
The cost difference here is the whole reason this matters at scale. Pulling a text layer off a digital PDF is milliseconds. Running OCR on a scanned page, even with something efficient like Tesseract, is measured in seconds per page, and that's before you factor in preprocessing steps like deskewing or contrast correction that scanned documents often need to OCR cleanly.
If you're processing a batch of 5,000 files and running every single one through OCR "just to be safe," you're paying that seconds-per-page tax on files that didn't need it at all. I've seen this mistake cut throughput by an order of magnitude on what should've been a simple batch job. The classification step isn't just about correctness, it's the difference between a job that finishes in ten minutes and one that finishes in three hours.
Where this gets tricky
A few edge cases worth planning for before they page you at 3 AM:
Fake text layers. Some scanning software adds an OCR text layer automatically, and it's often bad OCR, full of misread characters. Your char count check will say "digital" because technically there's text, but the text is garbage. Worth spot-checking extracted text against a basic dictionary or character-frequency sanity check if accuracy really matters downstream.
Vector graphics mistaken for images. Some PDF generators render text as vector paths instead of font glyphs, usually for print-perfect output. get_text() returns nothing, get_images() finds nothing either, since it's not a raster image; it's paths. This one doesn't fit neatly into either bucket, and honestly the most reliable fix is rendering the page to an image and running OCR on it regardless of what the "type" technically is.
Hybrid documents. As mentioned, per-page classification handles the common case of mixed-type files, but if you're stitching extracted tables across pages, make sure your merge logic doesn't assume every page in a document behaves the same way.
Why this matters beyond your own pipeline
If you're building this from scratch, budget real time for the classification layer; it's not a one-line if-statement in practice; it's the part of the system that determines whether your extraction is fast and accurate or slow and wrong. I underestimated it the first time and paid for it with that 3 AM page.
If you're not building this from scratch and just need PDFs, scanned or not, turned into usable spreadsheet data without maintaining this detection and OCR-routing logic yourself, this is basically the exact problem PDF Converter's PDF to Excel tool handles under the hood; it runs both paths and picks the right one per file so you don't have to.
Either way, the lesson holds: don't assume every PDF that lands in your pipeline is the same file format just because the extension matches. Check first, route second, extract third.
Top comments (0)