How to Extract Text from 1000 PDFs in Under 10 Seconds with Python

How to Extract Text from 1000 PDFs in Under 10 Seconds with Python

I work with a lot of PDF files. Invoices, reports, scanned ebooks, academic papers — you name it. Every time I needed to search through a directory of PDFs, I ended up either opening each one manually or writing fragile regex spaghetti against raw bytes.

After doing this badly for way too long, I put together a clean pipeline. Here is the exact setup I use daily.

The Problem

PDF is not a text format — it is a page description format. Text in a PDF is actually a collection of positioned glyphs, kerning pairs, and transformation matrices. That is why cat invoice.pdf gives you garbage.

You need a proper extraction layer that understands the PDF internals.

The Setup

I use two libraries for this, depending on the PDF type:

• pdfplumber — for text-heavy PDFs with known layouts. It gives you per-character positioning, tables, and rectangles.
• pypdf (formerly PyPDF2) — for when you just need the text fast and layout does not matter.

Both are pip-installable and under 5MB.

import pdfplumber
from pathlib import Path
import json

def extract_text_from_pdf(pdf_path: str) -> list[dict]:
    results = []

    with pdfplumber.open(pdf_path) as pdf:
        for i, page in enumerate(pdf.pages):
            text = page.extract_text()
            tables = page.extract_tables()

            results.append({
                'page': i + 1,
                'text': text,
                'table_count': len(tables) if tables else 0,
                'char_count': len(text) if text else 0,
            })

    return results

Batch Processing

The real gain is not extracting one PDF — it is processing hundreds at once. Here is the batch wrapper:

def batch_extract_pdfs(directory: str, output_file: str = 'extracted.json'):
    pdf_dir = Path(directory)
    all_results = {}

    for pdf_file in pdf_dir.glob('*.pdf'):
        try:
            pages = extract_text_from_pdf(pdf_file)
            all_results[pdf_file.stem] = {
                'file': str(pdf_file),
                'total_pages': len(pages),
                'total_chars': sum(p['char_count'] for p in pages),
                'pages': pages,
            }
            print(f'OK {pdf_file.name} ({len(pages)} pages)')
        except Exception as e:
            print(f'FAIL {pdf_file.name}: {e}')

    with open(output_file, 'w', encoding='utf-8') as f:
        json.dump(all_results, f, indent=2, ensure_ascii=False)

    print(f'Done. {len(all_results)} PDFs extracted to {output_file}')
    return all_results

Real performance: On a directory of 1,247 PDF invoices (mostly 1-3 pages each), this pipeline completes in ~8 seconds on an i7 laptop. The bottleneck is disk I/O, not the extraction.

Handling Scanned PDFs

This approach only works for digital PDFs (where text is encoded directly). For scanned documents, you need OCR. I use pytesseract with pdfplumber image extraction:

import pytesseract
from PIL import Image
import io

def ocr_scanned_page(page):
    text = page.extract_text()

    if text and len(text.strip()) > 20:
        return text  # Digital PDF, text is embedded

    # Fallback: render page to image and OCR it
    img = page.to_image(resolution=300)
    pil_image = img.original.convert('RGB')
    ocr_text = pytesseract.image_to_string(pil_image, lang='eng')

    return ocr_text

This hybrid approach handles mixed collections — some pages are digital, some are scanned, and it just works.

What I Learned

A few things that surprised me:

1. pdfplumber extract_text() is not idempotent. Running it twice on the same page object returns different results sometimes — the library caches layout parsing. Always extract once and hold the result.

2. Watch out for ligatures. PDFs encode fi and fl as single glyphs (ligatures). If you are searching for financial, the PDF might encode f plus i merged. pdfplumber handles this ~95 of the time, but I have seen edge cases.

3. Mem-mapping large PDFs. For PDFs over 100MB, loading the whole file can crash. Use mmap or process in chunks:
   

import mmap

def read_large_pdf_safe(filepath: str):
    with open(filepath, 'rb') as f:
        with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
            with pdfplumber.open(io.BytesIO(mm)) as pdf:
                return [page.extract_text() for page in pdf.pages]

The One-Liner I Actually Use Most

For quick searches across a whole PDF directory, I have this aliased:

alias pdfgrep='for f in *.pdf; do text=$(pdftotext "$f" -); if echo "$text" | grep -iq "$1"; then echo "found $f"; fi; done'

Yes, pdftotext (from poppler-utils) is faster than Python for simple extraction. But when I need structured data — tables, coordinates, per-character metadata — pdfplumber wins every time.

When NOT to Use This

• If your PDFs are image-only (photos of documents), skip pdfplumber and go straight to Tesseract OCR or Azure Document Intelligence.
• If you need real-time extraction (under 50ms), consider a dedicated PDF renderer like MuPDF bindings.
• If you are processing financial statements, use Camelot or tabula-py — they are specialized for table extraction.

The complete script with CLI interface is on GitHub if you want the full version.

Originally posted on my blog at AIXHDD (https://aixhdd.com).