Reading PDFs found during a crawl, including scanned ones

#python #pdf #ocr #rag

Half the useful content on government, academic, and enterprise sites is behind a .pdf link. A crawler that only understands HTML walks past it, and a RAG index built from that crawl has a hole exactly where the technical specifications live.

The MESSORA API handles PDFs in the same call as HTML. parse_pdf defaults to true, so a /scrape against a PDF URL returns Markdown without any special casing.

A PDF is just another URL

import os
import requests

API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}

resp = requests.post(
    f"{API}/scrape",
    headers=HEADERS,
    json={"url": "https://example.gov.br/edital-2026.pdf", "formats": ["markdown"]},
    timeout=180,
)
resp.raise_for_status()
data = resp.json()

print(data["metadata"]["title"])
print(data["markdown"][:500])

Raise the client timeout for PDFs. A 200-page document takes materially longer than an HTML page, and a 90-second client timeout on a job the server is still working through wastes the credit you already committed.

Turning it off on purpose

parse_pdf: false makes PDF URLs fail fast instead of consuming time and credits:

json={"url": url, "formats": ["markdown"], "parse_pdf": False}

That is the right setting for a crawl of a marketing site where the only PDFs are brochures you do not want in the index. You are trading coverage for predictable cost.

Scanned documents need OCR

A PDF produced by a scanner has no text layer — it is images wrapped in PDF structure. Text extraction returns empty, which surfaces as extraction_failed rather than an error, because the fetch worked fine.

On /crawl, pdf_ocr enables optical recognition for exactly this case:

resp = requests.post(
    f"{API}/crawl",
    headers=HEADERS,
    json={
        "url": "https://example.gov.br/transparencia/",
        "max_pages": 30,
        "url_regex": r".*\.pdf$",
        "parse_pdf": True,
        "pdf_ocr": True,
        "timeout_ms": 180_000,
    },
    timeout=30,
)
job_id = resp.json()["job_id"]

Two notes on this configuration:

url_regex restricted to .pdf turns the crawl into a document harvester. The frontier still discovers HTML links to traverse, but only PDFs are extracted.
timeout_ms raised to 180000. The default of 60000 is tuned for HTML. OCR over a multi-page scan will exceed it, and a timeout mid-crawl shows up as per-item timeout entries rather than a job failure.

OCR is slower and less accurate than a real text layer. Enable it when you know the corpus is scanned, not as a blanket default — on a digital-native PDF it adds latency for a worse result than plain extraction.

Detecting the empty-text case

def needs_ocr(item: dict) -> bool:
    """A PDF that fetched fine but yielded almost nothing is probably scanned."""
    if item["scrape_status"] == "extraction_failed":
        return True
    text = item.get("markdown") or ""
    return item["url"].lower().endswith(".pdf") and len(text.strip()) < 200


job = wait_for_job(job_id)
rescan = [item["url"] for item in job["results"] if needs_ocr(item)]
print(f"{len(rescan)} documents to reprocess with OCR")

The length threshold catches the worst case: a scanned PDF where extraction returned only the header text a text layer happened to contain, so scrape_status reads success while the body is missing. That silently poisons a RAG index with documents that look present and answer nothing.

Chunking extracted PDFs

PDF-derived Markdown has different structure from HTML-derived Markdown. Page breaks become paragraph breaks, and heading levels are inferred from font size rather than declared. Splitting on Markdown headers works less reliably than it does for web pages.

For PDFs, size-based splitting with overlap is more predictable:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1200,
    chunk_overlap=150,
    separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.split_text(data["markdown"])

The overlap matters more here than for HTML, because a table or a numbered clause split across a page boundary loses its subject line otherwise.

Cost

PDF extraction is billed as a page scrape: 1 credit, same as HTML, regardless of page count inside the document. A 300-page technical standard costs the same single credit as a landing page. For document-heavy corpora that pricing shape is the reason to prefer a PDF-aware crawl over building a separate download-and-parse pipeline.

DEV Community