Turn PDFs (even scanned ones) into RAG-ready chunks with one API call

#pdf #rag #python #ai

Every RAG project hits the same wall on day two: the documents. Half are born-digital PDFs, a quarter are scans someone ran through a photocopier in 2011, and the rest are Word files. Your embedding model wants clean text in sensible chunks, with page numbers so the answer can cite its source.

This post shows the two ways I do it: a small Python script you own, and a hosted API for when you don't want to run OCR yourself.

What "RAG-ready" actually means

Before code, the target. For each document you want:

Text in reading order, not columns glued together mid-sentence.
Structure kept as Markdown: headings, lists and tables survive, because they carry meaning for the LLM and help chunking.
OCR only where needed. Running OCR on a born-digital page is slower and worse than reading its text layer. Running nothing on a scanned page gives you an empty string, and your retriever silently never finds that page.
Chunks with overlap (1,000–2,000 characters with ~200 overlap works for most embedding models) and page numbers on each chunk.

Point 3 is where most pipelines break. A "PDF" is often mixed: 40 typed pages and 2 scanned signature pages.

Option 1: do it yourself in Python

For born-digital PDFs, pdfplumber gets you most of the way:

import pdfplumber

def pdf_pages(path):
    with pdfplumber.open(path) as pdf:
        for i, page in enumerate(pdf.pages, start=1):
            text = page.extract_text() or ""
            yield i, text

def chunk(text, size=1500, overlap=200):
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        yield text[start:end]
        if end == len(text):
            break
        start = end - overlap

for page_no, text in pdf_pages("paper.pdf"):
    if not text.strip():
        print(f"page {page_no}: no text layer -> needs OCR")
        continue
    for c in chunk(text):
        ...  # embed c, store with page_no

The if not text.strip() line is the important one: it's how you detect a scanned page. From there you render that page to an image (pypdfium2) and run Tesseract on it (pytesseract), with the right language packs installed. Then you'll want headings from font sizes, tables from ruled lines, DOCX support, language detection... It's a weekend, plus the fun of installing Tesseract language data in every environment you deploy to.

Option 2: one API call

Disclosure: I built this one. It's a pay-per-page Actor on the Apify Store, PDF Text Extractor, that does exactly the list above: text layer where there is one, OCR only on pages without one (13 languages), Markdown with headings/lists/ruled tables, per-page rows, chunks with overlap, and language detection. It also reads DOCX and image files.

curl -X POST "https://api.apify.com/v2/acts/malmurk~pdf-text-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "startUrls": [{"url": "https://arxiv.org/pdf/1706.03762"}],
        "outputMode": "page",
        "chunkSize": 1500,
        "chunkOverlap": 200
      }'

You get one JSON row per page (page 2 of that real run, text shortened):

{
  "file_name": "1706.03762v7.pdf",
  "page": 2,
  "page_count": 15,
  "markdown": "### 1 Introduction\n\nRecurrent neural networks, long short-term memory [13] and ...",
  "used_ocr": false,
  "language": "en",
  "chunks": [
    {"index": 0, "start": 0, "end": 1431, "text": "### 1 Introduction\n\nRecurrent neural networks ..."},
    {"index": 1, "start": 1234, "end": 2526, "text": "integral part of compelling sequence modeling ..."}
  ]
}

Feeding a vector store from Python:

from apify_client import ApifyClient

client = ApifyClient(APIFY_TOKEN)
run = client.actor("malmurk/pdf-text-extractor").call(run_input={
    "startUrls": [{"url": u} for u in pdf_urls],
    "outputMode": "page",
    "chunkSize": 1500,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    for c in row["chunks"] or []:
        store.add(text=c["text"], metadata={"file": row["file_name"], "page": row["page"]})

Pricing is $1 per 1,000 pages, plus $7 per 1,000 for pages that actually needed OCR. Files that fail (dead link, password-protected PDF, an HTML page instead of a PDF) cost nothing and come back as a row with an error field, which is handy for spotting the "download" links that are really web pages.

Gotchas either way

"PDF links" that are HTML pages. Many sites link to a viewer page with a Download button. Check the Content-Type before you parse.
Tables without ruling lines (most scientific papers) come out as text in reading order with any text-layer tool. If tables matter, check them by eye on a sample.
OCR language. Tesseract with only eng on a German scan produces confident garbage. Set the languages you actually have.
Rotated margin stamps (like arXiv's vertical ID) are noise; skip rotated text on text-layer pages.

Use it from an agent

If you work in Claude, Cursor or another MCP client, the same Actor is available as a tool through Apify's MCP server: add https://mcp.apify.com?tools=malmurk/pdf-text-extractor and ask it to "read this PDF and summarise section 3".

I'm Cătălin, a Python developer in Chișinău building scraping and document tools. Questions or a PDF that breaks it: leave a comment. Drafted with AI assistance, tested and edited by me.

DEV Community