SteadyScrape

Posted on Nov 8

Bulk-Convert Digital PDFs to Text and Markdown for a RAG Index (and Catch the Scanned Ones)

#python #pdf #rag #markdown

Most "chat with your documents" projects die on step one: getting text out of the PDFs. A handful of files is easy. A few hundred reports, manuals and specs from different sources is where you find the edge cases: encrypted files, 400-page monsters, and scans that look like documents but contain only images.

This post walks through:

a DIY extraction script and its traps,
the same batch job with a hosted Actor and real output,
chunking the result for a search or RAG index, plus cost and limits.

The DIY version

With pypdf the basic extraction is a few lines:

import io, requests
from pypdf import PdfReader

def pdf_text(url):
    data = requests.get(url, timeout=60).content
    reader = PdfReader(io.BytesIO(data))
    return "\n\n".join(page.extract_text() or "" for page in reader.pages)

What bites you in production:

Scanned PDFs return an empty string, silently. Your index now holds a "document" with no text, and nobody notices.
Encrypted files raise errors or return nothing.
Size and page limits. One huge file can stall a whole batch, so you need byte and page caps.
Bookkeeping. You want page counts, title/author metadata and a status per file, not a pile of strings.

The hosted version

PDF to Text Markdown Extractor (Digital, No OCR) does this batch job with explicit statuses. Disclosure: I build this Actor at SteadyScrape. It downloads each direct PDF URL and extracts text locally. There's no OCR in this version, so scanned files are labelled image_only and incur no extraction charge.

Input matching the current schema:

{
  "pdfUrls": [
    "https://www.orimi.com/pdf-test.pdf",
    "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
    "https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf"
  ],
  "format": "both",
  "maxFiles": 20,
  "includeMetadata": true
}

format is text, markdown or both. You can also upload files in the Apify Console instead of passing links.

From Python (token from an environment variable):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/pdf-text-markdown").call(run_input={
    "pdfUrls": [
        "https://www.orimi.com/pdf-test.pdf",
        "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
        "https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf",
    ],
    "format": "both",
})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Real output from a verified run of exactly those three files (text trimmed):

[
  {"source": "https://www.orimi.com/pdf-test.pdf", "status": "ok",
   "pageCount": 1, "chars": 367, "title": "PDF Test Page",
   "text": "PDF Test File\n\nCongratulations, your computer is equipped with a PDF ...",
   "chargeEvent": "pdf-extracted"},
  {"source": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
   "status": "image_only",
   "note": "No extractable text — PDF looks image-only/scanned. OCR is not included in v1. Not charged.",
   "text": null, "chargeEvent": null},
  {"source": "https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf", "status": "ok",
   "pageCount": 16, "chars": 23656,
   "text": "Internet Engineering Task Force (IETF) T. Bray, Ed.\nRequest for Comments: 8259 ...",
   "chargeEvent": "pdf-extracted"}
]

That run charged two PDF extractions plus the run fee and the standard start event; the image-only file added no charge, and the row tells you why.

Chunking for a search or RAG index

Keep the ok rows, route image_only rows to an OCR queue (or a human), and split the Markdown on headings, with a size fallback:

import re

def chunks(md, max_chars=2000):
    parts = re.split(r"\n(?=#{1,3} )", md)
    for part in parts:
        for i in range(0, len(part), max_chars):
            yield part[i:i + max_chars]

index, needs_ocr = [], []
for r in rows:
    if r["status"] == "ok":
        for n, c in enumerate(chunks(r["markdown"] or r["text"])):
            index.append({"source": r["source"], "chunk": n, "text": c})
    elif r["status"] == "image_only":
        needs_ocr.append(r["source"])

print(len(index), "chunks;", len(needs_ocr), "files need OCR")

Embed index with whatever model you use and store source with each chunk so answers can cite the file.

The Markdown headings are heuristic: the PDF format doesn't reliably mark headings, so treat them as hints for splitting, not as a guaranteed document outline. Tables and multi-column layouts can come out in reading-order text rather than structured tables.

What it costs

From the Store pricing:

$0.004 per successfully extracted digital PDF
$0.003 once per run, only if at least one PDF was extracted
Apify's standard $0.00005 Actor start event

200 successfully extracted digital PDFs in one run cost $0.80305 (200 × $0.004 + $0.003 + $0.00005). Scanned/image-only, empty, encrypted, oversized and failed files incur no extraction charge, and a run where no PDF is extracted doesn't pay the $0.003 run fee; Apify's $0.00005 start event still applies to every run.

Limits

No OCR. Scanned documents are flagged, not read.
Direct PDF links only. It doesn't crawl a website looking for PDFs.
Per-run file caps and per-file page and size caps apply. Oversized files return too_large / too_many_pages (no extraction charge).
Only process documents you're allowed to use.

Integrations

Run it via the Apify API, on a schedule, or from Make, Zapier or n8n through Apify's integrations. AI agents can call it through the Apify MCP server, which is handy as a "read this PDF" tool.

SteadyScrape (steady-scrape) builds small, pay-per-result data tools on Apify.

DEV Community