DEV Community

Caspian Data
Caspian Data

Posted on

Turn PDFs into Markdown chunks for RAG in 10 lines of Python (with OCR for scanned pages)

#llm #python #rag #tutorial

If you have ever fed PDFs into an LLM pipeline, you know the problem: pdftotext gives you one long wall of text. Headings are gone, page numbers and running headers are glued into sentences, two-column papers come out interleaved, and scanned pages come out empty.

For retrieval-augmented generation (RAG) that hurts twice: chunks cut through the middle of sections, and the embedding of a chunk has no idea which section it came from.

This post shows a small pipeline that fixes all of that:

  1. PDF (or DOCX / HTML) → clean Markdown with real headings and lists
  2. OCR only for pages that have no text layer
  3. Chunks split on heading and paragraph boundaries, each tagged with its nearest heading
  4. A chunks.jsonl file you can load into any vector database

The 10-line version

We use the Document to Markdown Actor on Apify, which does the parsing, OCR and chunking in the cloud. You need a free Apify account and its API token.

pip install apify-client
export APIFY_TOKEN=your_token
Enter fullscreen mode Exit fullscreen mode 
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("caspiandata/doc-to-markdown").call(run_input={
    "urls": ["https://arxiv.org/pdf/1706.03762"],
    "chunkSize": 1200,
    "chunkOverlap": 120,
})
for doc in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(doc["title"], doc["pageCount"], "pages,", len(doc["chunks"]), "chunks")
Enter fullscreen mode Exit fullscreen mode

For the "Attention Is All You Need" paper this prints:

Attention Is All You Need 15 pages, 48 chunks
Enter fullscreen mode Exit fullscreen mode

What the output looks like

Each document comes back with the full Markdown plus a chunks array:

{
  "index": 3,
  "heading": "Abstract",
  "text": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...",
  "chars": 1180
}
Enter fullscreen mode Exit fullscreen mode

The heading field is the part people usually miss. Store it as metadata next to the vector and you can show "found in section 3.2 Attention" in your answers, or boost matches from the right section.

Writing chunks for a vector database 

import json

with open("chunks.jsonl", "w", encoding="utf-8") as f:
    for doc in client.dataset(run["defaultDatasetId"]).iterate_items():
        for c in doc["chunks"]:
            f.write(json.dumps({
                "id": f"{doc['sourceUrl']}#{c['index']}",
                "text": c["text"],
                "metadata": {"source": doc["sourceUrl"], "heading": c["heading"]},
            }) + "\n")
Enter fullscreen mode Exit fullscreen mode

From here, embed text with the model of your choice and upsert into Pinecone, Qdrant, Weaviate or pgvector.

Scanned PDFs

Set "ocr": "auto" (the default) and pages without a text layer are OCR'd with Tesseract; normal pages are read directly, so you don't pay the OCR cost for them. For non-English scans add languages, e.g. "ocrLanguages": ["eng", "deu"].

What it handles (and what it doesn't)

Handles well: headings from font sizes, bullet lists, hyphenated line breaks, repeated headers/footers and page numbers, two-column academic papers, DOCX tables, the main content of HTML pages.

Not yet: complex PDF tables come out as text lines rather than Markdown tables, and math formulas stay plain text.

Cost

Pay per use: $3 per 1,000 documents (first 10 pages included), $0.30 per 1,000 extra pages, $3 per 1,000 OCR pages. A free Apify account includes monthly credit, which is enough to try this on a few hundred documents.

Full runnable examples (Python and JavaScript) are in this GitHub repo.

Disclosure: we (Caspian Data) built and maintain the Actor used here. This post was written with the help of AI tools and the code was tested on the real API before publishing.

Top comments (0)