PDF to Markdown in Python: build a RAG-ready pipeline

Kupa — Tue, 20 Oct 2026 16:53:17 +0000

Large language models read Markdown far better than raw PDF text. In this guide you convert a PDF to clean Markdown with the AnyToMarkdown API, get RAG-ready chunks with page numbers, and embed them for semantic search.

What you need

Python 3.9+ and pip install requests openai numpy
A RapidAPI key: subscribe to the free plan on AnyToMarkdown
An OPENAI_API_KEY for the embedding step

1. Convert the PDF and get chunks

import requests

HOST = "anytomarkdown-pdf-to-markdown-and-document-parser-api.p.rapidapi.com"
headers = {"X-RapidAPI-Key": "YOUR_RAPIDAPI_KEY", "X-RapidAPI-Host": HOST}

with open("handbook.pdf", "rb") as f:
    r = requests.post(
        f"https://{HOST}/v1/convert",
        headers=headers,
        files={"file": f},
        params={"includeChunks": "true", "includeContent": "false", "chunkSize": 400, "chunkOverlap": 40},
        timeout=180,
    )
r.raise_for_status()
chunks = r.json()["chunks"]
print(len(chunks), "chunks")

includeContent=false skips the full Markdown because you only need the chunks. Each chunk looks like this:

{ "index": 1, "content": "...", "tokens": 392, "headings": ["Employee Handbook", "2. Benefits"], "pageStart": 2, "pageEnd": 2 }

Chunks follow the document's sections: they never start in the middle of a table, and each one carries its heading path, so a chunk about vacation days still says it belongs to Benefits.

2. Embed the chunks

from openai import OpenAI
import numpy as np

client = OpenAI()
texts = [c["content"] for c in chunks]
response = client.embeddings.create(model="text-embedding-3-small", input=texts)
vectors = np.array([item.embedding for item in response.data])

Keep headings and pageStart next to each vector. They let your answers cite a section and page.

3. Search

def search(question, top=3):
    q = np.array(client.embeddings.create(model="text-embedding-3-small", input=question).data[0].embedding)
    scores = vectors @ q / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q))
    for i in scores.argsort()[::-1][:top]:
        c = chunks[i]
        print(f"{scores[i]:.2f}  {' > '.join(c['headings'])}  (page {c['pageStart']})")
        print(c["content"][:200], "\n")

search("How many vacation days do employees get?")

Tips

Scanned PDF? Leave ocr on auto, or force it with ocr=force.
Only need some pages? Add pages=1-10.
The free plan accepts 10 MB and 50 pages per request; higher plans raise the limits.
Want to see the Markdown first? Try the free PDF to Markdown converter.

Show DEV: Building Smartphone Specs API with .NET and Caddy

Kupa — Tue, 04 Aug 2026 15:13:48 +0000

Mobile hardware specification APIs have long been messy and hard to integrate, largely due to incomplete and unnormalized data.

Raw strings as specs are frustrating to use in program logic. So, I built Device Specs API.

1. The Problem: Messy & Unstructured Data

If you've ever tried to integrate a specs API into your project, you've likely faced messy data, missing fields, or raw strings. For example:

Screen Refresh Rates: You want to show the refresh rate of a device to the user properly. Instead of an integer, you get a raw string like "6.7 inches, 120Hz LTPO OLED". Parsing a complex string just to get one number is a nightmare.
CPU Clock Speeds: It gets even crazier. An API returns: Octa-core (1x3.4 GHz Cortex-A725 & 3x3.2 GHz Cortex-A725 & 4x2.2 GHz Cortex-A725) Congratulations! Your simple feature just turned into a Regex lecture.

2. The Solution: Strongly-Typed, Normalized Data

I shifted the parsing heavy lifting to the backend server. The data returned to the user is clean, easy to consume, and strongly typed: numbers are numbers, lists are lists, texts are texts, and booleans are booleans.

When you request data for a device, the server returns normalized JSON:

"battery_capacity": 5000 (integer, not "5000")
"has_nfc": true (boolean, not "yes")
Strongly-typed arrays for RAM and Storage options instead of raw blobs like "256GB 8GB RAM, 512GB 12GB RAM"

(If you ever need the raw format for legacy reasons, it's still available as an optional field!)

3. Deep Filtering

What if you want to query very specific hardware parameters without pulling unnecessary data?

That’s where Deep Filtering comes in. It allows you to query the dataset as deeply as possible directly through URL parameters.

For example, to find all devices with:

Battery capacity > 5000 mAh
RAM >= 8GB
Manufacturer is Samsung or Xiaomi
Model name contains "pro"

You just pass the pattern into the request URL:

?battery_gt=5000&ram_gte=8&manufacturer_in=samsung,xiaomi&model_contains=pro

Simple as that!

4. The Tech Stack & Architecture

Backend: ASP.NET Core Web API (.NET)
Frontend: Blazor WebAssembly / Server hybrid
Host: Ubuntu 24.04 + Docker
Reverse Proxy & SSL: Caddy
Data Format: Normalized JSON

5. Why Caddy Instead of Nginx?

I used Nginx for a while, but dealing with manual certbot setups and container configuration tweaks became tedious.

I switched to Caddy, which handles automatic HTTPS certificate management out of the box with zero manual cron jobs. The Caddyfile syntax is ultra-clean, minimal, and performance has been rock solid.

Try It Out

Full docs and live interactive testing are available here:

🔗 Device Specs API (ds.gtgroup.dev)

I’d love to get feedback from the DEV.to community on backend performance optimizations or features you'd like to see added!

DEV Community: Kupa