From PDF to Markdown: Why Document Parsing is Important For RAG.

LEI QIN — Sun, 15 Mar 2026 17:15:13 +0000

RAG (Retrieval Augmented Generation) is quickly becoming the default pattern for grounding LLMs in your own data. But the quality of your RAG system depends heavily on a step many teams overlook: how you turn documents into text before they ever hit the vector store.

If your source is PDF-heavy—technical docs, reports, contracts—the parsing layer can make or break retrieval. Here’s why it matters.

Why Parsing Quality Matters for Retrieval

RAG works by embedding chunks of text, storing them in a vector DB, and retrieving the most relevant chunks at query time. The better those chunks reflect the document’s structure and meaning, the better the model can answer questions.

Bad parsing (raw text extraction, naive PDF-to-text):

Broken tables → numbers and headers get mixed into paragraphs; retrieval returns incomplete or nonsensical rows
Lost headings → no semantic hierarchy; chunk boundaries ignore section logic
Garbled layout → multi-column or complex docs produce a jumbled reading order
Noise → headers, footers, page numbers pollute chunks and dilute relevance

Good parsing (structured Markdown with layout-aware extraction):

Clean tables → preserved as HTML or structured blocks; chunks can contain coherent rows and cells
Preserved hierarchy → headings become natural chunk boundaries; sections stay intact
Logical flow → reading order follows the document’s intended structure
Less noise → boilerplate can be filtered; chunks are more representative of actual content

The result: embeddings capture real meaning, and retrieval returns chunks that actually answer the question instead of fragments of garbage.

Markdown as an Ideal RAG Ingestion Format

Markdown is a natural fit for RAG pipelines because it:

Keeps structure (headings, lists, tables) explicit instead of hidden in layout
Is easy to chunk — split on ## or by block type without arbitrary character cuts
Preserves semantic units — a table row, a code block, or a section stays together
Plays well with embeddings — models typically train on Markdown; structured text tends to embed more predictably

Raw PDF text, by contrast, often looks like one long run-on with no clear boundaries. You either chunk by fixed token count (losing context) or try to infer structure yourself (expensive and fragile). Starting from clean Markdown gives you structure “for free.”

What to Look For in a Parser

When building a document ingestion pipeline for RAG, look for parsers that:

Preserve structure — headings, tables, lists, code blocks
Produce Markdown — or structured JSON you can convert to Markdown
Handle layout — multi-column, complex formatting, reading order
Expose block metadata — block types, bounding boxes, reading order for custom chunking

Claude Code Curl Alternative: When Websites Block Your Requests

LEI QIN — Wed, 11 Mar 2026 17:36:28 +0000

Claude Code Curl Alternative: When Websites Block Your Requests

You've got Claude Code doing research or pulling data—and it leans on curl to fetch pages. Works like a charm at first. Then you hit a site that slams the door: 403 Forbidden, access denied, or nothing but a blank screen. "Fine," you think, "I'll spin up Playwright locally." Real browser, real JavaScript—what could go wrong? Turns out: plenty. Local Playwright adds serious overhead and often runs into the same anti-bot walls. There's a smoother path.

Why Claude Code Curl Fails on Certain Websites

Claude Code fetches pages with curl: lightweight, straightforward GET requests. Nothing fancy. The catch? Modern sites are built to spot and block exactly that kind of traffic.

Why Sites Say No to Curl

Missing or suspicious headers — Curl keeps it minimal. Sites expect User-Agent, Accept-Language, Accept-Encoding, and more. Skip those, and you look like a bot.
No JavaScript execution — Plenty of sites ship skeleton HTML and load the real content via JS. Curl grabs the skeleton. You get empty shells.
IP-based rate limiting — One IP, lots of requests. Whether it's your machine or Claude's backend, you burn through limits fast.
Bot detection — TLS fingerprints, header order, timing, behavior. Curl's signature is easy to read.

When any of that kicks in, you see 403 Forbidden, 429 Too Many Requests, 503 Service Unavailable, or blank pages. No content means Claude Code has nothing to work with.

The Local Playwright Trap

You might think: real browser, real everything—problem solved. Playwright does handle JavaScript and feels more like a real browser. But it comes with trade-offs:

The Heavier Stack

Heavy dependencies — Chromium, Firefox, or WebKit. Installation and updates eat hundreds of MB.
Memory and CPU — Each headless instance chews RAM and CPU. Parallel crawls get expensive fast.
Environment headaches — Different OSes, Node versions, Playwright versions. "Works on my machine" becomes your new catchphrase.

Blocked Anyway

Even with a real browser, many high-value targets—e-commerce, news, travel, finance—use bot detection that still catches you:

Cloudflare challenges — Turnstile, JS challenges, "Checking your browser" screens. They stop curl, and often stop local headless browsers too.
Headless detection — navigator.webdriver, missing plugins, odd resolutions.
Behavioral analysis — No mouse moves, no natural scroll, inhuman click speeds.
Proxy and VPN detection — Datacenter IPs? Often flagged.

Result: same 403s, Cloudflare, and captchas. But now you're lugging a heavier stack.

AnyCrawl: Crawl for AI

AnyCrawl is crawl for AI—a cloud scraping API built for AI agents and LLMs. It gets around these limits, including Cloudflare challenges, without running browsers on your machine. You get LLM-optimized Markdown back, ready for Claude and others. When curl or local Playwright hits a wall, this is the drop-in replacement.

How It Handles What Curl Can't

Curl Problem	AnyCrawl Solution
Minimal headers, easy to fingerprint	Full browser-like headers and request patterns
No JavaScript execution	Cheerio, Playwright, and Puppeteer engines in the cloud
Single IP, easily rate-limited	Rotating proxy support to distribute traffic
Cloudflare challenges, 403, anti-bot	Bypass Cloudflare and managed anti-detection

AnyCrawl bypasses Cloudflare challenges (Turnstile, JS checks, and more), so sites that block curl or local browsers are still reachable. Send a URL, get clean Markdown. No local browser, no proxy setup, no maintenance.

Claude Code Integration via AnyCrawl CLI Skill

The best way to use AnyCrawl with Claude Code is the AnyCrawl CLI and its skill for AI coding agents:

# Install AnyCrawl CLI
npm install -g anycrawl-cli
# Or: npx -y anycrawl-cli init

# Authenticate (get API key at anycrawl.dev/dashboard)
anycrawl login --api-key <your-api-key>

# Install skill for Claude Code (and Cursor, Codex, etc.)
anycrawl setup skills

The skill teaches Claude Code when and how to use AnyCrawl for scraping, search, mapping, and crawling—swapping in AnyCrawl when curl gets blocked.

MCP integration — Add the AnyCrawl MCP server to Claude Code:

claude mcp add --transport stdio anycrawl -e ANYCRAWL_API_KEY=YOUR_API_KEY -- npx -y anycrawl-mcp@latest

Swap YOUR_API_KEY with your key from anycrawl.dev/dashboard. Or run anycrawl setup mcp after installing the CLI.

API Usage (Curl-Compatible)

Want to call the API directly from scripts or another AI? Same interface:

curl -X POST "https://api.anycrawl.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/page-that-blocks-curl",
    "engine": "playwright"
  }'

Response includes markdown or html (or both), optimized for AI—ready for Claude or any LLM.

1,500 Credits: Enough for Daily Use

Free tier gives you 1,500 credits per month—no card required. Credit use depends on page size and engine:

Cheerio (static HTML) — lowest cost, ideal when JS isn't needed
Playwright (JS-heavy sites) — moderate cost, handles most anti-bot cases
Puppeteer — similar to Playwright for Chrome-specific needs

For everyday research, docs lookup, and small-scale extraction—dozens to a few hundred pages a month—1,500 credits usually covers it. Need more? Paid tiers start at 2,500 credits (Hobby) and scale from there.

When to Use Each Approach

Scenario	Best Tool
Simple public pages, no anti-bot	Claude Code curl
JS-heavy but not protected	Local Playwright (if you're fine with the setup)
Protected sites, Cloudflare, 403/429, captchas	AnyCrawl
High volume, many domains	AnyCrawl (cloud, proxies, concurrency)
Claude Code integration	AnyCrawl CLI + skill (`anycrawl setup skills`)

Getting Started

Sign up at anycrawl.dev Promo code: STARTUPY
Grab your API key from the dashboard
Install the CLI and skill: anycrawl setup skills (see anycrawl-cli)
Or call the Scrape API directly from your own scripts

No credit card for Free. Try scraping a URL that blocks curl—the difference shows up right away.

Summary

Claude Code's curl is perfect for open, static pages. When you run into 403s, Cloudflare, blank responses, or rate limits, local Playwright adds complexity and often fails anyway. AnyCrawl is crawl for AI—cloud-based, bypasses Cloudflare and anti-bot, uses rotating proxies and multiple engines, and returns LLM-ready Markdown. For daily use, 1,500 free credits cover most research and extraction—without running browsers locally or wrestling anti-bot systems yourself.

DEV Community: LEI QIN

From PDF to Markdown: Why Document Parsing is Important For RAG.

Why Parsing Quality Matters for Retrieval

Markdown as an Ideal RAG Ingestion Format

What to Look For in a Parser

Claude Code Curl Alternative: When Websites Block Your Requests

Claude Code Curl Alternative: When Websites Block Your Requests

Why Claude Code Curl Fails on Certain Websites

Why Sites Say No to Curl

The Local Playwright Trap

The Heavier Stack

Blocked Anyway

AnyCrawl: Crawl for AI

How It Handles What Curl Can't

Claude Code Integration via AnyCrawl CLI Skill

API Usage (Curl-Compatible)

1,500 Credits: Enough for Daily Use

When to Use Each Approach

Getting Started

Summary