How To Convert Any Webpage to Markdown With /Scrape

For agents and RAG, converting webpages to Markdown is the easy part. Getting the page is harder. A plain fetch can return a bot-check page or an empty shell that needs JavaScript to fill in, and the request still looks successful. The model then gets no page content.

This guide shows how to get the real page as Markdown with a short Python script and the Decodo Web Data API.

TL;DR

• A POST to the Decodo Web Data API's /scrape endpoint with "markdown": true returns a page as Markdown. You can start on the free plan.
• The script starts with the cheapest request, checks each result for page content, and adds JavaScript rendering or the premium proxy pool only when needed.
• On 22 public test pages, the script converted 20, headless Chromium 18, and plain requests 15.

What you'll need

• Python 3.10+ and the Requests library
• A Decodo account on the Web Data API Free plan

The /scrape endpoint in 60 seconds

The /scrape endpoint handles proxy rotation, JavaScript rendering, and anti-bot protection.

Send the URL with "markdown": true. The page is returned in results, with the site's own status_code, so you can detect error pages.

Step 1: Get your API key

Activate the Web Data API free plan and copy your API key from the dashboard.

Install Requests and export the key:

pip install requests
export DECODO_API_KEY="your-api-key"

In PowerShell, use $env:DECODO_API_KEY="your-api-key" instead.

Step 2: Convert a page to Markdown

Save this as url_to_markdown.py. The script tries each request type in LADDER, cheapest first, and stops at the first usable result:

import os
import re
import sys
from pathlib import Path
from urllib.parse import urljoin, urlsplit

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

API_URL = "https://data.decodo.com/v1/scrape"
KEY = os.environ.get("DECODO_API_KEY") or sys.exit("Set DECODO_API_KEY")
HEADERS = {"Authorization": f"Bearer {KEY}"}
# A page with fewer words probably needs rendering or the premium pool
MIN_WORDS = 100
# Cheapest first, by the per-request prices on the billing page
LADDER = [
    ("standard", False),
    ("standard", True),
    ("premium", False),
    ("premium", True),
]
KNOWN_STEP = {}  # Domain -> index of the LADDER entry that worked
# Chinese and Japanese (about 2 characters a word) and Thai (about 6)
# don't put spaces between words
NO_SPACES = {r"[\u3040-\u30ff\u4e00-\u9fff]": 2, r"[\u0e00-\u0e7f]": 6}
SESSION = requests.Session()
# urllib3 skips POST retries by default
# (see Retry.DEFAULT_ALLOWED_METHODS).
# A /scrape call is safe to repeat.
SESSION.mount(
    "https://",
    HTTPAdapter(
        max_retries=Retry(
            total=3,
            backoff_factor=5,
            status_forcelist=[429, 500, 524],
            allowed_methods=None,
        )
    ),
)


def word_count(text):
    return len(text.split()) + sum(
        len(re.findall(p, text)) // n for p, n in NO_SPACES.items()
    )


class ScrapeError(Exception):
    def __init__(self, code, message):
        super().__init__(f"{code}: {message}")
        self.code = code


def scrape(url, pool, render_js):
    body = {"url": url, "markdown": True, "proxy_pool": pool}
    if render_js:
        body["headless"] = "html"
    resp = SESSION.post(API_URL, json=body, headers=HEADERS, timeout=180)
    data = resp.json()
    if resp.ok and data.get("status") != "failed":
        return next((r for r in data["results"] if r["type"] == "markdown"), None)
    raise ScrapeError(data.get("status_code", resp.status_code), data.get("message"))


def url_to_markdown(url, render_js=False, pool="standard"):
    domain = urlsplit(url).netloc
    step = max(LADDER.index((pool, render_js)), KNOWN_STEP.get(domain, 0))
    while step < len(LADDER):
        pool, render_js = LADDER[step]
        step += 1
        try:
            result = scrape(url, pool, render_js)
        except ScrapeError as err:
            # Some sites need the premium pool for Markdown. After a
            # 400 from standard, try premium. A 400 from premium raises.
            if err.code == 400 and pool == "standard":
                step = LADDER.index(("premium", render_js))
            # Codes for "try the next attempt", explained at
            # help.decodo.com/docs/web-data-api-status-codes
            elif err.code not in (612, 613, 999):
                raise
            continue
        if result is None:  # Try the next attempt
            continue
        if result["status_code"] in (404, 410):
            raise ScrapeError(result["status_code"], "page not found")
        ok = result["status_code"] < 400 and "help" not in result
        if ok and word_count(result["content"]) >= MIN_WORDS:
            KNOWN_STEP[domain] = step - 1
            break
    else:
        raise ScrapeError("all attempts", "no page content found")

    content = result["content"]
    # Remove the navigation above the title
    title = re.search(r"^# ", content, re.M)
    # A "#" after a code fence may be a code comment, not a title
    fence = re.search(r"^`{3}", content, re.M)
    if title and fence and fence.start() < title.start():
        title = None
    if title and word_count(content[title.start() :]) >= MIN_WORDS:
        content = content[title.start() :]
    # Absolute links keep citations working outside this site
    content = re.sub(r"\]\(([^)\s]+)", lambda m: "](" + urljoin(url, m[1]), content)
    slug = re.sub(r"\W+", "-", url.split("://")[-1]).strip("-")[:150]
    path = Path("pages") / (slug + ".md")
    path.parent.mkdir(exist_ok=True)
    path.write_text(content, encoding="utf-8")
    words = word_count(content)
    print(f"saved {path} ({words:,} words, {pool}, js={render_js})")
    return content


if __name__ == "__main__":
    for url in sys.argv[1:]:
        try:
            url_to_markdown(url)
        except (ScrapeError, requests.RequestException) as err:
            print(f"FAILED {url}: {err}", file=sys.stderr)

Run it on a static docs page:

python url_to_markdown.py https://docs.python.org/3/library/json.html

The script converts the page on the cheapest attempt:

saved pages/docs-python-org-3-library-json-html.md (4,359 words, standard, js=False)

The saved file starts at the page title, and its links are absolute:

# `json` — JSON encoder and decoder[¶](https://docs.python.org/3/library/json.html#module-json "Link to this heading")

**Source code:** [Lib/json/\_\_init\_\_.py](https://github.com/python/cpython/tree/3.15/Lib/json/__init__.py)

...

Steps 3 to 5 explain parts of this script. You don't need to add code.

Step 3: Handle JavaScript-rendered pages

Swagger UI renders API reference pages in the browser. Here's the Petstore demo:

The server's HTML body has only an empty div and scripts:

The /scrape response marks a page like this with a help field in its result. When a page has that field, or fewer than MIN_WORDS words, the script moves to the next attempt, which renders the page with "headless": "html":

python url_to_markdown.py https://petstore.swagger.io/

Rendering returns every endpoint:

saved pages/petstore-swagger-io.md (172 words, standard, js=True)

Step 4: Pick the proxy pool

Standard handles simple pages, and premium is built for complex anti-bot protection. Each LADDER entry sets the proxy_pool field to standard or premium. Standard costs less per request, so the script starts there. See the billing page for current prices. For a site that needs premium, call url_to_markdown(url, pool="premium").

The free plan includes both pools and JavaScript rendering, so the whole script runs on it.

Step 5: Run it on long URL lists

These 4 parts of the script keep the output clean and the cost low:

• Status checks. The script checks the site's own status_code and doesn't save error pages. For a site you scrape often, also check for a phrase that every real page has. Then you can safely lower MIN_WORDS for its short pages.
• Per-domain cache. KNOWN_STEP remembers the attempt that worked on each domain, so later pages there usually need 1 request. In a long-running process, clear KNOWN_STEP regularly, because sites change.
• Navigation removal. The script removes the navigation above the page's # title, so your chunks don't contain it.
• Absolute links. Relative links become absolute, so citations work outside the site.

Pass several URLs:

python url_to_markdown.py \
  https://www.law.cornell.edu/wex/contract \
  https://docs.python.org/3/library/no-such-page.html

The missing page is logged, not saved:

saved pages/www-law-cornell-edu-wex-contract.md (809 words, standard, js=False)
FAILED https://docs.python.org/3/library/no-such-page.html: 404: page not found

How it compares with Requests and Playwright

Each method fetched the same 22 public pages whose robots.txt allows AI crawlers such as GPTBot and ClaudeBot. Only this script used proxies. A page was converted if its Markdown contained a phrase from the main content.

Method	Pages converted
This script with Decodo's Web Data API	20
Headless Chromium with Playwright	18
Plain Requests with markdownify	15

The script also converted every page that the other methods did. Most of its 20 conversions needed only the cheapest attempt.

Edge cases and common problems

• Prompt injection. Attackers can use alt text, display:none blocks, or white-on-white text to hide instructions for models. Treat page Markdown as data, and put it between clear delimiters. Don't trigger a tool call only because page text asks for it.
• Choosing pages to convert. The Web Data API is built for publicly accessible pages. Check each site's terms of service.

Final thoughts

The Decodo Web Data API returns Markdown with 1 parameter, and the script checks each result before saving the page.

Next steps:

• Convert a whole docs site with /crawl, which also returns Markdown. To list its URLs first, use /map.
• Send large URL lists as asynchronous requests through /task/batch.
• Give an agent /scrape as a tool from our Model Context Protocol (MCP) server.
• Load pages with the LangChain integration or llama-index-readers-decodo, and set auth_mode="token" to authenticate with your API key.

Start with the Web Data API free plan, and compare the script's Markdown with your current pipeline's output.