Turning a web page into LLM-ready Markdown with one HTTP call

#rag #ai #python #api

Feeding raw HTML to a language model wastes tokens on markup that carries no meaning. A typical documentation page is 180 KB of HTML and roughly 6 KB of actual prose. The rest is <script>, inline CSS, navigation, cookie banners, and analytics beacons that inflate your context window and dilute retrieval quality.

The fix is to convert the page to Markdown server-side and send only the content. Here is the exact request against the MESSORA API, including the failure modes you have to handle in production.

The request

Authentication uses the X-API-Key header. There is no Authorization: Bearer variant — a bearer token returns 401.

import os
import requests

API = "https://api.messora.dev"


def to_markdown(url: str) -> str:
    resp = requests.post(
        f"{API}/scrape",
        headers={"X-API-Key": os.environ["MESSORA_API_KEY"]},
        json={
            "url": url,
            "formats": ["markdown"],
            "only_main_content": True,
        },
        timeout=90,
    )
    resp.raise_for_status()
    data = resp.json()

    if data["scrape_status"] != "success":
        raise RuntimeError(f"{url} -> {data['scrape_status']}: {data.get('error')}")

    return data["markdown"]


print(to_markdown("https://docs.python.org/3/library/asyncio-task.html")[:400])

only_main_content drops header, footer, nav, and sidebar nodes before conversion. Leave it off when you actually need the chrome — comparing navigation structures across competitor sites, for example.

What comes back

{
  "success": true,
  "scrape_status": "success",
  "markdown": "# asyncio — Task object\n...",
  "rawHtml": null,
  "json": null,
  "metadata": {
    "url": "https://docs.python.org/3/library/asyncio-task.html",
    "title": "Coroutines and Tasks",
    "statusCode": 200,
    "fetchedAt": "2026-09-02T13:04:11.882Z"
  },
  "credits_used": 1,
  "remaining_credits": 987
}

formats accepts markdown, raw, and json. Asking for raw alongside markdown populates rawHtml in the same response, which is useful when you want to keep the original for auditing and the Markdown for embedding.

The four statuses you must branch on

scrape_status is an enum with exactly four values. Treating a 200 OK as a successful scrape is the most common integration bug, because the HTTP layer succeeded while the fetch did not.

`scrape_status`	Meaning	Sensible reaction
`success`	Content extracted, 1 credit charged	Use `markdown`
`blocked_antibot`	Target refused the fetch	Retry later or drop the URL
`timeout`	Page exceeded the `timeout` budget	Raise `timeout`, or set `render_js: false`
`extraction_failed`	Fetch worked, conversion produced nothing usable	Fall back to `raw` and parse manually

Credits are only debited on success, so a blocked page costs nothing. That matters when you are crawling a domain of unknown difficulty and want to size the job before committing budget.

HTTP errors worth handling separately

STATUS_ACTIONS = {
    401: "X-API-Key header missing",
    402: "monthly credit balance exhausted",
    403: "API key invalid or revoked",
    422: "malformed payload or SSRF-blocked URL",
    429: "rate limit exceeded (10 req/min on /scrape)",
}


def scrape_with_diagnostics(url: str) -> dict:
    resp = requests.post(
        f"{API}/scrape",
        headers={"X-API-Key": os.environ["MESSORA_API_KEY"]},
        json={"url": url, "formats": ["markdown"], "only_main_content": True},
        timeout=90,
    )
    if resp.status_code in STATUS_ACTIONS:
        raise RuntimeError(f"{resp.status_code}: {STATUS_ACTIONS[resp.status_code]}")
    resp.raise_for_status()
    return resp.json()

The distinction between 402 and 429 drives very different retry logic. A 429 clears on its own within the minute, so backing off works. A 402 means the monthly balance is gone and retrying just burns requests — that one needs to page a human or switch plans.

Caching with max_age_ms

Re-scraping a page that has not changed is wasted credit. max_age_ms returns a cached copy when one exists within the window:

json={"url": url, "formats": ["markdown"], "max_age_ms": 86_400_000}

A cache hit still reports scrape_status: "success" and returns the stored Markdown. Use a wide window for reference documentation and a narrow one for pricing pages or anything with a timestamp your users will notice.

Token math

For a RAG index over 5,000 documentation pages, the difference between raw HTML and cleaned Markdown at roughly 4 characters per token is about 220 million tokens versus 7.5 million. That gap shows up twice: once at embedding time, and again on every retrieval that stuffs a chunk into the prompt.

Cleaning at fetch time is cheaper than cleaning at inference time, and it happens once instead of on every query.

DEV Community