My Smallest HTML-to-Markdown Pipeline—and What I'd Check Next

I start a website-to-Markdown workflow with one page and an explicit acceptance check. The converter is only one component: acquisition, main-content selection, link handling, and provenance all affect the final document.

Here is a deliberately small static-page baseline. It does not crawl recursively or handle authenticated pages.

Fetch, select, convert

Install the dependencies in a virtual environment:

python -m pip install httpx beautifulsoup4 markdownify

Save the following as page_to_markdown.py:

from datetime import datetime, timezone
from hashlib import sha256
from urllib.parse import urlparse

import httpx
from bs4 import BeautifulSoup
from markdownify import markdownify as to_markdown


def page_to_markdown(url: str) -> dict[str, str]:
    parsed = urlparse(url)
    if parsed.scheme not in {"http", "https"}:
        raise ValueError("Only HTTP(S) URLs are allowed")

    with httpx.Client(
        timeout=15.0,
        follow_redirects=True,
        headers={"User-Agent": "ContentResearchExample/1.0"},
    ) as client:
        response = client.get(url)
        response.raise_for_status()

    content_type = response.headers.get("content-type", "").lower()
    if "text/html" not in content_type:
        raise ValueError(f"Expected HTML, got {content_type}")

    soup = BeautifulSoup(response.text, "html.parser")
    for node in soup.select("script, style, nav, footer, aside, form"):
        node.decompose()

    main = soup.select_one("main") or soup.select_one("article") or soup.body
    if main is None:
        raise ValueError("No content container found")

    title = soup.title.get_text(" ", strip=True) if soup.title else "Untitled"
    markdown = to_markdown(str(main), heading_style="ATX").strip()
    if len(markdown) < 40:
        raise ValueError("Extracted content is unexpectedly short")

    return {
        "source_url": url,
        "final_url": str(response.url),
        "title": title,
        "fetched_at": datetime.now(timezone.utc).isoformat(),
        "content_hash": sha256(markdown.encode()).hexdigest(),
        "markdown": markdown,
    }


if __name__ == "__main__":
    record = page_to_markdown("https://example.com/")
    print(record["title"])
    print(record["markdown"][:300])

Run it with python page_to_markdown.py. The example requests one public demonstration page. That is enough to exercise the basic path, not enough to establish fidelity across a real site's layouts.

The HTTPX quickstart and markdownify project are useful references for the two library boundaries.

The broad cleanup rule is where I'd intervene

Deleting every aside is a demonstration shortcut. On documentation pages, an aside may contain a warning or prerequisite. Removing all forms can also remove meaningful content in unusual layouts.

For a real source, I inspect the DOM and choose a specific wrapper or reviewed extraction strategy. I test the first and last paragraphs, heading hierarchy, code indentation, and a critical table.

The short-output guard only catches obvious emptiness. It does not catch a long newsletter modal or a recommendation section mistakenly selected as the article.

Links need another pass

This baseline does not rewrite relative links. Before storing production documents, I resolve them against the final response URL and any applicable HTML base URL. I preserve useful image alt text and links where the image carries information.

A Markdown table can be syntactically valid while losing row or column semantics. I compare complex tables separately rather than relying on a parser's success return.

I would expand collection after validation

For multiple pages, I start with a reviewed list, bounded concurrency, per-host request limits, explicit failures, and durable writes. If source text is absent from the HTTP response, I would diagnose rendering before adding a browser path.

I would store source and final URLs, fetch time, content hash, representation type, and extraction version alongside each file. A retrieval index should inherit this metadata rather than receive anonymous chunks.

This function is also not a safe public URL-fetching endpoint. A service accepting arbitrary user URLs needs network restrictions and redirect validation. Here, I control the demonstration URL.

My next milestone would be four reviewed fixtures: an article, a code tutorial, a table reference, and a short page. Once those preserve their important content, I would scale the collection and only then build chunking or search on top.