Raw HTML is a poor thing to feed a model. A typical docs page is mostly navigation, scripts, styles and wrapper divs, and the actual text is a fraction of the bytes. Markdown keeps what matters (headings, lists, links, tables, code) and drops the rest, which makes it a good format for prompts, RAG chunks and chatbot knowledge files.
This post builds a small HTML-to-Markdown converter on Python's built-in html.parser, with no pip install. It reads a local file or a URL, skips page chrome, keeps tables and code blocks, and turns relative links into absolute ones.
Disclosure: Appie.run publishes this post. Appie has free HTML and webpage to Markdown converters, mentioned near the end. The script below does not depend on them.
Earlier posts in this series cover converting PDFs to Markdown for RAG and converting Notion to Markdown.
What a useful converter has to do
-
Drop the chrome.
<head>,<script>,<style>,<nav>,<header>,<footer>,<aside>, forms and inline SVGs add noise and no meaning. -
Catch navigation that is not a
<nav>. Many sites build sidebars from plain<div>s markedrole="navigation". -
Keep structure: headings, nested lists, bold and italic, inline code and
<pre>blocks. -
Make links absolute.
/docs/configmeans nothing once the text is pasted into a prompt. - Keep tables as tables, with pipes inside cells escaped.
- Collapse whitespace without breaking code blocks.
The script
Save this as html2md.py. It was tested with Python 3.13.5 on the cases below.
#!/usr/bin/env python3
"""Convert an HTML file or URL to Markdown with the standard library only."""
import sys
import urllib.request
from html.parser import HTMLParser
from urllib.parse import urljoin
SKIP = {"head", "script", "style", "nav", "footer", "header", "aside", "form", "noscript", "svg"}
SKIP_ROLES = {"navigation", "banner", "contentinfo", "search"}
BLOCK = {"p", "div", "section", "article", "main", "br", "table", "ul", "ol"}
class MarkdownParser(HTMLParser):
def __init__(self, base_url=""):
super().__init__(convert_charrefs=True)
self.base = base_url
self.out = []
self.skip_tag = None # tag that opened the skipped block
self.skip_count = 0 # nesting of that tag inside the block
self.lists = [] # stack of "ul" / "ol" with counters
self.href = None
self.link_text = []
self.in_pre = False
self.row = None
self.rows = []
def emit(self, text):
if self.href is not None:
self.link_text.append(text)
elif self.row is not None:
self.row[-1] += text
else:
self.out.append(text)
def handle_starttag(self, tag, attrs):
a = dict(attrs)
if self.skip_tag:
if tag == self.skip_tag:
self.skip_count += 1
return
if tag in SKIP or a.get("role") in SKIP_ROLES:
if tag not in {"img", "br", "input", "meta", "link"}:
self.skip_tag, self.skip_count = tag, 1
return
if tag in {"h1", "h2", "h3", "h4", "h5", "h6"}:
self.out.append("\n\n" + "#" * int(tag[1]) + " ")
elif tag in BLOCK:
if tag == "br":
self.out.append(" \n")
elif tag in {"ul", "ol"} and self.lists:
self.out.append("") # nested list: no blank line
else:
self.out.append("\n\n")
if tag in {"ul", "ol"}:
self.lists.append([tag, 0])
elif tag == "li":
kind = self.lists[-1] if self.lists else ["ul", 0]
kind[1] += 1
indent = " " * (len(self.lists) - 1)
marker = f"{kind[1]}." if kind[0] == "ol" else "-"
self.out.append(f"\n{indent}{marker} ")
elif tag in {"strong", "b"}:
self.emit("**")
elif tag in {"em", "i"}:
self.emit("*")
elif tag == "code" and not self.in_pre:
self.emit("`")
elif tag == "pre":
self.in_pre = True
self.out.append("\n\n" + "`" * 3 + "\n")
elif tag == "a" and a.get("href"):
self.href = urljoin(self.base, a["href"])
self.link_text = []
elif tag == "img" and a.get("src"):
self.emit(f"})")
elif tag == "tr":
self.row = []
elif tag in {"td", "th"} and self.row is not None:
self.row.append("")
def handle_endtag(self, tag):
if self.skip_tag:
if tag == self.skip_tag:
self.skip_count -= 1
if self.skip_count == 0:
self.skip_tag = None
return
if tag in {"ul", "ol"} and self.lists:
self.lists.pop()
self.out.append("\n")
elif tag in {"strong", "b"}:
self.emit("**")
elif tag in {"em", "i"}:
self.emit("*")
elif tag == "code" and not self.in_pre:
self.emit("`")
elif tag == "pre":
self.in_pre = False
self.out.append("\n" + "`" * 3 + "\n\n")
elif tag == "a" and self.href is not None:
text = "".join(self.link_text).strip() or self.href
href, self.href = self.href, None
self.emit(f"[{text}]({href})")
elif tag == "tr" and self.row is not None:
self.rows.append([c.strip().replace("|", "\\|") for c in self.row])
self.row = None
elif tag == "table" and self.rows:
width = max(len(r) for r in self.rows)
rows = [r + [""] * (width - len(r)) for r in self.rows]
lines = ["| " + " | ".join(rows[0]) + " |",
"|" + " --- |" * width]
lines += ["| " + " | ".join(r) + " |" for r in rows[1:]]
self.out.append("\n\n" + "\n".join(lines) + "\n\n")
self.rows = []
elif tag in {"h1", "h2", "h3", "h4", "h5", "h6", "p"}:
self.out.append("\n\n")
def handle_data(self, data):
if self.skip_tag:
return
if self.in_pre:
self.out.append(data)
return
text = " ".join(data.split())
if text:
# keep a space between inline pieces
if data[:1].isspace():
text = " " + text
if data[-1:].isspace():
text += " "
self.emit(text)
def markdown(self):
md = "".join(self.out)
lines = [line.rstrip() if not line.endswith(" ") else line for line in md.splitlines()]
cleaned, blank = [], 0
for line in lines:
if line.strip() == "":
blank += 1
if blank > 1:
continue
else:
blank = 0
line = line if line.startswith((" ", "|")) else line.lstrip()
cleaned.append(line)
return "\n".join(cleaned).strip() + "\n"
def load(source):
if source.startswith(("http://", "https://")):
req = urllib.request.Request(source, headers={"User-Agent": "html2md/1.0"})
with urllib.request.urlopen(req, timeout=20) as resp:
charset = resp.headers.get_content_charset() or "utf-8"
return resp.read().decode(charset, errors="replace"), source
with open(source, encoding="utf-8", errors="replace") as f:
return f.read(), ""
def main():
if len(sys.argv) < 2:
sys.exit("usage: html2md.py <file.html | URL> [base_url]")
html, base = load(sys.argv[1])
if len(sys.argv) > 2:
base = sys.argv[2]
parser = MarkdownParser(base)
parser.feed(html)
parser.close()
sys.stdout.write(parser.markdown())
if __name__ == "__main__":
main()
Usage:
python3 html2md.py page.html https://example.com/docs/start # local file, with a base URL for links
python3 html2md.py https://docs.python.org/3/library/html.parser.html > html-parser.md
How it works
-
HTMLParseris event based. It callshandle_starttag,handle_dataandhandle_endtagas it walks the document, and tolerates broken markup.convert_charrefs=Trueturns&and friends into plain characters. -
Skipping by element. When a tag in
SKIPopens, or any element has aroleinSKIP_ROLES, the parser records that tag name and counts nested tags of the same name. Everything is ignored until the matching close tag brings the count back to zero. Counting matters: a sidebar<div>usually contains more<div>s, and closing on the first</div>would leak the rest of the sidebar into your output. - Lists use a stack, so nested lists get two spaces of indentation per level and ordered lists keep their own counters.
-
Links are buffered: the text inside
<a>is collected and written as[text](absolute-url)on</a>.urljoinresolves relative paths against the base URL, which is the page URL when you pass a URL. -
Tables are collected row by row and written at
</table>. Short rows are padded so every row has the same number of cells, and the first row becomes the header. -
<pre>switches off whitespace collapsing and wraps the block in a fence. -
markdown()trims trailing spaces (keeping the two-space hard line break), removes repeated blank lines and returns the result.
Test it
A small page with every awkward part:
<html><head><title>Docs</title><style>body{}</style><script>var x=1;</script></head>
<body><nav><a href="/">Home</a> <a href="/pricing">Pricing</a></nav>
<div class="sidebar" role="navigation"><div><a href="/docs/a">Page A</a></div><a href="/docs/b">Page B</a></div>
<main>
<h1>Getting started</h1>
<p>Install the <strong>CLI</strong>, then read the <a href="/docs/config">config guide</a>.</p>
<h2>Plans</h2>
<table><tr><th>Plan</th><th>Seats</th></tr><tr><td>Free</td><td>1</td></tr><tr><td>Team | Pro</td><td>10</td></tr></table>
<ul><li>Fast setup</li><li>Works with <em>any</em> site<ol><li>Step one</li><li>Step two</li></ol></li></ul>
<pre><code>pip install example
example init</code></pre>
<img src="/img/flow.png" alt="Flow diagram">
</main>
<footer>Copyright 2026</footer></body></html>
python3 html2md.py sample.html https://example.com/docs/start
Output:
# Getting started
Install the **CLI**, then read the [config guide](https://example.com/docs/config).
## Plans
| Plan | Seats |
| --- | --- |
| Free | 1 |
| Team \| Pro | 10 |
- Fast setup
- Works with *any* site
1. Step one
2. Step two
```
pip install example
example init
```
The title, styles, script, both navigation blocks and the footer are gone, the nested sidebar <div> did not leak, the pipe in "Team | Pro" stayed inside its cell and every link is absolute.
On a real page, the Python docs page for html.parser was about 55 KB of HTML when we fetched it and came out at about 14.5 KB of Markdown. Your ratio will depend on the site.
Known limits (and how to extend it)
-
JavaScript-rendered pages come back nearly empty, because
urllibonly sees the initial HTML. Render them with a headless browser first and pass the saved HTML to the script. -
Header anchor links such as the
¶symbols on Sphinx docs survive as links. Skip<a>tags withclass="headerlink"if they bother you. -
Inline spacing around some tags can be off by one space (for example
**Source code:**[link]). It rarely matters to a model, but check output that people will read. -
Complex tables with
colspan,rowspanor nested tables are flattened. Markdown tables cannot represent them, so consider keeping those as HTML. -
Definition lists, footnotes and
<details>are treated as plain text. -
Pages without semantic markup may still include menus. If a site puts its content in
<main>or<article>, an easy extension is to only emit text while inside those elements. - Respect the site. Fetch pages you are allowed to use, keep request rates low, and check the site's terms before converting content you do not own.
Is Markdown the right format?
For prompts and chatbot knowledge, usually yes: headings give chunkers natural split points, links stay readable and tables survive. Plain text is smaller but loses structure; raw HTML keeps structure but spends most of its size on markup. If you are building a RAG index, split the Markdown on ## headings and keep the page URL with each chunk so answers can cite it.
When you do not want to run a script
Appie has two free browser tools for one-off conversions:
- Convert HTML to Markdown: upload an HTML file or paste markup. Headings, lists, links, images and tables are converted, and the page says input is not saved to an account.
- Convert a Webpage to Markdown: enter a public URL. Links come out absolute and navigation and scripts are left out. For private pages, the page suggests pasting the HTML into the HTML converter instead.
Each run asks for a Cloudflare Turnstile check. For repeatable jobs, many pages or anything behind a login, the script is the better fit.
If the Markdown is going into a chatbot, Appie's AI Chatbots app can also take URLs and sitemaps as knowledge directly, and you can add content from Claude Code or Cursor through Appie's MCP server, as shown in this MCP walkthrough.
Honest limits of Appie's converters
- One page or file per run, with a fresh captcha each time. There is no batch mode on the tool pages.
- Public URLs only for the webpage tool.
-
No options for what to skip are described on the pages; the script lets you edit
SKIPandSKIP_ROLESyourself.
FAQ
Why not BeautifulSoup plus markdownify?
They are good libraries and handle more edge cases. The standard-library version suits small containers, CI jobs and serverless functions where adding dependencies is a hassle, and it is short enough to read and change.
How do I convert a whole site?
Get the URL list from the sitemap, loop over it with a delay between requests, and write one .md file per URL. Our earlier sitemap posts on this account show how to pull URLs from a sitemap.
Does it handle non-UTF-8 pages?
For URLs it uses the charset from the Content-Type header and falls back to UTF-8, replacing undecodable bytes.
Can I keep the page title?
Read <title> separately (it is skipped with <head>) and write it as a # heading at the top if the page has no <h1>.
If you want a website chatbot trained on your pages, you can try Appie's AI Chatbots on the Free plan (500 credits, no credit card) at appie.run.
RG, Founder, Appie.run
Top comments (0)