hybridforgelab

Posted on Oct 27

Turn any web page into clean Markdown for your LLM, without ignoring robots.txt

#llm #rag #python #webdev

Disclosure: we're Hybrid Forge Lab and we built Distill, the API in this post. There's a free plan and a no-signup demo, so you can check everything below yourself.

If you feed web pages to an LLM, most of what you send isn't content. Navigation, footers, cookie banners, inline scripts, tracking pixels and sidebars all cost tokens and add noise to your embeddings.

A real example: the Python tutorial page on Data Structures is 111,008 bytes of HTML. The main content as Markdown is 28,453 characters, about 7,100 tokens, and keeps the headings, the code blocks and the 31 links.

That's roughly a quarter of the size, and none of what's gone is anything your model needed.

What Distill does

You send a URL, or HTML you already have. You get back:

the page's main content as Markdown: headings, lists, GFM tables, fenced code blocks with their language;
metadata: title, author, published date, language and site name;
counts: words, characters and an approximate token count;
the links list, with relative URLs already made absolute.

Here's the GitHub Blog's "What is Git?" post, as returned by the production API today (trimmed):

{
  "status": "ok",
  "title": "What is Git? Our beginner’s guide to version control",
  "author": "Kedasha Kerr",
  "published_date": "2024-05-27",
  "language": "en-US",
  "site_name": "The GitHub Blog",
  "markdown": "What is Git? Our beginner’s guide to version control\n\nLet’s get you started on your Git journey…",
  "truncated": true,
  "took_ms": 1825
}

Quick start

Through RapidAPI, with your key:

curl -G "https://distill-html-to-markdown-api.p.rapidapi.com/v1/markdown" \
  --data-urlencode "url=https://en.wikipedia.org/wiki/Markdown" \
  -d include_links=false -d max_chars=4000 \
  -H "X-RapidAPI-Key: YOUR_RAPIDAPI_KEY" \
  -H "X-RapidAPI-Host: distill-html-to-markdown-api.p.rapidapi.com"

The options you'll actually use:

max_chars cuts at a paragraph or word boundary, closes any open code fence and sets truncated: true;
include_links (default on) and include_images (default off);
include_tables (default on) keeps tables as GFM.

Already have the HTML, for example from your own crawler or a headless browser? POST it with a base_url, and the relative links come back resolved.

A small RAG ingest loop in Node.js

const url = new URL("https://distill-html-to-markdown-api.p.rapidapi.com/v1/markdown");
url.search = new URLSearchParams({ url: "https://docs.python.org/3/library/json.html", include_links: "false" });

const res = await fetch(url, {
  headers: {
    "X-RapidAPI-Key": process.env.RAPIDAPI_KEY,
    "X-RapidAPI-Host": "distill-html-to-markdown-api.p.rapidapi.com",
  },
});
if (!res.ok) throw new Error(`Distill ${res.status}`); // 401, 422, 429…
const page = await res.json();
if (page.status !== "ok") console.warn(page.status, page.detail); // robots_txt, blocked…
else chunks.push(...page.markdown.split("\n## ").map((s) => ({ text: s, url: page.final_url, title: page.title })));

Statuses, not exceptions

A page you can't use isn't an error in your code: it's an answer. Distill always returns HTTP 200 with a status:

status	what happened
`ok`	Markdown extracted
`no_content`	the page loaded, but there's no main content (often a JavaScript-built page)
`robots_txt`	robots.txt disallows the path
`opted_out`	the page says `noai`
`blocked`	the site's bot protection said no
`not_html`	PDF, image, JSON…
`http_error` / `unreachable`	the site errored or couldn't be reached

HTTP errors are kept for your mistakes (422 for a bad URL) and for limits (429).

What we refuse to do

This is the part we care about most. Scraping for AI has a bad reputation, mostly earned.

robots.txt is checked on every redirect hop, and if it answers 5xx we treat the path as disallowed (RFC 9309).
AI-crawler groups are honoured too. If a site blocks GPTBot, ClaudeBot, CCBot, Google-Extended or a dozen others, we don't fetch it either.
noai opt-outs in X-Robots-Tag or <meta name="robots"> are respected.
We identify ourselves as DistillBot/1.0 with a bot page. We never pretend to be a browser, and we never retry a bot wall from somewhere else.
At most 10 fresh fetches per minute per site, across all customers. Results are cached for an hour, so popular pages don't get hammered.

Limits, honestly

JavaScript isn't run. Pages that build their content in the browser come back as no_content. If you already render them, POST the HTML.
Pages are read up to 5 MB, with a 20-second total budget per request.
Extraction is heuristic (we build on trafilatura), so some layouts come out imperfect.

Try it

No signup: the demo on distill.hybridforgelab.com runs the real API in your browser.
Free plan: Basic is $0 for 15 requests a month, with a hard limit, so it never runs up a bill. Paid plans start at $9 for 10,000 requests.
Docs and plans: RapidAPI listing.

If a page comes out wrong, send us the URL: those reports are how the extraction gets better.

DEV Community