Convert any web page to LLM-ready Markdown

Many applications need real-time web access to deliver relevant information, especially those powered by large language models with static knowledge bases is generally frozen in time. Web HTML is typically noisy, cluttered with ads and navigation elements that inflate token usage and reduce LLM performance.

Gaffa’s Browser Request API is a simple REST interface for web automation and data extraction. Its generate_markdown action converts web pages into clean, LLM-ready markdown by removing clutter to reduce token usage whilst preserving key content.

HTML to Markdown Converter from Gaffa.dev

Convert any web page to clean markdown with this HTML to Markdown converter tool - powered by Gaffa's web scraping and proxy tech.

html2markdown.gaffa.dev

Converting a Web Page to Markdown

To extract a clean and readable markdown from a web page using Gaffa, you send a POST request to the /v1/browser/requests endpoint with the following JSON payload:

Generate Markdown Browser Request

{
  "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": false,
    "actions": [
      {
        "type": "wait",
        "selector": ".mw-content-container",
        "timeout": 10000,
        "continue_on_fail": true
      },
      {
        "type": "generate_markdown"
      }
    ]
  }
}

This instructs Gaffa to wait for the main content area to load and then generate markdown from it. This is the response:

Generate Markdown Response

{
    "data": {
        "id": "brq_VEmLpa5MAzkw6CWRhgU7seEbyFf2hB",
        "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
        "proxy_location": null,
        "state": "completed",
        "credit_usage": 1,
        "http_status_code": 200,
        "from_cache": false,
        "started_at": "2025-04-24T08:41:55.0836318Z",
        "completed_at": "2025-04-24T08:42:04.1878335Z",
        "running_time": "00:00:08.2352566",
        "page_load_time": "00:00:01.7245402",
        "actions": [
            {
                "id": "act_VEmLpahbSR8WRrwqiEKcaxjxitxxDQ",
                "type": "wait",
                "query": "wait?selector=.mw-content-container&timeout=10000&continue_on_fail=true",
                "timestamp": "2025-04-24T08:41:56.8081771Z"
            },
            {
                "id": "act_VEmLpXo1vxLMdCFQk3Pr6XBABUEBEn",
                "type": "generate_markdown",
                "query": "generate_markdown?continue_on_fail=false",
                "timestamp": "2025-04-24T08:41:57.1383759Z",
                "output": "https://storage.gaffa.dev/brq/md/brq_VEmLpa5MAzkw6CWRhgU7seEbyFf2hB/act_VEmLpXo1vxLMdCFQk3Pr6XBABUEBEn.md"
            }
        ]
    },
    "error": null
}

Gaffa returns a list of processed actions, with the generate_markdown action containing a markdown file that includes only meaningful content. The generate_markdown action removes lots of HTML information like styles, JavaScript, and headers that aren't relevant to the core content of the page. You can read more about this action in the Gaffa docs.

Building a Simple Python CLI Tool

To show how you can integrate Gaffa's markdown into your workflow, we built a simple Python CLI tool that:

1. Takes a URL input from the user
2. Sends a POST request to Gaffa, which will load the URL on our fleet of headless cloud browsers and return markdown, giving the output as a link.
3. Sends the resulting markdown to OpenAI’s API to enable question answering over the content.

You can find the complete Python implementation on GitHub if you'd like to build or modify the example yourself.

Try It Yourself

You can experiment with generate_markdown in the Gaffa API Playground by sending a URL and inspecting the Markdown returned.

The clean Markdown can then be passed to an LLM, used in a RAG pipeline, summarised, indexed, or used by an AI agent—without the unnecessary noise of raw HTML.

Frequently Asked Questions

What does Gaffa's generate_markdown action do?

It converts any web page into clean, LLM-ready Markdown by stripping ads, navigation, scripts, and styling, leaving only the meaningful content while reducing token usage.

Why use Markdown instead of raw HTML when feeding content into an LLM?

Raw HTML is cluttered with elements that inflate token counts and reduce LLM performance. Markdown removes that noise while preserving the core content, making it cheaper and more accurate to process.

Can generate_markdown handle pages that load content dynamically with JavaScript?

Yes. Gaffa renders pages in a real browser before generating Markdown, so JavaScript-loaded content is fully included, unlike simple HTTP requests that only capture the initial HTML shell.

How do I handle pages where content takes time to load?

Add a wait action before generate_markdown, specifying a CSS selector for the content area. Gaffa will pause until that element appears before generating the markdown.

Can I use the generated markdown to ask questions about a web page?

Yes. A common pattern is to fetch the markdown with Gaffa and pass it directly to an LLM API like OpenAI, allowing you to ask questions about the content of any live web page.

How do I access geo-restricted pages when generating markdown?

Set the proxy_location parameter in your request payload to a supported region, such as "us". Gaffa automatically routes the request through a residential proxy in that location.