DEV Community: Victory Nnaji

Turn Web Content Into Structured Data With Parse_JSON

Victory Nnaji — Mon, 19 Oct 2026 12:00:00 +0000

Traditional parsing libraries can only extract what's explicitly in the HTML structure. But what if you need to summarise content? Or convert unstructured text into a specific data format? That's where AI-based extraction becomes important.

Gaffa’s parse_json action uses AI to understand web content semantically, not structurally. Instead of using selectors that break with every design change, you simply describe what data you want, and the AI extracts it, even from unstructured content.

In this post, we’ll explore what parse_json does, how it works, and why it’s a powerful building block for web automation, data extraction, and AI-driven workflows.

The challenge of extracting unstructured data from the web

Most web scrapers are built using CSS selectors, and they work perfectly until the website design changes. Then your scraper breaks, often without warning, because companies don't announce site updates to scrapers (unfortunately!).

Industry data shows that developers spend 10-20% of their scraping time just fixing broken selectors. It's a maintenance nightmare that scales poorly as you scrape more sites.

Selectors aren’t the only challenge. The data you need might not be structured in an element. Take a job posting scenario where the salary info is buried inside a paragraph. "The position offers $80-120K annually depending on experience." In such a case, extracting those specific numbers requires a complex regex pattern, which is unreliable and may break with slight changes in wording. An alternative to this is manual extraction, which is itself costly and tedious.

PDF extraction is even harder. Tables span pages. Text flows in unpredictable ways. You need specialised libraries that could still miss content or fail in unexpected ways.

parse_json takes a completely different approach. Gaffa extracts the rendered page content, sends it to a language model along with your schema definition, and the AI structures the data according to your specifications. There are no selectors to maintain if the website layout changes - if the content remains, your extraction still works.

Using parse_json

To extract structured data from a web page or PDF, you send a POST request to the /v1/browser/requests endpoint with a JSON payload containing your schema definition.

Here's the basic structure

parse_json basic example

{
    "url": "https://demo.gaffa.dev/simulate/form?loadTime=3&showModal=true&modalDelay=5&formType=address",
    "async": false,
    "settings": {
        "record_request": false,
        "actions": [
            {
            "type": "parse_json",
            "data_schema": {
                // Your schema definition goes here
            },
            "model": "gpt-4o-mini",
            "output_type": "inline"
            }
        ]
    }
  }

A schema tells parse_json what data to extract and how to structure it. Every schema has three key components:

name: A descriptive identifier for your schema
description: Helps the AI understand what you're extracting
fields: Array of fields defining your data structure

Here's an example using the Gaffa demo site, where Gaffa analyses the form and tells you exactly which fields are present, their types, whether they're required, and their validation rules. This is incredibly useful for form automation workflows. For example, you could build a tool that analyses any web form to identify all required fields, fills them, and submits the form — like this one.

Auto-filling web forms with parse_json

parse_json address form

{
  "url": "https://demo.gaffa.dev/simulate/form?loadTime=3&showModal=true&modalDelay=5&formType=address",
  "async": false,
  "settings": {
    "record_request": false,
    "actions": [
      {
        "type": "wait",
        "selector": "form",
        "timeout": 10000
      },
      {
        "type": "parse_json",
        "data_schema": {
          "name": "FormFields",
          "description": "Structured representation of all visible fields in the address form.",
          "fields": [
            {
              "type": "string",
              "name": "form_title",
              "description": "The visible title displayed at the top of the form."
            },
            {
              "type": "array",
              "name": "fields",
              "description": "A collection of all visible input fields in the form, including text, number, telephone, and select elements.",
              "fields": [
                {
                  "type": "string",
                  "name": "label",
                  "description": "The human-readable label shown to the end user (e.g., 'First Name', 'Phone Number')."
                },
                {
                  "type": "string",
                  "name": "field_name",
                  "description": "The machine-readable 'name' attribute of the input element, used as the JSON key when submitting data programmatically."
                },
                {
                  "type": "string",
                  "name": "field_type",
                  "description": "The inferred logical data type of the field based on the HTML input type. Used to determine expected input format.",
                  "enum": ["string", "number"]
                },
                {
                  "type": "boolean",
                  "name": "required",
                  "description": "Indicates whether the field is required. Fields marked with an asterisk (*) in the UI should be set to true."
                }
              ]
            }
          ]
        },
        "instruction": "Extract all visible form fields with their labels, input names, inferred data types, required status, and placeholders.",
        "model": "gpt-4o-mini",
        "output_type": "inline"
      }
    ]
  }
}

In the examples above, we used "gpt-4o-mini", a lightweight model, and in our testing, it performs very well on structured extraction tasks such as form parsing and document analysis. The parse_json action lets you select the model for your use case. We’re also continuing to expand model support, so you’ll have even more options as the platform evolves.

This request returns a response containing all the requested information. No selectors. No HTML parsing. Simply describe what you want, and receive structured data in return.

output

{
    "data": {
        "id": "brq_VeMRv3knmWLUkXBSwy6JJgbBqV6gRU",
        "url": "https://demo.gaffa.dev/simulate/form?loadTime=3&showModal=true&modalDelay=5&formType=address",
        "state": "completed",
        "credit_usage": 0,
        "http_status_code": 200,
        "from_cache": false,
        "started_at": "2026-02-09T07:47:35.976084Z",
        "completed_at": "2026-02-09T07:47:46.9986663Z",
        "running_time": "00:00:11.0225823",
        "page_load_time": "00:00:00.4162253",
        "actions": [
            {
                "id": "act_VeMRvBCpeNAUqvW6927U1Hn6DY1FqT",
                "type": "wait",
                "timestamp": "2026-02-09T07:47:43.1450788Z"
            },
            {
                "id": "act_VeMRv8ToGnRhrmbt1phP9sMn2JqGCh",
                "type": "parse_json",
                "timestamp": "2026-02-09T07:47:46.9986446Z",
                "output": {
                    "form_title": "Form Submission Test",
                    "fields": [
                        {
                            "label": "First Name",
                            "field_name": "first_name",
                            "field_type": "text",
                            "required": true
                        },
                        {
                            "label": "Last Name",
                            "field_name": "last_name",
                            "field_type": "text",
                            "required": true
                        },
                        {
                            "label": "Email",
                            "field_name": "email",
                            "field_type": "email",
                            "required": true
                        },
                        {
                            "label": "Address Line 1",
                            "field_name": "address_line_1",
                            "field_type": "text",
                            "required": true
                        },
                        {
                            "label": "Address Line 2",
                            "field_name": "address_line_2",
                            "field_type": "text",
                            "required": false
                        },
                        {
                            "label": "City",
                            "field_name": "city",
                            "field_type": "text",
                            "required": true
                        },
                        {
                            "label": "State/Province",
                            "field_name": "state_province",
                            "field_type": "text",
                            "required": true
                        },
                        {
                            "label": "ZIP/Postal Code",
                            "field_name": "zip_postal_code",
                            "field_type": "text",
                            "required": true
                        },
                        {
                            "label": "Country",
                            "field_name": "country",
                            "field_type": "text",
                            "required": true
                        }
                    ]
                },
                "reference": "https://storage.gaffa.dev/brq/dom/brq_VeMRv3knmWLUkXBSwy6JJgbBqV6gRU/act_VeMRv8ToGnRhrmbt1phP9sMn2JqGCh_raw.txt"
            }
        ]
    }
}

One of the most important things to notice in the example above is that the output strictly follows the schema and instructions we provided. We didn’t tell Gaffa where elements lived in the DOM, nor did we hardcode selectors. All we did was define the fields we wanted, and the response followed those instructions. This schema-driven approach makes extraction easy. As long as the page still contains the information, your extraction holds up even when the site is redesigned.

What you can extract

Turning web content into structured data is a foundational capability for modern automation, and with parse_json, Gaffa provides a flexible, schema-driven way to extract meaningful data from real, rendered web pages and documents without the challenges of traditional scraping. Some other common use cases include:

E-commerce monitoring - Extract product names, prices, availability, and reviews in a consistent format, even as product pages evolve.
Content extraction and summarisation - Pull article titles, author information, and metadata, or ask the AI to generate summaries of page content in your desired format.
Structured data from tables - Convert HTML tables into clean JSON arrays, regardless of how they're styled or nested in the page structure.
Contact information extraction - Easily extract names, emails, phone numbers, and addresses from contact pages, even across sites with wildly different structures.

parse_json also works with PDF documents, making it easy to extract structured data from research papers, invoices, contracts, and reports without needing complex PDF parsing libraries.

Whether you’re automating form workflows, extracting data from PDFs, or building intelligent agents, parse_json helps bridge the gap between human-designed interfaces and machine-ready data.

For a real-world example of parse_json in production, see how we used it to build a personalised news plugin for TRMNL that turns any website into a feed on your e-ink display.

Why is Gaffa's JSON parsing different?

Unlike many JSON extraction tools that return loosely structured output, Gaffa allows you to rigidly define the schema and explicitly control the expected structure. The action validates the model’s output against your schema, ensuring predictable and reliable results.

Schemas can be defined inline for one-off requests, or saved to your account and reused across multiple workflows using the schema management endpoints. This makes it easy to standardise extraction logic across teams and projects.

The parse_json action is also priced to be accessible, especially when paired with lightweight models like "gpt-4o-mini," allowing you to balance cost and performance depending on your workload.

parse_json is one of Gaffa's most flexible actions and a powerful foundation for more advanced automation workflows. To see what else is possible, browse the full list of actions in the official documentation, or head to the API Playground to try pre-built requests and tweak them to suit your needs.

Frequently Asked Questions

What is Gaffa's parse_json action?

It's an AI-powered action that extracts structured JSON data from any web page or PDF. You define a schema describing what you want, and the AI extracts it, with no HTML selectors or custom scripts needed.

How is parse_json different from traditional web scrapers?

Traditional scrapers rely on CSS selectors that break when a site's layout changes. parse_json works semantically, so as long as the content is on the page, your extraction continues to work.

Can parse_json extract data from PDFs?

Yes. It works on PDFs as well as web pages, making it easy to pull structured data from invoices, contracts, research papers, and reports without specialised parsing libraries.

What AI models does parse_json support?

You can choose your model per request. Gaffa currently supports models such as gpt-4o-mini and is expanding its support over time, so you can balance cost and performance based on your workload.

How do I define what data parse_json should extract?

You write a schema with three parts: a name, a description, and a fields array. This schema is sent with your POST request to the /v1/browser/requests endpoint, specifying exactly what the AI should return.

What are the most common use cases for parse_json?

E-commerce monitoring, article metadata extraction, HTML table conversion, contact info scraping, PDF data extraction, and automated form analysis are the most popular use cases.

How to Extract & Simplify a Webpage DOM

Victory Nnaji — Mon, 12 Oct 2026 12:00:00 +0000

Building AI agents or AI-powered web automations, working with raw webpages can be difficult. Most HTML pages include useless elements that can become problematic when you are trying to extract meaningful data or feed clean content into an LLM. These elements include things like <style> tags, linked stylesheets, <script> blocks, tracking pixels, SVG icons, and empty or hidden nodes. These add weight to the raw DOM, increasing token usage and making LLM hallucinations more likely.

Gaffa uses a real browser to render pages by default, we call it headful as the opposite to headless which is the default with most other scraping or browser as a service products. This means the page loads exactly like it would on an actual device with JavaScript fully rendered and the DOM including all dynamic content, essential for scraping modern web apps.

Gaffa's Browser Request API has two actions that help you work with the DOM for a webpage, capture_dom and generate_simplified_dom.

capture_dom lets you capture the entire DOM of a webpage, including JavaScript-loaded content, which you can then process in your existing codebase or data extraction workflow
generate_simplified_dom distils that raw HTML down to just the useful parts, removing unnecessary tags, styles, scripts, and other clutter, leaving you with a clean, structured DOM ready for further processing

Simplifying the DOM is especially useful as it can reduce HTML size by 50 to 80 percent on some pages, saving thousands of tokens when feeding content into an LLM, without losing essential context. It's equally useful as input for further processing or data extraction in your scripts, such as extracting elements with Python and BeautifulSoup.

Using capture_dom to retrieve raw HTML

The capture_dom action lets you retrieve the complete Document Object Model (DOM) of a web page. This includes not just the static HTML served by the server, but also any JavaScript-generated content.

To retrieve raw HTML with fully rendered JavaScript from a web page using Gaffa, you send a POST request to the /v1/browser/requests endpoint with the following JSON payload.

Extracting the rendered DOM from a web page

{
  "url": "https://www.freecodecamp.org/news/how-to-contribute-to-open-source-handbook/",
  "proxy_location": "us",
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": false,
    "actions": [
      {
        "type": "capture_dom",
        "continue_on_fail": false
      }
    ]
  }
}

As a side note, Gaffa makes it easy to rotate IPs or geolocate requests using residential proxies. Just set the proxy_location field to your desired region, like "us", "de", or "sg".

This will return a .txt file containing the raw HTML DOM of the current page at the time the request is made, with this response.

Raw DOM response

{
    "data": {
        "id": "brq_VKi21MiGEKM6VMv5RKhXV1JQJychNr",
        "url": "https://www.freecodecamp.org/news/how-to-contribute-to-open-source-handbook/",
        "state": "completed",
        "credit_usage": 1,
        "http_status_code": 204,
        "from_cache": false,
        "started_at": "2025-06-24T07:58:09.4820891Z",
        "completed_at": "2025-06-24T07:58:19.8187492Z",
        "running_time": "00:00:09.4322002",
        "page_load_time": "00:00:04.8159109",
        "actions": [
            {
                "id": "act_VKi21LnenxQN2hoZs6hx1Li8X2ExUU",
                "type": "capture_dom",
                "query": "capture_dom?continue_on_fail=false",
                "timestamp": "2025-06-24T07:58:14.2980107Z",
                "output": "https://storage.gaffa.dev/brq/dom/brq_VKi21MiGEKM6VMv5RKhXV1JQJychNr/act_VKi21LnenxQN2hoZs6hx1Li8X2ExUU_raw.txt"
            }
        ]
    }
}

Getting the raw DOM means we are getting a lot of content including scripts, nav bars, stylesheets, sidebars, ads, and other elements that don't contribute to the actual content, and if sent to an LLM this will lead to extra token usage and potentially harm performance.

That's where the generate_simplified_dom action comes in.

Using generate_simplified_dom to clean the DOM

Using the generate_simplified_dom action, we can clean and distill the HTML down to just the essential parts. It removes elements that typically add unnecessary noise to the HTML without containing any meaningful information for later data extraction, helping you focus on what's valuable.

The JSON payload stays the same, the only change is to update the action type to match the current action.

Simplify DOM code

{
  "actions": [
    {
      "type": "generate_simplified_dom"
    }
  ]
}

This will output a cleaner version of the page's DOM to a .txt file, stripped of scripts, stylesheets, inline styles, and SVG paths. This reduction means fewer tokens used when feeding the data into an LLM, ideal for summarization, Q&A, or fine-tuning tasks. It could also be useful if training a model on the HTML structure of different sites.

Capturing and simplifying the DOM with Gaffa makes it easier to work with and extract data from fully rendered webpages. Depending on the task, capture_dom gives you the complete HTML structure of the rendered page, while generate_simplified_dom reduces it to a cleaner, more focused version that's easier to process.

These actions can also serve as building blocks for more advanced scraping and automation workflows.

Explore the full list of Gaffa actions in the official documentation
Experiment with pre-built requests in the API Playground and adapt them to your own use case

Frequently asked questions

How do I capture the full rendered HTML of a page using Gaffa?

Use the capture_dom action. It returns the complete DOM including JavaScript-loaded content, exactly as it appears in a real browser, not just the initial static HTML.

How do I get a clean, stripped-down version of a page's HTML?

Use the generate_simplified_dom action. It removes scripts, stylesheets, SVGs, tracking pixels, and empty nodes, leaving only the content that matters for extraction or LLM input.

Why does raw HTML cause problems when fed into an LLM?

Raw pages contain scripts, tracking pixels, and inline styles that add no useful context. This inflates token usage and increases the likelihood of LLM hallucinations.

How much can generate_simplified_dom reduce HTML size?

It can reduce HTML size by 50 to 80 percent on some pages, saving thousands of tokens without losing the essential content needed for extraction or LLM processing.

Does Gaffa render JavaScript before capturing the DOM?

Yes. Gaffa uses a headful browser by default, meaning pages load exactly as they would on a real device, with JavaScript fully executed and dynamic content included.

Can I use capture_dom output with my own parsing tools, such as BeautifulSoup?

Yes. Both capture_dom and generate_simplified_dom return a .txt file containing the HTML, which you can fetch and process with any library in your existing workflow.

How do I route requests through a specific country with Gaffa?

Set the proxy_location field in your request payload to your desired region, such as "us", "de", or "sg", and Gaffa routes the request through a residential proxy in that location.

How to Download Files From Any Website

Victory Nnaji — Mon, 05 Oct 2026 12:00:00 +0000

Downloading files from the web sounds simple until you try to do it at scale. You need a headless browser, proxy management, code to detect whether a file opens inline or triggers an automatic download, retry logic, and storage handling before you have written a single line of business logic. Gaffa's download_file action collapses all of that into a single API call. This guide covers three real-world download scenarios, how to handle each one, and how Gaffa compares to doing the same thing with a traditional tool like Playwright.

What is Gaffa's download_file action?

The download_file action tells Gaffa's browser to capture the most recently loaded file and return a link to it. It automatically handles both common download patterns: files that trigger an immediate download to disk, and files that open inline in the browser, such as a PDF rendered directly in a tab. Supported formats include .pdf, .jpg, .jpeg, .png, .gif, .bmp, .webp, .svg, .tiff, .tif, and .img. The maximum file size is 50MB.

Because Gaffa runs a real browser behind the scenes, it handles JavaScript rendering, cookie consent, and anti-bot measures without any additional configuration. You can also set _proxy_locatio_n at the request level to download files as if browsing from a specific country, which is useful for region-locked content.

Scenario 1: Downloading files — both direct and browser-rendered

Whether the file downloads automatically to disk or opens inline in the browser, the download_file action handles both patterns seamlessly without additional configuration.

Direct File Download (downloads to disk)

{
  "url": "https://demo.gaffa.dev/simulate/download?fileName=ReasoningAboutActionAndChange.pdf",
  "settings": {
    "actions": [
      {
        "type": "download_file",
        "timeout": 30000
      }
    ]
  }
}

Browser-Rendered File (opens inline)

{
  "url": "https://demo.gaffa.dev/simulate/pdf/ReasoningAboutActionAndChange.pdf",
  "settings": {
    "actions": [
      {
        "type": "download_file",
        "timeout": 30000
      }
    ]
  }
}

The optional timeout parameter sets how long Gaffa waits for the download to complete, in milliseconds. The default is 5,000ms (5 seconds); for larger files, you will want to increase this.

Gaffa returns the downloaded file as a hosted URL in the action output:

Download Action Response

{
  "actions": [
    {
      "id": "act_VHhrUbXjZSaYCPTqbBYD4acCzzeFGH",
      "type": "download_file",
      "output": "https://storage.gaffa.dev/brq/downloads/5845df07-3749-424e-9c64-9602be19a857.pdf"
    }
  ]
}

You can then fetch that URL, store it, or pipe it into another workflow.

Scenario 2: Submitting a form and downloading the result

Many workflows involve generating a file in response to a form submission, a report, a certificate, or an invoice. Gaffa handles the entire sequence in a single request: fill the form, click submit, download the resulting file.

Form Submission With PDF Download

{
  "url": "https://demo.gaffa.dev/simulate/form?loadTime=3&showModal=false&modalDelay=0&formType=address&firstName=John&lastName=Doe&address1=123%20Main%20Street&city=London&country=UK&pdfExport=true",
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "actions": [
      {
        "type": "type",
        "selector": "#email",
        "text": "johndoe@example.com",
        "timeout": 20000
      },
      {
        "type": "type",
        "selector": "#state",
        "text": "Brent",
        "timeout": 20000
      },
      {
        "type": "type",
        "selector": "#zipCode",
        "text": "12345",
        "timeout": 20000
      },
      {
        "type": "click",
        "selector": ".inline-flex",
        "timeout": 1000
      },
      {
        "type": "wait",
        "selector": "div.flex:nth-child(1)",
        "timeout": 1000,
        "continue_on_fail": true,
        "time": 1000
      },
      {
        "type": "download_file",
        "timeout": 1000
      }
    ]
  }
}

The click action submits the form by targeting the submit button via its CSS selector. Once the form is processed and the PDF is generated, download_file captures it. No polling, no callback handling, no separate request to retrieve the result. See the full Python example on GitHub.

How does this compare to doing it with Playwright?

Even for the simplest download scenario, the equivalent Playwright code requires launching a browser, configuring a context, listening for download events, waiting for the file to save, and handling storage before you have touched any proxy or retry logic. It will look something like this:

Playwright File Download (Python)

from playwright.sync_api import sync_playwright
import shutil

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    page = context.new_page()

    with page.expect_download() as download_info:
        page.goto("https://example.com/file.pdf")

    download = download_info.value
    download.save_as("/tmp/file.pdf")

    # Upload to storage
    # shutil.copy("/tmp/file.pdf", your_storage_destination)

    browser.close()

Adding a proxy to this requires passing proxy configuration into browser.new_context(). Parallelising downloads means managing multiple browser instances and their memory overhead.

With Gaffa, proxy support is a single top-level parameter (proxy_location), parallel downloads are handled by running concurrent API requests, and both inline and auto-download patterns use the same download_file interface.

Downloading files at scale

Gaffa is designed to handle volume. Higher-tier plans support multiple concurrent requests, so if you need to download hundreds of files, you can parallelise those requests across Gaffa's infrastructure rather than managing browser instances yourself.

If you're looking to download assets at scale, such as every image on a website, check out our guide on automatically scraping every image from a website. It walks through a practical approach for discovering and downloading large collections of files.

For other file types and more advanced extraction use cases, the parse_json action lets you extract structured data from any page rather than downloading the raw file. See our post on turning web content into structured data with parse_json for a detailed walkthrough.

Gaffa's download_file action handles the browser-side work involved in downloading files, including cases where files are served inline or triggered automatically by the page.

You can try the examples in the API Playground, or explore the download_file documentation for more details.

Frequently asked questions

What file types does download_file support?

The action supports .pdf, .jpg, .jpeg, .png, .gif, .bmp, .webp, .svg, .tiff, .tif, and .img. The maximum file size is 50MB.

Does it matter if the file downloads automatically or opens in the browser?

No. The download_file action automatically handles both patterns without requiring additional configuration or wait actions.

How do I download files from a specific country or region?

Set proxy_location at the top level of your request to route the browser to a specific geographic location, useful for region-locked content or testing regional behaviour.

Can I run multiple downloads at the same time?

Yes. Gaffa supports concurrent download requests, with the number of simultaneous jobs depending on your plan.

Is there a file size limit?

Yes, the current maximum is 50MB per file. Contact support to discuss higher limits for your use case.

What happens if a download fails?

Gaffa handles failures gracefully and returns an error response detailing what went wrong.

How to handle infinite scrolling and dynamic loading

Victory Nnaji — Mon, 28 Sep 2026 12:00:00 +0000

Automating scrolling on websites isn't simple. Modern web apps often use scrolling to control when and how content loads through lazy loading and infinite scrolling. Others add delays, pop-ups, or rate limits to slow down automation. If you're using frameworks like Playwright or Selenium, handling scrolling means writing dozens of lines of code, testing different methods, and still risking detection.

Gaffa gives you a huge amount of control over how your browser automation scrolls through a number of configurable settings, whether you want to immediately scroll to the desired location or mimic human behaviour, reducing the risk of being blocked.

Using the Gaffa demo site, the API Playground, and the following examples, you will be able to see how adjusting small parameter changes allows you to:

Control scrolling speed for smoother and more natural automation.
Use immediate scrolls with intervals to respect rate limits.
Scroll to the bottom of the page to load all content before capturing a screenshot.
Configure the wait times to handle slow or heavy pages.
Set maximum scroll times for infinite scroll, which ensures the scroll doesn’t run forever.

Why Scroll Handling Matters

It reduces the risk of detection and blocking - out-of-the-box web automation frameworks scroll unnaturally, which can trigger anti-bot systems. Gaffa lets you moderate speed and mimic human-like behaviour without writing any code.

Most modern sites only load items when scrolled into view. Without controlled scrolling, your automation might miss important content, and you'll see an incomplete DOM or unloaded images. The scroll action lets you tune waits and intervals to ensure you capture complete, clean data.

Practical Scrolling Scenarios with Gaffa

Now let's explore how Gaffa makes it easy to handle a variety of real-world scrolling needs. You can try all the examples below as you read using the API Playground.

These examples use Gaffa's Demo Site and the e-commerce storefront sample as a target for the automation. The site's behaviour can be tailored to reflect different scenarios using URL parameters and is also configurable from the home page. The following settings are configurable and used in the samples below:

Load time
Cookie modal display and delay - optionally display a cookie modal after a supplied time
Item count - a particular item count or infinite for infinite scrolling pages
Item load time - how long the items and subsequent batches take to load

Scenario 1: Changing Scroll Speed

One of the simplest ways to control automation is by adjusting scroll speed. Scrolling too fast can look “bot-like” or cause content to fail to load, while a natural pace makes your automation appear human and allows the page enough time to render all the content. Gaffa’s scroll action makes this easy with configurable scroll_speed and interval parameters, giving you 3 speed settings:

instant - smoothly scroll to the desired position immediately
medium - human-like scrolling at a normal speed
slow - very slow human-like scrolling, comparable to reading while scrolling

Note: All time-based parameters in Gaffa (e.g., timeout, wait_time, max_scroll_time) are specified in milliseconds.

Example: Instant Scroll

Here’s a request to scroll through an e-commerce storefront on the demo site at an instant speed:

Instant Scroll: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=20&itemLoadTime=0",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "scroll_speed": "instant",
        "timeout": 8000
      }
    ]
  }
}

This returns JSON output with a link to a screen recording of the action:

Instant Scroll: Response

{
  "data": {
    "id": "brq_VRs2cV9HEyvWd1xVLNwaejrZ7Wq8j9",
    "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=50&itemLoadTime=0",
    "state": "completed",
    "credit_usage": 0,
    "http_status_code": 200,
    "from_cache": false,
    "started_at": "2025-09-08T06:14:53.4042285Z",
    "completed_at": "2025-09-08T06:15:01.826619Z",
    "video": "https://storage.gaffa.dev/brq/video/brq_VRs2cV9HEyvWd1xVLNwaejrZ7Wq8j9.mp4"
  }
}

Example: Slow Scroll

The scroll behaviour can be changed by adjusting the scroll_speed value; here, we've changed it to a slow scroll.

Slow Scroll: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=20&itemLoadTime=0&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "scroll_speed": "slow",
        "timeout": 20000
      }
    ]
  }
}

Example: Instant Scroll with Interval

For pages that load content dynamically with rate limits, combine scroll_speed with interval to scroll and insert pauses between scroll events to have finer-grained control over the speed and avoid limits:

Instant Scroll with Interval: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=infinite&itemLoadTime=1&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "scroll_speed": "instant",
        "interval": 1000,
        "timeout": 20000
      }
    ]
  }
}

Tip: The timeout is used here to define the maximum time Gaffa will wait for the page to become scrollable.

Scenario 2: Full Page Screenshot with Image Loading

Taking screenshots is a common web automation task and can be quite simple, unless the page you want to take a screenshot of relies on lazy-loading; then taking a screenshot of the full height without scrolling first will leave you with missing content or placeholders.

With Gaffa, you can solve this by combining the scroll action with the screenshot action. By setting the scroll percentage to 100, Gaffa will automatically scroll to the bottom of the page, ensuring all items are loaded; then, using the capture_screenshot action, capture a full-screen screenshot of the entire page.

Scroll to Bottom for Screenshot: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=20&itemLoadTime=1&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "timeout": 30000
      },
      {
        "type": "capture_screenshot",
        "size": "fullscreen"
      }
    ]
  }
}

Here's the captured image.

Scenario 3: Handling a Slow Loading Site

Some sites have unpredictable loading times, where it’s not obvious when scrolling is complete. If you stop as soon as you reach the bottom of the page, you may miss items that are still being loaded. Gaffa can handle this with the wait_time parameter. Instead of ending the scroll request immediately, you can wait a certain amount of time for more items to load. If more items load before the full wait_time elapses, Gaffa will resume scrolling immediately.

Infinite Scroll on Slow Site: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=infinite&itemLoadTime=4&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "wait_time": 5000,
        "timeout": 30000
      }
    ]
  }
}

Scenario 4: Time-Limited Infinite Scrolling

For infinite scroll pages that could theoretically scroll forever, use max_scroll_time to set reasonable time limits for your request.

Infinite Scroll with Time Limit: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=infinite&itemLoadTime=0&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "max_scroll_time": 10000,
        "timeout": 30000
      }
    ]
  }
}

You can also use Gaffa's request-level time limits as an additional safety net to prevent runaway requests.

With tools like Playwright or Selenium, handling these scenarios often means writing custom scrolling logic, adding wait conditions, and testing site-specific behaviour. With Gaffa, you can control scroll distance, speed, wait times, screenshots, and infinite scrolling through a single scroll action and a few parameters.

Scrolling is just one of the browser actions available in Gaffa. You can explore the full list of actions in the documentation or try pre-built requests in the API Playground.

Frequently Asked Questions

Why does scroll handling matter for web scraping?

Most modern sites use lazy loading, meaning content only loads as you scroll. Without controlled scrolling, your automation captures an incomplete page, missing images, products, or articles that hadn't loaded yet.

What scroll speed options does Gaffa's scroll action support?

Gaffa offers three speeds: instant (scrolls immediately), medium (human-like pace), and slow (very slow, comparable to reading while scrolling). Slower speeds reduce the risk of triggering anti-bot detection.

How do I handle pages with rate limits during scrolling?

Use the interval parameter alongside scroll_speed. It adds a pause between scroll events, giving rate-limited pages time to load content before the next scroll.

How do I take a full-page screenshot of a lazy-loaded page?

Combine the scroll action (set percentage to 100) with capture_screenshot. Gaffa scrolls to the bottom first to trigger all lazy-loaded content, then captures the full page.

How do I prevent infinite scroll pages from running forever?

Set the max_scroll_time parameter to cap how long Gaffa will scroll. You can also use request-level time limits as an additional safety net.

How is Gaffa's scroll action different from doing it in Playwright or Selenium?

With Playwright or Selenium, you'd write custom scroll logic, sleep timers, and site-specific workarounds. Gaffa handles all of that with a single action and a few parameters; no extra code needed.

Convert any web page to LLM-ready Markdown

Victory Nnaji — Mon, 21 Sep 2026 12:00:00 +0000

Many applications need real-time web access to deliver relevant information, especially those powered by large language models with static knowledge bases is generally frozen in time. Web HTML is typically noisy, cluttered with ads and navigation elements that inflate token usage and reduce LLM performance.

Gaffa’s Browser Request API is a simple REST interface for web automation and data extraction. Its generate_markdown action converts web pages into clean, LLM-ready markdown by removing clutter to reduce token usage whilst preserving key content.

HTML to Markdown Converter from Gaffa.dev

Convert any web page to clean markdown with this HTML to Markdown converter tool - powered by Gaffa's web scraping and proxy tech.

html2markdown.gaffa.dev

Converting a Web Page to Markdown

To extract a clean and readable markdown from a web page using Gaffa, you send a POST request to the /v1/browser/requests endpoint with the following JSON payload:

Generate Markdown Browser Request

{
  "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": false,
    "actions": [
      {
        "type": "wait",
        "selector": ".mw-content-container",
        "timeout": 10000,
        "continue_on_fail": true
      },
      {
        "type": "generate_markdown"
      }
    ]
  }
}

This instructs Gaffa to wait for the main content area to load and then generate markdown from it. This is the response:

Generate Markdown Response

{
    "data": {
        "id": "brq_VEmLpa5MAzkw6CWRhgU7seEbyFf2hB",
        "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
        "proxy_location": null,
        "state": "completed",
        "credit_usage": 1,
        "http_status_code": 200,
        "from_cache": false,
        "started_at": "2025-04-24T08:41:55.0836318Z",
        "completed_at": "2025-04-24T08:42:04.1878335Z",
        "running_time": "00:00:08.2352566",
        "page_load_time": "00:00:01.7245402",
        "actions": [
            {
                "id": "act_VEmLpahbSR8WRrwqiEKcaxjxitxxDQ",
                "type": "wait",
                "query": "wait?selector=.mw-content-container&timeout=10000&continue_on_fail=true",
                "timestamp": "2025-04-24T08:41:56.8081771Z"
            },
            {
                "id": "act_VEmLpXo1vxLMdCFQk3Pr6XBABUEBEn",
                "type": "generate_markdown",
                "query": "generate_markdown?continue_on_fail=false",
                "timestamp": "2025-04-24T08:41:57.1383759Z",
                "output": "https://storage.gaffa.dev/brq/md/brq_VEmLpa5MAzkw6CWRhgU7seEbyFf2hB/act_VEmLpXo1vxLMdCFQk3Pr6XBABUEBEn.md"
            }
        ]
    },
    "error": null
}

Gaffa returns a list of processed actions, with the generate_markdown action containing a markdown file that includes only meaningful content. The generate_markdown action removes lots of HTML information like styles, JavaScript, and headers that aren't relevant to the core content of the page. You can read more about this action in the Gaffa docs.

Building a Simple Python CLI Tool

To show how you can integrate Gaffa's markdown into your workflow, we built a simple Python CLI tool that:

Takes a URL input from the user
Sends a POST request to Gaffa, which will load the URL on our fleet of headless cloud browsers and return markdown, giving the output as a link.
Sends the resulting markdown to OpenAI’s API to enable question answering over the content.

You can find the complete Python implementation on GitHub if you'd like to build or modify the example yourself.

Try It Yourself

You can experiment with generate_markdown in the Gaffa API Playground by sending a URL and inspecting the Markdown returned.

The clean Markdown can then be passed to an LLM, used in a RAG pipeline, summarised, indexed, or used by an AI agent—without the unnecessary noise of raw HTML.

Frequently Asked Questions

What does Gaffa's generate_markdown action do?

It converts any web page into clean, LLM-ready Markdown by stripping ads, navigation, scripts, and styling, leaving only the meaningful content while reducing token usage.

Why use Markdown instead of raw HTML when feeding content into an LLM?

Raw HTML is cluttered with elements that inflate token counts and reduce LLM performance. Markdown removes that noise while preserving the core content, making it cheaper and more accurate to process.

Can generate_markdown handle pages that load content dynamically with JavaScript?

Yes. Gaffa renders pages in a real browser before generating Markdown, so JavaScript-loaded content is fully included, unlike simple HTTP requests that only capture the initial HTML shell.

How do I handle pages where content takes time to load?

Add a wait action before generate_markdown, specifying a CSS selector for the content area. Gaffa will pause until that element appears before generating the markdown.

Can I use the generated markdown to ask questions about a web page?

Yes. A common pattern is to fetch the markdown with Gaffa and pass it directly to an LLM API like OpenAI, allowing you to ask questions about the content of any live web page.

How do I access geo-restricted pages when generating markdown?

Set the proxy_location parameter in your request payload to a supported region, such as "us". Gaffa automatically routes the request through a residential proxy in that location.

Stop right-click saving: How to automatically scrape every image from a website

Victory Nnaji — Tue, 15 Sep 2026 12:00:00 +0000

Manually saving images from the web is a tedious, time-consuming chore. This tutorial shows you how to automate it at scale using Gaffa's Mapping and Browser Requests endpoints. We'll use Python to find every page on a site, scrape every image from each page (including JavaScript-loaded ones), and download them all automatically.

Consider these scenarios: A marketing team needs to analyse the visual branding of five different competitors. Or perhaps a researcher is building a dataset of thousands of product images to train a machine learning model. Then there's the design agency migrating a client's old website that needs to securely archive all their original assets.

What do they all have in common? Someone, at some point, is facing the task of manually right-clicking and saving hundreds, maybe thousands, of images. It's error-prone, incredibly boring, and frankly, a waste of valuable time.

At Gaffa, we recently helped a communications company do exactly this, saving them days of manual work. In the process, we used two powerful features of our API: the Site Mapping endpoint and the enhanced download-file browser request action.

In this post, we'll show you how we did it. You'll learn how to build a Python script that automatically hunts down and downloads every image from any website, no manual effort required.

Why You Can’t Just "View Source" Anymore

If you've ever tried to scrape a modern website with a simple script, you've likely run into a wall. Why is it so hard?

JavaScript: A huge portion of the web is built with frameworks like React, Vue, and Angular. The HTML you get from a simple requests.get() call is often just a shell. The actual content, including images, is loaded asynchronously by JavaScript after the page renders.
Lazy-Loading: To improve performance, sites commonly use lazy-loading. Images only appear in the HTML (data-src attribute) and load into the src attribute as you scroll down the page.
Blocks & Rate-Limiting: Making too many requests from a single IP address will quickly get you blocked.

This is where a tool with a real browser engine becomes essential. It renders the page fully, executing all the JavaScript, just like a human user would see it. That's the power we'll leverage with Gaffa.

The Plan: How We'll Scrape All Images

Our strategy is straightforward:

Map the Territory: First, we use Gaffa's site/map endpoint to find every single page on the target website. It intelligently parses sitemaps and site structure to give us a list of URLs to process.
Render Each Page: For each URL, we launch a headless browser in the cloud. We tell it to wait for all images to load and then hand us the fully-rendered HTML (the Document Object Model, or DOM).
Find & Download: We parse that final DOM to find every <img> tag, extract their source URLs, and download the files using Gaffa's download_file action. This ensures each request uses a real browser fingerprint and residential proxy, making your scraping activity appear as legitimate user traffic while leveraging Gaffa's caching system to reduce server load.

The Code: Automated Image Scraping in Python

Let's break down the key parts of the script. You can find the complete, ready-to-run code in our GitHub repository.

Step 1: Fetch Every URL on the Site

The site/map endpoint is our starting point. It does the heavy lifting of discovery by reading the sitemap, traversing possible link-outs and retrieving every page available on the website you want to scrape.

Get Sitemap

def get_sitemap_urls(site_url, max_cache_age=86400):
    payload = {
        "url": site_url,
        "max_cache_age": max_cache_age
    }
    print("Retrieving sitemap URLs.")
    response = requests.post("https://api.gaffa.dev/v1/site/map", json=payload, headers=HEADERS)
    return response.json()["data"]["links"]

Step 2: Capturing the Real Page (after JavaScript)

This is where the real magic happens. We use a browser request with the capture_dom action to get the real HTML. The Gaffa API makes sure that the website is rendered first, which is often one of the main issues when using traditional web scraping frameworks.

Capture DOM with Gaffa

def get_dom(url):
    payload = {
        "url": url,
        "async": False,
        "settings": {
            "actions": [
                {"type": "wait", "selector": "img", "timeout": 20000},
                {"type": "capture_dom"}
            ],
            "time_limit": 40000
        }
    }
    print("Capturing DOM URL.")
    response = requests.post("https://api.gaffa.dev/v1/browser/requests", json=payload, headers=HEADERS)
    dom_url = response.json()["data"]["actions"][1]["output"]
    print("Retrieving DOM.")
    dom_response = requests.get(dom_url)
    return dom_response.text

Step 3: Finding and Saving the Images

With the real HTML in hand, we extract image URLs using a simple regex pattern and use Gaffa's download_file action for reliable downloads. This also allows us to use caching, which avoids downloading the same image over and over again and putting load onto the target server.

Download files using Gaffa

def extract_image_urls(dom_content, base_url):
    image_urls = []
    src_pattern = r'<img[^>]+(?:src|data-src)=["\']([^"\']+)["\']'
    matches = re.findall(src_pattern, dom_content)

    for src in matches:
        if not src.startswith(('http:', 'https:')):
            src = urljoin(base_url, src)
        image_urls.append(src)

    return image_urls

def download_image(image_url, filename):
    payload = {
        "url": image_url,
        "async": False,
        "settings": {
            "actions": [{"type": "download_file"}]
        }
    }
    print("Retrieving download URL.")
    response = requests.post("https://api.gaffa.dev/v1/browser/requests", json=payload, headers=HEADERS)
    actions = response.json()["data"]["actions"]
    download_url = actions[0]["output"]
    download_ext = os.path.splitext(download_url)[1]

    print("Downloading image.")
    img_response = requests.get(download_url)
    filepath = f"{filename}{download_ext}"
    with open(filepath, 'wb') as f:
        f.write(img_response.content)

Why Use Gaffa for Downloading?

The download_file action provides three crucial advantages over standard download methods:

Stealth & Success Rates: By using rotating residential proxies and real browser fingerprints, your requests appear as legitimate user traffic, dramatically reducing the chance of being blocked.
Responsible Caching: With max_cache_age set to 24 hours, repeated requests for the same image are served from cache, sparing the target website's servers from unnecessary load.
Built-in Reliability: Gaffa's infrastructure automatically manages request pacing and retries, but remember to always respect reasonable rate limits and check robots.txt before scraping.
Automatic Format Detection: Gaffa's API provides the correct file extension directly in the download URL, eliminating the need for complex content-type parsing and making the download process more reliable.

When you run the script, you'll see a satisfying log of progress as it systematically works through the site:

Retrieving sitemap URLs.
Capturing DOM URL.
Retrieving DOM.
Retrieving download URL.
Downloading image.

So, What Can You Actually Build With This?

The use cases go far beyond simple image collection. This kind of automation is a superpower for:

Competitive Intelligence & Market Research: Analyze product photography styles, branding consistency, and advertising campaigns across entire industries.
AI and Machine Learning: Build large-scale, curated image datasets for training computer vision models on anything from fashion to real estate.
Competitor Tracking & Trend Analysis: Create a timeline of how competitors' websites and visual content evolve by running automated scans weekly or monthly. Track branding changes, product launches, and marketing campaigns over time to stay ahead of market trends.
UX & Design Inspiration: Create massive mood boards or collect UI patterns from across the web to inspire your next project.
Performance Auditing: Scan your own site to find unoptimized images that are slowing down your page load times.

Scrape Responsibly, Build Amazing Things

It's important to use this power wisely. Gaffa’s built-in caching helps prevent overloading servers, but it's on all of us to be good citizens of the web and always check a website's robots.txt file and terms of service before scraping.

The useful part of this approach is that you can focus on what you want to extract rather than managing the browser infrastructure behind it. For something like image extraction, that means describing the actions you want to perform and working with the results in your application.

If you'd like to take this further, you can find a more detailed tutorial in the Gaffa docs

Frequently Asked Questions

Why can't I just use a simple Python script to scrape images from modern websites?

Most modern sites use JavaScript frameworks and lazy-loading, meaning images aren't in the initial HTML. A simple requests.get() call returns a shell page without the actual content; you need a real browser engine to render it fully.

How does Gaffa find every page on a website to scrape?

Gaffa's site/map endpoint reads the site's sitemap and traverses its link structure to return every available URL, saving you the manual work of discovering pages before scraping them.

What is the download_file action, and why use it over a standard download?

It downloads files using rotating residential proxies and real browser fingerprints, making requests appear as legitimate user traffic. It also handles caching, automatic file extension detection, and request retries, things you'd otherwise have to build yourself.

How does Gaffa handle lazy-loaded images that only appear on scroll?

The wait action tells Gaffa to pause until image elements appear in the DOM before capturing them, ensuring lazy-loaded images are included in the final HTML you parse.

Will scraping images at scale get my IP blocked?

Not with Gaffa. It uses rotating residential proxies and real browser fingerprints, so requests appear as normal user traffic rather than automated scraping.

Is there anything I should check before scraping images from a website?

Always review the site's robots.txt file and terms of service before scraping. Gaffa's built-in caching also helps reduce unnecessary load on target servers.

How to Scrape a Table with Python (The Easy Way)

Victory Nnaji — Thu, 10 Sep 2026 12:54:27 +0000

Web tables are goldmines of structured data, interest rate histories, sports standings, financial reports, and product comparisons. The problem is getting that data out cleanly. In this post, we'll walk through two ways to scrape a table using Python and Gaffa.

Manually, using capture_dom and BeautifulSoup
Automatically, using Gaffa's parse_table action.

You can find the full code for this post in our Python examples repository on GitHub.

The problem with traditional table scraping

Most tutorials recommend using Playwright or Selenium to load a page and retrieve the HTML. That works, but it means you're managing browser infrastructure yourself, dealing with headless Chrome setups, handling anti-bot detection, rotating proxies, and writing a non-trivial amount of boilerplate just to get to the data.

Gaffa handles all of that complexity for you. You send a POST request, and Gaffa spins up a real browser, handles anti-bot measures, optionally routes through proxies, and returns exactly what you ask for, no browser setup on your end.

We'll demonstrate two approaches to scraping tables using two different sites.

Getting Started

You'll need a Gaffa API key. Sign up at gaffa.dev and create your key in the API Keys section of the dashboard. For setup instructions, see the README in our GitHub samples repo.

Approach 1: Capture DOM + BeautifulSoup

The capture_dom approach gives you maximum control. Gaffa fetches the page and returns the raw HTML DOM. You then parse it locally with BeautifulSoup and extract the table data yourself.

Step 1: Fetch the DOM with Gaffa

Capturing a page DOM with Gaffa

import requests
import json
import os
from bs4 import BeautifulSoup

GAFFA_API_KEY = os.getenv("GAFFA_API_KEY")

def fetch_dom(url, proxy_location=None):
    payload = {
        "url": url,
        "proxy_location": proxy_location,
        "async": False,
        "max_cache_age": 0,
        "settings": {
            "record_request": False,
            "actions": [
                {
                    "type": "wait",
                    "selector": "table",
                    "timeout": 5000
                },
                {
                    "type": "capture_dom"
                }
            ]
        }
    }

    headers = {
        "x-api-key": GAFFA_API_KEY,
        "Content-Type": "application/json"
    }

    response = requests.post(
        "https://api.gaffa.dev/v1/browser/requests",
        json=payload,
        headers=headers
    )
    response.raise_for_status()

    dom_url = response.json()["data"]["actions"][1]["output"]
    dom_response = requests.get(dom_url)
    dom_response.raise_for_status()

    return dom_response.text

The wait action tells Gaffa to wait until a table element appears in the DOM, which is useful for pages where the table loads dynamically. The capture_dom action then returns the full HTML as a file URL, which we fetch separately.

Step 2: Parse the table with BeautifulSoup

Parsing a Table from the DOM using BeautifulSoup

def parse_table_from_dom(html, table_selector="table"):
    soup = BeautifulSoup(html, "html.parser")
    table = soup.select_one(table_selector)

    if not table:
        raise ValueError(f"No table found for selector: {table_selector}")

    headers = [th.get_text(strip=True) for th in table.select("thead th")]
    rows = []

    for tr in table.select("tbody tr"):
        cells = [td.get_text(strip=True) for td in tr.select("td")]
        if cells:
            rows.append(dict(zip(headers, cells)))

    return rows

Step 3: Run it and save the output

Saving the parsed table

if __name__ == "__main__":
    html = fetch_dom("https://demo.gaffa.dev/simulate/table?loadTime=1&rowCount=10")
    data = parse_table_from_dom(html)

    # Print to console
    print(json.dumps(data, indent=2))

    # Save to file
    with open("table_data.json", "w") as f:
        json.dump(data, f, indent=2)

    print(f"\nSaved {len(data)} rows to table_data.json")

Sample output:

Sample parsed table output

[
  {
    "ID": "1",
    "Name": "Item 1",
    "Quantity": "87",
    "Price": "$25.29"
  },
  {
    "ID": "2",
    "Name": "Item 2",
    "Quantity": "12",
    "Price": "$8.96"
  }
]

See the full sample output on GitHub.

This approach is flexible. If you need to clean up values, handle merged cells, or do any custom transformation before saving, BeautifulSoup gives you full control over the parsed data.

Approach 2: Using the parse_table action to get JSON with no processing

If you just want the data and don't need custom processing, Gaffa's parse_table action eliminates the need for the BeautifulSoup step entirely. It finds the table on the page, reads the headers, and returns a ready-to-use JSON object directly.

Here's what the action does internally:

It locates the table using your CSS selector
Converts the header row into property names (lowercased, non-alphanumeric characters replaced with underscores)
Then maps each cell value to its corresponding header for every row, returning a clean JSON array with no post-processing required on your end.

Using parse_table on the Demo Site

The Gaffa demo site at demo.gaffa.dev is a simple test environment with pre-built pages designed for trying out Gaffa actions before pointing them at a real site

Using Gaffa's parse_table action

import requests
import json
import os

GAFFA_API_KEY = os.getenv("GAFFA_API_KEY")

def fetch_parsed_table(url, selector="table", proxy_location=None):
    payload = {
        "url": url,
        "proxy_location": proxy_location,
        "async": False,
        "max_cache_age": 0,
        "settings": {
            "record_request": False,
            "actions": [
                {
                    "type": "parse_table",
                    "selector": selector,
                    "timeout": 5000
                }
            ]
        }
    }

    headers = {
        "x-api-key": GAFFA_API_KEY,
        "Content-Type": "application/json"
    }

    response = requests.post(
        "https://api.gaffa.dev/v1/browser/requests",
        json=payload,
        headers=headers
    )
    response.raise_for_status()

    result_url = response.json()["data"]["actions"][0]["output"]
    result_response = requests.get(result_url)
    result_response.raise_for_status()

    return result_response.json()


if __name__ == "__main__":
    data = fetch_parsed_table(
        url="https://demo.gaffa.dev/simulate/table?loadTime=1&rowCount=10"
    )
    print(json.dumps(data, indent=2))

That's the entire script: no HTML parsing, no BeautifulSoup, no column mapping. The output is already shaped as a list of objects you can use immediately.

Real-World Example

Let's apply parse_table to Wikipedia's List of Countries by GDP (Nominal), a clean, publicly accessible table of financial data. Wikipedia uses a consistent CSS class on all its data tables, making it straightforward and reliable to target with a selector.

Still using the same fetch_parsed_table function we wrote for the demo site, just swap the if name block with this:

Wikipedia table parsing using Gaffa

if __name__ == "__main__":
    data = fetch_parsed_table(
        url="https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)",
        selector="table.wikitable",
    )

    with open("gdp_data.json", "w") as f:
        json.dump(data, f, indent=2)

    print(f"Fetched {len(data)} records")
    print(json.dumps(data[:3], indent=2))

Sample output:

Sample table data

[
  {
    "country_territory": "World",
    "imf__2026__1": "123,584,494",
    "world_bank__2024__6": "111,326,370",
    "united_nations__2024__7": "100,834,796"
  },
  {
    "country_territory": "United States",
    "imf__2026__1": "31,821,293",
    "world_bank__2024__6": "28,750,956",
    "united_nations__2024__7": "29,298,000"
  }
]

See the full sample output on GitHub.

Notice how the original column headers, like “Country/Territory” and “IMF 2026”, are automatically normalised into “country_territory” and “imf_2026”. Spaces and special characters are replaced with underscores, and everything is lowercased, so the output is immediately usable without any cleanup.

No proxy_location is needed here, since Wikipedia is globally accessible, but for sites that restrict access by geography, you can simply add proxy_location="us" or another supported region to route the request through the appropriate IP address.

Which Approach Should You Use?

	`capture_dom`	`parse_table`
Setup	More code	Minimal code
Control	Full	Limited to what Gaffa returns
Best for	Complex tables, complex logic	Standard tables, fast extraction
Post-processing	Yes	None needed

If the table is straightforward and you just need the data, use parse_table. If you need to do any custom processing, such as merging columns, skipping rows, or reformatting values, use capture_dom with BeautifulSoup.

Either way, you're not managing browser infrastructure, rotating proxies, or writing anti-bot workarounds. Gaffa handles that layer so your Python code stays focused on what matters: the data.

Don't Want to Write Code Yet? Use the Playground

If you want to test parse_table or capture_dom before writing any Python, the Gaffa Playground lets you run requests directly from your browser with no code and no setup.

Just paste in your JSON payload, hit Send Request, and see the output instantly. It's a great way to confirm your selector is working and the table is returning the right data before you wire it up in a script.

Ultimately, the right approach depends on how much control you need. Use capture_dom with BeautifulSoup when you need full control over how the table is processed, or parse_table when you want structured table data without the extra parsing step.

Frequently Asked Questions

What is the easiest way to scrape a table from a website using Python?

Use Gaffa's parse_table action. Send a POST request with your target URL and CSS selector, and it returns a clean JSON array, no HTML parsing, no BeautifulSoup, no browser setup required.

How do I scrape a table and get raw HTML I can process myself?

Use the capture_dom action. It returns the fully rendered HTML of the page, which you can then parse manually with BeautifulSoup for custom processing, column merging, or data transformation.

How do I scrape a table and get clean JSON without any processing?

Use the parse_table action. It automatically finds the table, maps each row to its headers, and returns ready-to-use JSON, with no post-processing required.

Do I need to set up a headless browser or manage proxies to use Gaffa?

No. Gaffa handles browser infrastructure, anti-bot measures, and proxy routing on its end. Your Python code only needs to send an HTTP request.

How do I scrape tables from pages that load content dynamically?

Add a wait action before your capture_dom or parse_table action. It tells Gaffa to pause until the table element appears in the DOM before proceeding with extraction.

Can Gaffa scrape tables from geo-restricted websites?

Yes. Add proxy_location="us" (or another supported region) to your request payload to route the request through the appropriate IP address.

How does parse_table format the extracted data?

It automatically normalises column headers to lowercase, with underscores replacing spaces and special characters, so "Country/Territory" becomes country_territory, immediately usable with no cleanup needed.