DEV Community: Victory Nnaji

How to handle infinite scrolling and dynamic loading

Victory Nnaji — Mon, 28 Sep 2026 12:00:00 +0000

Automating scrolling on websites isn't simple. Modern web apps often use scrolling to control when and how content loads through lazy loading and infinite scrolling. Others add delays, pop-ups, or rate limits to slow down automation. If you're using frameworks like Playwright or Selenium, handling scrolling means writing dozens of lines of code, testing different methods, and still risking detection.

Gaffa gives you a huge amount of control over how your browser automation scrolls through a number of configurable settings, whether you want to immediately scroll to the desired location or mimic human behaviour, reducing the risk of being blocked.

Using the Gaffa demo site, the API Playground, and the following examples, you will be able to see how adjusting small parameter changes allows you to:

Control scrolling speed for smoother and more natural automation.
Use immediate scrolls with intervals to respect rate limits.
Scroll to the bottom of the page to load all content before capturing a screenshot.
Configure the wait times to handle slow or heavy pages.
Set maximum scroll times for infinite scroll, which ensures the scroll doesn’t run forever.

Why Scroll Handling Matters

It reduces the risk of detection and blocking - out-of-the-box web automation frameworks scroll unnaturally, which can trigger anti-bot systems. Gaffa lets you moderate speed and mimic human-like behaviour without writing any code.

Most modern sites only load items when scrolled into view. Without controlled scrolling, your automation might miss important content, and you'll see an incomplete DOM or unloaded images. The scroll action lets you tune waits and intervals to ensure you capture complete, clean data.

Practical Scrolling Scenarios with Gaffa

Now let's explore how Gaffa makes it easy to handle a variety of real-world scrolling needs. You can try all the examples below as you read using the API Playground.

These examples use Gaffa's Demo Site and the e-commerce storefront sample as a target for the automation. The site's behaviour can be tailored to reflect different scenarios using URL parameters and is also configurable from the home page. The following settings are configurable and used in the samples below:

Load time
Cookie modal display and delay - optionally display a cookie modal after a supplied time
Item count - a particular item count or infinite for infinite scrolling pages
Item load time - how long the items and subsequent batches take to load

Scenario 1: Changing Scroll Speed

One of the simplest ways to control automation is by adjusting scroll speed. Scrolling too fast can look “bot-like” or cause content to fail to load, while a natural pace makes your automation appear human and allows the page enough time to render all the content. Gaffa’s scroll action makes this easy with configurable scroll_speed and interval parameters, giving you 3 speed settings:

instant - smoothly scroll to the desired position immediately
medium - human-like scrolling at a normal speed
slow - very slow human-like scrolling, comparable to reading while scrolling

Note: All time-based parameters in Gaffa (e.g., timeout, wait_time, max_scroll_time) are specified in milliseconds.

Example: Instant Scroll

Here’s a request to scroll through an e-commerce storefront on the demo site at an instant speed:

Instant Scroll: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=20&itemLoadTime=0",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "scroll_speed": "instant",
        "timeout": 8000
      }
    ]
  }
}

This returns JSON output with a link to a screen recording of the action:

Instant Scroll: Response

{
  "data": {
    "id": "brq_VRs2cV9HEyvWd1xVLNwaejrZ7Wq8j9",
    "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=50&itemLoadTime=0",
    "state": "completed",
    "credit_usage": 0,
    "http_status_code": 200,
    "from_cache": false,
    "started_at": "2025-09-08T06:14:53.4042285Z",
    "completed_at": "2025-09-08T06:15:01.826619Z",
    "video": "https://storage.gaffa.dev/brq/video/brq_VRs2cV9HEyvWd1xVLNwaejrZ7Wq8j9.mp4"
  }
}

Example: Slow Scroll

The scroll behaviour can be changed by adjusting the scroll_speed value; here, we've changed it to a slow scroll.

Slow Scroll: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=20&itemLoadTime=0&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "scroll_speed": "slow",
        "timeout": 20000
      }
    ]
  }
}

Example: Instant Scroll with Interval

For pages that load content dynamically with rate limits, combine scroll_speed with interval to scroll and insert pauses between scroll events to have finer-grained control over the speed and avoid limits:

Instant Scroll with Interval: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=infinite&itemLoadTime=1&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "scroll_speed": "instant",
        "interval": 1000,
        "timeout": 20000
      }
    ]
  }
}

Tip: The timeout is used here to define the maximum time Gaffa will wait for the page to become scrollable.

Scenario 2: Full Page Screenshot with Image Loading

Taking screenshots is a common web automation task and can be quite simple, unless the page you want to take a screenshot of relies on lazy-loading; then taking a screenshot of the full height without scrolling first will leave you with missing content or placeholders.

With Gaffa, you can solve this by combining the scroll action with the screenshot action. By setting the scroll percentage to 100, Gaffa will automatically scroll to the bottom of the page, ensuring all items are loaded; then, using the capture_screenshot action, capture a full-screen screenshot of the entire page.

Scroll to Bottom for Screenshot: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=20&itemLoadTime=1&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "timeout": 30000
      },
      {
        "type": "capture_screenshot",
        "size": "fullscreen"
      }
    ]
  }
}

Here's the captured image.

Scenario 3: Handling a Slow Loading Site

Some sites have unpredictable loading times, where it’s not obvious when scrolling is complete. If you stop as soon as you reach the bottom of the page, you may miss items that are still being loaded. Gaffa can handle this with the wait_time parameter. Instead of ending the scroll request immediately, you can wait a certain amount of time for more items to load. If more items load before the full wait_time elapses, Gaffa will resume scrolling immediately.

Infinite Scroll on Slow Site: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=infinite&itemLoadTime=4&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "wait_time": 5000,
        "timeout": 30000
      }
    ]
  }
}

Scenario 4: Time-Limited Infinite Scrolling

For infinite scroll pages that could theoretically scroll forever, use max_scroll_time to set reasonable time limits for your request.

Infinite Scroll with Time Limit: Request

{
  "url": "https://demo.gaffa.dev/simulate/ecommerce?loadTime=3&showModal=false&modalDelay=5&itemCount=infinite&itemLoadTime=0&isVirtualScroll=false",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": true,
    "actions": [
      {
        "type": "scroll",
        "percentage": 100,
        "max_scroll_time": 10000,
        "timeout": 30000
      }
    ]
  }
}

You can also use Gaffa's request-level time limits as an additional safety net to prevent runaway requests.

With tools like Playwright or Selenium, handling these scenarios often means writing custom scrolling logic, adding wait conditions, and testing site-specific behaviour. With Gaffa, you can control scroll distance, speed, wait times, screenshots, and infinite scrolling through a single scroll action and a few parameters.

Scrolling is just one of the browser actions available in Gaffa. You can explore the full list of actions in the documentation or try pre-built requests in the API Playground.

Frequently Asked Questions

Why does scroll handling matter for web scraping?

Most modern sites use lazy loading, meaning content only loads as you scroll. Without controlled scrolling, your automation captures an incomplete page, missing images, products, or articles that hadn't loaded yet.

What scroll speed options does Gaffa's scroll action support?

Gaffa offers three speeds: instant (scrolls immediately), medium (human-like pace), and slow (very slow, comparable to reading while scrolling). Slower speeds reduce the risk of triggering anti-bot detection.

How do I handle pages with rate limits during scrolling?

Use the interval parameter alongside scroll_speed. It adds a pause between scroll events, giving rate-limited pages time to load content before the next scroll.

How do I take a full-page screenshot of a lazy-loaded page?

Combine the scroll action (set percentage to 100) with capture_screenshot. Gaffa scrolls to the bottom first to trigger all lazy-loaded content, then captures the full page.

How do I prevent infinite scroll pages from running forever?

Set the max_scroll_time parameter to cap how long Gaffa will scroll. You can also use request-level time limits as an additional safety net.

How is Gaffa's scroll action different from doing it in Playwright or Selenium?

With Playwright or Selenium, you'd write custom scroll logic, sleep timers, and site-specific workarounds. Gaffa handles all of that with a single action and a few parameters; no extra code needed.

Convert any web page to LLM-ready Markdown

Victory Nnaji — Mon, 21 Sep 2026 12:00:00 +0000

Many applications need real-time web access to deliver relevant information, especially those powered by large language models with static knowledge bases is generally frozen in time. Web HTML is typically noisy, cluttered with ads and navigation elements that inflate token usage and reduce LLM performance.

Gaffa’s Browser Request API is a simple REST interface for web automation and data extraction. Its generate_markdown action converts web pages into clean, LLM-ready markdown by removing clutter to reduce token usage whilst preserving key content.

HTML to Markdown Converter from Gaffa.dev

Convert any web page to clean markdown with this HTML to Markdown converter tool - powered by Gaffa's web scraping and proxy tech.

html2markdown.gaffa.dev

Converting a Web Page to Markdown

To extract a clean and readable markdown from a web page using Gaffa, you send a POST request to the /v1/browser/requests endpoint with the following JSON payload:

Generate Markdown Browser Request

{
  "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "proxy_location": null,
  "async": false,
  "max_cache_age": 0,
  "settings": {
    "record_request": false,
    "actions": [
      {
        "type": "wait",
        "selector": ".mw-content-container",
        "timeout": 10000,
        "continue_on_fail": true
      },
      {
        "type": "generate_markdown"
      }
    ]
  }
}

This instructs Gaffa to wait for the main content area to load and then generate markdown from it. This is the response:

Generate Markdown Response

{
    "data": {
        "id": "brq_VEmLpa5MAzkw6CWRhgU7seEbyFf2hB",
        "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
        "proxy_location": null,
        "state": "completed",
        "credit_usage": 1,
        "http_status_code": 200,
        "from_cache": false,
        "started_at": "2025-04-24T08:41:55.0836318Z",
        "completed_at": "2025-04-24T08:42:04.1878335Z",
        "running_time": "00:00:08.2352566",
        "page_load_time": "00:00:01.7245402",
        "actions": [
            {
                "id": "act_VEmLpahbSR8WRrwqiEKcaxjxitxxDQ",
                "type": "wait",
                "query": "wait?selector=.mw-content-container&timeout=10000&continue_on_fail=true",
                "timestamp": "2025-04-24T08:41:56.8081771Z"
            },
            {
                "id": "act_VEmLpXo1vxLMdCFQk3Pr6XBABUEBEn",
                "type": "generate_markdown",
                "query": "generate_markdown?continue_on_fail=false",
                "timestamp": "2025-04-24T08:41:57.1383759Z",
                "output": "https://storage.gaffa.dev/brq/md/brq_VEmLpa5MAzkw6CWRhgU7seEbyFf2hB/act_VEmLpXo1vxLMdCFQk3Pr6XBABUEBEn.md"
            }
        ]
    },
    "error": null
}

Gaffa returns a list of processed actions, with the generate_markdown action containing a markdown file that includes only meaningful content. The generate_markdown action removes lots of HTML information like styles, JavaScript, and headers that aren't relevant to the core content of the page. You can read more about this action in the Gaffa docs.

Building a Simple Python CLI Tool

To show how you can integrate Gaffa's markdown into your workflow, we built a simple Python CLI tool that:

Takes a URL input from the user
Sends a POST request to Gaffa, which will load the URL on our fleet of headless cloud browsers and return markdown, giving the output as a link.
Sends the resulting markdown to OpenAI’s API to enable question answering over the content.

You can find the complete Python implementation on GitHub if you'd like to build or modify the example yourself.

Try It Yourself

You can experiment with generate_markdown in the Gaffa API Playground by sending a URL and inspecting the Markdown returned.

The clean Markdown can then be passed to an LLM, used in a RAG pipeline, summarised, indexed, or used by an AI agent—without the unnecessary noise of raw HTML.

Frequently Asked Questions

What does Gaffa's generate_markdown action do?

It converts any web page into clean, LLM-ready Markdown by stripping ads, navigation, scripts, and styling, leaving only the meaningful content while reducing token usage.

Why use Markdown instead of raw HTML when feeding content into an LLM?

Raw HTML is cluttered with elements that inflate token counts and reduce LLM performance. Markdown removes that noise while preserving the core content, making it cheaper and more accurate to process.

Can generate_markdown handle pages that load content dynamically with JavaScript?

Yes. Gaffa renders pages in a real browser before generating Markdown, so JavaScript-loaded content is fully included, unlike simple HTTP requests that only capture the initial HTML shell.

How do I handle pages where content takes time to load?

Add a wait action before generate_markdown, specifying a CSS selector for the content area. Gaffa will pause until that element appears before generating the markdown.

Can I use the generated markdown to ask questions about a web page?

Yes. A common pattern is to fetch the markdown with Gaffa and pass it directly to an LLM API like OpenAI, allowing you to ask questions about the content of any live web page.

How do I access geo-restricted pages when generating markdown?

Set the proxy_location parameter in your request payload to a supported region, such as "us". Gaffa automatically routes the request through a residential proxy in that location.

Stop right-click saving: How to automatically scrape every image from a website

Victory Nnaji — Tue, 15 Sep 2026 12:00:00 +0000

Manually saving images from the web is a tedious, time-consuming chore. This tutorial shows you how to automate it at scale using Gaffa's Mapping and Browser Requests endpoints. We'll use Python to find every page on a site, scrape every image from each page (including JavaScript-loaded ones), and download them all automatically.

Consider these scenarios: A marketing team needs to analyse the visual branding of five different competitors. Or perhaps a researcher is building a dataset of thousands of product images to train a machine learning model. Then there's the design agency migrating a client's old website that needs to securely archive all their original assets.

What do they all have in common? Someone, at some point, is facing the task of manually right-clicking and saving hundreds, maybe thousands, of images. It's error-prone, incredibly boring, and frankly, a waste of valuable time.

At Gaffa, we recently helped a communications company do exactly this, saving them days of manual work. In the process, we used two powerful features of our API: the Site Mapping endpoint and the enhanced download-file browser request action.

In this post, we'll show you how we did it. You'll learn how to build a Python script that automatically hunts down and downloads every image from any website, no manual effort required.

Why You Can’t Just "View Source" Anymore

If you've ever tried to scrape a modern website with a simple script, you've likely run into a wall. Why is it so hard?

JavaScript: A huge portion of the web is built with frameworks like React, Vue, and Angular. The HTML you get from a simple requests.get() call is often just a shell. The actual content, including images, is loaded asynchronously by JavaScript after the page renders.
Lazy-Loading: To improve performance, sites commonly use lazy-loading. Images only appear in the HTML (data-src attribute) and load into the src attribute as you scroll down the page.
Blocks & Rate-Limiting: Making too many requests from a single IP address will quickly get you blocked.

This is where a tool with a real browser engine becomes essential. It renders the page fully, executing all the JavaScript, just like a human user would see it. That's the power we'll leverage with Gaffa.

The Plan: How We'll Scrape All Images

Our strategy is straightforward:

Map the Territory: First, we use Gaffa's site/map endpoint to find every single page on the target website. It intelligently parses sitemaps and site structure to give us a list of URLs to process.
Render Each Page: For each URL, we launch a headless browser in the cloud. We tell it to wait for all images to load and then hand us the fully-rendered HTML (the Document Object Model, or DOM).
Find & Download: We parse that final DOM to find every <img> tag, extract their source URLs, and download the files using Gaffa's download_file action. This ensures each request uses a real browser fingerprint and residential proxy, making your scraping activity appear as legitimate user traffic while leveraging Gaffa's caching system to reduce server load.

The Code: Automated Image Scraping in Python

Let's break down the key parts of the script. You can find the complete, ready-to-run code in our GitHub repository.

Step 1: Fetch Every URL on the Site

The site/map endpoint is our starting point. It does the heavy lifting of discovery by reading the sitemap, traversing possible link-outs and retrieving every page available on the website you want to scrape.

Get Sitemap

def get_sitemap_urls(site_url, max_cache_age=86400):
    payload = {
        "url": site_url,
        "max_cache_age": max_cache_age
    }
    print("Retrieving sitemap URLs.")
    response = requests.post("https://api.gaffa.dev/v1/site/map", json=payload, headers=HEADERS)
    return response.json()["data"]["links"]

Step 2: Capturing the Real Page (after JavaScript)

This is where the real magic happens. We use a browser request with the capture_dom action to get the real HTML. The Gaffa API makes sure that the website is rendered first, which is often one of the main issues when using traditional web scraping frameworks.

Capture DOM with Gaffa

def get_dom(url):
    payload = {
        "url": url,
        "async": False,
        "settings": {
            "actions": [
                {"type": "wait", "selector": "img", "timeout": 20000},
                {"type": "capture_dom"}
            ],
            "time_limit": 40000
        }
    }
    print("Capturing DOM URL.")
    response = requests.post("https://api.gaffa.dev/v1/browser/requests", json=payload, headers=HEADERS)
    dom_url = response.json()["data"]["actions"][1]["output"]
    print("Retrieving DOM.")
    dom_response = requests.get(dom_url)
    return dom_response.text

Step 3: Finding and Saving the Images

With the real HTML in hand, we extract image URLs using a simple regex pattern and use Gaffa's download_file action for reliable downloads. This also allows us to use caching, which avoids downloading the same image over and over again and putting load onto the target server.

Download files using Gaffa

def extract_image_urls(dom_content, base_url):
    image_urls = []
    src_pattern = r'<img[^>]+(?:src|data-src)=["\']([^"\']+)["\']'
    matches = re.findall(src_pattern, dom_content)

    for src in matches:
        if not src.startswith(('http:', 'https:')):
            src = urljoin(base_url, src)
        image_urls.append(src)

    return image_urls

def download_image(image_url, filename):
    payload = {
        "url": image_url,
        "async": False,
        "settings": {
            "actions": [{"type": "download_file"}]
        }
    }
    print("Retrieving download URL.")
    response = requests.post("https://api.gaffa.dev/v1/browser/requests", json=payload, headers=HEADERS)
    actions = response.json()["data"]["actions"]
    download_url = actions[0]["output"]
    download_ext = os.path.splitext(download_url)[1]

    print("Downloading image.")
    img_response = requests.get(download_url)
    filepath = f"{filename}{download_ext}"
    with open(filepath, 'wb') as f:
        f.write(img_response.content)

Why Use Gaffa for Downloading?

The download_file action provides three crucial advantages over standard download methods:

Stealth & Success Rates: By using rotating residential proxies and real browser fingerprints, your requests appear as legitimate user traffic, dramatically reducing the chance of being blocked.
Responsible Caching: With max_cache_age set to 24 hours, repeated requests for the same image are served from cache, sparing the target website's servers from unnecessary load.
Built-in Reliability: Gaffa's infrastructure automatically manages request pacing and retries, but remember to always respect reasonable rate limits and check robots.txt before scraping.
Automatic Format Detection: Gaffa's API provides the correct file extension directly in the download URL, eliminating the need for complex content-type parsing and making the download process more reliable.

When you run the script, you'll see a satisfying log of progress as it systematically works through the site:

Retrieving sitemap URLs.
Capturing DOM URL.
Retrieving DOM.
Retrieving download URL.
Downloading image.

So, What Can You Actually Build With This?

The use cases go far beyond simple image collection. This kind of automation is a superpower for:

Competitive Intelligence & Market Research: Analyze product photography styles, branding consistency, and advertising campaigns across entire industries.
AI and Machine Learning: Build large-scale, curated image datasets for training computer vision models on anything from fashion to real estate.
Competitor Tracking & Trend Analysis: Create a timeline of how competitors' websites and visual content evolve by running automated scans weekly or monthly. Track branding changes, product launches, and marketing campaigns over time to stay ahead of market trends.
UX & Design Inspiration: Create massive mood boards or collect UI patterns from across the web to inspire your next project.
Performance Auditing: Scan your own site to find unoptimized images that are slowing down your page load times.

Scrape Responsibly, Build Amazing Things

It's important to use this power wisely. Gaffa’s built-in caching helps prevent overloading servers, but it's on all of us to be good citizens of the web and always check a website's robots.txt file and terms of service before scraping.

The useful part of this approach is that you can focus on what you want to extract rather than managing the browser infrastructure behind it. For something like image extraction, that means describing the actions you want to perform and working with the results in your application.

If you'd like to take this further, you can find a more detailed tutorial in the Gaffa docs

Frequently Asked Questions

Why can't I just use a simple Python script to scrape images from modern websites?

Most modern sites use JavaScript frameworks and lazy-loading, meaning images aren't in the initial HTML. A simple requests.get() call returns a shell page without the actual content; you need a real browser engine to render it fully.

How does Gaffa find every page on a website to scrape?

Gaffa's site/map endpoint reads the site's sitemap and traverses its link structure to return every available URL, saving you the manual work of discovering pages before scraping them.

What is the download_file action, and why use it over a standard download?

It downloads files using rotating residential proxies and real browser fingerprints, making requests appear as legitimate user traffic. It also handles caching, automatic file extension detection, and request retries, things you'd otherwise have to build yourself.

How does Gaffa handle lazy-loaded images that only appear on scroll?

The wait action tells Gaffa to pause until image elements appear in the DOM before capturing them, ensuring lazy-loaded images are included in the final HTML you parse.

Will scraping images at scale get my IP blocked?

Not with Gaffa. It uses rotating residential proxies and real browser fingerprints, so requests appear as normal user traffic rather than automated scraping.

Is there anything I should check before scraping images from a website?

Always review the site's robots.txt file and terms of service before scraping. Gaffa's built-in caching also helps reduce unnecessary load on target servers.

How to Scrape a Table with Python (The Easy Way)

Victory Nnaji — Thu, 10 Sep 2026 12:54:27 +0000

Web tables are goldmines of structured data, interest rate histories, sports standings, financial reports, and product comparisons. The problem is getting that data out cleanly. In this post, we'll walk through two ways to scrape a table using Python and Gaffa.

Manually, using capture_dom and BeautifulSoup
Automatically, using Gaffa's parse_table action.

You can find the full code for this post in our Python examples repository on GitHub.

The problem with traditional table scraping

Most tutorials recommend using Playwright or Selenium to load a page and retrieve the HTML. That works, but it means you're managing browser infrastructure yourself, dealing with headless Chrome setups, handling anti-bot detection, rotating proxies, and writing a non-trivial amount of boilerplate just to get to the data.

Gaffa handles all of that complexity for you. You send a POST request, and Gaffa spins up a real browser, handles anti-bot measures, optionally routes through proxies, and returns exactly what you ask for, no browser setup on your end.

We'll demonstrate two approaches to scraping tables using two different sites.

Getting Started

You'll need a Gaffa API key. Sign up at gaffa.dev and create your key in the API Keys section of the dashboard. For setup instructions, see the README in our GitHub samples repo.

Approach 1: Capture DOM + BeautifulSoup

The capture_dom approach gives you maximum control. Gaffa fetches the page and returns the raw HTML DOM. You then parse it locally with BeautifulSoup and extract the table data yourself.

Step 1: Fetch the DOM with Gaffa

Capturing a page DOM with Gaffa

import requests
import json
import os
from bs4 import BeautifulSoup

GAFFA_API_KEY = os.getenv("GAFFA_API_KEY")

def fetch_dom(url, proxy_location=None):
    payload = {
        "url": url,
        "proxy_location": proxy_location,
        "async": False,
        "max_cache_age": 0,
        "settings": {
            "record_request": False,
            "actions": [
                {
                    "type": "wait",
                    "selector": "table",
                    "timeout": 5000
                },
                {
                    "type": "capture_dom"
                }
            ]
        }
    }

    headers = {
        "x-api-key": GAFFA_API_KEY,
        "Content-Type": "application/json"
    }

    response = requests.post(
        "https://api.gaffa.dev/v1/browser/requests",
        json=payload,
        headers=headers
    )
    response.raise_for_status()

    dom_url = response.json()["data"]["actions"][1]["output"]
    dom_response = requests.get(dom_url)
    dom_response.raise_for_status()

    return dom_response.text

The wait action tells Gaffa to wait until a table element appears in the DOM, which is useful for pages where the table loads dynamically. The capture_dom action then returns the full HTML as a file URL, which we fetch separately.

Step 2: Parse the table with BeautifulSoup

Parsing a Table from the DOM using BeautifulSoup

def parse_table_from_dom(html, table_selector="table"):
    soup = BeautifulSoup(html, "html.parser")
    table = soup.select_one(table_selector)

    if not table:
        raise ValueError(f"No table found for selector: {table_selector}")

    headers = [th.get_text(strip=True) for th in table.select("thead th")]
    rows = []

    for tr in table.select("tbody tr"):
        cells = [td.get_text(strip=True) for td in tr.select("td")]
        if cells:
            rows.append(dict(zip(headers, cells)))

    return rows

Step 3: Run it and save the output

Saving the parsed table

if __name__ == "__main__":
    html = fetch_dom("https://demo.gaffa.dev/simulate/table?loadTime=1&rowCount=10")
    data = parse_table_from_dom(html)

    # Print to console
    print(json.dumps(data, indent=2))

    # Save to file
    with open("table_data.json", "w") as f:
        json.dump(data, f, indent=2)

    print(f"\nSaved {len(data)} rows to table_data.json")

Sample output:

Sample parsed table output

[
  {
    "ID": "1",
    "Name": "Item 1",
    "Quantity": "87",
    "Price": "$25.29"
  },
  {
    "ID": "2",
    "Name": "Item 2",
    "Quantity": "12",
    "Price": "$8.96"
  }
]

See the full sample output on GitHub.

This approach is flexible. If you need to clean up values, handle merged cells, or do any custom transformation before saving, BeautifulSoup gives you full control over the parsed data.

Approach 2: Using the parse_table action to get JSON with no processing

If you just want the data and don't need custom processing, Gaffa's parse_table action eliminates the need for the BeautifulSoup step entirely. It finds the table on the page, reads the headers, and returns a ready-to-use JSON object directly.

Here's what the action does internally:

It locates the table using your CSS selector
Converts the header row into property names (lowercased, non-alphanumeric characters replaced with underscores)
Then maps each cell value to its corresponding header for every row, returning a clean JSON array with no post-processing required on your end.

Using parse_table on the Demo Site

The Gaffa demo site at demo.gaffa.dev is a simple test environment with pre-built pages designed for trying out Gaffa actions before pointing them at a real site

Using Gaffa's parse_table action

import requests
import json
import os

GAFFA_API_KEY = os.getenv("GAFFA_API_KEY")

def fetch_parsed_table(url, selector="table", proxy_location=None):
    payload = {
        "url": url,
        "proxy_location": proxy_location,
        "async": False,
        "max_cache_age": 0,
        "settings": {
            "record_request": False,
            "actions": [
                {
                    "type": "parse_table",
                    "selector": selector,
                    "timeout": 5000
                }
            ]
        }
    }

    headers = {
        "x-api-key": GAFFA_API_KEY,
        "Content-Type": "application/json"
    }

    response = requests.post(
        "https://api.gaffa.dev/v1/browser/requests",
        json=payload,
        headers=headers
    )
    response.raise_for_status()

    result_url = response.json()["data"]["actions"][0]["output"]
    result_response = requests.get(result_url)
    result_response.raise_for_status()

    return result_response.json()


if __name__ == "__main__":
    data = fetch_parsed_table(
        url="https://demo.gaffa.dev/simulate/table?loadTime=1&rowCount=10"
    )
    print(json.dumps(data, indent=2))

That's the entire script: no HTML parsing, no BeautifulSoup, no column mapping. The output is already shaped as a list of objects you can use immediately.

Real-World Example

Let's apply parse_table to Wikipedia's List of Countries by GDP (Nominal), a clean, publicly accessible table of financial data. Wikipedia uses a consistent CSS class on all its data tables, making it straightforward and reliable to target with a selector.

Still using the same fetch_parsed_table function we wrote for the demo site, just swap the if name block with this:

Wikipedia table parsing using Gaffa

if __name__ == "__main__":
    data = fetch_parsed_table(
        url="https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)",
        selector="table.wikitable",
    )

    with open("gdp_data.json", "w") as f:
        json.dump(data, f, indent=2)

    print(f"Fetched {len(data)} records")
    print(json.dumps(data[:3], indent=2))

Sample output:

Sample table data

[
  {
    "country_territory": "World",
    "imf__2026__1": "123,584,494",
    "world_bank__2024__6": "111,326,370",
    "united_nations__2024__7": "100,834,796"
  },
  {
    "country_territory": "United States",
    "imf__2026__1": "31,821,293",
    "world_bank__2024__6": "28,750,956",
    "united_nations__2024__7": "29,298,000"
  }
]

See the full sample output on GitHub.

Notice how the original column headers, like “Country/Territory” and “IMF 2026”, are automatically normalised into “country_territory” and “imf_2026”. Spaces and special characters are replaced with underscores, and everything is lowercased, so the output is immediately usable without any cleanup.

No proxy_location is needed here, since Wikipedia is globally accessible, but for sites that restrict access by geography, you can simply add proxy_location="us" or another supported region to route the request through the appropriate IP address.

Which Approach Should You Use?

	`capture_dom`	`parse_table`
Setup	More code	Minimal code
Control	Full	Limited to what Gaffa returns
Best for	Complex tables, complex logic	Standard tables, fast extraction
Post-processing	Yes	None needed

If the table is straightforward and you just need the data, use parse_table. If you need to do any custom processing, such as merging columns, skipping rows, or reformatting values, use capture_dom with BeautifulSoup.

Either way, you're not managing browser infrastructure, rotating proxies, or writing anti-bot workarounds. Gaffa handles that layer so your Python code stays focused on what matters: the data.

Don't Want to Write Code Yet? Use the Playground

If you want to test parse_table or capture_dom before writing any Python, the Gaffa Playground lets you run requests directly from your browser with no code and no setup.

Just paste in your JSON payload, hit Send Request, and see the output instantly. It's a great way to confirm your selector is working and the table is returning the right data before you wire it up in a script.

Ultimately, the right approach depends on how much control you need. Use capture_dom with BeautifulSoup when you need full control over how the table is processed, or parse_table when you want structured table data without the extra parsing step.

Frequently Asked Questions

What is the easiest way to scrape a table from a website using Python?

Use Gaffa's parse_table action. Send a POST request with your target URL and CSS selector, and it returns a clean JSON array, no HTML parsing, no BeautifulSoup, no browser setup required.

How do I scrape a table and get raw HTML I can process myself?

Use the capture_dom action. It returns the fully rendered HTML of the page, which you can then parse manually with BeautifulSoup for custom processing, column merging, or data transformation.

How do I scrape a table and get clean JSON without any processing?

Use the parse_table action. It automatically finds the table, maps each row to its headers, and returns ready-to-use JSON, with no post-processing required.

Do I need to set up a headless browser or manage proxies to use Gaffa?

No. Gaffa handles browser infrastructure, anti-bot measures, and proxy routing on its end. Your Python code only needs to send an HTTP request.

How do I scrape tables from pages that load content dynamically?

Add a wait action before your capture_dom or parse_table action. It tells Gaffa to pause until the table element appears in the DOM before proceeding with extraction.

Can Gaffa scrape tables from geo-restricted websites?

Yes. Add proxy_location="us" (or another supported region) to your request payload to route the request through the appropriate IP address.

How does parse_table format the extracted data?

It automatically normalises column headers to lowercase, with underscores replacing spaces and special characters, so "Country/Territory" becomes country_territory, immediately usable with no cleanup needed.