How to Scrape a Table with Python (The Easy Way)

Web tables are goldmines of structured data, interest rate histories, sports standings, financial reports, and product comparisons. The problem is getting that data out cleanly. In this post, we'll walk through two ways to scrape a table using Python and Gaffa.

1. Manually, using capture_dom and BeautifulSoup
2. Automatically, using Gaffa's parse_table action.

You can find the full code for this post in our Python examples repository on GitHub.

The problem with traditional table scraping

Most tutorials recommend using Playwright or Selenium to load a page and retrieve the HTML. That works, but it means you're managing browser infrastructure yourself, dealing with headless Chrome setups, handling anti-bot detection, rotating proxies, and writing a non-trivial amount of boilerplate just to get to the data.

Gaffa handles all of that complexity for you. You send a POST request, and Gaffa spins up a real browser, handles anti-bot measures, optionally routes through proxies, and returns exactly what you ask for, no browser setup on your end.

We'll demonstrate two approaches to scraping tables using two different sites.

Getting Started

You'll need a Gaffa API key. Sign up at gaffa.dev and create your key in the API Keys section of the dashboard. For setup instructions, see the README in our GitHub samples repo.

Approach 1: Capture DOM + BeautifulSoup

The capture_dom approach gives you maximum control. Gaffa fetches the page and returns the raw HTML DOM. You then parse it locally with BeautifulSoup and extract the table data yourself.

Step 1: Fetch the DOM with Gaffa

Capturing a page DOM with Gaffa

import requests
import json
import os
from bs4 import BeautifulSoup

GAFFA_API_KEY = os.getenv("GAFFA_API_KEY")

def fetch_dom(url, proxy_location=None):
    payload = {
        "url": url,
        "proxy_location": proxy_location,
        "async": False,
        "max_cache_age": 0,
        "settings": {
            "record_request": False,
            "actions": [
                {
                    "type": "wait",
                    "selector": "table",
                    "timeout": 5000
                },
                {
                    "type": "capture_dom"
                }
            ]
        }
    }

    headers = {
        "x-api-key": GAFFA_API_KEY,
        "Content-Type": "application/json"
    }

    response = requests.post(
        "https://api.gaffa.dev/v1/browser/requests",
        json=payload,
        headers=headers
    )
    response.raise_for_status()

    dom_url = response.json()["data"]["actions"][1]["output"]
    dom_response = requests.get(dom_url)
    dom_response.raise_for_status()

    return dom_response.text

The wait action tells Gaffa to wait until a table element appears in the DOM, which is useful for pages where the table loads dynamically. The capture_dom action then returns the full HTML as a file URL, which we fetch separately.

Step 2: Parse the table with BeautifulSoup

Parsing a Table from the DOM using BeautifulSoup

def parse_table_from_dom(html, table_selector="table"):
    soup = BeautifulSoup(html, "html.parser")
    table = soup.select_one(table_selector)

    if not table:
        raise ValueError(f"No table found for selector: {table_selector}")

    headers = [th.get_text(strip=True) for th in table.select("thead th")]
    rows = []

    for tr in table.select("tbody tr"):
        cells = [td.get_text(strip=True) for td in tr.select("td")]
        if cells:
            rows.append(dict(zip(headers, cells)))

    return rows

Step 3: Run it and save the output

Saving the parsed table

if __name__ == "__main__":
    html = fetch_dom("https://demo.gaffa.dev/simulate/table?loadTime=1&rowCount=10")
    data = parse_table_from_dom(html)

    # Print to console
    print(json.dumps(data, indent=2))

    # Save to file
    with open("table_data.json", "w") as f:
        json.dump(data, f, indent=2)

    print(f"\nSaved {len(data)} rows to table_data.json")

Sample output:

Sample parsed table output

[
  {
    "ID": "1",
    "Name": "Item 1",
    "Quantity": "87",
    "Price": "$25.29"
  },
  {
    "ID": "2",
    "Name": "Item 2",
    "Quantity": "12",
    "Price": "$8.96"
  }
]

See the full sample output on GitHub.

This approach is flexible. If you need to clean up values, handle merged cells, or do any custom transformation before saving, BeautifulSoup gives you full control over the parsed data.

Approach 2: Using the parse_table action to get JSON with no processing

If you just want the data and don't need custom processing, Gaffa's parse_table action eliminates the need for the BeautifulSoup step entirely. It finds the table on the page, reads the headers, and returns a ready-to-use JSON object directly.

Here's what the action does internally:

1. It locates the table using your CSS selector
2. Converts the header row into property names (lowercased, non-alphanumeric characters replaced with underscores)
3. Then maps each cell value to its corresponding header for every row, returning a clean JSON array with no post-processing required on your end.

Using parse_table on the Demo Site

The Gaffa demo site at demo.gaffa.dev is a simple test environment with pre-built pages designed for trying out Gaffa actions before pointing them at a real site

Using Gaffa's parse_table action

import requests
import json
import os

GAFFA_API_KEY = os.getenv("GAFFA_API_KEY")

def fetch_parsed_table(url, selector="table", proxy_location=None):
    payload = {
        "url": url,
        "proxy_location": proxy_location,
        "async": False,
        "max_cache_age": 0,
        "settings": {
            "record_request": False,
            "actions": [
                {
                    "type": "parse_table",
                    "selector": selector,
                    "timeout": 5000
                }
            ]
        }
    }

    headers = {
        "x-api-key": GAFFA_API_KEY,
        "Content-Type": "application/json"
    }

    response = requests.post(
        "https://api.gaffa.dev/v1/browser/requests",
        json=payload,
        headers=headers
    )
    response.raise_for_status()

    result_url = response.json()["data"]["actions"][0]["output"]
    result_response = requests.get(result_url)
    result_response.raise_for_status()

    return result_response.json()


if __name__ == "__main__":
    data = fetch_parsed_table(
        url="https://demo.gaffa.dev/simulate/table?loadTime=1&rowCount=10"
    )
    print(json.dumps(data, indent=2))

That's the entire script: no HTML parsing, no BeautifulSoup, no column mapping. The output is already shaped as a list of objects you can use immediately.

Real-World Example

Let's apply parse_table to Wikipedia's List of Countries by GDP (Nominal), a clean, publicly accessible table of financial data. Wikipedia uses a consistent CSS class on all its data tables, making it straightforward and reliable to target with a selector.

Still using the same fetch_parsed_table function we wrote for the demo site, just swap the if name block with this:

Wikipedia table parsing using Gaffa

if __name__ == "__main__":
    data = fetch_parsed_table(
        url="https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)",
        selector="table.wikitable",
    )

    with open("gdp_data.json", "w") as f:
        json.dump(data, f, indent=2)

    print(f"Fetched {len(data)} records")
    print(json.dumps(data[:3], indent=2))

Sample output:

Sample table data

[
  {
    "country_territory": "World",
    "imf__2026__1": "123,584,494",
    "world_bank__2024__6": "111,326,370",
    "united_nations__2024__7": "100,834,796"
  },
  {
    "country_territory": "United States",
    "imf__2026__1": "31,821,293",
    "world_bank__2024__6": "28,750,956",
    "united_nations__2024__7": "29,298,000"
  }
]

See the full sample output on GitHub.

Notice how the original column headers, like “Country/Territory” and “IMF 2026”, are automatically normalised into “country_territory” and “imf_2026”. Spaces and special characters are replaced with underscores, and everything is lowercased, so the output is immediately usable without any cleanup.

No proxy_location is needed here, since Wikipedia is globally accessible, but for sites that restrict access by geography, you can simply add proxy_location="us" or another supported region to route the request through the appropriate IP address.

Which Approach Should You Use?

	capture_dom	parse_table
Setup	More code	Minimal code
Control	Full	Limited to what Gaffa returns
Best for	Complex tables, complex logic	Standard tables, fast extraction
Post-processing	Yes	None needed

If the table is straightforward and you just need the data, use parse_table. If you need to do any custom processing, such as merging columns, skipping rows, or reformatting values, use capture_dom with BeautifulSoup.

Either way, you're not managing browser infrastructure, rotating proxies, or writing anti-bot workarounds. Gaffa handles that layer so your Python code stays focused on what matters: the data.

Don't Want to Write Code Yet? Use the Playground

If you want to test parse_table or capture_dom before writing any Python, the Gaffa Playground lets you run requests directly from your browser with no code and no setup.

Just paste in your JSON payload, hit Send Request, and see the output instantly. It's a great way to confirm your selector is working and the table is returning the right data before you wire it up in a script.

Ultimately, the right approach depends on how much control you need. Use capture_dom with BeautifulSoup when you need full control over how the table is processed, or parse_table when you want structured table data without the extra parsing step.

Frequently Asked Questions

What is the easiest way to scrape a table from a website using Python?

Use Gaffa's parse_table action. Send a POST request with your target URL and CSS selector, and it returns a clean JSON array, no HTML parsing, no BeautifulSoup, no browser setup required.

How do I scrape a table and get raw HTML I can process myself?

Use the capture_dom action. It returns the fully rendered HTML of the page, which you can then parse manually with BeautifulSoup for custom processing, column merging, or data transformation.

How do I scrape a table and get clean JSON without any processing?

Use the parse_table action. It automatically finds the table, maps each row to its headers, and returns ready-to-use JSON, with no post-processing required.

Do I need to set up a headless browser or manage proxies to use Gaffa?

No. Gaffa handles browser infrastructure, anti-bot measures, and proxy routing on its end. Your Python code only needs to send an HTTP request.

How do I scrape tables from pages that load content dynamically?

Add a wait action before your capture_dom or parse_table action. It tells Gaffa to pause until the table element appears in the DOM before proceeding with extraction.

Can Gaffa scrape tables from geo-restricted websites?

Yes. Add proxy_location="us" (or another supported region) to your request payload to route the request through the appropriate IP address.

How does parse_table format the extracted data?

It automatically normalises column headers to lowercase, with underscores replacing spaces and special characters, so "Country/Territory" becomes country_territory, immediately usable with no cleanup needed.