DEV Community: circobit

Building a Type Inference System for Messy Web Data

circobit — Wed, 15 Apr 2026 14:00:00 +0000

Web tables are strings. Everything is a string. But when you export to JSON or SQL, you want:

"1,234.56" → 1234.56 (number)
"2024-03-15" → date type
"Yes" → true (boolean)
"N/A" → null

Building a reliable type inference system means handling the chaos of real-world data. Here's how I approached it for HTML Table Exporter.

The Problem: Ambiguity Everywhere

Consider these values:

Value	Could be...
`"1,234"`	Number 1234 (US) or 1.234 (EU)
`"01/02/03"`	Jan 2, 2003 (US) or Feb 1, 2003 (EU) or 2001-02-03
`"1"`	Integer 1 or boolean true
`"N/A"`	String or null
`"0"`	Integer 0 or boolean false

Type inference isn't about parsing—it's about guessing intent from ambiguous evidence.

Architecture: Column-Level Inference

Inferring types cell-by-cell is unreliable. The value "1" alone could be anything. But a column containing ["1", "2", "3", "4"] is clearly integers.

My approach:

Sample values from the column (up to 100)
Infer type for each non-null value
Aggregate to determine the column type
Apply 90% threshold—if 90%+ values match a type, use it

const DATA_TYPES = {
  STRING: "string",
  INTEGER: "integer", 
  NUMBER: "number",
  BOOLEAN: "boolean",
  DATE: "date",
  NULL: "null",
};

function inferColumnType(columnValues, maxSamples = 100) {
  const nonNullValues = columnValues.filter(v => v != null && v !== "");

  if (nonNullValues.length === 0) {
    return { type: DATA_TYPES.STRING, confidence: 0 };
  }

  const sample = nonNullValues.slice(0, maxSamples);

  // Count occurrences of each type
  const typeCounts = {
    [DATA_TYPES.INTEGER]: 0,
    [DATA_TYPES.NUMBER]: 0,
    [DATA_TYPES.BOOLEAN]: 0,
    [DATA_TYPES.DATE]: 0,
    [DATA_TYPES.STRING]: 0,
  };

  for (const val of sample) {
    const type = inferValueType(val);
    if (type === DATA_TYPES.INTEGER) {
      // INTEGER is a subset of NUMBER
      typeCounts[DATA_TYPES.INTEGER]++;
      typeCounts[DATA_TYPES.NUMBER]++;
    } else if (type !== DATA_TYPES.NULL) {
      typeCounts[type]++;
    }
  }

  const total = sample.length;
  const threshold = 0.9;

  // Check types in priority order
  if (typeCounts[DATA_TYPES.BOOLEAN] >= total * threshold) {
    return { type: DATA_TYPES.BOOLEAN, confidence: typeCounts[DATA_TYPES.BOOLEAN] / total };
  }
  if (typeCounts[DATA_TYPES.DATE] >= total * threshold) {
    return { type: DATA_TYPES.DATE, confidence: typeCounts[DATA_TYPES.DATE] / total };
  }
  if (typeCounts[DATA_TYPES.INTEGER] >= total * threshold) {
    return { type: DATA_TYPES.INTEGER, confidence: typeCounts[DATA_TYPES.INTEGER] / total };
  }
  if (typeCounts[DATA_TYPES.NUMBER] >= total * threshold) {
    return { type: DATA_TYPES.NUMBER, confidence: typeCounts[DATA_TYPES.NUMBER] / total };
  }

  return { type: DATA_TYPES.STRING, confidence: 1 };
}

Why 90%? Real data has noise. A column of 100 integers might have one "N/A". Requiring 100% match is too strict.

Value-Level Type Detection

The inferValueType function handles individual values:

function inferValueType(value) {
  if (value == null || value === "") {
    return DATA_TYPES.NULL;
  }

  const str = String(value).trim();
  if (str === "") return DATA_TYPES.NULL;

  // Boolean check
  const lowerStr = str.toLowerCase();
  if (["true", "false", "yes", "no", "sí", "si"].includes(lowerStr)) {
    return DATA_TYPES.BOOLEAN;
  }

  // Date check (ISO format preferred)
  if (/^\d{4}-\d{2}-\d{2}/.test(str)) {
    return DATA_TYPES.DATE;
  }
  if (/^\d{1,2}[/\-\.]\d{1,2}[/\-\.]\d{2,4}$/.test(str)) {
    return DATA_TYPES.DATE;
  }

  // Integer check (strict)
  if (/^-?\d+$/.test(str)) {
    return DATA_TYPES.INTEGER;
  }

  // Number check (with decimals and separators)
  const cleanedForNumber = str
    .replace(/[$€£¥%\s]/g, "")
    .replace(/,/g, ".");

  if (/^-?\d+(\.\d+)?$/.test(cleanedForNumber)) {
    return DATA_TYPES.NUMBER;
  }

  return DATA_TYPES.STRING;
}

The Number Normalization Challenge

European vs US number formats are the hardest ambiguity:

Value	US interpretation	EU interpretation
`"1,234"`	1234	1.234
`"1.234"`	1.234	1234
`"1,234.56"`	1234.56	Invalid
`"1.234,56"`	Invalid	1234.56

My heuristic for detecting format:

function normalizeNumber(value) {
  if (value == null) return value;

  let str = String(value).trim();

  // Remove currency symbols and whitespace
  str = str.replace(/[$€£¥\s]/g, "");

  // Handle percentage
  const isPercent = str.endsWith("%");
  if (isPercent) str = str.slice(0, -1);

  // Detect format by analyzing separators
  const commaCount = (str.match(/,/g) || []).length;
  const dotCount = (str.match(/\./g) || []).length;
  const lastComma = str.lastIndexOf(",");
  const lastDot = str.lastIndexOf(".");

  let normalized;

  if (commaCount === 0 && dotCount === 0) {
    // Plain integer: "1234"
    normalized = str;
  } else if (commaCount === 0 && dotCount === 1) {
    // Either "1.234" (EU thousand) or "1.23" (decimal)
    // Heuristic: 3 digits after dot = thousand separator
    const afterDot = str.slice(lastDot + 1);
    if (afterDot.length === 3 && /^\d+$/.test(afterDot)) {
      // Likely EU thousand separator
      normalized = str.replace(".", "");
    } else {
      // Likely decimal
      normalized = str;
    }
  } else if (commaCount === 1 && dotCount === 0) {
    // Either "1,234" (US thousand) or "1,23" (EU decimal)
    const afterComma = str.slice(lastComma + 1);
    if (afterComma.length === 3 && /^\d+$/.test(afterComma)) {
      // Likely US thousand separator
      normalized = str.replace(",", "");
    } else {
      // Likely EU decimal
      normalized = str.replace(",", ".");
    }
  } else if (lastDot > lastComma) {
    // "1,234.56" - US format
    normalized = str.replace(/,/g, "");
  } else if (lastComma > lastDot) {
    // "1.234,56" - EU format
    normalized = str.replace(/\./g, "").replace(",", ".");
  } else {
    // Ambiguous, return as-is
    return value;
  }

  const num = parseFloat(normalized);
  if (Number.isNaN(num)) return value;

  return isPercent ? num / 100 : num;
}

Key insight: The position and count of separators resolves most ambiguity. The last separator is usually the decimal separator.

Boolean Detection (With Gotchas)

Obvious booleans: "true", "false", "yes", "no"

But what about "0" and "1"?

// Problematic: This converts tax rates to booleans
// Column: [0, 0.05, 0.1, 0.2]  -> [false, 0.05, 0.1, 0.2]

My solution: Don't convert purely numeric values to booleans.

function applyBooleanNormalization(rows, booleanConfig) {
  const trueValues = new Set(booleanConfig.true.map(v => v.toLowerCase()));
  const falseValues = new Set(booleanConfig.false.map(v => v.toLowerCase()));

  return rows.map((row, rowIndex) => {
    if (rowIndex === 0) return row; // Skip header

    return row.map(cell => {
      if (cell == null) return cell;

      const cellStr = String(cell).toLowerCase().trim();

      // Skip if purely numeric (prevents 0 -> false in number columns)
      if (/^-?\d+(\.\d+)?$/.test(cellStr)) {
        return cell;
      }

      if (trueValues.has(cellStr)) return "true";
      if (falseValues.has(cellStr)) return "false";

      return cell;
    });
  });
}

Null Value Handling

Web data uses many representations for "no value":

"" (empty string)
"N/A", "n/a", "NA"
"-", "--"
"null", "NULL"
"none", "None"
"." (seen in government data)

Configurable null detection:

function applyNullValues(rows, nullPatterns) {
  const nullSet = new Set(nullPatterns.map(v => v.toLowerCase().trim()));

  return rows.map((row, rowIndex) => {
    if (rowIndex === 0) return row; // Skip header

    return row.map(cell => {
      if (cell == null) return null;

      const cellStr = String(cell).toLowerCase().trim();
      return nullSet.has(cellStr) ? null : cell;
    });
  });
}

// Usage
applyNullValues(rows, ["N/A", "n/a", "-", "--", "null", "none", "."]);

The Pipeline

Full type inference and cleaning pipeline:

function cleanTable(tableInfo, config) {
  let rows = cloneRows(tableInfo.rows);

  // 1. Trim all strings
  if (config.trimStrings) {
    rows = applyTrimStrings(rows);
  }

  // 2. Convert null patterns to actual null
  if (config.nullValues?.length) {
    rows = applyNullValues(rows, config.nullValues);
  }

  // 3. Normalize booleans (before numbers, to avoid 0->false issues)
  if (config.booleans) {
    rows = applyBooleanNormalization(rows, config.booleans);
  }

  // 4. Normalize numbers (handles EU/US formats)
  if (config.normalizeNumbers) {
    rows = applyNumberNormalization(rows);
  }

  // 5. Normalize dates (optional, format-specific)
  if (config.normalizeDates) {
    rows = applyDateNormalization(rows, config.dateFormat);
  }

  return { ...tableInfo, rows };
}

Testing Edge Cases

Type inference has many edge cases. I maintain a test suite:

// Number normalization tests
assertEqual(normalizeNumber("1,234.56"), 1234.56);    // US format
assertEqual(normalizeNumber("1.234,56"), 1234.56);    // EU format
assertEqual(normalizeNumber("€1.234,56"), 1234.56);   // With currency
assertEqual(normalizeNumber("45,5%"), 0.455);         // Percentage
assertEqual(normalizeNumber("1.234"), 1234);          // EU thousand
assertEqual(normalizeNumber("1.23"), 1.23);           // Decimal

// Boolean tests
assertEqual(inferValueType("Yes"), DATA_TYPES.BOOLEAN);
assertEqual(inferValueType("0"), DATA_TYPES.INTEGER);  // NOT boolean
assertEqual(inferValueType("0.5"), DATA_TYPES.NUMBER); // NOT boolean

Integration with Export

When exporting to SQL, use inferred types:

function inferSqlColumnTypes(rows) {
  const header = rows[0];
  const dataRows = rows.slice(1);

  return header.map((_, colIndex) => {
    const columnValues = dataRows.map(row => row[colIndex]);
    const { type } = inferColumnType(columnValues);

    switch (type) {
      case DATA_TYPES.INTEGER: return "INTEGER";
      case DATA_TYPES.NUMBER: return "REAL";
      case DATA_TYPES.BOOLEAN: return "BOOLEAN";
      case DATA_TYPES.DATE: return "DATE";
      default: return "TEXT";
    }
  });
}

Lessons Learned

Column context beats cell analysis. A single "1" is ambiguous; a column of integers is not.
Be conservative with conversion. It's better to leave something as a string than corrupt it with a wrong conversion.
Make it configurable. Different domains have different null values, boolean representations, and number formats.
Test with real data. Synthetic tests miss the chaos of actual web tables.

This system powers the data cleaning in HTML Table Exporter. For a practical example of these challenges, see how copying tables from websites to Excel often results in broken formatting that proper type inference can fix.

The PRO version lets users configure these rules per export profile. Learn more at gauchogrid.com/html-table-exporter or try it on the Chrome Web Store.

Building type inference for another domain? I'd love to hear about your edge cases.

Web Scraping vs Browser Extensions: When to Use Each for Data Extraction

circobit — Wed, 08 Apr 2026 14:00:00 +0000

You need data from a website. Do you write a Python scraper? Spin up Playwright? Use a browser extension? After extracting data from hundreds of different sites, I've developed a framework for choosing the right tool.

The Options

Approach	Runs	Handles JS	Login Support	Setup Time
Python + requests	Server	❌	Manual cookies	5 min
Python + BeautifulSoup	Server	❌	Manual cookies	5 min
Playwright/Puppeteer	Server	✅	Scriptable	15 min
Browser Extension	User's browser	✅	Automatic	0 min
Copy-paste	User's browser	✅	Automatic	0 min

Each has tradeoffs. Let's break them down.

Option 1: Python + Requests/BeautifulSoup

Best for: Static HTML pages, APIs, automated pipelines

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com/data")
soup = BeautifulSoup(response.text, "html.parser")

table = soup.find("table")
rows = []
for tr in table.find_all("tr"):
    row = [td.get_text(strip=True) for td in tr.find_all(["td", "th"])]
    rows.append(row)

Advantages:

Fast execution
Easy to schedule (cron, Airflow)
No browser overhead

Limitations:

Doesn't execute JavaScript
Many modern sites render tables client-side
Authentication requires manual cookie handling

When it fails: Try scraping a React-based dashboard. The HTML you get is an empty <div id="root"></div>.

Option 2: Headless Browsers (Playwright/Puppeteer)

Best for: JavaScript-rendered pages, automated pipelines, sites requiring interaction

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com/dashboard');

  // Wait for the table to render
  await page.waitForSelector('table');

  const data = await page.evaluate(() => {
    const table = document.querySelector('table');
    return Array.from(table.rows).map(row =>
      Array.from(row.cells).map(cell => cell.textContent.trim())
    );
  });

  console.log(data);
  await browser.close();
})();

Advantages:

Executes JavaScript
Can interact with pages (clicks, scrolls, form fills)
Scriptable authentication flows

Limitations:

Slower than direct HTTP
Resource intensive (memory, CPU)
More complex error handling
Sites may detect headless browsers

When it fails: Many sites detect Puppeteer/Playwright via navigator properties, WebGL fingerprinting, or behavioral analysis. Stealth plugins help but aren't foolproof.

Option 3: Browser Extensions

Best for: Ad-hoc extraction, authenticated sites, user-initiated workflows

A browser extension runs in the user's actual browser session, with full access to:

Authenticated sessions
JavaScript-rendered content
The exact DOM the user sees

For a comparison of the best options, see 5 Best Chrome Extensions to Export Tables from Websites.

// Content script in a Chrome extension
chrome.runtime.onMessage.addListener((msg, sender, sendResponse) => {
  if (msg.type === "EXTRACT_TABLE") {
    const table = document.querySelector("table");
    const data = Array.from(table.rows).map(row =>
      Array.from(row.cells).map(cell => cell.textContent.trim())
    );
    sendResponse({ data });
  }
});

Advantages:

Zero authentication handling—you're already logged in
Sees exactly what you see (no JS rendering issues)
Works on any site (no bot detection)
No server infrastructure needed

Limitations:

Requires manual trigger (user clicks)
Can't run on a schedule
Limited to browser context

When it fails: You need to extract data from 10,000 pages automatically. Extensions are for user-initiated workflows, not batch processing.

Decision Framework

Use Python + requests when:

✅ Page is server-rendered HTML
✅ You need automated/scheduled extraction
✅ You're building a data pipeline
✅ Authentication is via API keys or simple cookies

Use Playwright/Puppeteer when:

✅ Page requires JavaScript rendering
✅ You need automated/scheduled extraction
✅ You need to interact (scroll, click, paginate)
✅ You can handle bot detection countermeasures

Use a Browser Extension when:

✅ You're already logged in to the site
✅ You need data occasionally (not automated)
✅ The site has strong bot detection
✅ You want data NOW without writing code

Just copy-paste when:

✅ One-time extraction
✅ Simple table structure
✅ You don't need it in a specific format

Real-World Examples

Example 1: Wikipedia Tables

Best approach: Browser extension or Python

Wikipedia is server-rendered HTML with no authentication. Python works perfectly:

import pandas as pd

url = "https://en.wikipedia.org/wiki/List_of_countries_by_population"
tables = pd.read_html(url)
df = tables[0]

But Wikipedia tables often have complex rowspans and navigation rows. A dedicated extension handles these automatically.

For a step-by-step guide, see Export Wikipedia Tables to Excel in 30 Seconds.

Example 2: Financial Dashboard (Behind Login)

Best approach: Browser extension

Your brokerage shows portfolio data after login. Options:

Python: Reverse-engineer the authentication flow, handle 2FA, maintain session cookies. Possible but fragile.
Playwright: Script the login, handle 2FA prompts, navigate to the data. Works but complex.
Extension: Log in normally, click "Export Table." Done.

For occasional exports, the extension wins on time-to-data.

Example 3: Daily Price Monitoring (1000 Pages)

Best approach: Playwright + queue

You need to check prices across 1000 product pages daily. This requires:

// Pseudocode for batch extraction
const urls = loadUrlsFromDatabase();

for (const url of urls) {
  const page = await browser.newPage();
  await page.goto(url);
  await page.waitForSelector('.price');

  const price = await page.evaluate(() => 
    document.querySelector('.price').textContent
  );

  await saveToDatabase(url, price);
  await page.close();

  // Rate limiting
  await sleep(randomBetween(1000, 3000));
}

Extensions can't do this—they require user interaction. Playwright is the right tool.

Example 4: One-Time Sports Stats Export

Best approach: Browser extension

FBREF has complex two-level headers. You need this season's stats once.

Python approach: 30 minutes writing a custom parser for their table structure.

Extension approach: Click export. 10 seconds.

For one-time extraction, don't over-engineer it.

Hybrid Approaches

Sometimes you need both:

Use extension to export authentication cookies
- Export cookies from a logged-in session
- Import into Python/Playwright for automation
Use extension to inspect structure, Python to scale
- Manually examine the DOM with extension tools
- Write a targeted scraper once you understand the structure
Use Playwright for navigation, extension for extraction
- Script navigates to the page
- Calls extension API for complex table parsing

My Stack

For the 80% of cases where I need web table data:

One-time, authenticated: HTML Table Exporter (browser extension I built)
One-time, public data: pd.read_html() or extension
Automated pipeline: Playwright with custom parsers
API available: Direct API calls (always preferred)

The best tool depends on the specific extraction. Match the complexity of your solution to the complexity of your problem.

What's your go-to extraction approach? Learn more at gauchogrid.com/html-table-exporter or try the extension free on the Chrome Web Store.

How I Handle Nested Tables and Rowspans (The Hard Parts of HTML Table Parsing)

circobit — Wed, 01 Apr 2026 14:00:00 +0000

Parsing HTML tables seems straightforward until you encounter real-world data. Wikipedia tables have navigation rows. Financial sites use complex rowspans. Sports statistics sites nest headers two levels deep.

After building HTML Table Exporter, a table extraction tool used on thousands of different sites, I've catalogued the edge cases that break most parsers. Here's how to handle each one.

Problem 1: Rowspan Expansion

A cell with rowspan="3" occupies vertical space in the current row and the next two rows. If you iterate through row.cells naively, your columns misalign.

The broken output:

| Country | 2020 | 2021 | 2022 |    <- Header
| USA     | 100  | 200  | 300  |    <- Expected
| 150     | 250  | 350  |           <- Missing "USA" (rowspan continued)

The fix: Track occupied positions in a virtual grid.

function expandRowspans(table) {
  const rows = Array.from(table.rows);
  const grid = [];

  rows.forEach((rowEl, rowIndex) => {
    if (!grid[rowIndex]) grid[rowIndex] = [];
    let colIndex = 0;

    Array.from(rowEl.cells).forEach(cell => {
      // Find next unoccupied column
      while (grid[rowIndex][colIndex] !== undefined) {
        colIndex++;
      }

      const text = cell.textContent.trim();
      const rowSpan = parseInt(cell.rowSpan, 10) || 1;
      const colSpan = parseInt(cell.colSpan, 10) || 1;

      // Mark all cells this element spans
      for (let r = 0; r < rowSpan; r++) {
        const targetRow = rowIndex + r;
        if (!grid[targetRow]) grid[targetRow] = [];

        for (let c = 0; c < colSpan; c++) {
          grid[targetRow][colIndex + c] = text;
        }
      }

      colIndex += colSpan;
    });
  });

  // Normalize row lengths
  const maxCols = Math.max(...grid.map(r => r.length));
  return grid.map(row => {
    const normalized = new Array(maxCols).fill("");
    row.forEach((val, i) => normalized[i] = val ?? "");
    return normalized;
  });
}

Key insight: The virtual grid is the source of truth. The DOM cells are just instructions for populating it.

Problem 2: Nested Tables

Wikipedia infoboxes often contain tables within table cells. A recursive approach extracts garbage:

<table>
  <tr>
    <td>Country</td>
    <td>
      <table>  <!-- Nested! -->
        <tr><td>Population</td><td>330M</td></tr>
      </table>
    </td>
  </tr>
</table>

Detection strategy: Check if a table's ancestor is also a table.

function isNestedTable(table) {
  let parent = table.parentElement;

  while (parent) {
    if (parent.tagName === "TABLE") {
      return true;
    }
    parent = parent.parentElement;
  }

  return false;
}

// When scanning a page
function getTopLevelTables() {
  const all = document.querySelectorAll("table");
  return Array.from(all).filter(t => !isNestedTable(t));
}

But what about the nested table's content?

For the outer table, I flatten nested tables to their text content:

function extractCellText(cell) {
  const clone = cell.cloneNode(true);

  // Remove nested tables (their text is already included via textContent)
  clone.querySelectorAll("table").forEach(t => t.remove());

  // Remove invisible elements
  clone.querySelectorAll("style, script").forEach(el => el.remove());

  return (clone.textContent || "").replace(/\s+/g, " ").trim();
}

Problem 3: Wikipedia Navigation Rows

Wikipedia tables often start with a navigation row:

| v t e  List of countries by population |
| Rank | Country | Population |
| 1    | China   | 1.4B       |

That "v t e" row (View/Talk/Edit links) isn't data—it's UI. A parser that treats it as the header row produces garbage.

For a practical guide on handling Wikipedia tables, see Export Wikipedia Tables to Excel in 30 Seconds.

Detection:

function isWikipediaNavRow(row) {
  const firstCell = row[0] || "";

  // Common patterns for nav rows
  const patterns = [
    /^v\s+t\s+e\s/i,           // "v t e "
    /^\s*v\s*\|\s*t\s*\|\s*e/i, // "v | t | e"
    /^\[v\]\s*\[t\]\s*\[e\]/i   // "[v] [t] [e]"
  ];

  return patterns.some(p => p.test(firstCell));
}

function detectHeaderRowIndex(matrix) {
  for (let i = 0; i < Math.min(3, matrix.length - 1); i++) {
    if (isWikipediaNavRow(matrix[i])) {
      return i + 1;  // Header is the next row
    }
  }
  return 0;  // Default: first row is header
}

Problem 4: Title Rows (Spanning All Columns)

Some tables have a title row that spans the entire width:

<table>
  <tr><td colspan="4">Quarterly Revenue ($ millions)</td></tr>
  <tr><td>Q1</td><td>Q2</td><td>Q3</td><td>Q4</td></tr>
  <tr><td>100</td><td>120</td><td>115</td><td>130</td></tr>
</table>

After rowspan expansion, the first row becomes ["Quarterly Revenue...", "Quarterly Revenue...", ...]—the same value repeated.

Detection:

function isTitleRow(row, nextRow) {
  if (!row || !nextRow) return false;

  const uniqueValues = new Set(row.filter(v => v.trim()));
  const nextUniqueValues = new Set(nextRow.filter(v => v.trim()));

  // Title row characteristics:
  // 1. Only one unique value (repeated via colspan)
  // 2. Next row has multiple unique values (actual headers)
  // 3. The single value is long text (>30 chars typically)

  return (
    uniqueValues.size === 1 &&
    nextUniqueValues.size > 2 &&
    row[0] && row[0].length > 30
  );
}

Problem 5: Grouped Column Headers (FBREF Style)

Sports statistics sites like FBREF use two-level headers:

|        |        | Playing Time      | Performance    |
| Player | Nation | MP | Starts | Min | Gls | Ast | xG |
| Haaland| Norway | 35 | 33     | 2950| 36  | 8   | 32 |

The first row contains group names. The second row contains actual column names. Both are "headers."

The challenge: After colspan expansion, row 0 looks like:

["", "", "Playing Time", "Playing Time", "Playing Time", "Performance", "Performance", "Performance"]

Detection heuristics:

function isGroupHeaderRow(row, nextRow) {
  if (!row || !nextRow || row.length !== nextRow.length) return false;

  // Count how many cells have the same value as their neighbor
  let repeatCount = 0;
  for (let i = 1; i < row.length; i++) {
    if (row[i] && row[i] === row[i-1]) repeatCount++;
  }

  const repeatRatio = repeatCount / (row.length - 1);

  // Group header rows typically have 40%+ repeated values
  // AND the next row has more unique values
  const uniqueInRow = new Set(row.filter(v => v.trim())).size;
  const uniqueInNext = new Set(nextRow.filter(v => v.trim())).size;

  return repeatRatio > 0.4 && uniqueInNext > uniqueInRow;
}

Merging group + sub-headers:

function mergeGroupAndSubHeaders(groupRow, subHeaderRow) {
  return subHeaderRow.map((subHeader, idx) => {
    const group = (groupRow[idx] || "").trim();
    const sub = (subHeader || "").trim();

    if (!group) return sub;
    if (!sub) return group;
    if (sub.toLowerCase() === group.toLowerCase()) return sub;

    return `${group} - ${sub}`;
  });
}

// Result: ["Player", "Nation", "Playing Time - MP", "Playing Time - Starts", ...]

Problem 6: Horizontally Duplicated Tables

Wikipedia population tables often have this structure:

| Rank | Name   | Pop   | Rank | Name    | Pop   |
| 1    | Tokyo  | 37M   | 11   | Paris   | 11M   |
| 2    | Delhi  | 32M   | 12   | Cairo   | 10M   |

This is ONE logical table displayed in two columns to save vertical space.

Detection:

function detectHorizontalDuplication(headers) {
  const half = Math.floor(headers.length / 2);
  if (half < 2) return null;

  const firstHalf = headers.slice(0, half);
  const secondHalf = headers.slice(half, half * 2);

  // Check if second half matches first half
  const matches = firstHalf.every((h, i) => 
    h.toLowerCase() === secondHalf[i]?.toLowerCase()
  );

  if (matches) {
    return { detected: true, repeatCount: 2, baseColumns: half };
  }

  return null;
}

Normalization: Split each row and stack vertically:

function normalizeHorizontallyDuplicatedTable(matrix, baseColumns) {
  const header = matrix[0].slice(0, baseColumns);
  const normalizedRows = [header];

  for (let i = 1; i < matrix.length; i++) {
    const row = matrix[i];
    // First half
    normalizedRows.push(row.slice(0, baseColumns));
    // Second half (if not empty)
    const secondHalf = row.slice(baseColumns, baseColumns * 2);
    if (secondHalf.some(cell => cell.trim())) {
      normalizedRows.push(secondHalf);
    }
  }

  return normalizedRows;
}

The Combined Algorithm

Real-world parsing requires checking all these cases in sequence:

function parseTable(table) {
  // 1. Expand rowspans/colspans to virtual grid
  let matrix = expandRowspans(table);

  // 2. Detect and skip nav/title rows
  const headerIndex = detectHeaderRowIndex(matrix);
  if (headerIndex > 0) {
    matrix = matrix.slice(headerIndex);
  }

  // 3. Handle grouped headers (FBREF style)
  const groupedHeaders = detectGroupedColumnHeaders(matrix);
  if (groupedHeaders) {
    const mergedHeaders = mergeGroupAndSubHeaders(matrix[0], matrix[1]);
    matrix = [mergedHeaders, ...matrix.slice(2)];
  }

  // 4. Handle horizontal duplication
  const duplication = detectHorizontalDuplication(matrix[0]);
  if (duplication) {
    matrix = normalizeHorizontallyDuplicatedTable(matrix, duplication.baseColumns);
  }

  return matrix;
}

Testing These Edge Cases

Every pattern above came from a real bug report. I maintain a test suite with HTML fixtures for each:

// Test: Wikipedia-style nav row
const navRowHtml = `
  <table>
    <tr><td colspan="3">v t e Countries</td></tr>
    <tr><td>Rank</td><td>Country</td><td>Pop</td></tr>
    <tr><td>1</td><td>China</td><td>1.4B</td></tr>
  </table>
`;

const result = parseTable(parseHtml(navRowHtml));
assert(result[0][0] === "Rank");  // Header correctly identified
assert(result[1][1] === "China"); // Data correctly aligned

The test suite has 24 cases covering combinations of these patterns. New bug reports become new test cases.

Try It Yourself

If you're building table extraction, I hope this saves you debugging time. If you just need to export tables without writing code, HTML Table Exporter handles all these cases automatically.

Learn more at gauchogrid.com/html-table-exporter or try it free on the Chrome Web Store.

Found a table that breaks your parser? Share the URL; I collect these edge cases.

Automating Web Table Exports with JavaScript: A Practical Guide

circobit — Wed, 25 Mar 2026 15:00:00 +0000

You've found a table on a website. You need that data in a spreadsheet. The obvious path—copy, paste, clean up in Excel—works once. But what if you need this data weekly? Or from 50 different pages?

This guide shows you how to extract HTML tables programmatically with JavaScript, handle the edge cases that break naive approaches, and export to formats your tools actually accept.

The Naive Approach (And Why It Fails)

The simplest extraction looks like this:

function extractTable(table) {
  return Array.from(table.rows).map(row => 
    Array.from(row.cells).map(cell => cell.textContent.trim())
  );
}

This works for simple tables. It breaks immediately when you hit:

Rowspan/colspan — Cells that span multiple rows or columns
Nested tables — Tables inside table cells
Hidden content — <style>, <script>, or display:none elements
Special characters — Newlines, tabs, and quotes in cell content

Let's fix each one.

Handling Rowspan and Colspan

When a cell has rowspan="2", it occupies space in the current row AND the next row. A naive extractor sees fewer cells than expected and misaligns columns.

The solution: build a virtual grid that tracks occupied positions.

function extractTableMatrix(table) {
  const rows = Array.from(table.rows);
  const grid = [];

  rows.forEach((rowEl, rowIndex) => {
    if (!grid[rowIndex]) grid[rowIndex] = [];

    let colIndex = 0;

    Array.from(rowEl.cells).forEach(cell => {
      // Skip columns already occupied by previous rowspans
      while (grid[rowIndex][colIndex] !== undefined) {
        colIndex++;
      }

      const text = cell.textContent.trim();
      const rowSpan = parseInt(cell.rowSpan, 10) || 1;
      const colSpan = parseInt(cell.colSpan, 10) || 1;

      // Fill the rectangular block this cell occupies
      for (let r = 0; r < rowSpan; r++) {
        const targetRow = rowIndex + r;
        if (!grid[targetRow]) grid[targetRow] = [];

        for (let c = 0; c < colSpan; c++) {
          const targetCol = colIndex + c;
          if (grid[targetRow][targetCol] === undefined) {
            grid[targetRow][targetCol] = text;
          }
        }
      }

      colIndex += colSpan;
    });
  });

  return grid;
}

Now a table like this:

<table>
  <tr><td rowspan="2">A</td><td>B</td></tr>
  <tr><td>C</td></tr>
</table>

Correctly becomes:

[
  ["A", "B"],
  ["A", "C"]  // "A" appears in both rows
]

Extracting Clean Text

textContent grabs everything—including CSS rules in <style> tags and JavaScript in <script> tags that some pages inject into table cells.

Clean extraction requires filtering:

function extractCellText(cell) {
  if (!cell) return "";

  // Clone to avoid modifying the DOM
  const clone = cell.cloneNode(true);

  // Remove invisible elements
  const invisibleSelectors = "style, script, noscript, template, link";
  clone.querySelectorAll(invisibleSelectors).forEach(el => el.remove());

  // Normalize whitespace
  return (clone.textContent || "").replace(/\s+/g, " ").trim();
}

Detecting Nested Tables

When a table contains another table in a cell, you typically want the outer table's data, not a recursive mess.

Detection is straightforward:

function isNestedTable(table, allTables) {
  let parent = table.parentElement;

  while (parent) {
    if (parent.tagName === "TABLE") {
      return true;  // This table is inside another table
    }
    parent = parent.parentElement;
  }

  return false;
}

// Filter when scanning a page
const allTables = document.querySelectorAll("table");
const topLevelTables = Array.from(allTables)
  .filter(t => !isNestedTable(t, allTables));

Converting to CSV

CSV looks simple until you need to handle:

Commas inside values
Quotes inside values
Newlines inside values

The RFC 4180 compliant approach:

function toCSV(rows, delimiter = ",") {
  return rows.map(row =>
    row.map(cell => {
      if (cell == null) cell = "";
      const str = String(cell);

      // Quote if contains delimiter, quotes, or newlines
      const needsQuotes = str.includes(delimiter) || /["\r\n]/.test(str);
      const escaped = str.replace(/"/g, '""');

      return needsQuotes ? `"${escaped}"` : escaped;
    }).join(delimiter)
  ).join("\r\n");
}

This correctly handles the nightmare case:

toCSV([['Say "Hello, World"', "Normal"]])
// '"Say ""Hello, World""",Normal'

For a complete guide on CSV exports, see How to Export HTML Tables to CSV in Chrome.

Converting to JSON

For JSON export, the first row becomes keys:

function toJSON(rows) {
  if (rows.length < 2) return "[]";

  const headers = rows[0].map((h, i) => sanitizeKey(h, i));
  const dataRows = rows.slice(1);

  const objects = dataRows.map(row => {
    const obj = {};
    headers.forEach((key, i) => {
      obj[key] = row[i] ?? "";
    });
    return obj;
  });

  return JSON.stringify(objects, null, 2);
}

function sanitizeKey(header, index) {
  let key = (header || "").toString().trim();

  if (!key) return `col_${index + 1}`;

  // Normalize to lowercase snake_case
  return key
    .normalize("NFD")
    .replace(/[\u0300-\u036f]/g, "")  // Remove accents
    .toLowerCase()
    .replace(/[^a-z0-9]+/g, "_")
    .replace(/^_+|_+$/g, "");
}

Input:

| Product Name | Price ($) |
|--------------|-----------|
| Widget       | 29.99     |

Output:

[
  {
    "product_name": "Widget",
    "price": "29.99"
  }
]

Triggering the Download

In a browser context, you can trigger a download without a server:

function downloadFile(content, filename, mimeType) {
  const blob = new Blob([content], { type: mimeType });
  const url = URL.createObjectURL(blob);

  const link = document.createElement("a");
  link.href = url;
  link.download = filename;
  link.click();

  URL.revokeObjectURL(url);
}

// Usage
const csv = toCSV(extractTableMatrix(table));
downloadFile(csv, "data.csv", "text/csv;charset=utf-8");

Putting It All Together

Here's a minimal bookmarklet that exports the first table on any page:

javascript:(function(){
  const table = document.querySelector("table");
  if (!table) { alert("No table found"); return; }

  function extractTableMatrix(table) {
    const rows = Array.from(table.rows);
    const grid = [];
    rows.forEach((rowEl, ri) => {
      if (!grid[ri]) grid[ri] = [];
      let ci = 0;
      Array.from(rowEl.cells).forEach(cell => {
        while (grid[ri][ci] !== undefined) ci++;
        const text = cell.textContent.trim();
        const rs = parseInt(cell.rowSpan) || 1;
        const cs = parseInt(cell.colSpan) || 1;
        for (let r = 0; r < rs; r++) {
          if (!grid[ri+r]) grid[ri+r] = [];
          for (let c = 0; c < cs; c++) {
            if (grid[ri+r][ci+c] === undefined) grid[ri+r][ci+c] = text;
          }
        }
        ci += cs;
      });
    });
    return grid;
  }

  const data = extractTableMatrix(table);
  const csv = data.map(row => 
    row.map(c => c.includes(",") ? `"${c}"` : c).join(",")
  ).join("\n");

  const blob = new Blob([csv], {type: "text/csv"});
  const link = document.createElement("a");
  link.href = URL.createObjectURL(blob);
  link.download = "table.csv";
  link.click();
})();

When to Use a Browser Extension Instead

This code works, but maintaining it across different sites is tedious. If you're extracting tables regularly, a browser extension handles:

Multiple tables per page
Format selection (CSV, JSON, Excel)
Data cleaning (number normalization, null handling)
Column selection and reordering

I built HTML Table Exporter for exactly this workflow. The core algorithms are similar to what's shown here, packaged in a usable UI.

Learn more at gauchogrid.com/html-table-exporter or try it free on the Chrome Web Store.

Questions about table extraction edge cases? Drop a comment; I've probably hit it.

From Web Table to Pandas DataFrame in 30 Seconds

circobit — Wed, 18 Mar 2026 15:00:00 +0000

You found the perfect dataset on a website. Now you need it in Pandas.

The traditional approach:

import pandas as pd

# Hope the website structure is simple
tables = pd.read_html('https://example.com/data')

# Guess which table you want
df = tables[0]  # Maybe? Let's see...

# Discover the problems
print(df.dtypes)
# Everything is 'object' (string)
# Numbers have commas
# Dates are unparseable
# Column names have spaces

# Spend 30 minutes cleaning...

Let me show you a faster way.

The Problem with `pd.read_html()`

Pandas' read_html() is convenient but limited:

No table selection — It grabs all tables. You guess which index you need.
No cleaning — Numbers like "1,234,567" stay as strings.
CORS issues — Many sites block programmatic access.
JavaScript rendering — Dynamic tables don't exist in the raw HTML.
Authentication — Can't access logged-in content.

For quick scripts, it works. For real analysis, you need something better.

The 30-Second Workflow

Here's what I actually do:

Step 1: Export from Browser (5 seconds)

Using HTML Table Exporter:

Click the table I want
Click the extension icon
Select "For Pandas" profile
Export as CSV from the highlighted table within the extension

The extension sees exactly what your browser sees—JavaScript-rendered content, authenticated pages, everything.

Step 2: Load in Pandas (5 seconds)

import pandas as pd

df = pd.read_csv('export.csv')
print(df.dtypes)

That's it. The data is already clean.

What "Clean" Actually Means

When I export with the "For Pandas" profile, the extension handles:

Number Normalization

Before: "1.234.567,89" (European format)
After:  1234567.89     (float)

Before: "$1,234.56"
After:  1234.56

The CSV contains normalized numbers that Pandas parses correctly:

# Without cleaning:
df['revenue'].sum()  # TypeError: can only concatenate str

# With cleaning:
df['revenue'].sum()  # 4892341.50 ✓

Boolean Conversion

Before: "Yes", "No", "Y", "N", "True", "False"
After:  true, false

# Filter works immediately
active_users = df[df['is_active'] == True]

Null Handling

Before: "-", "N/A", "n/a", "", "null", "—"
After:  (empty, parsed as NaN)

# Null detection works
df['optional_field'].isna().sum()  # Correct count

Snake Case Headers

Before: "Revenue ($M)", "User Count", "Growth Rate %"
After:  revenue_m, user_count, growth_rate

# Clean column access
df['revenue_m']  # Instead of df['Revenue ($M)']

Real Example: FBRef Football Stats

Let's say I want Premier League player statistics from FBRef.

The Old Way

import pandas as pd
import requests
from bs4 import BeautifulSoup

url = 'https://fbref.com/en/comps/9/stats/Premier-League-Stats'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

# Find the right table (there are many)
table = soup.find('table', {'id': 'stats_standard'})

# Parse it manually because the headers are complex
# FBRef uses grouped headers: "Playing Time" spans multiple columns
# This breaks pd.read_html()

# 45 minutes later...

The New Way

Open FBRef in browser
Click extension → select table → export with "For Pandas" profile
Load:

df = pd.read_csv('fbref_stats.csv')
print(df.columns.tolist())
# ['player', 'nation', 'squad', 'playing_time_mp', 
#  'playing_time_starts', 'performance_gls', ...]

The grouped headers ("Playing Time", "Performance") are merged with sub-headers automatically.

The Code I No Longer Write

Here's cleanup code I used to write for every web scrape:

def clean_web_data(df):
    """The function I no longer need."""

    # Fix number columns
    for col in df.select_dtypes(include='object'):
        # Try to convert to numeric
        try:
            # Remove currency symbols
            cleaned = df[col].str.replace(r'[$€£¥]', '', regex=True)
            # Remove thousands separators
            cleaned = cleaned.str.replace(',', '')
            # Convert
            df[col] = pd.to_numeric(cleaned, errors='ignore')
        except:
            pass

    # Fix boolean columns
    bool_map = {
        'yes': True, 'no': False,
        'true': True, 'false': False,
        'y': True, 'n': False,
        '1': True, '0': False,
    }
    for col in df.columns:
        if df[col].str.lower().isin(bool_map.keys()).all():
            df[col] = df[col].str.lower().map(bool_map)

    # Fix null values
    null_values = ['', '-', 'N/A', 'n/a', 'null', 'NULL', '—', '–']
    df = df.replace(null_values, np.nan)

    # Fix column names
    df.columns = (df.columns
        .str.lower()
        .str.replace(r'[^a-z0-9]+', '_', regex=True)
        .str.strip('_'))

    return df

This function ran on every dataset. Now the export handles it.

When to Use What

Scenario	Best Approach
One-time analysis	Browser export → CSV → Pandas
Repeated scraping	Python script with `requests`
JavaScript-heavy sites	Browser export (sees rendered content)
Authenticated data	Browser export (uses your session)
API available	Use the API directly
Simple static table	`pd.read_html()` is fine

Pro Tip: JSON for Complex Data

For nested or typed data, export as JSON:

import pandas as pd
import json

with open('export.json') as f:
    data = json.load(f)

df = pd.DataFrame(data)

The JSON export preserves types:

Numbers as numbers (not strings)
Booleans as booleans
Nulls as null

print(df.dtypes)
# player         object
# goals          int64   # Already numeric!
# is_starter     bool    # Already boolean!
# injury_date    object  # Can parse as datetime

For more details on JSON workflows, see Export Web Tables to JSON for Python & Pandas.

The Workflow Summary

Old workflow (30+ minutes):

Write scraping script
Handle CORS/auth issues
Parse complex HTML
Clean numbers
Clean booleans
Clean nulls
Fix column names
Debug edge cases
Finally: analyze

New workflow (30 seconds):

Click extension
Export with cleaning profile
pd.read_csv()
Analyze

Try It

Install HTML Table Exporter
Find a table you want to analyze
Export with cleaning presets
Load in Pandas

The free version handles basic exports. PRO adds the cleaning presets and profiles for Pandas-optimized output.

Learn more at gauchogrid.com/html-table-exporter or try it on the Chrome Web Store.

What's your current workflow for getting web data into Pandas? I'm curious how much time you spend on the cleaning step. Drop a comment below.

Why Data Analysts Hate Copy-Paste from Websites

circobit — Wed, 11 Mar 2026 15:00:00 +0000

Every data analyst has done it. You find the perfect dataset on a website, select the table, Ctrl+C, switch to Excel, Ctrl+V.

And then the pain begins.

What Actually Happens When You Copy-Paste

Let me walk you through what should be a 30-second task.

Step 1: Find the table on a website

Step 2: Select it (carefully avoiding the surrounding text, navigation, and ads)

Step 3: Copy

Step 4: Paste into Excel

Step 5: Discover that:

Numbers are text, not numbers
Dates are in the wrong format
Some columns merged incorrectly
There's invisible characters breaking your formulas
The formatting is a mess

Step 6-20: Fix everything manually

I tracked my time once. A "quick" copy-paste of a 50-row table took 23 minutes to clean up. Multiply that by the dozens of tables analysts work with weekly, and you're losing hours to data entry—not analysis.

The Hidden Problems

Problem 1: Numbers as Text

You paste 1,234 and Excel sees text, not the number 1234.

Why? The comma. In US format, it's a thousands separator. In European format, it's a decimal. Excel doesn't know which you meant, so it plays it safe and keeps it as text.

Now your =SUM() formula returns 0, and you spend 10 minutes figuring out why.

Original:    1,234,567.89
Pasted as:   "1,234,567.89" (text)
You wanted:  1234567.89 (number)

Problem 2: European vs US Decimals

Half the world uses . for decimals. The other half uses ,.

US format:       1,234.56
European format: 1.234,56

Copy from a German website, paste into US Excel: nothing works.

Problem 3: Hidden Characters

Websites love invisible characters:

Non-breaking spaces ( )
Zero-width spaces
Tab characters
Newlines inside cells

Your cell looks empty but =ISBLANK() returns FALSE. Your VLOOKUP fails because " John" ≠ "John".

// What the cell contains:
"\u00a0John Smith\u200b"

// What you see:
"John Smith"

// Why your formulas break:
// The invisible characters are still there

Problem 4: Merged Cells

Tables with rowspan/colspan paste incorrectly. Merged cells become single values in the wrong position:

Original table:

| Category  | Q1  | Q2  |
| Electronics | $1M | $2M |
|           | Phones: $500K | Phones: $800K |

After paste:

| Category | Q1 | Q2 |
| Electronics | $1M | $2M |
| Phones: $500K | Phones: $800K | (empty) |

The sub-category row shifted left because the merged "Category" cell wasn't repeated.

For a detailed guide on handling these issues without code, see How to Scrape Tables from Websites Without Code.

Problem 5: Multi-Row Headers

Many data tables have grouped headers:

|           | Q1      | Q2      |
| Region    | Sales   | Sales   |
|           | ($)     | (units) |

Copy-paste flattens this. You lose the context that the first "Sales" is dollars and the second is units.

Problem 6: Dates From Hell

Web tables display dates however they want:

02/03/2024 — Is this Feb 3 or March 2?
2024.02.03
Feb 3, 2024
3-Feb-24

Excel guesses. Excel guesses wrong.

Original:    03/02/2024
Your locale: US (MM/DD/YYYY)
You wanted:  February 3, 2024
You got:     March 2, 2024

One wrong date cascades through your entire analysis.

The Real Cost

Let's do the math.

Conservative estimate:

5 tables per week
15 minutes average cleanup per table
75 minutes per week

Per year: 65 hours of copy-paste cleanup

That's almost two full work weeks spent on data entry, not analysis.

And that's assuming you catch all the errors. The errors you don't catch? Those become wrong conclusions, bad decisions, embarrassing corrections.

The Alternatives

Option 1: Web Scraping (Overkill)

You could write a Python script:

import pandas as pd

tables = pd.read_html('https://example.com/data')
df = tables[0]

But now you need:

Python environment set up
Dependencies installed
Script maintenance when the site changes
10 minutes of setup for a 30-second task

Web scraping is powerful but overkill for "I just need this one table."

Option 2: Browser DevTools (Technical)

Open DevTools, find the table element, copy the HTML, parse it yourself.

Great if you're a developer. Terrible if you just want data.

Option 3: Browser Extension (One Click)

This is why I built HTML Table Exporter.

For a step-by-step walkthrough, see Copy Any Table from a Website to Excel.

Click the extension icon
Select the table
Choose format (CSV, Excel, JSON)
Click Export

The extension handles:

✅ Rowspan/colspan (builds a proper grid)
✅ Number normalization (European and US formats)
✅ Hidden characters (strips invisible content)
✅ Multi-row headers (merges them intelligently)
✅ Clean text extraction (no style tags, no scripts)

Time spent: 5 seconds.

When Copy-Paste Is Fine

To be fair, copy-paste works for:

Simple tables with no merged cells
Plain text with no special formatting
One-off tasks where cleanup time doesn't matter
Tables you'll manually review anyway

But if you're doing this regularly, with real data, for actual analysis—stop suffering.

The Workflow That Actually Works

Here's what I do now:

Find the data on any website
Click the extension → select table → export as CSV
Open in Excel/Sheets → data is already clean
Start analyzing immediately

No cleanup. No formula debugging. No invisible character hunting.

The 23-minute task becomes 30 seconds.

For Power Users: Cleaning Presets

If you're exporting data regularly for Python/Pandas analysis, the PRO version includes cleaning presets:

Original:      "1.234.567,89"  (European)
Normalized:    "1234567.89"   (Standard)

Original:      "Yes", "No", "N/A"
Normalized:    true, false, null

Original:      "Revenue ($M)"
Normalized:    "revenue_m"    (snake_case)

One profile configured, every export is analysis-ready.

The Bottom Line

Copy-paste from websites is a tax on your time. It feels quick, but the cleanup adds up.

If you work with web data regularly:

Stop accepting the pain as normal
Use a proper extraction tool
Spend your time on analysis, not data entry

HTML Table Exporter is free for basic exports (CSV, JSON, Excel). PRO adds advanced cleaning and automation for power users. Try it on the Chrome Web Store.

How much time do you spend cleaning pasted data? I'm curious if my 65 hours/year estimate resonates. Share your horror stories below.

The Hidden Complexity of HTML Tables

circobit — Wed, 04 Mar 2026 15:00:00 +0000

HTML tables look simple. Rows and cells. What could go wrong?

Everything, it turns out.

I've spent months building HTML Table Exporter, a table extraction tool, and I've learned that HTML tables are deceptively complex. Here's what nobody tells you when you start parsing them.

The Naive Approach (That Breaks Immediately)

Your first attempt probably looks like this:

function extractTable(table) {
  return Array.from(table.rows).map(row =>
    Array.from(row.cells).map(cell => cell.textContent.trim())
  );
}

Clean, simple, wrong.

This works for basic tables. But the real web is full of tables that will break this code in creative ways.

Problem 1: Rowspan and Colspan

HTML cells can span multiple rows or columns:

<table>
  <tr>
    <td rowspan="3">Category A</td>
    <td>Item 1</td>
    <td>$10</td>
  </tr>
  <tr>
    <td>Item 2</td>
    <td>$20</td>
  </tr>
  <tr>
    <td>Item 3</td>
    <td>$30</td>
  </tr>
</table>

The naive approach gives you:

Row 0: ["Category A", "Item 1", "$10"]
Row 1: ["Item 2", "$20"]           // Only 2 cells!
Row 2: ["Item 3", "$30"]           // Only 2 cells!

Rows 1 and 2 are missing a column. Your column alignment is now broken.

The Fix: Build a Virtual Grid

You need to track which cells are "occupied" by spans from previous rows:

function extractTableMatrix(table) {
  const rows = Array.from(table.rows);
  const grid = [];

  rows.forEach((rowEl, rowIndex) => {
    if (!grid[rowIndex]) grid[rowIndex] = [];
    let colIndex = 0;

    // Skip columns already occupied by rowspans
    while (grid[rowIndex][colIndex] !== undefined) {
      colIndex++;
    }

    Array.from(rowEl.cells).forEach((cell) => {
      // Keep advancing past occupied columns
      while (grid[rowIndex][colIndex] !== undefined) {
        colIndex++;
      }

      const text = cell.textContent.trim();
      const rowSpan = cell.rowSpan || 1;
      const colSpan = cell.colSpan || 1;

      // Fill the entire span region
      for (let r = 0; r < rowSpan; r++) {
        for (let c = 0; c < colSpan; c++) {
          const targetRow = rowIndex + r;
          const targetCol = colIndex + c;

          if (!grid[targetRow]) grid[targetRow] = [];
          if (grid[targetRow][targetCol] === undefined) {
            grid[targetRow][targetCol] = text;
          }
        }
      }

      colIndex += colSpan;
    });
  });

  return grid;
}

Now you get:

Row 0: ["Category A", "Item 1", "$10"]
Row 1: ["Category A", "Item 2", "$20"]
Row 2: ["Category A", "Item 3", "$30"]

The spanning cell's value is duplicated into each position it occupies.

Problem 2: Nested Tables

Some sites use tables inside tables for layout:

<table>
  <tr>
    <td>
      <table>
        <tr><td>Nested content</td></tr>
      </table>
    </td>
    <td>Regular cell</td>
  </tr>
</table>

Using cell.textContent captures everything, including nested table content. Using cell.innerText might help, but it's inconsistent across browsers.

The Fix: Clone and Remove

function extractCellText(cell) {
  if (!cell) return "";

  // Clone to avoid modifying the DOM
  const clone = cell.cloneNode(true);

  // Remove invisible and nested content
  const removeSelectors = "style, script, noscript, template, table";
  clone.querySelectorAll(removeSelectors).forEach(el => el.remove());

  // Normalize whitespace
  return (clone.textContent || "").replace(/\s+/g, " ").trim();
}

Problem 3: Headers Aren't Always Row 0

Many tables have title rows, navigation rows, or other content before the actual headers:

<table>
  <tr>
    <td colspan="3">Quarterly Sales Report 2024</td>
  </tr>
  <tr>
    <th>Region</th>
    <th>Q1</th>
    <th>Q2</th>
  </tr>
  <tr>
    <td>North</td>
    <td>$1.2M</td>
    <td>$1.4M</td>
  </tr>
</table>

If you assume row 0 is headers, you'll get "Quarterly Sales Report 2024" as your column name.

The Fix: Detect Header Row

function detectHeaderRowIndex(matrix) {
  for (let i = 0; i < Math.min(matrix.length - 1, 3); i++) {
    const row = matrix[i];
    const nextRow = matrix[i + 1];

    // Count unique values
    const uniqueValues = new Set(row.filter(c => c && c.trim()));
    const uniqueNext = new Set(nextRow.filter(c => c && c.trim()));

    // Title row: 1 unique value (spans all columns), next row has more
    const isTitleRow = 
      uniqueValues.size === 1 && 
      uniqueNext.size > 1 &&
      row[0]?.length > 30;

    if (isTitleRow) {
      return i + 1; // Headers are in the next row
    }
  }

  return 0;
}

Problem 4: Empty Cells and Inconsistent Columns

Some rows have fewer cells than others. Some cells are completely empty. Some have just whitespace or  .

// Normalize all rows to the same length
function normalizeMatrix(grid) {
  const maxCols = Math.max(...grid.map(row => row.length));

  return grid.map(row => {
    const normalized = new Array(maxCols);
    for (let i = 0; i < maxCols; i++) {
      normalized[i] = row[i] ?? "";
    }
    return normalized;
  });
}

Problem 5: Invisible Content

Tables can contain content that's visually hidden but present in the DOM:

<style> blocks for scoped CSS
<script> tags
display: none elements
Zero-width characters

const invisibleSelectors = [
  "style",
  "script", 
  "noscript",
  "template",
  "[hidden]",
  "[style*='display: none']",
  "[style*='display:none']"
].join(", ");

clone.querySelectorAll(invisibleSelectors).forEach(el => el.remove());

Problem 6: Character Encoding

Tables from different sources use different encodings:

  (non-breaking space)
— and – (dashes)
Smart quotes vs straight quotes
UTF-8 vs ISO-8859-1

Always normalize:

function normalizeText(text) {
  return text
    .replace(/\u00a0/g, " ")           // Non-breaking space
    .replace(/[\u2018\u2019]/g, "'")   // Smart single quotes
    .replace(/[\u201c\u201d]/g, '"')   // Smart double quotes
    .replace(/[\u2013\u2014]/g, "-")   // En/em dashes
    .trim();
}

The Complete Pipeline

After handling all these cases, my extraction pipeline looks like this:

function extractTable(tableElement) {
  // 1. Build virtual grid (handles rowspan/colspan)
  let matrix = extractTableMatrix(tableElement);

  // 2. Normalize to consistent column count
  matrix = normalizeMatrix(matrix);

  // 3. Find actual header row
  const headerIndex = detectHeaderRowIndex(matrix);

  // 4. Remove title rows
  if (headerIndex > 0) {
    matrix = matrix.slice(headerIndex);
  }

  // 5. Clean all cell text
  matrix = matrix.map(row => 
    row.map(cell => normalizeText(extractCellText(cell)))
  );

  return matrix;
}

Edge Cases I've Encountered

After processing thousands of tables, here are some real-world edge cases:

Source	Issue	Solution
Wikipedia	"v t e" navigation rows	Pattern detection + skip
Wikipedia	Horizontally duplicated columns	Detect repeating headers, unstack
FBRef	Grouped column headers	Merge group + sub-header
Financial sites	Numbers as `1.234.567,89`	Locale-aware normalization
Government sites	Tables inside `<form>`	Ignore form wrapper

Lessons Learned

Never trust the DOM structure. Build your own normalized representation.
Test with edge cases early. Wikipedia, FBRef, and government sites are great stress tests.
Normalize everything. Whitespace, encoding, column count—make it consistent.
Headers need detection. Don't assume row 0 is the header.
Spans are the enemy. The grid-building approach handles them cleanly.

If you want to skip all this complexity, HTML Table Exporter handles these edge cases automatically. One click to export any table to CSV, JSON, or Excel.

Learn more at gauchogrid.com/html-table-exporter or try it free on the Chrome Web Store.

But if you're building your own parser, I hope this saves you some debugging time. The rabbit hole is deep.

What's the weirdest table structure you've encountered? Share in the comments. I collect edge cases like trading cards.

From Web Table to Pandas DataFrame in 30 Seconds

circobit — Wed, 25 Feb 2026 15:00:00 +0000

You found the perfect dataset. It's sitting right there on a webpage, neatly formatted in an HTML table. You just need to get it into Pandas.

How hard could it be?

The One-Liner (When It Works)

Pandas has a built-in function for this:

import pandas as pd

tables = pd.read_html('https://example.com/page-with-table')
df = tables[0]  # First table on the page

This is beautiful when it works. Three lines, done.

But here's what the tutorials don't tell you: pd.read_html() fails on a surprising number of real-world websites.

JavaScript-rendered tables? Pandas can't see them. It only reads the raw HTML.

Tables that require authentication? You'll need to handle sessions and cookies first.

Complex nested structures? The parsing might produce garbage.

Anti-scraping measures? You'll get blocked or served different content.

For simple, static HTML tables on public pages, pd.read_html() is great. For everything else, you need alternatives.

The Requests + BeautifulSoup Approach

When pd.read_html() fails, the next step is usually:

import requests
from bs4 import BeautifulSoup
import pandas as pd

response = requests.get('https://example.com/page')
soup = BeautifulSoup(response.content, 'html.parser')
table = soup.find('table', {'class': 'data-table'})

# Extract headers
headers = [th.text.strip() for th in table.find_all('th')]

# Extract rows
rows = []
for tr in table.find_all('tr')[1:]:
    row = [td.text.strip() for td in tr.find_all('td')]
    if row:
        rows.append(row)

df = pd.DataFrame(rows, columns=headers)

This gives you more control. You can target specific tables, handle edge cases, clean data during extraction.

The downsides:

More code to write and maintain
Still can't handle JavaScript-rendered content
Breaks when the website structure changes

The Selenium Nuclear Option

For JavaScript-heavy sites:

from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd

driver = webdriver.Chrome()
driver.get('https://example.com/page')

# Wait for JavaScript to render
import time
time.sleep(3)

# Now parse the rendered HTML
table = driver.find_element(By.CSS_SELECTOR, 'table.data-table')
# ... extraction logic similar to BeautifulSoup

driver.quit()

This works on almost anything. The browser renders the page fully, JavaScript and all, then you extract the data.

The cost:

Slow (seconds per page instead of milliseconds)
Requires browser driver setup
Resource-heavy
Feels like overkill for grabbing one table

The Lazy Way (My Favorite)

Here's what I actually do most of the time:

Open the page in my browser
Export the table to CSV with a browser extension
Load it in Pandas:

df = pd.read_csv('exported_table.csv')

Done. No scraping code. No debugging HTML selectors. No handling edge cases in Python.

The browser already rendered the JavaScript. The browser extension handles the HTML parsing and data cleaning. I get a clean CSV file that Pandas reads without issues.

I use HTML Table Exporter for this: it detects tables automatically and exports to CSV, JSON, or Excel with one click. Runs locally, no data uploaded anywhere.

For more details on JSON workflows, see Export Web Tables to JSON for Python & Pandas.

But there are other tools too. The point is: sometimes the fastest path to a DataFrame isn't through Python.

When to Use What

Here's my decision tree:

Use pd.read_html() when:

Simple static HTML table
Public page, no authentication
You need to automate repeated fetches

Use BeautifulSoup when:

pd.read_html() fails
You need precise control over extraction
Table structure is unusual

Use Selenium when:

JavaScript renders the table
You need to interact with the page first
Automation is required

Use browser export when:

One-time data grab
Complex page that would be painful to scrape
You want the data in 30 seconds, not 30 minutes

A Practical Example

Let's say you want World Bank GDP data that's displayed in a table on their site.

The pd.read_html() approach:

import pandas as pd

url = 'https://data.worldbank.org/indicator/NY.GDP.MKTP.CD'
tables = pd.read_html(url)

# Pray that it worked and figure out which table index you need
for i, table in enumerate(tables):
    print(f"Table {i}: {table.shape}")

Sometimes this works. Sometimes the table is loaded via JavaScript and you get nothing useful.

The browser export approach:

Open the page
Click the export button in your extension
Choose CSV
df = pd.read_csv('world_bank_gdp.csv')

Total time: about 30 seconds.

I used to feel like this was "cheating"—real data engineers write scrapers, right? But then I realized that getting the data quickly and accurately is the actual goal. The method is just a means to an end.

The Takeaway

pd.read_html() is underrated for simple cases. Browser-based export is underrated for complex cases. And writing a custom scraper should be your last resort, not your first instinct.

Match the tool to the job. Your time is worth more than proving you can parse HTML in Python.

Learn more about HTML Table Exporter or try it free on the Chrome Web Store. What's your go-to method for grabbing web tables? Let me know in the comments.

Why Data Analysts Hate Copy-Paste from Websites

circobit — Wed, 18 Feb 2026 15:00:00 +0000

Last week I spent 40 minutes fixing a spreadsheet that should have taken 5 minutes to build.

The task was simple: grab a table from a government statistics website, paste it into Excel, run some quick analysis. I'd done it hundreds of times before.

But this time, the numbers wouldn't sort correctly. Percentages showed as text. Dates were scrambled. And there were invisible characters breaking my formulas.

If you work with web data, you've been here.

The Problem Isn't You

When you copy a table from a website, you're not copying data. You're copying a visual representation of data wrapped in HTML formatting, CSS styles, hidden spans, and sometimes JavaScript-generated content.

Your spreadsheet receives all of this and tries to make sense of it. Sometimes it works. Often it doesn't.

Here's what's actually happening:

Numbers that aren't numbers. That "1,234" might contain a non-breaking space (Unicode 160) instead of a regular space. Excel sees it as text. Your SUM formula returns zero, and you stare at the screen wondering what went wrong.

Dates in disguise. "01/02/2024" could be January 2nd or February 1st, depending on the source website's locale. Excel guesses. It guesses wrong about 50% of the time.

Hidden formatting. Websites use <span> tags, zero-width characters, and CSS tricks to display data. When you paste, these come along. You can't see them, but they break everything.

Merged cells chaos. That nicely formatted table with headers spanning multiple columns? Paste it and watch your data structure collapse.

The Manual Fixes (And Why They're Painful)

Experienced analysts develop rituals. Paste into Notepad first to strip formatting. Use "Paste Special > Values" in Excel. Run Find & Replace to catch common invisible characters.

These work. But they're slow, error-prone, and you have to remember to do them every single time.

I've seen analysts build elaborate VBA macros just to clean pasted web data. I've seen teams dedicate hours each week to "data cleaning" that's really just "fixing copy-paste problems."

This isn't analysis. It's janitorial work.

What Actually Works

There are three real solutions:

1. APIs (when they exist)

If the website offers an API, use it. You'll get clean, structured JSON or CSV. No formatting issues. No invisible characters.

The problem: most websites don't have public APIs. Government data portals, financial sites, sports statistics, e-commerce comparisons—they show you the data in tables but don't let you export it cleanly.

2. Web scraping

You can write a Python script with BeautifulSoup or Selenium to extract table data programmatically. You control the output format. You can clean the data as you extract it.

The problem: this requires coding skills, setup time, and maintenance. When the website changes its HTML structure, your script breaks. For a one-time data grab, it's overkill.

3. Browser-based extraction

This is the middle ground. Tools that run in your browser, detect tables on the page, and export them directly to clean CSV, Excel, or JSON.

No coding. No API needed. The tool handles the HTML parsing, character normalization, and format conversion.

For a step-by-step guide on this approach, see our tutorial on copying tables from websites to Excel.

I built one of these tools because I got tired of the copy-paste dance. It's called HTML Table Exporter and it runs entirely in your browser—no servers, no uploads, your data stays local.

But honestly, the specific tool matters less than the approach. Stop copying and pasting tables manually. The time you waste fixing broken data adds up fast.

The Real Cost

Here's a calculation I did recently:

If you copy-paste web tables 3 times per week, and spend an average of 10 extra minutes per table fixing formatting issues, that's 30 minutes per week. Over a year, that's 26 hours spent on preventable problems.

Twenty-six hours of your life, deleting invisible characters.

Find a better way. Your future self will thank you.

Learn more at gauchogrid.com/html-table-exporter or try it free on the Chrome Web Store. What's your worst copy-paste horror story? I'd love to hear it in the comments.

The Hidden Complexity of HTML Tables (Why Parsing Them Is Harder Than You Think)

circobit — Wed, 11 Feb 2026 15:00:00 +0000

HTML tables look simple. <table>, <tr>, <td>. What could go wrong?

After building HTML Table Exporter, a table export tool that's processed thousands of real-world tables, I can tell you: a lot. This post covers the edge cases that break naive parsers and how to handle them.

The Deceptively Simple Case

A perfect table looks like this:

<table>
  <thead>
    <tr>
      <th>Name</th>
      <th>Revenue</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Acme Inc</td>
      <td>$1.2M</td>
    </tr>
  </tbody>
</table>

Parsing this is trivial:

const rows = table.querySelectorAll('tr');
const data = [...rows].map(row => 
  [...row.querySelectorAll('td, th')].map(cell => cell.textContent.trim())
);

Done, right? Not even close.

Problem 1: Merged Cells (colspan/rowspan)

Real tables have merged cells. A lot of them.

<tr>
  <td rowspan="3">Q1 2024</td>
  <td>January</td>
  <td>$100k</td>
</tr>
<tr>
  <td>February</td>
  <td>$120k</td>
</tr>
<tr>
  <td>March</td>
  <td>$90k</td>
</tr>

If you parse this naively, you get:

Row 1: ["Q1 2024", "January", "$100k"]
Row 2: ["February", "$120k"]          // Missing first column!
Row 3: ["March", "$90k"]              // Missing first column!

The Fix: Build a Position Matrix

You need to track which cells are "occupied" by rowspans from previous rows:

function parseTableWithMergedCells(table) {
  const rows = table.querySelectorAll('tr');
  const matrix = [];
  const rowspanTracker = []; // Track active rowspans per column

  rows.forEach((row, rowIndex) => {
    matrix[rowIndex] = [];
    let colIndex = 0;

    // Skip columns occupied by previous rowspans
    while (rowspanTracker[colIndex] > 0) {
      matrix[rowIndex][colIndex] = matrix[rowIndex - 1]?.[colIndex] || '';
      rowspanTracker[colIndex]--;
      colIndex++;
    }

    row.querySelectorAll('td, th').forEach(cell => {
      // Skip occupied columns
      while (rowspanTracker[colIndex] > 0) {
        matrix[rowIndex][colIndex] = matrix[rowIndex - 1]?.[colIndex] || '';
        rowspanTracker[colIndex]--;
        colIndex++;
      }

      const colspan = parseInt(cell.getAttribute('colspan')) || 1;
      const rowspan = parseInt(cell.getAttribute('rowspan')) || 1;
      const value = cell.textContent.trim();

      // Fill colspan
      for (let c = 0; c < colspan; c++) {
        matrix[rowIndex][colIndex] = value;

        // Track rowspan for future rows
        if (rowspan > 1) {
          rowspanTracker[colIndex] = rowspan - 1;
        }
        colIndex++;
      }
    });
  });

  return matrix;
}

This is simplified—the real implementation needs to handle nested rowspans within colspans, which gets ugly fast.

Problem 2: Tables That Aren't Data Tables

Not every <table> contains data. Many sites (yes, still in 2024) use tables for layout:

<table>
  <tr>
    <td><nav>Menu here</nav></td>
    <td><main>Content here</main></td>
  </tr>
</table>

Or for forms:

<table>
  <tr>
    <td><label>Email:</label></td>
    <td><input type="email"></td>
  </tr>
</table>

The Fix: Heuristics

I use several signals to detect "real" data tables:

function isDataTable(table) {
  const rows = table.querySelectorAll('tr');
  const cells = table.querySelectorAll('td, th');

  // Too few rows or cells
  if (rows.length < 2 || cells.length < 4) return false;

  // Contains form elements (probably a form layout)
  if (table.querySelector('input, select, textarea, button')) return false;

  // Mostly navigation links
  const links = table.querySelectorAll('a');
  const textContent = table.textContent.length;
  const linkText = [...links].reduce((sum, a) => sum + a.textContent.length, 0);
  if (linkText / textContent > 0.7) return false;

  // Check column consistency
  const colCounts = [...rows].map(row => 
    row.querySelectorAll('td, th').length
  );
  const variance = Math.max(...colCounts) - Math.min(...colCounts);
  if (variance > 3) return false; // Inconsistent columns = probably layout

  return true;
}

None of these are perfect. You'll always have edge cases.

Problem 3: Hidden Content

Cells often contain more than visible text:

<td>
  <span class="value">1,234</span>
  <span class="sort-key" style="display:none">1234</span>
</td>

Wikipedia does this a lot for sortable tables. If you just grab textContent, you get "1,234 1234".

The Fix: Extract Visible Text Only

function getVisibleText(element) {
  // Clone to avoid modifying original
  const clone = element.cloneNode(true);

  // Remove hidden elements
  clone.querySelectorAll('[style*="display: none"], [style*="display:none"], .hidden, [hidden]').forEach(el => el.remove());

  // Also check computed style for dynamically hidden elements
  // (more expensive, use sparingly)

  return clone.textContent.trim();
}

Problem 4: Numbers That Aren't Numbers

"$1,234.56" is a number. So is "1.234,56" (European format). So is "(1,234)" (accounting negative). So is "1,234 M" (with suffix).

Your spreadsheet needs actual numbers to do math.

The Fix: Locale-Aware Parsing

function parseNumber(value) {
  if (!value || typeof value !== 'string') return value;

  // Remove currency symbols and whitespace
  let cleaned = value.replace(/[$€£¥₹\s]/g, '').trim();

  // Handle accounting negatives: (1,234) -> -1234
  if (cleaned.startsWith('(') && cleaned.endsWith(')')) {
    cleaned = '-' + cleaned.slice(1, -1);
  }

  // Handle suffixes: 1.5M, 2.3B, 100K
  const suffixes = { 'K': 1e3, 'M': 1e6, 'B': 1e9, 'T': 1e12 };
  const suffixMatch = cleaned.match(/([0-9.,]+)\s*([KMBT])$/i);
  if (suffixMatch) {
    cleaned = suffixMatch[1];
    var multiplier = suffixes[suffixMatch[2].toUpperCase()];
  }

  // Detect European vs US format
  // European: 1.234,56 (dot for thousands, comma for decimal)
  // US: 1,234.56 (comma for thousands, dot for decimal)
  const lastComma = cleaned.lastIndexOf(',');
  const lastDot = cleaned.lastIndexOf('.');

  if (lastComma > lastDot && lastComma > cleaned.length - 4) {
    // European format
    cleaned = cleaned.replace(/\./g, '').replace(',', '.');
  } else {
    // US format
    cleaned = cleaned.replace(/,/g, '');
  }

  let num = parseFloat(cleaned);
  if (multiplier) num *= multiplier;

  return isNaN(num) ? value : num;
}

This handles maybe 90% of cases. The other 10% will surprise you.

Problem 5: Character Encoding Hell

You'd think UTF-8 solved this. It didn't.

Real tables contain:

Non-breaking spaces (  / \u00A0) that look like spaces but aren't
Zero-width characters that break string comparison
Windows-1252 characters that got mangled into UTF-8
Emoji that break older parsers
Right-to-left marks in mixed-language tables

The Fix: Normalize Everything

function normalizeText(text) {
  return text
    // Normalize unicode (handles composed vs decomposed characters)
    .normalize('NFC')
    // Replace non-breaking spaces with regular spaces
    .replace(/\u00A0/g, ' ')
    // Remove zero-width characters
    .replace(/[\u200B-\u200D\uFEFF]/g, '')
    // Normalize whitespace
    .replace(/\s+/g, ' ')
    .trim();
}

And when exporting to CSV for Excel, prepend the UTF-8 BOM:

const BOM = '\uFEFF';
const csvContent = BOM + generateCSV(data);

Without the BOM, Excel may interpret your UTF-8 file as Windows-1252 and mangle special characters.

Problem 6: Nested Tables

Yes, tables inside tables. Usually for layout, but sometimes for data:

<table>
  <tr>
    <td>Product A</td>
    <td>
      <table>
        <tr><td>Size S</td><td>$10</td></tr>
        <tr><td>Size M</td><td>$12</td></tr>
      </table>
    </td>
  </tr>
</table>

The Fix: Decide Your Strategy

Options:

Flatten: Convert nested table to text ("Size S: $10, Size M: $12")
Extract separately: Treat nested tables as separate exports
Expand rows: Create multiple parent rows, one per nested row

I went with option 2 (extract separately) with option 1 as fallback for deeply nested cases. There's no perfect answer—it depends on use case.

The Reality

After handling all these cases, my table parser is ~800 lines of JavaScript. And it still doesn't handle everything perfectly.

Some hard truths:

No parser is perfect. Real-world HTML is messy.
Heuristics fail. You'll always need escape hatches for users.
Performance matters. Some pages have 50+ tables. Parsing needs to be fast.
Edge cases are infinite. Ship something that works for 95% of cases, then iterate.

Tools and Resources

If you're building something similar:

SheetJS (xlsx) - Solid library for generating Excel files
Papa Parse - Fast CSV parsing and generation
Chrome DevTools - $('table') in console to quickly inspect tables

Or if you just need to export tables without building anything: I made HTML Table Exporter specifically because I got tired of writing one-off scrapers. It handles all the edge cases above.

Learn more at gauchogrid.com/html-table-exporter or try it free on the Chrome Web Store.

What weird table edge cases have you encountered? I'm always looking for new test cases to break my parser.

5 Ways to Extract Tables from Websites (Compared)

circobit — Sun, 08 Feb 2026 15:45:38 +0000

If you've ever needed to grab data from an HTML table and put it into a spreadsheet or database, you know it's rarely as simple as copy-paste. Here's a practical comparison of the most common methods, with pros and cons for each.

1. Copy-Paste (The Classic)

The most obvious approach: select the table, Ctrl+C, paste into Excel or Google Sheets.

Pros:

No setup required
Works for simple tables

Cons:

Formatting often breaks
Merged cells cause chaos
Doesn't work on tables rendered with JavaScript
Manual and tedious for multiple tables

Best for: One-off extractions from simple, static tables.

2. Excel's Web Query (Get Data from Web)

Excel has a built-in feature to import data from web pages: Data → Get Data → From Web.

Pros:

Native Excel feature, no extensions needed
Can refresh data automatically
Handles multiple tables on a page

Cons:

Struggles with JavaScript-rendered tables
Can't handle authentication/login walls
Sometimes imports garbage along with the table
Limited data cleaning options

Best for: Recurring imports from static, public pages (government data, Wikipedia).

3. Python + BeautifulSoup/Pandas

For developers, Python is the Swiss army knife of data extraction:

import pandas as pd

tables = pd.read_html('https://example.com/page-with-tables')
df = tables[0]  # First table on the page
df.to_csv('output.csv', index=False)

Pros:

Maximum flexibility
Can handle authentication, pagination, complex logic
Easy to automate and schedule
Great for large-scale scraping

Cons:

Requires coding knowledge
Setup overhead for simple tasks
Need to handle headers, sessions, rate limiting
Breaks when site structure changes

Best for: Developers doing recurring or complex extractions.

4. Browser Extensions

Chrome extensions like Table Capture, Data Miner, or HTML Table Exporter let you export tables directly from the browser with a few clicks.

For a detailed comparison of these tools, see our guide to the best Chrome extensions for table export.

Pros:

Works on JavaScript-rendered content
No coding required
See what you're exporting (WYSIWYG)
Fast for ad-hoc extractions
Some offer data cleaning and format options

Cons:

Manual process (not ideal for automation)
Quality varies between extensions
Some have privacy concerns (send data to servers)

Best for: Non-developers who need clean exports quickly, or developers who want to avoid writing throwaway scripts.

5. Dedicated Scraping Tools (Octoparse, ParseHub, etc.)

Visual scraping tools that let you point-and-click to define extraction rules.

Pros:

No coding required
Can handle complex multi-page scraping
Often include scheduling and cloud features

Cons:

Learning curve for the interface
Usually paid for serious usage
Overkill for simple table extraction
Data often goes through their servers

Best for: Non-technical users doing large-scale or complex scraping projects.

Quick Comparison

Method	Coding?	JS Tables?	Speed	Best For
Copy-paste	No	No	Fast	Simple one-offs
Excel Web Query	No	No	Medium	Recurring static data
Python	Yes	Yes*	Slow setup	Complex/automated
Browser Extensions	No	Yes	Fast	Quick clean exports
Scraping Tools	No	Yes	Medium	Large projects

*With Selenium or Playwright

My Recommendation

For most people: Start with a browser extension. It's the fastest path from "I need this data" to "I have this data in a spreadsheet."

If you're a developer: Python is unbeatable for automation, but for quick one-offs, an extension saves you from writing (and debugging) throwaway code.

If you need to scrape at scale: Look into dedicated tools or build a proper Python pipeline.

What I Built

After years of copying tables manually and writing one-off Python scripts, I built HTML Table Exporter, a Chrome extension focused on clean exports with data normalization built-in.

It's free for basic exports (CSV, Excel, JSON). The Pro version adds features like reusable profiles for Pandas/SQL workflows and automatic data cleaning.

Learn more at gauchogrid.com/html-table-exporter or try it free on the Chrome Web Store.

What's your go-to method for extracting web tables? Let me know in the comments.

DEV Community: circobit

Building a Type Inference System for Messy Web Data

The Problem: Ambiguity Everywhere

Architecture: Column-Level Inference

Value-Level Type Detection

The Number Normalization Challenge

Boolean Detection (With Gotchas)

Null Value Handling

The Pipeline

Testing Edge Cases

Integration with Export

Lessons Learned

Web Scraping vs Browser Extensions: When to Use Each for Data Extraction

The Options

Option 1: Python + Requests/BeautifulSoup

Option 2: Headless Browsers (Playwright/Puppeteer)

Option 3: Browser Extensions

Decision Framework

Use Python + requests when:

Use Playwright/Puppeteer when:

Use a Browser Extension when:

Just copy-paste when:

Real-World Examples

Example 1: Wikipedia Tables

Example 2: Financial Dashboard (Behind Login)

Example 3: Daily Price Monitoring (1000 Pages)

Example 4: One-Time Sports Stats Export

Hybrid Approaches

My Stack

How I Handle Nested Tables and Rowspans (The Hard Parts of HTML Table Parsing)

Problem 1: Rowspan Expansion

Problem 2: Nested Tables

Problem 3: Wikipedia Navigation Rows

Problem 4: Title Rows (Spanning All Columns)

Problem 5: Grouped Column Headers (FBREF Style)

Problem 6: Horizontally Duplicated Tables

The Combined Algorithm

Testing These Edge Cases

Try It Yourself

Automating Web Table Exports with JavaScript: A Practical Guide

The Naive Approach (And Why It Fails)

Handling Rowspan and Colspan

Extracting Clean Text

Detecting Nested Tables

Converting to CSV

Converting to JSON

Triggering the Download

Putting It All Together

When to Use a Browser Extension Instead

From Web Table to Pandas DataFrame in 30 Seconds

The Problem with pd.read_html()

The 30-Second Workflow

Step 1: Export from Browser (5 seconds)

Step 2: Load in Pandas (5 seconds)

What "Clean" Actually Means

Number Normalization

Boolean Conversion

Null Handling

Snake Case Headers

Real Example: FBRef Football Stats

The Old Way

The New Way

The Code I No Longer Write

When to Use What

Pro Tip: JSON for Complex Data

The Workflow Summary

Try It

Why Data Analysts Hate Copy-Paste from Websites

What Actually Happens When You Copy-Paste

The Hidden Problems

Problem 1: Numbers as Text

Problem 2: European vs US Decimals

Problem 3: Hidden Characters

Problem 4: Merged Cells

Problem 5: Multi-Row Headers

Problem 6: Dates From Hell

The Real Cost

The Alternatives

Option 1: Web Scraping (Overkill)

Option 2: Browser DevTools (Technical)

The Problem with `pd.read_html()`