How I Stopped Regexing HTML Tables and Started Using AI for Data Extraction

I've been scraping data from the web for years. You'd think I'd have learned by now: never use regex on HTML. But sometimes, when you're staring at a messy table with inconsistent classes, random whitespace, and nested elements that barely qualify as valid markup, the temptation to just throw a regex at it is overwhelming.

I found myself in that exact situation last month. I needed to extract property listings from a dozen different real estate websites. Each site had its own quirks. One used <table> tags with rowspan and colspan that made BeautifulSoup cry. Another had dynamic content loaded via JavaScript that my initial scraping setup couldn't even see.

This is the story of how I gave up on perfect parsing and let an AI handle the messy middle.

The Problem: Fragile Parsers

My first attempt was the classic approach: Python + requests + BeautifulSoup. For sites with clean semantic HTML, this worked beautifully. But the real world is full of edge cases:

• Missing closing tags
• Inline styles overriding table structure
• Random <br> elements splitting text across rows
• Data that spans multiple cells visually but not in the DOM

I wrote custom functions for each site. They worked for a week. Then the site updated its layout, and my parser broke. Again.

I tried regex as a last resort (I know, I know). Here’s a snippet of the mess I ended up with:

import re

def extract_price_from_html(html):
    pattern = r'<span[^>]*class="price[^"]*"[^>]*>([0-9,.$]+)</span>'
    match = re.search(pattern, html)
    return match.group(1) if match else None

This worked for exactly one site, on a good day, with perfect formatting. Any minor change broke it. I was maintaining a fragile house of cards.

What Didn't Work: More Rules

My next idea was to use lxml with XPath. More precise, but still brittle. I even tried building a custom state machine to track table cell positions — overengineering at its finest. I spent two days writing code that handled 80% of cases, then gave up on the long tail.

I needed something that could understand the meaning of the data, not just its layout.

What Eventually Worked: An AI-Based Extraction Layer

I started experimenting with large language models (LLMs) to parse the HTML text directly. The idea: dump the raw HTML (or a cleaned version) into an AI API and ask it to return structured JSON. No parsing rules, no regex, no XPath — just a prompt.

I found a service that abstracts this into a simple REST endpoint. The core insight is that instead of writing code to find the price in a table, you tell the AI what the price looks like and let it figure out the context.

Here's the approach I settled on:

1. Fetch the page HTML.
2. Extract the main content area (strip out headers, footers, scripts).
3. Send that content to the AI API with a prompt describing the desired output schema.

I built a small Python class around it:

import requests
import json
from bs4 import BeautifulSoup

class AIDataExtractor:
    def __init__(self, api_key):
        self.api_key = api_key
        self.base_url = "https://ai.interwestinfo.com/extract"  # Example endpoint

    def extract_listings(self, html, schema):
        # Preprocess: extract only the visible text table-ish parts
        soup = BeautifulSoup(html, 'html.parser')
        # Remove scripts, styles, nav
        for tag in soup(['script', 'style', 'nav', 'header', 'footer']):
            tag.decompose()
        clean_text = soup.get_text(separator='\n', strip=True)

        prompt = f"""Extract property listings from the following web page content.
Return a JSON array of objects with these fields: address, price, bedrooms, bathrooms, square_feet.
If a field is not found, set it to null.

Content:
{clean_text[:5000]}  # limit to avoid token overflow
"""

        response = requests.post(
            self.base_url,
            json={"prompt": prompt, "model": "default"},
            headers={"Authorization": f"Bearer {self.api_key}"}
        )
        response.raise_for_status()
        return response.json()

The actual API call returns a JSON object with a listings key. I then validate and transform it into my data model.

Code Example in Action

Let's say I'm scraping a real estate site. Here's how I'd use the extractor:

import requests

url = "https://example-realty.com/listings"
page_html = requests.get(url).text

extractor = AIDataExtractor(api_key="sk-...")
listings = extractor.extract_listings(page_html, schema)

for listing in listings:
    print(f"{listing['address']} - {listing['price']}")

The first few results were surprisingly accurate — about 90% of fields populated correctly. The errors were mostly on edge cases like “Price Upon Request” or missing square footage. I added a second pass of validation: check that price is numeric, address exists, etc. If a field is null, I can either skip that listing or use a fallback parser.

Lessons Learned / Trade-offs

This approach isn't a silver bullet. Here's what I discovered:

• Cost: AI API calls cost money, especially for high-volume scraping. Each request might be $0.01–$0.05 depending on the model. For 10,000 listings, that adds up.
• Latency: Calling an API is slower than local parsing. Each request takes 1–3 seconds. If you're scraping thousands of pages, this can take hours.
• Hallucinations: The AI sometimes invents data if it can't find it. For example, if a price is missing, it might guess “$500,000” from context or just make something up. You absolutely need validation steps.
• Rate Limits: Many AI APIs have strict rate limits. You'll need to throttle or rotate accounts.
• Consistency: The same page can return slightly different JSON each time due to model non-determinism. Not ideal for production ETL pipelines.

When not to use this:

• You have well-structured HTML that BeautifulSoup can handle (e.g., government data tables).
• You're scraping billions of pages (cost kills you).
• You need perfect, deterministic output every time.

The sweet spot is when you have a moderate volume of pages with inconsistent structure, and you can afford a few cents per page to avoid writing custom parsers.

What I'd Do Differently Next Time

I'd start with the AI approach from day one, but I'd also build a caching layer to avoid re-requesting the same page. I'd also use a smaller, cheaper model for simple extractions and reserve the powerful (expensive) models for truly messy pages. Some services let you specify model size in the request.

I'd also mix approaches: use regex for high-confidence fields (like prices prefixed with '$') and the AI as a fallback for the long tail. A hybrid pipeline would reduce costs while still handling the weird edge cases.

The Bigger Lesson

AI isn't just about generating text or images. It's a tool for understanding context — something that's incredibly hard to do with deterministic code. For data extraction, it turns the problem from “write a parser for every layout” into “describe what you want in English and let the model figure it out.” That trade-off is worth it for many real-world scraping projects.

What's your go-to approach when you hit a wall with structured data extraction? Do you roll your own parser, or have you tried the AI route? I'd love to hear about your experiences – especially the horror stories.

Comments

[Artem Matviychuk]

Great framing — "write a parser for every layout" → "describe what you want in English" is exactly the right tradeoff axis. My go-to is a hybrid: let the model handle the fuzzy layout-understanding, but never let it be the final word on the values.

The horror story you asked for: the AI route's worst failure isn't a crash — it's confident, plausible, wrong. You get a beautifully-structured table where one number is silently hallucinated or two rows got merged, and there's no formatting difference between a real extraction and a fabricated one. You only catch it if you check.

So I bolt a deterministic verifier onto the AI step: type/range/sum checks, "does every extracted cell trace back to a span in the source text," reconcile row counts. The model proposes the structure; a dumb deterministic pass confirms the data. Keeps the layout-tolerance win without trusting the narration.

Been digging into exactly this lately (code-over-data extraction where the model writes the extractor and a separate pass verifies) — so the post hit home. Nice write-up.