I spent 3 days scraping a site until I tried LLMs for data extraction

I’m not proud of it, but I spent three days building a scraper for an e‑commerce site that kept changing its HTML classes overnight. The first version used BeautifulSoup with CSS selectors. It worked for exactly four hours. Then the site pushed a new build, all the class names became hashed, and my carefully crafted selectors turned into wet cardboard. I patched it with regex. That held for another day until they changed the ordering of fields in the product cards. I was losing my mind.

This story isn’t about that specific site, and it’s not about any single tool. It’s about the moment I stopped trying to outsmart inconsistent markup and started treating the whole page as a blob of text that a language model could parse. It was a shift from “find the pattern” to “understand the meaning”. That changed everything.

The problem: semi‑structured web data

I needed to extract product name, price, description, and inventory status from dozens of product listing pages – some with pagination, some with infinite scroll, all with different HTML structures. The data was there, but the containers were unpredictable. One page used <div class="price">, another used <span class="final-amount">, and a third had the price inside a <meta> tag. Writing a universal parser was like playing Whac‑A‑Mole.

What I tried (and why it hurt)

1. BeautifulSoup + CSS selectors – brittle. One class name change and the whole script broke.

2. lxml with XPath – slightly more robust, but still relied on structural assumptions. When the site added an extra wrapper div, my XPath expressions stopped matching.

3. Regex over raw HTML – I know, I know. It was a desperate move. It worked for a few pages, then failed on pages with nested attributes or JavaScript‑rendered content.

4. Headless browser (Playwright) – solved the JS rendering problem, but I still had to write selectors to extract each field. Same fragility, now with a 300ms overhead per page.

After three days, I had a script that worked on exactly the pages I had tested. Anything new required a manual tweak. I knew there had to be a better way.

The lightbulb: ask the model, don't parse the HTML

I had been using LLMs for text generation, but it never occurred to me to use them for extraction until I saw a blog post about zero‑shot named entity recognition. The idea is simple: feed the model the text content of the page (stripped of markup) and ask it to return a JSON object with the fields you need.

No selectors. No class names. No regex. Just the raw text and a prompt.

Here’s what a first attempt looked like in Python:

import requests
from bs4 import BeautifulSoup
from openai import OpenAI

client = OpenAI()

# Fetch and strip page
def get_page_text(url):
    resp = requests.get(url)
    soup = BeautifulSoup(resp.text, 'html.parser')
    # Remove script/style elements
    for tag in soup(['script', 'style', 'noscript']):
        tag.decompose()
    return soup.get_text(separator=' ', strip=True)

url = 'https://example.com/products/123'
text = get_page_text(url)

prompt = f"""Extract the product name, price, description, and stock status from the following text. Return a JSON object with keys: name, price, description, in_stock (boolean).

Text:
{text}
"""

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    response_format={"type": "json_object"}
)

print(response.choices[0].message.content)

This worked on the first try. The model correctly identified the price even when it was buried in a paragraph of reviews. It understood that “out of stock” meant in_stock: false. I didn’t have to tell it anything about HTML.

Making it production‑ready (the hard part)

That initial script felt magical, but magic doesn’t scale. I ran into three real‑world problems:

1. Token limits and cost

Entire page text can be 10,000+ tokens. The cost of sending every page to an API adds up fast. My workaround: extract only the most relevant parts of the DOM. I still use BeautifulSoup to remove headers, footers, and sidebars by looking for common selectors (#footer, .sidebar, etc.) – not for exact fields, just to trim noise. This cut token usage by 60%.

2. Hallucinations and missing fields

The model sometimes invented a price if none was present, or returned a random number. I added a few‑shot example with a “not available” marker and parsed the JSON in a try/except block. If the price field was missing or looked like a date, I flagged the result for manual review.

import json

def safe_extract(text):
    prompt = f"""Extract product info from the text below. If a field is not present, set it to null. Return JSON with keys: name, price, description, in_stock.

Examples:
Text: "The Widget Pro is $29.99. In stock."
Output: {{"name": "Widget Pro", "price": 29.99, "description": null, "in_stock": true}}

Text: "Out of stock, no price listed."
Output: {{"name": null, "price": null, "description": null, "in_stock": false}}

Now do this for the following text:
{text}
"""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    try:
        data = json.loads(response.choices[0].message.content)
        # basic validation
        if "name" not in data or "price" not in data:
            raise ValueError("Missing required fields")
        if not isinstance(data.get("in_stock"), bool):
            raise ValueError("in_stock must be boolean")
        return data
    except (json.JSONDecodeError, ValueError):
        return None

3. Speed and retries

An API call takes 1–3 seconds per page. For 1,000 pages, that’s 15–30 minutes. I parallelized with concurrent.futures.ThreadPoolExecutor and implemented exponential backoff. Still, if your pipeline needs sub‑second extraction, this approach won’t cut it. For my use case (nightly batch jobs), it was fine.

The pattern, not the tool

Everything I just described works with any LLM – OpenAI, Claude, local models via Ollama, or a dedicated extraction API. The key insight is: shift the parsing burden from code to language understanding. Instead of reverse‑engineering HTML, you describe what you want and let a model that understands human language do the work.

Once I had this pattern down, I explored a few hosted services that specialise in this (the one I ended up using is at https://ai.interwestinfo.com/, but the pattern is the same regardless of the endpoint). I chose a managed service to avoid handling API keys, retry logic, and prompt optimisation myself – but you can absolutely roll your own with a few dozen lines of code.

When NOT to do this

• High‑frequency, low‑latency (e.g. real‑time pricing updates). LLM inference is still too slow for sub‑100ms responses. Use traditional selectors for data you can control.
• Strict budget. If you’re scraping thousands of pages per day, the API cost might exceed your AWS bill. For my 500‑page batch, it was about $2 per run – acceptable for a business tool, but not for a hobby project.
• Deterministic requirements. If the extraction must be 100% repeatable and auditable, LLMs introduce variance. A regex‑based parser is deterministic; an LLM might shift its output format slightly between models.

What I’d do differently next time

I should have tested the LLM approach on day one, not after three days of fighting selectors. I assumed it would be too expensive or too slow, but the development speed gain alone made it worth the API cost. Next time I’ll start with a hybrid: use simple selectors for the easy 80% of fields, and fall back to an LLM for the tricky 20%.

Also, I’d cache the raw page text so I can re‑prompt without re‑downloading – especially useful when tuning the prompt.

Lessons learned

• HTML structure is ephemeral; language meaning is stable. An LLM can find a price even if it’s wrapped in a <span> or a <div> or plain text.
• Prompt engineering is the new parser. Spend time crafting good few‑shot examples – it’s the equivalent of writing robust selectors, but more forgiving.
• Always validate output. Even a good model can spit out garbage. Build a schema checker.

Three days of frustration turned into two hours of wiring up a prompt. I’m not going back.

What’s your approach when scraping sites that refuse to keep still? Do you trust selectors, or have you started leaning on LLMs?