DEV Community

Cover image for Stop Silent Failures: Using LLMs to Validate Web Scraper Output
Jerry A. Henley
Jerry A. Henley

Posted on

Stop Silent Failures: Using LLMs to Validate Web Scraper Output

You’ve likely experienced the "silent failure" nightmare. You build a web scraper, test your CSS selectors, and everything works perfectly. You deploy it to production, and it runs for three days without a single crash. You check your database, expecting thousands of rows of clean data, only to find the price column is empty or contains the text "Sign up for our newsletter."

The website’s layout changed just enough to break your selectors, but not enough to trigger a 404 error or a script crash. Your scraper thought it was doing its job, but it was actually collecting garbage.

This guide covers how to automate scraper QA by implementing AI schema validation. We’ll move beyond simple data-type checks and use Large Language Models (LLMs) to perform semantic validation, ensuring the data you extract actually matches the context of the source page.

The Problem: Structural vs. Semantic Validation

In a traditional data pipeline, we use structural validation. Tools like Pydantic in Python or JSON Schema are excellent at ensuring a field named price is a float and a field named sku is a string.

from pydantic import BaseModel

class Product(BaseModel):
    title: str
    price: float
    sku: str
Enter fullscreen mode Exit fullscreen mode

If your scraper extracts the string "Free Shipping" into the price field, Pydantic will throw an error because "Free Shipping" cannot be cast to a float. This is helpful, but it doesn't solve the semantic problem.

What if the scraper extracts "$19.99" from a "Recommended Products" sidebar instead of the main product price? Structurally, it’s a valid float. Semantically, it’s a failure. Traditional code cannot easily "read" the page to know if a piece of text is the correct piece of text. This is where an AI Judge comes in.

The Solution: The "AI Judge" Architecture

The "AI Judge" pattern introduces a secondary validation step in your scraping loop. Instead of trusting the parser implicitly, take a small sample of the raw HTML and the extracted JSON and pass them to an LLM.

The workflow works as follows:

  1. Extraction: Your scraper (Playwright, BeautifulSoup, etc.) extracts data using selectors.
  2. Contextual Sampling: Isolate the HTML block where the data was found.
  3. Verification: An LLM compares the raw HTML to the JSON.
  4. Decision: If the LLM flags a mismatch, the system alerts the developer or triggers a retry.

By using an LLM, you can use its ability to understand unstructured text and visual hierarchy without writing thousands of lines of fragile regex or manual checks.

Step 1: The Setup (The Fragile Scraper)

Let’s start with a standard extraction script. We’ll target a typical e-commerce product page, similar to those in the BestBuy.com-Scrapers repository.

import requests
from bs4 import BeautifulSoup

def extract_product_data(html_content):
    soup = BeautifulSoup(html_content, 'html.parser')

    # These selectors break easily if the site updates
    return {
        "title": soup.select_one(".product-title").get_text(strip=True),
        "price": soup.select_one(".price-value").get_text(strip=True),
        "sku": soup.select_one(".model-number").get_text(strip=True)
    }

# Imagine this HTML is fetched via requests
sample_html = '<div class="product-title">Sony Alpha 7 IV</div><div class="price-value">$2,499.99</div>'
data = extract_product_data(sample_html)
print(data)
Enter fullscreen mode Exit fullscreen mode

This works today, but if the site changes .price-value to .price-display-v2, your scraper will return None or pull data from an unrelated element.

Step 2: Building the AI Validator

To build the validator, construct a prompt that asks the LLM to act as a QA Engineer. The LLM should return a structured response, specifically a boolean and a reason for failure.

We’ll use the openai library and JSON Mode to ensure the output is machine-readable.

import openai
import json

client = openai.OpenAI(api_key="YOUR_API_KEY")

def validate_extraction(html_snippet, extracted_data):
    prompt = f"""
    You are a Data Quality Auditor. Compare extracted JSON data 
    against a raw HTML snippet to ensure accuracy.

    RAW HTML:
    {html_snippet}

    EXTRACTED JSON:
    {json.dumps(extracted_data)}

    Rules:
    1. Check if the 'title' in JSON matches the main product title in HTML.
    2. Check if the 'price' in JSON matches the actual product price.
    3. Ignore minor whitespace or formatting differences.
    4. If the data is missing or incorrect, set 'is_valid' to false.

    Return ONLY a JSON object with this structure:
    {{"is_valid": boolean, "reason": "string explaining the error if invalid"}}
    """

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )

    return json.loads(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Why this works

  • Context Isolation: Sending the entire 100KB HTML file is expensive and noisy. We only send the relevant container.
  • Semantic Comparison: The LLM understands that "$2,499.99" in the HTML is the same as "2499.99" in your JSON, even if the formatting changed.
  • Reasoning: If it fails, the "reason" field provides an immediate debugging hint.

Step 3: Implementing the Feedback Loop

Now, let's integrate the validator into the scraping logic. In a production environment, you shouldn't stop the entire crawl for one error, but you should log it and stop the spider if the error rate exceeds a specific threshold.

def run_scraper(url):
    html = requests.get(url).text
    extracted_data = extract_product_data(html)

    # Pass only the relevant part of the HTML to save tokens
    soup = BeautifulSoup(html, 'html.parser')
    container = str(soup.select_one(".product-main-area"))

    validation_result = validate_extraction(container, extracted_data)

    if not validation_result['is_valid']:
        print(f"CRITICAL: Validation failed for {url}")
        print(f"Reason: {validation_result['reason']}")
        # Log to your monitoring system (e.g., Sentry or ScrapeOps)
        return None

    return extracted_data
Enter fullscreen mode Exit fullscreen mode

Optimization: Cost and Performance

Sending every request to an LLM makes your scraper slow and expensive. If you scrape 100,000 pages, a $0.01 API call per page adds up to $1,000. Use Statistical Sampling to optimize this.

1. Sampling

You don't need to validate every row. Checking 1% of your data is often enough to catch site-wide layout changes.

import random

def should_validate(rate=0.01):
    return random.random() < rate

# In your loop
if should_validate(rate=0.05): # Validate 5% of requests
    validation_result = validate_extraction(html, data)
Enter fullscreen mode Exit fullscreen mode

2. Model Selection

Avoid using GPT-4o for simple comparisons. Models like gpt-4o-mini or claude-3-haiku are significantly cheaper and more than capable of comparing JSON to HTML. They also have much lower latency.

3. Confidence-Based Triggers

Trigger the AI Judge only when your local code is "unsure." For example, if a selector returns an empty string or if a regex pattern fails, pass the HTML to the LLM and ask it to find the missing data.

To Wrap Up

Automating schema validation with AI moves web scraping from a "fingers crossed" approach to a rigorous engineering discipline. By using LLMs as a semantic QA layer, you can catch silent failures before they corrupt your datasets.

Key Takeaways:

  • Structural validation (Pydantic) catches data type errors, while Semantic validation (AI) catches context errors.
  • Context Isolation is vital. Only send relevant HTML snippets to the LLM to save on costs and improve accuracy.
  • Use Sampling to keep your pipeline performant and cost-effective.
  • Structured Outputs allow you to integrate AI feedback directly into your code logic.

As a next step, consider using the ScrapeOps Proxy Provider to ensure you're getting high-quality HTML back from your targets before you begin the validation process. Successful data extraction starts with the right tools and ends with reliable verification.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.