How Scraping AI Extracts Structured Data from Any Webpage Without CSS Selectors

How Scraping AI Extracts Structured Data from Any Webpage Without CSS Selectors

Stop maintaining fragile CSS selectors. Turn any webpage into validated JSON with the Python SDK.

[!NOTE]
TL;DR

• The problem: Traditional scrapers break when a site changes its CSS classes, such as .price becoming ._3xP9z. That means more maintenance and broken data pipelines.
• The solution: The scraping-ai Python SDK uses semantic extraction instead of relying on fixed DOM selectors: URL → Dynamic Render → Markdown Distillation → LLM Schema Matching → Validated JSON.
• AI agents & RAG: JSON Schema output makes it easy to use Scraping AI as a web tool with LangChain or LlamaIndex.
• Python SDK: pip install scraping-ai
• Try it free: Get 200 free tokens with no credit card required: https://pig-data.jp/service/scraping-ai/

Why DOM-Based Scraping Breaks

Most web scrapers built with BeautifulSoup, Cheerio, or Selenium depend on one basic assumption:

The structure of the website won't change.

For example:

# The fragile approach (BeautifulSoup)
soup = BeautifulSoup(html_content, "html.parser")

title = soup.select_one(
    ".product-container > .title-wrapper > h1.title"
).text

price = soup.select_one(
    ".price-box span.current-price"
).text

This works until the site changes its frontend.

Maybe the company moves to Tailwind CSS. Maybe it replaces its component library. Maybe a developer renames a class during a redesign.

Your selectors stop matching. Sometimes you get an obvious error. Other times, you just get empty data.

Either way, your data pipeline needs fixing.

The Semantic Approach: Define What You Need

Instead of telling your scraper how to navigate the DOM, you describe what data you want:

{
  "title": "string",
  "price": "number",
  "in_stock": "boolean"
}

The extraction engine finds the relevant information on the page and maps it to your schema.

You don't need to know which CSS class contains the price. You just need to define what a price is.

How Scraping AI Works

The extraction process has four main steps:

┌─────────────────┐       ┌─────────────────┐       ┌─────────────────┐
│ 1. Smart Render │ ─────▶│ 2. Markdown     │ ─────▶│ 3. LLM Semantic │
│ (Auto JS Exec)  │       │ Distillation    │       │ Schema Match    │
└─────────────────┘       └─────────────────┘       └─────────────────┘
                                                             │
                                                             ▼
                                                    ┌─────────────────┐
                                                    │ 4. Validated    │
                                                    │ JSON Output     │
                                                    └─────────────────┘

1. Smart Rendering and JavaScript Execution

Simple HTML pages can be rendered directly.

For JavaScript-heavy sites, including React, Next.js, and Vue applications, Scraping AI can use headless Chromium to execute client-side JavaScript and render the page before extraction.

2. Markdown Distillation

A raw webpage can contain a lot of content that isn't useful for extraction: inline SVGs, tracking pixels, CSS, scripts, and other presentation-related markup.

Scraping AI converts the page into a cleaner Markdown representation while keeping the text, structure, and context needed for extraction.

3. LLM Semantic Matching

The engine uses models such as GPT-4o and Gemini to understand the page and match its content to your schema.

The location of the data doesn't have to be consistent.

A price could appear in a product card, a table cell, or a header. The model looks at the meaning of the content rather than relying on a specific CSS selector.

4. JSON Schema Validation

The extracted data is validated against a JSON Schema before it's returned to your application.

That gives your Python code structured, typed output instead of another block of raw HTML to parse.

Quickstart: Python SDK

Install the SDK:

pip install scraping-ai

Synchronous Extraction

Here's a basic extraction with error handling:

from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="YOUR_API_KEY")

try:
    data = client.extract(
        url="https://example.com/products/headphones",
        schema={
            "title": "string",
            "price": "number",
            "in_stock": "boolean",
            "rating": "number"
        }
    )

    print(data.results)

except Exception as e:
    print(f"Extraction error handled gracefully: {e}")

Asynchronous Extraction

For higher-throughput workloads, you can use the async client:

import asyncio

from scraping_ai import AsyncScrapingAIClient


async def main():
    async with AsyncScrapingAIClient(
        api_key="YOUR_API_KEY"
    ) as client:

        data = await client.extract(
            url="https://example.com/products/headphones",
            schema={
                "title": "string",
                "price": "number"
            }
        )

        print(data.results)


asyncio.run(main())

Sample Output

The result is structured JSON:

{
  "results": [
    {
      "data": {
        "title": "Wireless Noise Cancelling Headphones",
        "price": 89.99,
        "in_stock": true,
        "rating": 4.7
      },
      "target_url": "https://example.com/products/headphones"
    }
  ]
}

Using Scraping AI with AI Agents and LangChain

If you're building an LLM agent or RAG pipeline, you can expose Scraping AI as a web extraction tool.

For example:

from langchain.tools import tool
from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="YOUR_API_KEY")


@tool
def web_data_extractor(
    url: str,
    required_schema_description: str
) -> dict:
    """Fetch clean, structured JSON from a URL."""

    result = client.extract(
        url=url,
        schema={
            "extracted_info": "string",
            "summary": "string"
        }
    )

    return result.results

The agent gets structured data instead of having to reason over a page full of HTML, styles, scripts, and other noise.

Honest Limitations

Scraping AI isn't a replacement for every scraping tool.

A few things to keep in mind:

• Bot protection: Automated stealth handling works against many bot-defense systems, but some aggressive Cloudflare Turnstile configurations can still require manual intervention.
• Excluded targets: Social platforms such as X, Instagram, and LinkedIn are excluded, as is the extraction of personal private data.
• Simple sites: If you're scraping a single personal blog once, BeautifulSoup is probably all you need.

The point is not to replace traditional scraping everywhere. It's to reduce the maintenance work that comes with extracting structured data from websites that keep changing.

Get Started

1. Sign up: https://pig-data.jp/service/scraping-ai/

2. Get 200 free tokens with no credit card required.

3. Install the SDK:
   

   pip install scraping-ai

1. Run your first extraction.

Pricing

• Free: 200 tokens
• Starter: $10 / 1,600 tokens
• Growth: $30 / 5,000 tokens
• Pro: $100 / 20,000 tokens

API Documentation: https://pig-data.jp/service/scraping-ai/docs/

About Scraping AI

Scraping AI (https://pig-data.jp/service/scraping-ai/) is developed and operated by indigodata Inc., an AI venture subsidiary of SMS DataTech Co., Ltd. in Tokyo, Japan.

The product is based on PigData's experience with 500+ enterprise data extraction projects and provides a self-serve LLM extraction API for developers.

Full disclosure

I’m a software developer at Indigodata, the team behind Scraping AI. I'm sharing the architecture behind how we built this because dealing with broken CSS selectors is a pain we've all faced.

Note: This article was co-authored with my colleague Harsh Tripathi and originally published on [Medium]. I’m sharing our team's work here with the Dev.to community!