Extracting typed JSON from any web page with a Pydantic schema

#python #pydantic #ai #webscraping

Parsing product pages with CSS selectors breaks the first time a designer renames a class. The selector-based approach encodes the page's current DOM shape into your code, and that shape is not part of any contract the site owner agreed to.

Schema-driven extraction inverts the dependency: you declare the fields you want, and the extraction layer figures out where they live. Here is how to do it with a Pydantic model, and what it costs.

Declare the shape you want

import json
import os
import requests
from pydantic import BaseModel, Field

API = "https://api.messora.dev"


class Product(BaseModel):
    name: str = Field(description="Product display name")
    price_brl: float = Field(description="Price in BRL, numeric only")
    in_stock: bool = Field(description="True when purchasable right now")
    sku: str | None = Field(default=None, description="Manufacturer SKU if shown")


def extract(url: str, model: type[BaseModel]) -> dict:
    resp = requests.post(
        f"{API}/scrape",
        headers={"X-API-Key": os.environ["MESSORA_API_KEY"]},
        json={
            "url": url,
            "formats": ["json"],
            "json_schema": model.model_json_schema(),
            "only_main_content": True,
        },
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()

    if data["scrape_status"] != "success":
        raise RuntimeError(f"{url} -> {data['scrape_status']}")

    if data.get("json_extraction_error_code"):
        raise ValueError(f"extraction rejected: {data['json_extraction_error_code']}")

    return data["json"]


payload = extract("https://example.com/produto/teclado-mecanico", Product)
print(json.dumps(payload, indent=2, ensure_ascii=False))

model_json_schema() from Pydantic emits standard JSON Schema, which is exactly what the json_schema field expects. The description on each field is not decoration — it is the disambiguation signal when a page shows three numbers that could all be prices.

Validate on the way back in

The response is a dict, not a model instance. Round-trip it through Pydantic so a missing or mistyped field fails loudly at the boundary instead of three layers deeper:

product = Product.model_validate(payload)
assert product.price_brl > 0

This is the step most integrations skip. Without it, a null price silently becomes 0.0 somewhere downstream and your pricing dashboard reports a free keyboard.

Cost model

Structured extraction is billed at a flat 10 credits per request, regardless of page size. Markdown and raw scraping cost 1 credit per page. The gap reflects the inference pass that maps page content onto your schema.

Two consequences for how you design jobs:

Do not request json when you only need text. Ten credits for a page you were going to embed anyway is a 10x overspend.
Batch fields, not requests. One schema with twelve fields costs 10 credits. Twelve single-field requests cost 120 credits for the same page.

When there is no schema yet

json_prompt accepts a natural-language description instead of a schema. It is the right tool while you are still exploring what a page even contains:

json={
    "url": url,
    "formats": ["json"],
    "json_prompt": "Return the author name, publication date in ISO 8601, and a list of cited URLs.",
}

The tradeoff is that the output shape is not guaranteed across calls. Use json_prompt to discover the shape, then freeze it into a Pydantic model for anything that runs on a schedule.

Failure signal to watch

json_extraction_error_code is populated when the fetch succeeded but the mapping did not. That is a different problem from scrape_status and needs different handling: a blocked_antibot means try another URL, while an extraction error usually means your schema asks for a field the page does not have.

Checking only scrape_status and then reading data["json"] gives you None with no explanation. Check both.

Practical field design

Fields that work well are the ones a human could point at on the rendered page. Fields that fail are the ones requiring inference across pages — "is this cheaper than competitors" is not on the page, so it does not belong in the schema.

Keep Optional on anything that is genuinely absent on some pages. A required field that is missing forces the extractor to invent a value, and an invented SKU is worse than a null one.

DEV Community