How to extract structured JSON from dynamic websites using typed Pydantic schemas

When scraping e-commerce sites, job boards, or financial portals, converting unstructured HTML into valid, typed JSON usually requires fragile CSS selectors or multi-step LLM extraction prompts that hallucinate or miss required fields.

A more reliable pattern is schema-constrained extraction at the gateway level, where the extraction engine enforces a JSON Schema directly.

Defining the target schema in Pydantic

from pydantic import BaseModel, Field
from typing import List, Optional

class ProductItem(BaseModel):
    title: str = Field(description="Product name without promotional badges")
    price_cents: int = Field(description="Price in integer cents (e.g. 1999 for $19.99)")
    currency: str = Field(default="USD", description="Three-letter ISO currency code")
    in_stock: bool = Field(description="Stock availability status")
    features: List[str] = Field(default_factory=list, description="Bullet points of key product specifications")

class ProductCatalog(BaseModel):
    store_name: str
    products: List[ProductItem]

Extracting typed data with MESSORA

Instead of writing custom BeautifulSoup parsers or regexes for every site layout, pass the JSON Schema directly to the extraction endpoint:

import os
import json
import requests

api_key = os.environ.get("MESSORA_API_KEY")

payload = {
    "url": "https://example-store.com/electronics",
    "json_schema": ProductCatalog.model_json_schema(),
}

response = requests.post(
    "https://api.messora.dev/v1/extract",
    headers={"Authorization": f"Bearer {api_key}"},
    json=payload,
    timeout=45,
)
response.raise_for_status()

# Parse directly into Pydantic model
catalog_data = response.json().get("json")
catalog = ProductCatalog.model_validate(catalog_data)

for product in catalog.products:
    print(f"{product.title}: ${product.price_cents / 100:.2f} (In Stock: {product.in_stock})")

Why schema-first extraction matters

1. Deterministic types: Booleans, integers, and nested lists conform to the schema type contracts on the first pass.
2. Zero parsing maintenance: When sites change class names or DOM hierarchies, extraction continues working without breaking selectors.
3. No prompt engineering required: Field descriptions in the schema serve as extraction instructions.