Turn any web page into typed JSON with an LLM and a JSON Schema (Python)

Writing CSS selectors for every site you want data from gets old fast, and the selectors break when the site changes its markup. For many jobs (a product list, an events page, a team page) it's simpler to give an LLM the page text and a schema and let it fill in the fields.

The naive version (paste the HTML, ask for JSON) works in a demo and fails in production: the output shape drifts, links vanish, and long pages blow through the context window. This post shows a version that holds up, using a real page you can try: the Travel category of books.toscrape.com, a sandbox site built for scraping practice.

The pattern

1. Fetch and clean. Strip scripts, styles, navigation and footers. They cost tokens and add noise.
2. Keep the link targets. After you convert to text, the model can't return a URL it never saw, so write text (url) in place of each link.
3. Cap the size. Trim to a fixed budget before sending.
4. Force the shape. Use the API's strict JSON Schema mode, not "please answer in JSON".

The script

import json, os, re, requests  # pip install requests beautifulsoup4
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
html = requests.get(URL, timeout=30).text
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "header", "footer"]):
    tag.decompose()
# Keep link targets: the model can't return a URL it never saw.
for a in soup.find_all("a", href=True):
    a.replace_with(f"{a.get_text(strip=True)} ({requests.compat.urljoin(URL, a['href'])})")
text = re.sub(r"\n\s*\n+", "\n", soup.get_text("\n"))[:30000]

schema = {
    "type": "object",
    "properties": {"books": {"type": "array", "items": {
        "type": "object",
        "properties": {"title": {"type": "string"}, "price": {"type": "number"},
                       "currency": {"type": "string"}, "inStock": {"type": "boolean"},
                       "url": {"type": "string"}},
        "required": ["title", "price", "currency", "inStock", "url"],
        "additionalProperties": False}}},
    "required": ["books"],
    "additionalProperties": False,
}
r = requests.post(
    "https://api.openai.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
    json={
        "model": "gpt-6-luna",
        "messages": [
            {"role": "system", "content": "Extract data from the page text. Use only what is on the page; never guess."},
            {"role": "user", "content": f"Every book on this page.\n\nPAGE:\n{text}"},
        ],
        "response_format": {"type": "json_schema",
                            "json_schema": {"name": "page", "strict": True, "schema": schema}},
    },
    timeout=120,
)
r.raise_for_status()
data = json.loads(r.json()["choices"][0]["message"]["content"])
print(len(data["books"]), "books")
print(data["books"][0])

Output:

11 books
{'title': "It's Only the Himalayas", 'price': 45.17, 'currency': '£', 'inStock': True, 'url': 'https://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html'}

All 11 books on the page, with absolute URLs and real numbers, from one request. (The model name is whatever your account uses; any recent model that supports structured outputs works.)

The gotchas

1. Say what format you want, not only what field. Notice 'currency': '£'. The schema says string, and the page shows £, so that's a valid answer. If you need GBP, write it in the instruction: "currency as an ISO 4217 code". The same goes for dates ("ISO 8601") and numbers ("price as a number without the symbol").

2. Strict mode has rules. With "strict": true, every object needs "additionalProperties": false and every property must appear in required. For an optional field, make it nullable ("type": ["string", "null"]) and keep it in required.

3. The model only sees text. Books to Scrape shows the star rating as a CSS class (<p class="star-rating Three">), not as text. Ask for a rating and the model either guesses or returns nothing. When a value lives in an attribute, carry it into the text yourself, as the script does for href. Here that means reading the star-rating class. Tell the model "never guess", so the field comes back empty instead of made up.

4. Long pages need a budget. Truncating at 30,000 characters keeps cost predictable, but anything past the cut is lost. If the data sits in one region of the page, select it first (soup.select_one("main")) rather than sending everything.

5. Validate anyway. Strict mode guarantees the shape, not the truth. Check that prices are positive and URLs are on the expected domain, and log pages where a list came back empty.

6. Crawling multiplies everything. For a whole catalog, you need link following, deduplication, robots.txt, retries, rate limiting and a cost cap per run, which is more code than the extraction itself.

If you'd rather not maintain it

Disclosure: I packaged this as a hosted tool, AI Extract on Apify, with the source on GitHub. You pass URLs, a plain-English instruction and/or a schema, and it returns one JSON row per page. An AI model is included, or you can bring your own OpenAI, Anthropic, Gemini, DeepSeek, Groq, OpenRouter or Together key. It also handles crawling (maxCrawlDepth, URL include/exclude patterns), robots.txt, and limits for input and output tokens.

The schema can be a full JSON Schema or a shorthand, including nested lists:

{
  "startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"}],
  "instructions": "For every book on the page: title, price as a number, currency code, in stock (boolean), star rating 1-5, and the book page URL.",
  "jsonSchema": {
    "books": [{"title": "string", "price": "number", "currency": "string",
                "inStock": "boolean", "rating": "integer", "url": "string"}]
  },
  "maxPages": 1
}

The row it returned, trimmed to two of the 11 books:

{
  "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html",
  "status": "ok",
  "title": "Travel | Books to Scrape - Sandbox",
  "data": {
    "books": [
      {"title": "It's Only the Himalayas", "price": 45.17, "currency": "GBP", "inStock": true, "rating": 2,
       "url": "https://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html"},
      {"title": "Under the Tuscan Sun", "price": 37.33, "currency": "GBP", "inStock": true, "rating": 3,
       "url": "https://books.toscrape.com/catalogue/under-the-tuscan-sun_504/index.html"}
    ]
  }
}

Two of the gotchas above, handled. The instruction asked for a "currency code", so the result is GBP, not £. And rating is filled in, even though the page shows it only as a CSS class. Before converting the page to text, the tool writes values that live only in attributes into the text: star-rating classes (star-rating Three, stars-4-5), data-rating, schema.org microdata (itemprop="ratingValue" content="4.7") and aria-labels on icon links. All 11 ratings matched the page. When a value really isn't on the page, the field is left out rather than guessed.

Call it from Python

import os
from apify_client import ApifyClient  # pip install apify-client

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("swillz/ai-extract").call(
    run_input={
        "startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"}],
        "instructions": "Every book: title, price as a number, ISO currency code, in stock, book URL.",
        "jsonSchema": {"books": [{"title": "string", "price": "number", "currency": "string",
                                   "inStock": "boolean", "url": "string"}]},
        "maxPages": 1,
    },
    max_total_charge_usd=0.05,
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    for book in row["data"].get("books", []):
        print(book["title"], book["price"], book["currency"])

max_total_charge_usd caps what the run can cost. (On 1.x versions of the client, use run["defaultDatasetId"].)

Useful options: contentSelector (send only main or .product to the model), removeSelectors (strip reviews or related products), maxCrawlDepth plus includeUrlPatterns (e.g. /catalogue/) to crawl a section, and includeMarkdown if you also want the cleaned page text for RAG.

Let an AI agent extract pages (MCP)

Add Apify's hosted MCP server with just this tool:

{
  "mcpServers": {
    "apify": { "url": "https://mcp.apify.com?tools=swillz/ai-extract" }
  }
}

Then ask:

"From these three product pages, get name, price, currency code and availability, and put them in a table."

Tips for agents:

• Keep runs short. Leave maxPages at 10 or below for answers within the MCP call-actor wait (about 45 seconds).
• Give a schema when the shape matters. The agent can pass the shorthand ({"items": [{"name": "string", "price": "number"}]}), and the rows come back validated against it.
• Failures are explicit. Pages that can't be loaded, or a schema with no fields, return a clear error instead of an empty result.

Wrap-up

• Clean the HTML, keep the link targets, and cap the size before you call the model.
• Use strict JSON Schema mode, and spell out formats (ISO codes, numbers without symbols) in the instruction.
• The model sees text only, so copy values that exist only in attributes or CSS classes into the text first.
• Validate the values, not just the shape.