DEV Community

XSron Hou
XSron Hou

Posted on • Originally published at scrapio.dev

How to Extract Structured JSON from Any Website

#webscraping #api #ai #showdev

Originally posted on the Scrapio blog — sharing here too.

Every data pipeline eventually hits the same wall: the information you need is on a web page, but it comes back as a blob of HTML. Writing CSS selectors works until the site redesigns. Regex works until it doesn't.

Scrapio's extract parameter takes a different approach — you describe the shape of the data you want, and the API figures out how to pull it from any page.

What you'll need

  • A Scrapio API key
  • The URL of a page with structured content (product page, article, listing)
  • curl or Python

Define a schema

Pass output: ["json"] and an extract object with mode: "schema". The schema maps field names to their types — the API handles figuring out where those values live on the page.

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/wireless-headphones",
    "output": ["json"],
    "extract": {
      "mode": "schema",
      "schema": {
        "name": "string",
        "price": "number",
        "currency": "string",
        "availability": "string",
        "rating": "number",
        "review_count": "integer"
      }
    }
  }'
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "json": {
      "name": "Sony WH-1000XM5 Wireless Headphones",
      "price": 279.99,
      "currency": "USD",
      "availability": "In Stock",
      "rating": 4.7,
      "review_count": 12483
    }
  },
  "usage": { "credits": 5 }
}
Enter fullscreen mode Exit fullscreen mode

Schema and instruction extraction run through the LLM extractor by default (5 credits) whenever an LLM provider is configured on your account; it only falls back to the cheaper heuristic extractor (2 credits) if no provider is configured or the LLM call fails. Set "engine": "heuristic" in the extract object if you want to guarantee the 2-credit rate.

Extract with a natural-language instruction

For more complex extractions, use mode: "instruction" to describe what you want in plain English:

import httpx

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/search?q=headphones",
        "output": ["json"],
        "extract": {
            "mode": "instruction",
            "instruction": "Extract all product listings. For each product return: title, price, URL, and rating.",
        },
    },
    timeout=30,
)
items = resp.json()["outputs"]["json"]
print(f"Found {len(items)} products")
Enter fullscreen mode Exit fullscreen mode

Extract using CSS selectors

When you know the page structure, mode: "selectors" is the most precise option:

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/products/headphones",
        "output": ["json"],
        "extract": {
            "mode": "selectors",
            "fields": {
                "title":  {"selector": "h1.product-title", "type": "text"},
                "price":  {"selector": ".price-now",       "type": "text"},
                "rating": {"selector": "[data-rating]",    "type": "attr", "attribute": "data-rating"},
            },
        },
    },
    timeout=30,
)
data = resp.json()["outputs"]["json"]
Enter fullscreen mode Exit fullscreen mode

Tips

  • Keep schemas focused. Only request fields you need — the more fields, the harder the extraction.
  • Use render_js: true for pages that load prices or content via JavaScript.
  • Validate downstream. For production pipelines, validate the output shape before inserting into a database.

Next steps

Top comments (0)