Getting stable JSON from a page with a schema

Markdown is useful for reading and indexing, but downstream code usually needs a typed record: a product name, a price, a release date, or a list of features. The MESSORA /scrape endpoint can return structured JSON when the request includes "json" in formats and supplies a json_schema.

The schema is the boundary between page text and application data. Keep it narrow. A small contract is easier to validate, compare across pages, and repair when a site changes its layout.

Send a schema with the request

import os
import requests

API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}

product_schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "price": {"type": ["number", "null"]},
        "currency": {"type": ["string", "null"]},
        "availability": {"type": "string"},
    },
    "required": ["name", "price", "currency", "availability"],
    "additionalProperties": False,
}

response = requests.post(
    f"{API}/scrape",
    headers=HEADERS,
    json={
        "url": "https://example.com/products/widget",
        "formats": ["json"],
        "json_schema": product_schema,
        "only_main_content": True,
    },
    timeout=90,
)
response.raise_for_status()
payload = response.json()

if payload["scrape_status"] != "success":
    raise RuntimeError(payload.get("error") or payload["scrape_status"])

record = payload["json"]
print(record["name"], record["price"], record["currency"])

json_schema is required whenever formats includes json. Omitting it is a request validation error, not a reason to accept an unstructured response. The response places the extracted object under json; the usual fields such as scrape_status, credits_used, and remaining_credits remain outside that object.

Keep extraction failures visible

A successful HTTP response does not guarantee that structured extraction produced useful values. When extraction fails completely, the API sets json_extraction_error_code to extraction_failed and keeps the schema properties in json with null values. Check that field before writing the record to a database.

def extract_record(payload: dict) -> dict:
    if payload["scrape_status"] != "success":
        raise RuntimeError(payload.get("error") or payload["scrape_status"])

    if payload.get("json_extraction_error_code") == "extraction_failed":
        raise ValueError("page fetched, but no structured record was extracted")

    record = payload.get("json")
    if not isinstance(record, dict):
        raise ValueError("response did not contain a JSON object")
    return record

Do not use a missing json value as the only failure signal. A page can be fetched successfully while a model cannot locate the requested fields. The explicit error code preserves that distinction for metrics and retry policy.

Make null a deliberate value

If a source sometimes omits a price, define price as number or null instead of forcing a fake zero. Zero means a free product; null means the page did not provide a usable price. The same rule applies to dates, ratings, and availability. Your schema should represent what the source can actually say.

For fields that need a controlled vocabulary, describe the values in the schema or in json_prompt. Do not put parser instructions only in a comment in your application. The request should carry the extraction contract that explains how the result is meant to be interpreted.

Validate before persistence

The API returns a JSON object, but the receiving service still owns its database invariants. Validate types, required fields, and domain constraints locally. Record the source URL and the extraction status alongside the normalized row, so a later review can distinguish a changed product from a failed extraction.

For a mixed pipeline, request both markdown and json during development. Markdown lets an operator inspect the source interpretation; JSON feeds the application. Once the schema is stable, request only the format you consume to keep payloads smaller. The cost remains one successful scrape, while the response contract becomes explicit instead of relying on positional text parsing.