DEV Community

neuralbyte
neuralbyte

Posted on

Website to JSON API: Build a Schema-Validated Extraction Pipeline

TL;DR

  • Build website-to-JSON as retrieve → map → validate → store, not as one opaque prompt.
  • Define a versioned JSON Schema before crawling and keep provenance outside the model-generated fields.
  • Use Nstproxy Crawl to retrieve a clean page representation, then apply deterministic selectors, an extraction model, or a hybrid mapper.
  • Validate twice: JSON Schema checks shape; semantic checks compare important values with source evidence.
  • Scale with bounded per-host queues and measure accepted records rather than HTTP successes.

A “website to JSON API” sounds like a single operation. Production systems work better when the operation is split into four observable stages:

URL
  → retrieved page evidence
  → mapped business object
  → validated record
  → durable storage
Enter fullscreen mode Exit fullscreen mode

This separation tells you whether a failure came from page access, rendering, extraction, validation, or persistence. It also lets you replace one component without redesigning the entire pipeline.

This tutorial uses a public product-page schema and Nstproxy Crawl as the retrieval layer. The API request requires your own key; the schema validation code is runnable locally.

1. Define the JSON Contract

Start with the object your downstream system needs, not the HTML currently available on one sample page.

Create product-page.schema.json:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.org/schemas/product-page.json",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "name",
    "price",
    "currency",
    "availability",
    "source_url",
    "extracted_at"
  ],
  "properties": {
    "name": {"type": "string", "minLength": 1},
    "sku": {"type": ["string", "null"]},
    "price": {"type": "string", "pattern": "^[0-9]+(\\.[0-9]{2})?$"},
    "currency": {"type": "string", "pattern": "^[A-Z]{3}$"},
    "availability": {
      "type": "string",
      "enum": ["in_stock", "out_of_stock", "preorder", "unknown"]
    },
    "source_url": {"type": "string", "format": "uri"},
    "extracted_at": {"type": "string", "format": "date-time"}
  }
}
Enter fullscreen mode Exit fullscreen mode

Prices are strings here to avoid binary floating-point surprises. sku is nullable because some legitimate product pages do not expose one. additionalProperties: false prevents an extractor from quietly expanding the contract.

The JSON Schema specification documents the available validation vocabulary. Treat changes to required fields or field meaning as schema versions, not silent edits.

2. Retrieve Page Evidence With Nstproxy Crawl

Nstproxy Crawl can act as the access and rendering layer. Its current documentation describes synchronous and asynchronous page scraping, browser rendering, main-content extraction, task state, and multiple output formats.

The following request uses the synchronous quick-start route shown in the current documentation:

curl --request POST \
  --url 'https://api.nstproxy.com/api/v1/crawl/scrape/submit-sync' \
  --header "x-api-key: $NSTPROXY_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://example.com/product/coffee-grinder",
    "formats": ["markdown", "html"],
    "onlyMainContent": true,
    "timeout": 60000
  }'
Enter fullscreen mode Exit fullscreen mode

Use an environment variable or secret manager for the API key. Never place it in source control or logs.

Request Markdown when a model will interpret headings and prose. Request HTML when deterministic selectors, attributes, tables, or embedded JSON-LD matter. Avoid requesting formats you will not consume because every extra artifact creates storage and processing work.

Inspect both the outer envelope and the nested task result. A successful HTTP response does not guarantee that the target page contains meaningful content. Record the target status code, final URL, title, language, and task-level success state when present.

3. Map Fields Using the Right Extractor

Choose the mapping method by field stability rather than applying an LLM to everything.

Method Best fit Main failure mode
Embedded JSON-LD Sites publishing accurate structured metadata Metadata can be stale or incomplete
CSS selectors Stable templates you control or monitor closely Layout changes break selectors
Extraction model Heterogeneous prose and page layouts Unsupported or inferred values
Hybrid mapper Critical identifiers plus descriptive fields More orchestration and validation

A robust hybrid pipeline might use JSON-LD for SKU and product identity, a deterministic price selector for the selected variant, and a model for descriptive attributes.

If an extraction model is used, give it the schema and the retrieved evidence. Require it to return null or an explicit unknown value when the source does not support a field. Do not let the model generate source_url or extracted_at; those belong to the pipeline.

4. Validate the Object Locally

Install the validator:

python3 -m pip install 'jsonschema[format]'
Enter fullscreen mode Exit fullscreen mode

Create validate_product.py:

import json
from pathlib import Path

from jsonschema import Draft202012Validator, FormatChecker

schema = json.loads(Path("product-page.schema.json").read_text())

record = {
    "name": "Example Coffee Grinder",
    "sku": "GRIND-01",
    "price": "89.00",
    "currency": "USD",
    "availability": "in_stock",
    "source_url": "https://example.com/product/coffee-grinder",
    "extracted_at": "2026-09-07T08:00:00Z",
}

validator = Draft202012Validator(
    schema,
    format_checker=FormatChecker(),
)

errors = sorted(
    validator.iter_errors(record),
    key=lambda error: list(error.path),
)

if errors:
    for error in errors:
        print(f"{list(error.path)}: {error.message}")
    raise SystemExit(1)

print("valid product record")
Enter fullscreen mode Exit fullscreen mode

Run it:

python3 validate_product.py
Enter fullscreen mode Exit fullscreen mode

Expected output:

valid product record
Enter fullscreen mode Exit fullscreen mode

Change currency to lowercase or remove name; the validator should reject the object.

5. Add Semantic Evidence Checks

Schema validity is necessary but insufficient. The string 9999.00 can satisfy the price pattern even when the page displays 89.00.

Add domain checks after structural validation:

from decimal import Decimal, InvalidOperation


def validate_semantics(record: dict, visible_text: str) -> list[str]:
    errors = []

    try:
        price = Decimal(record["price"])
        if price < 0:
            errors.append("price_negative")
    except InvalidOperation:
        errors.append("price_invalid_decimal")

    if record["price"] not in visible_text:
        errors.append("price_not_in_evidence")

    if record["name"].casefold() not in visible_text.casefold():
        errors.append("name_not_in_evidence")

    return errors
Enter fullscreen mode Exit fullscreen mode

Real commerce pages need more careful normalization for thousands separators, currency symbols, sale prices, and variants. The important pattern is that critical values must be connected to retrieved evidence.

6. Store Data and Provenance Separately

A useful storage envelope separates business fields from processing history:

{
  "data": {
    "name": "Example Coffee Grinder",
    "price": "89.00",
    "currency": "USD",
    "availability": "in_stock"
  },
  "provenance": {
    "requested_url": "https://example.com/product/coffee-grinder",
    "final_url": "https://example.com/product/coffee-grinder",
    "retrieved_at": "2026-09-07T07:59:40Z",
    "extractor_version": "product-v3",
    "content_hash": "sha256:illustrative-value"
  },
  "quality": {
    "schema_valid": true,
    "evidence_check": "passed"
  }
}
Enter fullscreen mode Exit fullscreen mode

This envelope supports reprocessing. When product-v4 ships, the team can map the retained evidence again while preserving the earlier record and its provenance.

7. Scale With Bounded Queues

Scale by domain, not by firing an unlimited number of requests.

Track at least four counters:

submitted_urls
pages_with_meaningful_content
schema_valid_objects
semantically_accepted_objects
Enter fullscreen mode Exit fullscreen mode

Calculate cost per accepted object from the final counter. Separately record rejection reasons such as retrieval_empty, challenge_page, schema_required_field, unsupported_locale, duplicate_canonical, and evidence_mismatch.

Use bounded exponential backoff for transient failures and a per-host circuit breaker when errors rise. Deduplicate using normalized canonical URLs and content hashes. Do not automatically retry schema failures: a missing required field is usually a mapping or source problem, not a network problem.

Current Nstproxy billing options are described on the Crawl pricing page. For LLM extraction workflows, the URL-to-Markdown guide explains why clean page representations are easier to process than raw HTML.

8. Protect User-Supplied URL Inputs

If users can submit arbitrary URLs, treat the retrieval service as an SSRF boundary. Block localhost, private and link-local address ranges, cloud metadata hosts, unsafe schemes, suspicious ports, and redirects outside the approved scope.

The OWASP SSRF guidance provides a practical checklist. Also respect authentication boundaries, site terms, privacy requirements, copyright, and data-retention rules.

9. Make Extraction Idempotent and Reproducible

The same page should not create a new logical record every time a worker retries. Give each extraction an idempotency key derived from stable inputs such as the canonical URL, schema version, extractor version, and content hash.

idempotency_key = hash(
  canonical_url
  + content_hash
  + schema_version
  + extractor_version
)
Enter fullscreen mode Exit fullscreen mode

This design separates two different events. If the page content is unchanged and only a transport retry occurs, the key remains the same and storage can safely upsert the result. If the source content or extraction contract changes, the new key preserves a separate version for review.

Reproducibility also requires freezing the inputs used by the mapper. Store the cleaned Markdown or HTML reference, the schema version, deterministic parsing rules, model identifier when applicable, prompt version, and normalization configuration. A record that cannot be reproduced becomes difficult to correct when a customer challenges it or a template changes.

Avoid using retrieval time alone as the record identity. Two workers can fetch the same page seconds apart and create duplicate objects. Use retrieval time as provenance, while canonical identity and content determine whether the business record changed.

10. Observe Every Pipeline Boundary

Production monitoring should reveal where accepted records are being lost. Track latency, throughput, and rejection counts independently for retrieval, mapping, schema validation, semantic validation, and storage.

Useful service-level metrics include:

retrieval_success_rate
meaningful_content_rate
schema_valid_rate
semantic_acceptance_rate
duplicate_rate
cost_per_accepted_record
p95_end_to_end_latency
Enter fullscreen mode Exit fullscreen mode

Break the metrics down by domain, page template, schema version, and extractor version. An overall success rate can remain stable while one important source silently deteriorates.

Keep representative rejected examples for debugging, but apply a retention policy and remove sensitive values from logs. Alert on changes in rejection distribution rather than only on total request failures. A rise in evidence_mismatch is often more important than a small increase in HTTP errors because it indicates that the pipeline is returning plausible but unreliable data.

Create a small golden dataset for every important template. Each example should contain the source representation, expected object, and expected rejection outcome when data is missing. Run it whenever the mapper, schema, normalization rules, or extraction model changes. Golden cases will not replace live monitoring, but they catch obvious regressions before a new version processes thousands of pages.

Add a manual-review lane for ambiguous high-value records. Reviewers should see the extracted object beside the exact source evidence and choose a structured rejection reason. Feed those decisions back into tests and mapping rules rather than treating manual corrections as isolated edits. This creates a measurable improvement loop without allowing unreviewed guesses into the production dataset.

Final Take

A production website-to-JSON pipeline is not a converter. It is an evidence system.

Retrieve a reproducible page representation, map only supported fields, validate the object’s shape, verify important values against evidence, and store provenance with the result. Once those boundaries are observable, the pipeline becomes easier to debug, scale, and trust.

FAQ

Is JSON Schema enough for website data extraction?

No. JSON Schema verifies structure and allowed values, but it cannot prove that extracted values are factually supported by a page. Add semantic evidence checks for important fields.

Should I use Markdown or HTML for extraction?

Use Markdown when a model needs clean headings and prose. Use HTML when selectors, tables, attributes, or embedded structured data are important. Some pipelines benefit from retaining both.

Can I use an LLM for every field?

You can, but deterministic sources are usually better for stable identifiers and critical numeric values. A hybrid mapper often provides a stronger balance between flexibility and auditability.

How should extraction failures be retried?

Retry transient retrieval failures with bounded exponential backoff. Do not blindly retry schema or semantic failures; route them to revised mapping rules or manual review.

What should be stored with each JSON record?

Store the requested and final URL, retrieval time, content hash, extractor version, validation outcome, and enough source evidence to reproduce important fields.

Top comments (0)