Turn scraped HTML into clean JSON with your own schema

#webscraping #ai #json #python

If you've scraped anything, you know the boring part isn't fetching pages. It's what comes after: prices written as £51.77, $135k - $160k / year or 2,350/mo, cookie banners mixed into the text, star ratings that only exist as CSS classes, and every site laid out differently. You end up writing a pile of selectors and regexes per site, and they break the next time the site changes.

I built a small tool for exactly that step: spruce_the_data. You describe the JSON you want with a JSON Schema, hand it messy text, HTML or page links, and it returns data in that shape. It runs on Apify, so you can try it in the browser or call it from code.

In this post I'll walk through three ways to use it:

Clean web pages straight from their URLs
Clean HTML you already have, with your own fields
Run it after a scraper you already use, or from Python

1. From a link to JSON in one step

Open spruce_the_data on the Apify Store and switch the input to JSON. Paste this:

{
  "urls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  ],
  "outputSchema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "price": { "type": ["number", "null"] },
      "currency": { "type": ["string", "null"], "description": "ISO 4217 code, e.g. GBP" },
      "in_stock": { "type": ["boolean", "null"] },
      "rating": { "type": ["integer", "null"], "description": "Star rating from 1 to 5" }
    },
    "required": ["title", "price", "currency", "in_stock", "rating"]
  }
}

Click Start. A few seconds per page later, the Output tab has one row per link:

{
  "index": 0,
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "status": "success",
  "data": {
    "title": "A Light in the Attic",
    "price": 51.77,
    "currency": "GBP",
    "in_stock": true,
    "rating": 3
  },
  "droppedFields": []
}

The second book comes back as "Tipping the Velvet", 53.74, "GBP", true, 1.

The rating is the interesting part. On that site it isn't written anywhere as text; it's a <p class="star-rating Three"> with five icons inside. The cleanup step keeps hints like that class while throwing away scripts, styles and layout, so the model can still read it.

(books.toscrape.com is a practice site made for testing scrapers, so it's a safe one to try.)

2. Your HTML, your fields

The schema is the whole interface. Here's a messy job card, with breadcrumbs, a cookie banner and a salary written as $135k - $160k / year:

{
  "texts": [
    "<div class=\"job-card\"><nav>Home &gt; Jobs &gt; Engineering</nav><h2 class=\"title\">Senior Data Engineer</h2><div class=\"company\">Northwind Analytics</div><span class=\"loc\">Austin, TX (Hybrid, 2 days in office)</span><ul><li>Full-time</li><li>$135k - $160k / year</li></ul><p>We're hiring a data engineer to own our ingestion pipelines.</p><div class=\"cookie\">We use cookies. Accept?</div></div>"
  ],
  "outputSchema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "company": { "type": ["string", "null"] },
      "location": { "type": ["string", "null"], "description": "City and region as written" },
      "work_mode": { "type": ["string", "null"], "enum": ["onsite", "hybrid", "remote", null] },
      "employment_type": { "type": ["string", "null"], "enum": ["full-time", "part-time", "contract", "internship", null] },
      "salary_min": { "type": ["number", "null"], "description": "Yearly amount as a plain number, e.g. 135000" },
      "salary_max": { "type": ["number", "null"], "description": "Yearly amount as a plain number, e.g. 160000" },
      "currency": { "type": ["string", "null"], "description": "ISO 4217 code, e.g. USD" }
    },
    "required": ["title", "company", "location", "work_mode", "employment_type", "salary_min", "salary_max", "currency"]
  }
}

Result:

{
  "title": "Senior Data Engineer",
  "company": "Northwind Analytics",
  "location": "Austin, TX",
  "work_mode": "hybrid",
  "employment_type": "full-time",
  "salary_min": 135000,
  "salary_max": 160000,
  "currency": "USD"
}

A few things that make a real difference to accuracy:

Describe your fields. "ISO 4217 code, e.g. USD" or "Yearly amount as a plain number" tells the model exactly what format you want.
Use enums for anything with a fixed set of values, like work_mode above.
Make fields nullable (["number", "null"]) when the value might be missing, so it can say "not there" instead of guessing.
Use formats. "format": "email", "uri", "date" and "phone" are checked after extraction. If a nullable field fails the check, it comes back as null and is listed in droppedFields, so you know which values to look at.

3. After your scraper, or from code

Chaining with a scraper. If you already run a scraper on Apify, copy its run's dataset ID, paste it into Source dataset ID, and set Text field in the source dataset to the field holding the page text or HTML (for example text or html). Every item gets cleaned into your schema. With Apify's integrations you can trigger this automatically whenever the scraper finishes.

From Python. Install the client with pip install apify-client (version 3 or later), then:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")  # Apify Console > Settings > API & Integrations

run = client.actor("sprucethedata/clean-json").call(run_input={
    "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    "outputSchema": {
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "price": {"type": ["number", "null"]},
            "currency": {"type": ["string", "null"], "description": "ISO 4217 code, e.g. GBP"},
        },
        "required": ["title", "price", "currency"],
    },
})

for item in client.dataset(run.default_dataset_id).iterate_items():
    if item["status"] == "success":
        print(item["data"])
    else:
        print("failed:", item["error"])

What it's not good at

I'd rather you know this up front:

It's not a crawler. URL mode does a plain download. Pages that need JavaScript to show their content, need a login, or block bots won't load. For those, run a proper scraper first and feed its results in (option 3).
One item per page works best. A product, a job ad, a listing. Long list pages work but are slower, and very long pages are rejected with a message.
It doesn't read link addresses or image URLs inside a page, only the visible text.
It processes items one at a time, about 5 seconds each, so big batches take a while.

Pricing

It's pay per result: you're charged for items that come back clean, and failed items are free. Apify free-plan accounts can clean 5 items per run to try it out. Current prices are on the Store page.

Feedback welcome

This is a new tool and I'm actively improving it. If you try it on a page where it gets something wrong, open an issue on the Actor's Issues tab with the input (minus anything private). Those reports are the fastest way it gets better.

Top comments (1)

magia c'c • Oct 22

The CSS star-rating example shows why cleanup must keep semantic hints. For JS-rendered sources, BrowserAct Cloud can supply reusable structured extraction before validation.