Scraping E-commerce Product Data: Variants, Prices and Stock Without Losing Accuracy

E-commerce data scraping is the collection of public product information from online stores (titles, identifiers, variants, prices, availability and pack sizes) into structured records you can compare across sites and over time. Fetching the pages is rarely the hard part. Accuracy is lost in the modelling: a price attached to the wrong variant, a sale price read as the list price, or "Only 2 left" stored as free text.

These are the five fields that break most often:

1. Variant price: the page shows one price, but each size or colour has its own.
2. Sale versus list price: two numbers on the page, and the parser picks the wrong one.
3. Currency and number format: 1.299,00 and 1,299.00 are the same amount.
4. Stock status: a dozen phrasings for four or five real states.
5. Pack size: a 6 x 330ml multipack compared with a single can.

This guide works through each one, schema first, with Python you can adapt. Every snippet below was run against sample data.

1. Decide the grain: one record per variant

The most important decision comes before any parsing. If a T-shirt comes in four sizes and three colours, are you storing one product or twelve?

For price and stock work, store one record per variant (SKU), linked to its parent. Prices and availability live at variant level, so a parent-level record will eventually average away the thing you care about.

{
  "parent_id": "TS-100",
  "sku": "TS-100-NAVY-L",
  "title": "Organic Cotton T-Shirt",
  "varies_by": ["colour", "size"],
  "attributes": { "colour": "Navy", "size": "L" },
  "gtin": "05012345678900",
  "price": "19.99",
  "list_price": "24.99",
  "currency": "GBP",
  "availability": "in_stock",
  "unit_price": null,
  "source_url": "https://example-shop.co.uk/p/ts-100?colour=navy&size=l",
  "collected_at": "2026-10-22T06:00:00Z",
  "parser_version": "3.2.0"
}

Three fields here are not product data at all: source_url, collected_at and parser_version. Keep them on every record. When a number looks wrong three weeks later, they tell you where it came from, when, and which code produced it.

2. Read the most structured source first

Most product pages expose the same data in up to three places. In order of preference:

1. Embedded structured data, usually schema.org Product, ProductGroup and Offer in JSON-LD.
2. Embedded application state, the JSON a JavaScript front end uses to render the page.
3. The visible page, parsed from the HTML.

Schema.org's ProductGroup type is built for variants: hasVariant lists each variant, variesBy names the properties that distinguish them, and productGroupID identifies the parent. Google added support for this markup in its product structured data in early 2024, so more stores now publish it.

import json
from bs4 import BeautifulSoup

def iter_nodes(data):
    """Yield every dict in a JSON-LD blob, including @graph members."""
    if isinstance(data, list):
        for item in data:
            yield from iter_nodes(item)
    elif isinstance(data, dict):
        yield data
        for item in data.get("@graph", []):
            yield from iter_nodes(item)

def extract_products(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    products = []
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            blob = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue  # malformed JSON-LD is common; fall back to the DOM
        for node in iter_nodes(blob):
            kind = node.get("@type")
            if kind == "ProductGroup":
                for variant in node.get("hasVariant", []):
                    variant.setdefault("_parent_id", node.get("productGroupID"))
                    products.append(variant)
            elif kind == "Product":
                products.append(node)
    return products

One caution: structured data is not always what the shopper sees. Some stores only mark up the default variant, and some cache it longer than the visible price. Spot-check a sample of records against the rendered page for each source, and treat a mismatch as a parser bug until proven otherwise.

3. Prices: two numbers, one currency, no floats

When a page shows a struck-through price and a lower one, store both: price (what the shopper pays now) and list_price (the original). In schema.org markup, the original is usually an Offer price specification with priceType set to StrikethroughPrice, the convention Google documents for sale pricing.

Take the currency from priceCurrency or the store's locale, never from the symbol: "$" is used by many currencies. Store amounts as Decimal, not floats, and parse with the source's decimal separator:

from decimal import Decimal, InvalidOperation
import re

def parse_price(raw: str, decimal_sep: str = ".") -> Decimal | None:
    """Parse '1.299,00' or '1,299.00' into Decimal('1299.00')."""
    if raw is None:
        return None
    digits = re.sub(r"[^\d,.]", "", str(raw))
    thousands_sep = "," if decimal_sep == "." else "."
    digits = digits.replace(thousands_sep, "").replace(decimal_sep, ".")
    try:
        return Decimal(digits)
    except InvalidOperation:
        return None

Set decimal_sep per source in configuration rather than guessing from the string. 1.299 is ambiguous on its own.

4. Stock: map many phrasings to a few states

Schema.org's ItemAvailability has twelve values, and visible pages add their own wording on top. Downstream users usually need five or six states. Map explicitly, keep the raw value alongside, and treat anything unmapped as unknown so it surfaces rather than hides:

AVAILABILITY = {
    "InStock": "in_stock",
    "LimitedAvailability": "in_stock",
    "OnlineOnly": "in_stock",
    "InStoreOnly": "store_only",
    "OutOfStock": "out_of_stock",
    "SoldOut": "out_of_stock",
    "BackOrder": "backorder",
    "PreOrder": "preorder",
    "PreSale": "preorder",
    "MadeToOrder": "made_to_order",
    "Discontinued": "discontinued",
    "Reserved": "unavailable",
}

def parse_availability(raw: str | None) -> str:
    if not raw:
        return "unknown"
    key = raw.rstrip("/").rsplit("/", 1)[-1]  # handles full schema.org URLs
    return AVAILABILITY.get(key, "unknown")

5. Pack sizes: compare like with like

Comparing a £3.60 six-pack with a £0.75 single can is meaningless until both are a price per litre. Parse the pack size and compute a unit price:

PACK = re.compile(r"(?:(\d+)\s*[x×]\s*)?(\d+(?:[.,]\d+)?)\s*(ml|l|g|kg)\b", re.I)
TO_BASE = {"ml": ("l", Decimal("0.001")), "l": ("l", Decimal(1)),
           "g": ("kg", Decimal("0.001")), "kg": ("kg", Decimal(1))}

def unit_price(title: str, price: Decimal) -> tuple[Decimal, str] | None:
    """'Sparkling Water 6 x 330ml' at 3.60 -> (Decimal('1.82'), 'l')"""
    m = PACK.search(title)
    if not m:
        return None
    count = int(m.group(1) or 1)
    size = Decimal(m.group(2).replace(",", "."))
    base_unit, factor = TO_BASE[m.group(3).lower()]
    quantity = count * size * factor
    return (price / quantity).quantize(Decimal("0.01")), base_unit

Prefer a dedicated size field or structured attributes when the store provides them. The title is a fallback, and a regex like this one should be tested against each source's naming habits.

6. Validate every record, then every run

Parsing produces data. Validation tells you whether to trust it. Check each record against rules:

REQUIRED = ("sku", "source_url", "collected_at", "currency", "price", "availability")

def validate(rec: dict) -> list[str]:
    issues = [f"missing:{f}" for f in REQUIRED if rec.get(f) in (None, "")]
    price, list_price = rec.get("price"), rec.get("list_price")
    if price is not None and price <= 0:
        issues.append("price_not_positive")
    if price is not None and list_price is not None and price > list_price:
        issues.append("sale_above_list")
    if rec.get("currency") and not re.fullmatch(r"[A-Z]{3}", rec["currency"]):
        issues.append("currency_not_iso4217")
    if rec.get("availability") == "unknown":
        issues.append("availability_unmapped")
    for attr in rec.get("varies_by", []):
        if not rec.get("attributes", {}).get(attr):
            issues.append(f"variant_attr_missing:{attr}")
    return issues

Then check each run as a whole, because the worst failures (a site redesign that silently drops half the catalogue) look fine record by record:

def check_run(today: list[dict], yesterday_count: int, tolerance: float = 0.15) -> list[str]:
    alerts = []
    if yesterday_count and abs(len(today) - yesterday_count) / yesterday_count > tolerance:
        alerts.append(f"record_count_shift:{yesterday_count}->{len(today)}")
    null_share = sum(r.get("price") is None for r in today) / max(len(today), 1)
    if null_share > 0.02:
        alerts.append(f"null_price_share:{null_share:.1%}")
    return alerts

Flag large price jumps for review rather than dropping them. A 60% drop is sometimes a parser error and sometimes a genuine clearance sale, and only one of those should be deleted.

Why the effort pays off

Accurate variant-level records are what make the data usable for the work it is collected for: price comparison, assortment analysis and keeping a catalogue complete and consistent. Teams that manage e-commerce product data across many sources find that the cost of a modelling mistake grows with every downstream system that trusts it. Fix it at extraction, where it is cheapest.

Two practical notes to finish. Collect from public product pages only, at a polite request rate, and respect each site's terms. And store the raw extraction next to the parsed record, so that when you fix a parser you can reprocess history instead of recollecting it.

FAQ

What data can you scrape from e-commerce sites?

Typically public product information: titles, brand, identifiers such as GTIN or MPN, variant attributes (size, colour), current and original prices, currency, availability, pack size, category and product URL. Collect from publicly visible product pages only.

How do you handle product variants when scraping?

Store one record per variant (SKU), linked to a parent product ID, with the attributes that distinguish it. Where a store publishes schema.org ProductGroup markup, hasVariant and variesBy provide this structure directly.

Should you scrape JSON-LD or the visible page?

Start with JSON-LD or other embedded structured data because it is more stable and better structured, then spot-check it against the visible page. Some stores only mark up the default variant or update markup less often than the displayed price.

Comments

[Rave Digital]

Accurate product data is essential for making informed ecommerce decisions, especially when pricing and availability change across variants. We see how even small data inconsistencies can affect catalog management, customer trust, and purchasing decisions. Appreciate the focus on data accuracy as a foundation for reliable ecommerce operations.