DEV Community

Dodo Data
Dodo Data

Posted on

Every Shopify store publishes its whole catalogue at one URL

#webscraping #ecommerce #api #json

If you have ever written a scraper for a Shopify store, you probably parsed the HTML. You did not have to.

Every Shopify storefront serves its own catalogue as JSON at /products.json. It is not a private API and not a leak — it is the storefront's own public endpoint, the same data the shop renders for any visitor:

https://www.allbirds.com/products.json?limit=250&page=1
https://www.allbirds.com/collections/mens/products.json?limit=250&page=1
https://www.allbirds.com/meta.json          → store name, currency, country
Enter fullscreen mode Exit fullscreen mode

250 products per page, page=N until you get an empty list. Collections work the same way, so you can scope a run to one part of a shop.

What a product actually contains

The interesting part is not the title — it is variants. Each variant is a real SKU with its own price, its own crossed-out price and its own stock flag:

{
  "id": 7205191974992,
  "title": "Daily Grind Women's Corduroy Workwear Jacket",
  "vendor": "Death Wish Coffee",
  "variants": [
    { "sku": "DWC-JKT-S", "price": "65.00", "compare_at_price": "90.00", "available": true,  "option1": "S" },
    { "sku": "DWC-JKT-M", "price": "65.00", "compare_at_price": "90.00", "available": false, "option1": "M" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Two things follow from that shape, and both are easy to get wrong.

A discount only means something inside one variant. If you take the lowest price in the product and compare it with the highest compare_at_price in the product, a shop that sells a $10 accessory and a $100 jacket marked down from $120 will report a 92% discount. It is not on sale by 92%; those are two different things being subtracted from each other. Compute the discount per variant, then take the best one:

const discounts = variants
    .filter((v) => v.on_sale && v.price !== null && v.compare_at_price)
    .map((v) => Math.round((1 - v.price / v.compare_at_price) * 100));
const bestDiscount = discounts.length ? Math.max(...discounts) : null;
Enter fullscreen mode Exit fullscreen mode

Stock is per variant too. "In stock" for a product usually means some size is in stock. If you are monitoring availability, one row per variant is the only honest shape.

Prices are strings, and the currency is somewhere else

price arrives as "65.00", a string. /products.json never tells you the currency — that lives in /meta.json alongside the shop's name and country. Fetch it once per store and attach it to every row, or your dataset silently mixes USD, EUR and GBP in one column.

A real run

Two stores (Allbirds and Death Wish Coffee), one run on 2026-09-17: 439 products, 3,282 variants, 9 seconds, 5 requests. 23 products were on sale, 246 had at least one variant in stock, and every row carried the store's currency. The deepest genuine discount was a pair of socks at $4 against a $16 compare-at price — computed inside one variant, which is why it can be trusted.

What the endpoint will not give you

Some shops turn /products.json off or put it behind a bot check; Gymshark, for one, returns 403. Treat that as a normal outcome, log it and move on — it is a per-store decision by the merchant, not a puzzle to defeat. And a site that is not on Shopify simply 404s.

Use it

If you would rather not maintain the pagination, the variant maths and the per-store currency, it is on Apify Store as Shopify Products Scraper & API — paste store or collection URLs, choose one row per product or per variant, filter to in-stock or on-sale only, $1 per 1,000 rows:

https://apify.com/dododata/shopify-products-scraper

The id field is stable across runs (shopify:store:product:variant), so a daily run plus a diff is a price and stock monitor in about one line of code. If something changes, it is fixed within 48 hours — that promise is on the listing.

Top comments (0)