Every Shopify store has a public products.json: how to export a full catalog (and the currency trap)

#shopify #ecommerce #javascript #api

If a store runs on Shopify, you can usually get its whole public catalog without scraping a single HTML page. Storefronts expose /products.json: the same product data the store shows to every visitor, as JSON.

curl "https://www.allbirds.com/products.json?limit=250&page=1"

You get up to 250 products per page, each with title, vendor, product type, tags, images, options and every variant (size, color) with price, compare-at price, SKU and an in-stock flag.

Paging through a whole catalog

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function allProducts(store) {
  const out = [];
  for (let page = 1; ; page++) {
    const res = await fetch(`${store}/products.json?limit=250&page=${page}`, {
      headers: { 'User-Agent': 'my-catalog-export/1.0 (contact@example.com)' },
    });
    if (res.status === 429) { await sleep(10000); page--; continue; } // too fast: wait, retry
    if (!res.ok) break;
    const { products } = await res.json();
    out.push(...products);
    if (products.length < 250) break; // last page
    await sleep(800); // one request at a time, with a pause
  }
  return out;
}

Two more endpoints are handy:

/collections/<handle>/products.json: only one collection, e.g. sale or new-arrivals.
/meta.json: the store's name, country, main currency and number of published products.

Gotcha 1: "on sale" lives in compare_at_price

There's no "discount" field. A variant is on sale when compare_at_price is set and higher than price. Both are strings like "87.50", so convert them before comparing ("125.00" > "87.50" is false as strings).

Gotcha 2: prices can change currency based on where you are

This one cost me an hour. I ran the same export from Japan and got a sneaker priced at 14300. From a US server it was 87.50. Same product, same endpoint.

Stores selling internationally (Shopify Markets) localize prices to the visitor's country, and the JSON doesn't say which currency you got. Ways to stay sane:

Run from the country whose prices you want (servers in the US usually get USD).
Read the main currency from /meta.json and sanity-check a known product.
Never compare prices between runs made from different locations.

Gotcha 3: stock is a yes/no flag

available: true/false per variant is all you get. There are no exact quantities in this data, so don't promise "units in stock" in your dashboard.

Is this OK to use?

It's public data the storefront publishes for everyone, and Shopify's default robots.txt allows crawling product and collection pages. Still: go one request at a time, back off on 429, identify yourself, and respect each store's terms. Some stores switch the endpoint off or password-protect the shop; move on when they do.

The code

The full script (CSV export, paging, 429 backoff, sale flags; MIT, no dependencies) is on GitHub: swiftkit-dev/shopify-catalog-export.

The packaged version

I put this on Apify as Shopify Store Products Scraper: paste store or collection links and get products or one row per variant, the store's currency, an "only new products" mode for daily alerts, and a free summary per store (price range and median, share on sale, top vendors, products added in the last 30 days). It handles the paging and 429s for you.

What do you use catalog data for: price tracking, competitor research, or something else?

Written with AI assistance; the code and numbers were tested before publishing.

DEV Community