How to Scrape Any Shopify Store and Get Structured Product Data

#webscraping #python #ecommerce #shopify

Most Shopify stores expose a public /products.json endpoint. No API key needed, no authentication, no anti-bot measures. Just append /products.json to any Shopify store URL and you get structured product data back.

I built a tool around this that handles pagination, variant extraction, and batch processing across multiple stores. Here's how it works and why you might want it.

The /products.json Endpoint

Every Shopify store has this:

https://any-store.myshopify.com/products.json?limit=250&page=1

It returns JSON with product titles, descriptions, pricing, variants (sizes, colors, SKUs), images, availability, and tags. Max 250 products per page, so you paginate through larger catalogs.

This works on custom domains too — if a store runs on Shopify, the endpoint exists.

What You Get Back

Here's what the structured output looks like per product:

{
  "title": "Classic Cotton T-Shirt",
  "vendor": "BrandName",
  "product_type": "Clothing",
  "price": 29.99,
  "price_max": 34.99,
  "compare_at_price": 49.99,
  "available": true,
  "variants_count": 3,
  "variants": [
    {"title": "S", "price": "29.99", "sku": "TS-S", "available": true},
    {"title": "M", "price": "29.99", "sku": "TS-M", "available": true},
    {"title": "L", "price": "34.99", "sku": "TS-L", "available": false}
  ],
  "tags": ["cotton", "summer", "sale"],
  "image_url": "https://cdn.shopify.com/...",
  "product_url": "https://store.com/products/classic-cotton-t-shirt",
  "store_url": "https://store.com"
}

You get min/max pricing across variants, availability status, SKUs, compare-at prices (for sale items), and direct product URLs.

Running It at Scale

I packaged this as an Apify Actor so you can run it in the cloud without setting up infrastructure:

{
  "storeUrls": [
    "https://competitor-a.com",
    "https://competitor-b.myshopify.com",
    "https://competitor-c.com"
  ],
  "maxProducts": 500,
  "includeVariants": true,
  "includeCollections": true
}

It handles:

Batch processing — scrape multiple stores in one run
Pagination — automatically walks through all pages
Variant extraction — every size/color/SKU with individual pricing
Collections — store categories and their metadata
Error handling — retries on failures, skips non-Shopify stores

Use Cases

Price monitoring — Run it daily on competitor stores. Track when prices drop, when items go on sale, when new products appear.

Product research — Analyzing a niche? Scrape 10 stores in that category and compare product ranges, pricing strategies, and inventory depth.

Dropshipping research — Find products across suppliers, compare margins, check availability.

Market analysis — Compare how different brands in the same space position their products and pricing.

The Technical Details

Under the hood it uses httpx for async HTTP with retry logic and redirect following (important because many Shopify stores use custom domains that redirect). It validates that a store is actually running Shopify before trying to scrape it, so you don't waste time on non-Shopify URLs.

The pagination follows Shopify's pattern: keep requesting pages until you get fewer than 250 products back, which means you've reached the end.

Try It

The quickest way to test: run it on Apify with a single store URL. The default input scrapes Shopify's own demo store so you can see the output format immediately.

If you want to integrate it programmatically, call the Apify API:

curl -X POST "https://api.apify.com/v2/acts/ambitious_door~shopify-scraper/runs" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"storeUrls": ["https://your-target-store.com"], "maxProducts": 100}'