agenthustler

Posted on Apr 9

Shopify Store Scraping: Extract Product Data, Reviews and Inventory

#webdev #programming #javascript #webscraping

Web scraping Shopify stores has become one of the most common data extraction tasks in e-commerce intelligence. Whether you're monitoring competitor prices, building a product comparison engine, or analyzing market trends, Shopify's predictable structure makes it an excellent target for automated data collection.

In this comprehensive guide, I'll walk you through everything you need to know about extracting product data, reviews, and inventory information from Shopify-powered stores.

Understanding Shopify's Storefront Architecture

Shopify powers over 4.6 million websites globally. What makes it particularly interesting for web scraping is its consistent, well-structured data layer. Every Shopify store follows the same underlying architecture, which means a scraper built for one store can often work across thousands of others with minimal modifications.

The JSON Product API

Every Shopify store exposes a built-in JSON API that doesn't require authentication for public product data. This is the single most important thing to understand about Shopify scraping:

https://store-domain.com/products.json
https://store-domain.com/products.json?page=2&limit=250
https://store-domain.com/collections/all/products.json

This endpoint returns structured JSON containing product titles, descriptions, variants, prices, images, tags, and more. No API key required for public storefronts.

Product Page Structure

Individual product pages follow this pattern:

https://store-domain.com/products/product-handle.json

Each product page also embeds structured data in JSON-LD format within the HTML, which search engines use and which provides another reliable extraction point.

Extracting Product Catalog Data

Let's start with the most fundamental task: pulling the complete product catalog from a Shopify store.

JavaScript Approach (Node.js)

const axios = require('axios');

async function scrapeShopifyProducts(storeUrl) {
    const products = [];
    let page = 1;
    const limit = 250; // Maximum allowed by Shopify

    while (true) {
        try {
            const url = `${storeUrl}/products.json?page=${page}&limit=${limit}`;
            const response = await axios.get(url, {
                headers: {
                    'User-Agent': 'Mozilla/5.0 (compatible; ProductResearch/1.0)'
                }
            });

            const pageProducts = response.data.products;

            if (!pageProducts || pageProducts.length === 0) {
                break;
            }

            for (const product of pageProducts) {
                products.push({
                    id: product.id,
                    title: product.title,
                    vendor: product.vendor,
                    productType: product.product_type,
                    handle: product.handle,
                    createdAt: product.created_at,
                    updatedAt: product.updated_at,
                    tags: product.tags,
                    variants: product.variants.map(v => ({
                        id: v.id,
                        title: v.title,
                        price: v.price,
                        compareAtPrice: v.compare_at_price,
                        sku: v.sku,
                        available: v.available,
                        inventoryQuantity: v.inventory_quantity
                    })),
                    images: product.images.map(img => img.src)
                });
            }

            console.log(`Page ${page}: Found ${pageProducts.length} products`);
            page++;

            // Respectful delay between requests
            await new Promise(resolve => setTimeout(resolve, 1000));
        } catch (error) {
            if (error.response && error.response.status === 429) {
                console.log('Rate limited. Waiting 30 seconds...');
                await new Promise(resolve => setTimeout(resolve, 30000));
                continue;
            }
            break;
        }
    }

    console.log(`Total products scraped: ${products.length}`);
    return products;
}

// Usage
scrapeShopifyProducts('https://example-store.myshopify.com')
    .then(products => {
        const fs = require('fs');
        fs.writeFileSync('products.json', JSON.stringify(products, null, 2));
    });

Python Approach

import requests
import json
import time

def scrape_shopify_products(store_url):
    products = []
    page = 1
    limit = 250

    session = requests.Session()
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (compatible; ProductResearch/1.0)'
    })

    while True:
        url = f"{store_url}/products.json?page={page}&limit={limit}"

        try:
            response = session.get(url, timeout=30)

            if response.status_code == 429:
                print("Rate limited. Waiting 30 seconds...")
                time.sleep(30)
                continue

            response.raise_for_status()
            data = response.json()

            page_products = data.get('products', [])
            if not page_products:
                break

            for product in page_products:
                products.append({
                    'id': product['id'],
                    'title': product['title'],
                    'vendor': product['vendor'],
                    'product_type': product['product_type'],
                    'handle': product['handle'],
                    'tags': product.get('tags', ''),
                    'variants': [{
                        'id': v['id'],
                        'title': v['title'],
                        'price': v['price'],
                        'compare_at_price': v.get('compare_at_price'),
                        'sku': v.get('sku'),
                        'available': v.get('available', False),
                    } for v in product.get('variants', [])],
                    'images': [img['src'] for img in product.get('images', [])]
                })

            print(f"Page {page}: {len(page_products)} products")
            page += 1
            time.sleep(1)

        except requests.exceptions.RequestException as e:
            print(f"Error on page {page}: {e}")
            break

    print(f"Total: {len(products)} products")
    return products

# Usage
products = scrape_shopify_products("https://example-store.myshopify.com")
with open("products.json", "w") as f:
    json.dump(products, f, indent=2)

Handling Pagination and Large Catalogs

Shopify's /products.json endpoint has a limit of 250 products per page. For stores with thousands of products, you need proper pagination handling.

Cursor-Based Pagination

Some Shopify stores use cursor-based pagination through the Storefront API. Here's how to handle it:

async function scrapeWithCursor(storeUrl) {
    let cursor = null;
    const allProducts = [];

    while (true) {
        let url = `${storeUrl}/products.json?limit=250`;
        if (cursor) {
            url += `&page_info=${cursor}`;
        }

        const response = await axios.get(url);
        const products = response.data.products;

        if (products.length === 0) break;

        allProducts.push(...products);

        // Check for Link header with next page cursor
        const linkHeader = response.headers['link'];
        if (linkHeader && linkHeader.includes('rel="next"')) {
            const match = linkHeader.match(/page_info=([^>&]*)/);
            cursor = match ? match[1] : null;
        } else {
            break;
        }

        await new Promise(r => setTimeout(r, 1000));
    }

    return allProducts;
}

Collection-Based Approach

For very large stores, scraping by collection often yields better results:

import requests
import time

def get_collections(store_url):
    # Fetch all collections from a Shopify store
    response = requests.get(f"{store_url}/collections.json")
    return response.json().get('collections', [])

def scrape_by_collection(store_url):
    # Scrape products organized by collection
    collections = get_collections(store_url)
    all_products = {}

    for collection in collections:
        handle = collection['handle']
        page = 1

        while True:
            url = f"{store_url}/collections/{handle}/products.json?page={page}&limit=250"
            response = requests.get(url)
            products = response.json().get('products', [])

            if not products:
                break

            for product in products:
                product_id = product['id']
                if product_id not in all_products:
                    all_products[product_id] = product
                    all_products[product_id]['collections'] = []
                all_products[product_id]['collections'].append(handle)

            page += 1
            time.sleep(1)

    return list(all_products.values())

Extracting Product Reviews

Reviews are crucial for competitive analysis and sentiment monitoring. Shopify stores typically use third-party review apps, each with their own data format.

Common Review Apps and Their Endpoints

Judge.me Reviews:

async function scrapeJudgeMeReviews(shopDomain, apiToken) {
    const reviews = [];
    let page = 1;

    while (true) {
        const url = `https://judge.me/api/v1/reviews?` +
            `shop_domain=${shopDomain}&page=${page}&per_page=100`;

        const response = await axios.get(url, {
            headers: { 'Authorization': `Bearer ${apiToken}` }
        });

        const data = response.data.reviews;
        if (!data || data.length === 0) break;

        reviews.push(...data.map(r => ({
            id: r.id,
            rating: r.rating,
            title: r.title,
            body: r.body,
            reviewer: r.reviewer.name,
            createdAt: r.created_at,
            productId: r.product_external_id,
            verified: r.verified_buyer
        })));

        page++;
        await new Promise(r => setTimeout(r, 500));
    }

    return reviews;
}

Extracting Reviews from HTML (Generic Approach):

from bs4 import BeautifulSoup
import requests
import re

def extract_reviews_from_page(product_url):
    # Generic review extraction from product page HTML
    response = requests.get(product_url, headers={
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
    })
    soup = BeautifulSoup(response.text, 'html.parser')

    reviews = []

    # Look for common review containers
    review_selectors = [
        '.spr-review',           # Shopify Product Reviews
        '.jdgm-rev',             # Judge.me
        '.loox-review',          # Loox
        '.yotpo-review',         # Yotpo
        '[data-review-id]',      # Generic data attribute
    ]

    for selector in review_selectors:
        review_elements = soup.select(selector)
        if review_elements:
            for elem in review_elements:
                review = {}

                # Extract rating
                stars = elem.select_one('[class*="star"]')
                if stars:
                    rating_match = re.search(r'(\d)', str(stars.get('style', '')))
                    review['rating'] = rating_match.group(1) if rating_match else None

                # Extract review text
                body = elem.select_one('[class*="body"], [class*="content"], [class*="text"]')
                review['body'] = body.get_text(strip=True) if body else ''

                # Extract author
                author = elem.select_one('[class*="author"], [class*="name"]')
                review['author'] = author.get_text(strip=True) if author else 'Anonymous'

                if review.get('body'):
                    reviews.append(review)

            break

    return reviews

Price Monitoring and Inventory Tracking

One of the highest-value applications of Shopify scraping is automated price monitoring.

Building a Price Monitor

const axios = require('axios');
const fs = require('fs');

class ShopifyPriceMonitor {
    constructor(stores) {
        this.stores = stores;
        this.historyFile = 'price_history.json';
        this.history = this.loadHistory();
    }

    loadHistory() {
        try {
            return JSON.parse(fs.readFileSync(this.historyFile, 'utf8'));
        } catch {
            return {};
        }
    }

    saveHistory() {
        fs.writeFileSync(this.historyFile, JSON.stringify(this.history, null, 2));
    }

    async checkPrices() {
        const changes = [];
        const timestamp = new Date().toISOString();

        for (const store of this.stores) {
            try {
                const response = await axios.get(
                    `${store.url}/products.json?limit=250`
                );

                for (const product of response.data.products) {
                    for (const variant of product.variants) {
                        const key = `${store.name}:${variant.id}`;
                        const currentPrice = parseFloat(variant.price);
                        const previousEntry = this.history[key];

                        if (previousEntry) {
                            const previousPrice = previousEntry.price;
                            if (currentPrice !== previousPrice) {
                                const change = {
                                    store: store.name,
                                    product: product.title,
                                    variant: variant.title,
                                    oldPrice: previousPrice,
                                    newPrice: currentPrice,
                                    change: ((currentPrice - previousPrice) / previousPrice * 100).toFixed(2),
                                    timestamp
                                };
                                changes.push(change);
                                console.log(
                                    `PRICE CHANGE: ${product.title} ` +
                                    `$${previousPrice} -> $${currentPrice} ` +
                                    `(${change.change}%)`
                                );
                            }
                        }

                        this.history[key] = {
                            price: currentPrice,
                            available: variant.available,
                            product: product.title,
                            variant: variant.title,
                            lastChecked: timestamp
                        };
                    }
                }

                await new Promise(r => setTimeout(r, 2000));
            } catch (err) {
                console.error(`Error checking ${store.name}: ${err.message}`);
            }
        }

        this.saveHistory();
        return changes;
    }
}

// Usage
const monitor = new ShopifyPriceMonitor([
    { name: 'Store A', url: 'https://store-a.myshopify.com' },
    { name: 'Store B', url: 'https://store-b.myshopify.com' }
]);

monitor.checkPrices().then(changes => {
    console.log(`Found ${changes.length} price changes`);
});

Inventory Level Detection

import requests

def check_inventory_status(store_url):
    # Check inventory availability across all products
    response = requests.get(f"{store_url}/products.json?limit=250")
    products = response.json().get('products', [])

    inventory_report = []

    for product in products:
        for variant in product.get('variants', []):
            status = {
                'product': product['title'],
                'variant': variant['title'],
                'price': variant['price'],
                'available': variant.get('available', False),
                'inventory_policy': variant.get('inventory_policy', 'deny'),
                'inventory_quantity': variant.get('inventory_quantity', 'N/A')
            }
            inventory_report.append(status)

    out_of_stock = [item for item in inventory_report if not item['available']]
    low_stock = [item for item in inventory_report
                 if item['available']
                 and isinstance(item['inventory_quantity'], int)
                 and item['inventory_quantity'] < 10]

    return {
        'total_variants': len(inventory_report),
        'out_of_stock': len(out_of_stock),
        'low_stock': len(low_stock),
        'details': inventory_report
    }

Using Apify Store for Shopify Scraping

While building your own scraper gives you full control, using pre-built actors on the Apify Store can save significant development time.

Apify offers several ready-to-use Shopify scrapers that handle all the edge cases — rate limiting, pagination, proxy rotation, and data formatting. These actors run in the cloud, so you don't need to manage infrastructure.

Key Benefits of Using Apify Actors

Proxy management: Automatic rotation to avoid IP blocks
Scheduled runs: Set up daily, hourly, or custom monitoring schedules
Cloud execution: No local infrastructure needed
Data export: Direct export to CSV, JSON, Excel, or webhooks
Monitoring dashboards: Track run success rates and data quality

Example: Using Apify SDK

const { ApifyClient } = require('apify-client');

const client = new ApifyClient({
    token: 'YOUR_APIFY_TOKEN',
});

async function runShopifyScraper(storeUrl) {
    const run = await client.actor('apify/shopify-scraper').call({
        startUrls: [{ url: storeUrl }],
        maxProducts: 1000,
        includeReviews: true,
        proxy: {
            useApifyProxy: true,
            apifyProxyGroups: ['RESIDENTIAL']
        }
    });

    const { items } = await client.dataset(run.defaultDatasetId).listItems();
    console.log(`Scraped ${items.length} products`);
    return items;
}

Best Practices and Rate Limiting

When scraping Shopify stores, following best practices ensures reliability and avoids issues:

Respectful Scraping Guidelines

Add delays between requests: At minimum 1 second between requests. Shopify's rate limit is typically 2 requests per second for unauthenticated access.
Use meaningful User-Agent strings: Identify your bot clearly rather than pretending to be a browser.
Handle 429 responses gracefully: Implement exponential backoff when rate limited.
Cache responses: Don't re-scrape data that hasn't changed. Check updated_at timestamps.
Respect robots.txt: While /products.json is generally accessible, check the store's robots.txt for any specific restrictions.

Error Handling Pattern

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def create_resilient_session():
    session = requests.Session()
    retries = Retry(
        total=3,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET"]
    )
    adapter = HTTPAdapter(max_retries=retries)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session

Data Storage and Analysis

Once you've collected product data, proper storage enables analysis:

import sqlite3
import json

def store_products(products, db_path="shopify_data.db"):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    cursor.execute('''
        CREATE TABLE IF NOT EXISTS products (
            id INTEGER PRIMARY KEY,
            title TEXT,
            vendor TEXT,
            product_type TEXT,
            handle TEXT,
            tags TEXT,
            created_at TEXT,
            updated_at TEXT,
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    ''')

    cursor.execute('''
        CREATE TABLE IF NOT EXISTS variants (
            id INTEGER PRIMARY KEY,
            product_id INTEGER,
            title TEXT,
            price REAL,
            compare_at_price REAL,
            sku TEXT,
            available BOOLEAN,
            FOREIGN KEY (product_id) REFERENCES products(id)
        )
    ''')

    for product in products:
        cursor.execute('''
            INSERT OR REPLACE INTO products
            (id, title, vendor, product_type, handle, tags)
            VALUES (?, ?, ?, ?, ?, ?)
        ''', (
            product['id'], product['title'], product['vendor'],
            product['product_type'], product['handle'],
            json.dumps(product.get('tags', []))
        ))

        for variant in product.get('variants', []):
            cursor.execute('''
                INSERT OR REPLACE INTO variants
                (id, product_id, title, price, compare_at_price, sku, available)
                VALUES (?, ?, ?, ?, ?, ?, ?)
            ''', (
                variant['id'], product['id'], variant['title'],
                float(variant['price']),
                float(variant['compare_at_price']) if variant.get('compare_at_price') else None,
                variant.get('sku'), variant.get('available', False)
            ))

    conn.commit()
    conn.close()

Conclusion

Shopify's consistent architecture makes it one of the most accessible e-commerce platforms for web scraping. The built-in JSON API eliminates much of the complexity you'd face with other platforms, and the predictable URL structure means your scrapers are reliable and maintainable.

Key takeaways:

Start with /products.json — it's the fastest path to product data
Handle pagination properly — use cursor-based pagination for large catalogs
Respect rate limits — 1-2 second delays between requests prevent blocks
Monitor prices over time — single snapshots are less valuable than trend data
Use cloud platforms like Apify for production workloads that need proxy rotation and scheduling
Store data in structured formats for analysis and comparison

Whether you're building a price comparison tool, tracking competitor inventory, or analyzing market trends, the techniques in this guide give you a solid foundation for extracting value from Shopify's vast ecosystem of online stores.

DEV Community