DEV Community

mamangwarkop
mamangwarkop

Posted on Originally published at github.com

How to Extract Shopify Competitor Product Catalogs & Real Discount Margins with Python

#shopify #python #webscraping #ecommerce

How to Extract Shopify Competitor Product Catalogs & Real Discount Margins with Python

Whether you run an e-commerce brand, a dropshipping store, or a digital marketing agency, competitor price monitoring is often tedious. Manually clicking through hundreds of category pages or paying \$200/month for specialized e-commerce intelligence tools is both slow and expensive.

Fortunately, Shopify stores have a standardized architecture that makes extraction straightforward.

In this tutorial, you'll learn:

  1. How Shopify's hidden JSON endpoints work.
  2. How to extract product titles, variants, and live inventory status.
  3. How to calculate actual discount percentages programmatically.
  4. How to detect installed marketing pixels (Klaviyo, Meta, TikTok) on any store.

1. The Power of Shopify's Standardized API

Almost 99% of Shopify merchants leave their public catalog endpoints exposed under standard routing:

  • Catalog Endpoint: https://[store-domain].com/products.json?limit=250&page=1
  • Collections Endpoint: https://[store-domain].com/collections.json

This means you don't need heavy headless browsers (like Selenium or Puppeteer) just to read product data. Pure HTTP requests with JSON deserialization work instantly.

2. Python Script to Scrape and Calculate Discount Margins

Here is an example in Python that fetches products, resolves all SKU variants, and detects real promotional markdowns:

import urllib.request
import json

def fetch_shopify_products(store_url: str, limit: int = 50):
    endpoint = f"{store_url.rstrip('/')}/products.json?limit={limit}"
    req = urllib.request.Request(
        endpoint,
        headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
    )

    with urllib.request.urlopen(req, timeout=15) as resp:
        data = json.loads(resp.read().decode("utf-8"))

    enriched_products = []
    for item in data.get("products", []):
        variants = item.get("variants", [])
        for v in variants:
            price = float(v.get("price") or 0.0)
            compare_price = float(v.get("compare_at_price") or 0.0) if v.get("compare_at_price") else None

            discount_pct = 0.0
            if compare_price and compare_price > price:
                discount_pct = round(((compare_price - price) / compare_price) * 100, 2)

            enriched_products.append({
                "productTitle": item.get("title"),
                "variantTitle": v.get("title"),
                "sku": v.get("sku"),
                "currentPrice": price,
                "originalPrice": compare_price,
                "discountPercent": discount_pct,
                "inStock": v.get("available", False)
            })

    return enriched_products

# Example test:
# products = fetch_shopify_products("https://allbirds.com", limit=10)
# print(products[:2])
Enter fullscreen mode Exit fullscreen mode

3. Detecting Marketing Stack (Pixels & CRM)

If you are an agency pitching e-commerce brands, knowing their tech stack gives you immediate leverage. By fetching the homepage HTML and scanning script tags, you can check:

  • Klaviyo: Look for klaviyo.com/onsite/js
  • Meta Pixel: Look for connect.facebook.net/en_US/fbevents.js
  • TikTok Pixel: Look for analytics.tiktok.com/i18n/pixel

If a merchant has high-volume traffic but is missing TikTok Pixel or Klaviyo, that's your exact \$2,000/month pitch right there.

4. Ready-to-Use Cloud Tool & n8n Blueprint

If you want an automated tool that handles pagination, export to CSV/JSON, and store lead qualification without having to write or host the code yourself:

Happy building, and feel free to connect or drop suggestions below!

Top comments (0)