Competitor price tracking, product research and catalog migrations all start with the same question: how do I get every product of a store, with prices and stock, without copying it by hand? For the two most common e-commerce platforms you usually do not need to parse HTML at all.
Option 1: WooCommerce's public Store API in Python
Most WooCommerce stores answer at /wp-json/wc/store/v1/products (the Store API that WooCommerce's block-based cart and checkout use). It needs no key, returns up to 100 products per page, and includes prices, stock and categories.
import json
import urllib.request
def get(url):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (price-check-script)"})
with urllib.request.urlopen(req) as r:
return json.load(r)
def woo_products(store, pages=2):
out = []
for page in range(1, pages + 1):
items = get(f"https://{store}/wp-json/wc/store/v1/products?per_page=100&page={page}")
if not items:
break
for p in items:
pr = p["prices"]
unit = 10 ** pr["currency_minor_unit"] # prices come in cents as strings
out.append({
"name": p["name"],
"price": int(pr["price"]) / unit,
"regular_price": int(pr["regular_price"]) / unit,
"currency": pr["currency_code"],
"in_stock": p["is_in_stock"],
"url": p["permalink"],
})
return out
for p in woo_products("woocommerce.com", pages=1)[:5]:
print(p["price"], p["currency"], "|", p["name"], "|", "in stock" if p["in_stock"] else "out of stock")
Two details that trip people up: prices are strings in the currency's minor unit (cents), and product names come HTML-escaped (– and friends), so unescape them with html.unescape before you compare titles.
Shopify: products.json, and the 429 wall
Every Shopify store serves its catalog at /products.json?limit=250&page=N: title, handle, vendor, product type, tags, images and a variants list with price, compare_at_price, sku and available. For a single collection, use /collections/<handle>/products.json.
The catch is rate limiting. While writing this post, four different Shopify stores answered 429 Too Many Requests to the very first request from my own connection. Once you scrape more than a few stores, you need retries with back-off, pauses between pages, and ideally a different network than your laptop.
Other things you will end up handling:
- Which platform is it? A list of competitor domains does not tell you whether a store runs Shopify or WooCommerce.
- Variants: a T-shirt is one product with many sizes and colors, each with its own price, SKU and stock.
-
Sale prices:
compare_at_priceon Shopify andregular_pricevspriceon WooCommerce. - Change tracking: to know what changed since yesterday, you have to keep yesterday's prices.
Option 2: a ready-made Actor that also tracks changes
I packaged all of that as an Apify Actor: Shopify & WooCommerce Products Scraper. You paste store domains or URLs, it detects the platform, and it returns one clean row per product. A real item from a run (shortened):
{
"store": "www.beardbrand.com",
"platform": "shopify",
"title": "Bold Fortune Classic Beard Bundle",
"brand": "Beardbrand",
"productType": "Bundles",
"url": "https://www.beardbrand.com/products/bold-fortune-classic-beard-bundle",
"currency": "USD",
"price": 119.85,
"compareAtPrice": 141,
"onSale": true,
"available": false,
"variantCount": 1,
"change": null
}
What it adds:
- Filters for keywords, excluded words, minimum and maximum price, in stock only and on sale only
- Variants with their own price, SKU and stock, and the product description as plain text, when you turn them on
-
Return only changes: schedule it daily and each run returns only new products and products whose price or stock changed, with
previousPriceandpreviousAvailable - Export to JSON, CSV or Excel, or send results to Google Sheets, Slack or a webhook
From code:
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("webobsidians/shopify-woocommerce-products-scraper").call(
run_input={"stores": ["www.beardbrand.com", "woocommerce.com"], "maxProductsPerStore": 100, "onlyOnSale": True}
)
for p in client.dataset(run["defaultDatasetId"]).iterate_items():
print(p["store"], "|", p["title"], "|", p["price"], p["currency"])
It costs $1.00 per 1,000 products plus $0.00005 per run start, with platform usage included, and Apify's free plan includes monthly credit to try it.
Which one should you use?
- One WooCommerce store, once: the Python script above.
- Many stores, Shopify at scale, or daily price-change alerts: the Actor.
Either way, you are reading catalogs that the stores publish for every visitor, so keep request rates polite.
Top comments (0)