DEV Community: Jerry A. Henley

Automate Competitor Price Tracking: Turn One-Off Scrapes into Weekly Audits

Jerry A. Henley — Fri, 03 Apr 2026 07:36:27 +0000

Manual competitor analysis is a silent productivity killer. For growth and revenue teams, keeping tabs on a competitor like Zappos often involves an exhausting cycle of spot-checking top-selling items or waiting weeks for a busy engineering team to prioritize a custom scraper. By the time you get the data, the sale is over or the stock has shifted.

The problem with spot-checking is that it only captures a moment, not a trend. To truly understand a competitor's strategy, you need to monitor entire categories—like "Nike Running Shoes"—to see how prices fluctuate and when items go out of stock.

This guide shows you how to build a repeatable Zappos price monitor using Node.js and Playwright. We’ll use production-ready patterns from the ScrapeOps Zappos Scraper Bank to extract entire category listings and create a "Diff" engine that alerts you to price drops and new arrivals.

The Strategy: From Spot-Check to Audit

Most developers start by scraping a single product page. While useful, the real business value lies in Category Snapshotting.

Instead of looking at one shoe, we scrape the entire "Men's Running" category on Day 1. We then repeat the process on Day 7. By comparing these two snapshots, we can calculate the delta (the change) between the datasets. This allows us to identify:

Price Drops: Items where the current price is lower than last week.
Stock-outs: Items that were present last week but are missing now.
New Arrivals: Fresh inventory that didn't exist in the previous snapshot.

Prerequisites

To follow this tutorial, you'll need:

Node.js installed on your machine.
A text editor like VS Code.
A ScrapeOps API Key for proxy rotation (you can get a free one here).
Basic familiarity with the terminal.

Setup: Configuring the Category Scraper

We’ll use Playwright because Zappos uses dynamic content that simple HTML parsers often miss. Playwright renders the page fully, ensuring we capture prices even if they load via JavaScript.

Start by cloning the repository and installing the dependencies:

# Clone the repository
git clone https://github.com/scraper-bank/Zappos.com-Scrapers.git

# Navigate to the Playwright category scraper directory
cd Zappos.com-Scrapers/node/playwright/product_category

# Install dependencies
npm install playwright-extra puppeteer-extra-plugin-stealth cheerio

Open scraper/zappos_scraper_product_category_v1.js and replace YOUR-API_KEY with your actual ScrapeOps API key. This routes your requests through residential proxies to avoid anti-bot detection.

Code Walkthrough: Ensuring Data Integrity

When building a price monitor, data reliability is everything. If your scraper saves the same product twice because of a page refresh or a sorting shift, your analysis will be skewed.

The ScrapeOps implementation uses a DataPipeline class to handle this. It uses a Set to track itemsSeen. Even if Zappos shifts product order during pagination, the script only records each item once.

class DataPipeline {
    constructor(outputFile = CONFIG.outputFile) {
        this.itemsSeen = new Set();
        this.outputFile = outputFile;
        this.writeFile = promisify(fs.appendFile);
    }

    isDuplicate(data) {
        // Use the unique Product ID or URL as the key
        const itemKey = data.productId || data.url;
        if (this.itemsSeen.has(itemKey)) {
            console.warn('Duplicate item found, skipping');
            return true;
        }
        this.itemsSeen.add(itemKey);
        return false;
    }

    async addData(scrapedData) {
        if (!this.isDuplicate(scrapedData)) {
            const jsonLine = JSON.stringify(scrapedData) + '\n';
            await this.writeFile(this.outputFile, jsonLine, 'utf8');
        }
    }
}

Why JSONL?

The script outputs to .jsonl (JSON Lines). Unlike a standard JSON array, JSONL stores one object per line. This is a recommended approach for web scraping because:

It’s streamable: You can read the file line-by-line without loading massive files into memory.
It’s reliable: If the scraper crashes on page 10, the first 9 pages of data are already safely saved on disk.

Execution: Running the Scrape

To monitor a specific category, find the URL on Zappos, such as a filtered search for "Nike Running Shoes," and pass it to the script.

Run this command in your terminal:

node scraper/zappos_scraper_product_category_v1.js

The scraper will navigate through the pagination. You'll see logs as it saves items:
Saved item to zappos_com_product_category_page_scraper_data_20240214_120000.jsonl

Pro Tip: Rename your output files with the date, like nike_running_week_01.jsonl. This makes the comparison step much easier.

The "Diff" Strategy: Finding Insights

Once you have two files (e.g., week1.jsonl and week2.jsonl), you need to find the differences. We can use a simple Node.js utility script to perform this audit.

This script loads the old data into a Map for fast lookup and then iterates through the new data to find changes.

const fs = require('fs');

function loadData(filePath) {
    const data = new Map();
    const lines = fs.readFileSync(filePath, 'utf8').split('\n');
    lines.forEach(line => {
        if (line) {
            const item = JSON.parse(line);
            data.set(item.productId, item);
        }
    });
    return data;
}

const oldWeek = loadData('nike_running_week_01.jsonl');
const newWeek = loadData('nike_running_week_02.jsonl');

newWeek.forEach((newItem, pid) => {
    const oldItem = oldWeek.get(pid);

    if (oldItem) {
        if (newItem.price < oldItem.price) {
            console.log(`🚨 PRICE DROP: ${newItem.name} is now $${newItem.price} (was $${oldItem.price})`);
        }
    } else {
        console.log(`✨ NEW ARRIVAL: ${newItem.name} added at $${newItem.price}`);
    }
});

oldWeek.forEach((oldItem, pid) => {
    if (!newWeek.has(pid)) {
        console.log(`❌ STOCK OUT: ${oldItem.name} is no longer listed.`);
    }
});

This logic transforms raw text files into actionable business intelligence. You can now see exactly which products your competitors are discounting and what new inventory they are betting on.

Operationalizing: Making it Repeatable

Running a script manually every Monday is still manual work. To turn this into a true monitor, you should automate the execution.

Using Cron (Linux/Mac)

You can schedule the scraper to run every Monday at 9:00 AM by adding a line to your crontab:
0 9 * * 1 /usr/local/bin/node /path/to/zappos_scraper.js

Storing Data

Create a dedicated folder for your snapshots. Over six months, these timestamped JSONL files become a powerful historical dataset. You can use them to identify seasonal pricing trends. For example, does Zappos always drop prices on specific brands in October?

To wrap up

Building a professional-grade price monitor doesn't require a massive engineering team. By using robust open-source scrapers and a "snapshot and diff" strategy, you can automate your competitor research.

Key Takeaways:

Category Scrapes > Product Scrapes: Monitoring whole categories provides context that single-product tracking lacks.
Data Integrity: Use unique identifiers like productId and pipelines to prevent duplicates from ruining your analysis.
JSONL is better: Use streamable formats for long-running scrapes to ensure data safety.
Automate the Audit: Use a simple comparison script to highlight price drops and stock changes automatically.

Stop Paying Twice: How to Cut Web Scraping Proxy Costs with Smart Caching

Jerry A. Henley — Fri, 20 Mar 2026 17:13:00 +0000

Imagine your growth team just launched a new pricing dashboard. It’s a hit, but there is a problem: every time a user refreshes the page, your backend triggers a fresh scrape of a dozen competitor sites. Within 48 hours, your residential proxy bill has doubled, even though the data on the dashboard hasn't changed once.

This is "Scraping Shock." In data extraction, redundancy isn't just inefficient; it’s expensive. Unlike traditional web development where caching primarily improves speed, web scraping optimization focuses on unit economics.

Treat every successful request to a high-quality proxy as a financial asset. This guide covers how to build an intelligent caching layer using Python and Redis to protect both your budget and your infrastructure.

The Proxy Paradox: Why Redundancy is Expensive

To bypass modern anti-bot systems, developers often rely on residential proxies. These provide IPs tied to real home devices, making them nearly impossible for websites to distinguish from legitimate traffic. However, these proxies come with a catch: the Proxy Paradox.

The more "human" a proxy behaves, the more it costs. While datacenter proxies are often sold at flat monthly rates, residential and mobile proxies are billed by bandwidth, often ranging from $10 to $20 per GB.

Consider the math:

Without Caching: 1,000 requests for a 1MB page = 1GB of bandwidth used.
With Caching (80% redundancy): 200 requests for that page = 200MB of bandwidth used.

If you scrape the same product page 50 times an hour to check for a price change that only happens once a day, you are effectively throwing money away. Implementing a cache turns a linear cost curve into a logarithmic one.

Designing the Strategy: Stale-While-Revalidate

A common mistake in scraping is using a "Hard TTL" (Time To Live). If you set a cache for 60 minutes, the 61st user must wait for a slow, synchronous scrape to finish before they see any data. In scraping, where requests can take 10–30 seconds due to proxy rotation and browser rendering, this creates a poor user experience.

The Stale-While-Revalidate (SWR) pattern is a better approach. This strategy categorizes data into three tiers:

Fresh: The data is recent. Serve it from the cache immediately.
Stale: The data is slightly old. Serve it from the cache immediately to keep the UI snappy, but trigger a background scrape to update the cache for the next user.
Expired: The data is too old to be useful. Block the request, perform the scrape, and return the new data.

This approach balances data freshness with performance and cost savings.

Technical Implementation: The Stack

To build this caching layer, we’ll use a reliable, simple stack:

Python: The primary logic engine.
Redis: An in-memory key-value store. Redis handles TTLs naturally and allows multiple scraping workers to share the same cache.
Redis-py: The standard Python client for Redis.

First, ensure Redis is running and the library is installed:

pip install redis

Step-by-Step: Building the Caching Decorator

We want the caching logic to be reusable. The cleanest way to do this in Python is with a decorator, which allows us to add caching to any scraping function with one line of code.

1. Creating a Deterministic Cache Key

Before storing data, we need a unique fingerprint for the request. If we scrape the same URL but with different headers or parameters, the cache key must reflect those differences.

import hashlib
import json

def generate_cache_key(func_name, url, params=None):
    """Creates a unique MD5 hash for a specific request."""
    key_data = {
        "func": func_name,
        "url": url,
        "params": params or {}
    }
    # Sort keys to ensure the hash is identical regardless of dict order
    key_string = json.dumps(key_data, sort_keys=True)
    return f"scrape:{hashlib.md5(key_string.encode()).hexdigest()}"

2. The Smart Caching Decorator

This decorator handles basic caching and serves as the foundation for SWR logic.

import functools
import redis
import time

# Connect to your local Redis instance
cache = redis.Redis(host='localhost', port=6379, db=0)

def smart_cache(ttl_seconds=3600):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(url, *args, **kwargs):
            cache_key = generate_cache_key(func.__name__, url, kwargs.get('params'))

            # Try to get data from Redis
            cached_val = cache.get(cache_key)
            if cached_val:
                print(f"Cache Hit for {url}")
                return json.loads(cached_val)

            # Cache Miss: Execute the expensive scraping function
            print(f"Cache Miss for {url}. Fetching fresh data...")
            result = func(url, *args, **kwargs)

            # Store in Redis with an expiration
            if result:
                cache.setex(cache_key, ttl_seconds, json.dumps(result))

            return result
        return wrapper
    return decorator

Handling Edge Cases: Errors and Compression

Production-grade scraping requires more than just saving strings. We need to address two major issues: cache poisoning and memory bloat.

Don't Cache Errors

If your scraper hits a 403 Forbidden or a Captcha, do not cache that result. Otherwise, you'll serve an error page to users for the duration of the TTL. Always validate the response before saving.

Use Compression

Scraped HTML is often repetitive and bulky. Storing raw HTML for 100,000 pages in Redis will quickly exhaust your RAM. Using zlib can often compress HTML by 80–90%.

Here is an advanced version of the decorator:

import zlib

def production_cache(ttl_seconds=3600):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(url, *args, **kwargs):
            cache_key = f"z:{generate_cache_key(func.__name__, url)}"

            compressed_data = cache.get(cache_key)
            if compressed_data:
                # Decompress and return
                return json.loads(zlib.decompress(compressed_data))

            result = func(url, *args, **kwargs)

            # Validate: Don't cache empty results or common error patterns
            if result and "error" not in result:
                # Compress before storing
                json_data = json.dumps(result).encode('utf-8')
                compressed = zlib.compress(json_data)
                cache.setex(cache_key, ttl_seconds, compressed)

            return result
        return wrapper
    return decorator

The Economics: ROI Calculation

Suppose you use a residential proxy provider charging $15 per GB. Here is how the numbers look:

Metric	No Caching	With Smart Caching (60% Hit Rate)
Daily Requests	50,000	50,000
Proxy Requests	50,000	20,000
Bandwidth (Avg 500KB/req)	25 GB	10 GB
Daily Cost	$375	$150
Monthly Savings	$0	$6,750

By implementing a one-hour cache, you make the app faster and save over $6,000 a month. This "Freshness Tax"—the cost of wanting data exactly now versus recently—is the most important metric for scraping teams to track.

To Wrap Up

Smart caching is the most effective way to optimize a web scraping pipeline. It reduces reliance on expensive residential proxies, protects target sites from unnecessary load, and provides a better experience for end users.

Key Takeaways:

Focus on Unit Economics: In scraping, bandwidth is money. Every cache hit is a direct cost saving.
Use SWR Patterns: Don't make users wait for a proxy rotation if you have "good enough" data available.
Validate Before Storing: Never cache 403s, 404s, or Captcha pages.
Compress Your Data: Use zlib to keep your Redis memory footprint small.

To take this further, consider using a background task queue like Celery to handle the revalidation portion of the SWR strategy. This allows the scraper to update the cache in the background without making the user wait.

Stop Silent Failures: Using Python Dataclasses to Catch Stale CSS Selectors

Jerry A. Henley — Mon, 09 Mar 2026 17:30:00 +0000

The nightmare of every web scraping developer is the silent failure. Your scraper runs perfectly, the logs show a 200 OK status for every request, and your database is growing. But when you check the data, you realize that for the last three days, every product price has been recorded as None or $0.00.

What happened? The target website changed a single <div> class, and your CSS selector stopped working. Because your code was designed to handle missing elements gracefully, it didn't crash; it just started collecting empty data.

We can solve this by transforming data containers from passive storage bins into active validation layers. This guide explores how to use Python dataclasses to detect DOM changes in real-time, using the CrateandbarrelScrapers repository as a practical case study.

Prerequisites

To follow along, you should have:

Intermediate knowledge of Python.
Experience with web scraping libraries like Selenium or Playwright.
A solid understanding of CSS selectors.

The Anatomy of a Passive Dataclass

Let's look at a common implementation found in the repository's Selenium product scraper: python/selenium/product_data/scraper/crateandbarrel_scraper_product_data_v1.py.

The scraper defines a ScrapedData class to hold the extracted information:

@dataclass
class ScrapedData:
    aggregateRating: Dict[str, Any] = field(default_factory=dict)
    availability: str = "out_of_stock"
    brand: str = "Crate & Barrel"
    name: str = ""
    price: Optional[float] = None
    productId: str = ""
    url: str = ""
    # ... other fields

This is a passive dataclass. It’s excellent for organization, but it has a major flaw: it permits silent failures. If the extract_data function fails to find the product name or price due to a DOM change, the dataclass is instantiated with default values like "" or None.

The pipeline then saves this empty object to your JSONL file without raising an alarm. This assumes the DOM is static, which is a dangerous assumption in production scraping.

Strategy 1: Implementing Mandatory Field Validation

The first step to robust scraping is defining which fields are critical. For a site like Crate & Barrel, a product without a productId or a name is useless.

The __post_init__ method in Python dataclasses runs validation logic immediately after the object is created. If a critical field is missing, we can raise a custom exception.

class DataValidationError(Exception):
    """Raised when scraped data fails business logic validation."""
    pass

@dataclass
class ScrapedData:
    name: str = ""
    productId: str = ""
    price: Optional[float] = None
    availability: str = "out_of_stock"

    def __post_init__(self):
        self.validate()

    def validate(self):
        critical_fields = {
            "name": self.name,
            "productId": self.productId,
        }

        missing = [field for field, value in critical_fields.items() if not value]

        if missing:
            raise DataValidationError(f"Missing critical fields: {', '.join(missing)}")

With this added, the moment your CSS selector for h1.product-name fails, the scraper raises a DataValidationError. This transforms a silent data quality issue into a trackable error.

Strategy 2: Context-Aware Logic

DOM changes are often subtle. A site might change the price layout for "In Stock" items but keep it the same for "Clearance" items. We can use business logic to detect when a specific selector is stale.

At Crate & Barrel, if a product is listed as "In Stock," it must have a price greater than zero. If the price is missing but the product is available, your price selector is almost certainly broken.

    def validate(self):
        # ... previous validation ...

        # Context-aware check: Price vs Availability
        if self.availability == "in_stock":
            if self.price is None or self.price <= 0:
                 raise DataValidationError(
                     f"Product {self.productId} is 'in_stock' but price is {self.price}. "
                     "Selector likely broken."
                 )

This logic catches the scenario where the scraper successfully finds the page and the availability but fails to extract the specific price value.

Strategy 3: The "Error Budget" Pattern

While we want to catch errors, a single malformed product page shouldn't crash a 10,000-item scrape. In production, use an Error Budget.

We can modify the DataPipeline class from the Crate & Barrel repo to track the failure rate. If only 1% of products fail validation, it might be a data edge case. If 15% fail, the website has likely updated its layout.

class DataPipeline:
    def __init__(self, jsonl_filename="output.jsonl", failure_threshold=0.05):
        self.jsonl_filename = jsonl_filename
        self.total_attempted = 0
        self.failure_count = 0
        self.threshold = failure_threshold

    def add_data(self, scraped_data: ScrapedData):
        self.total_attempted += 1
        try:
            scraped_data.validate()
            # ... save to file logic ...
        except DataValidationError as e:
            self.failure_count += 1
            logger.error(f"Validation failed: {e}")
            self.check_error_budget()

    def check_error_budget(self):
        if self.total_attempted > 10:  # Wait for a small sample size
            failure_rate = self.failure_count / self.total_attempted
            if failure_rate > self.threshold:
                # Trigger a critical alert
                raise SystemExit(f"CRITICAL: Failure rate {failure_rate:.2%} exceeds budget! DOM change detected.")

Implementation Guide: Integrating into the Scraper

To implement this in the existing Crate & Barrel Selenium scraper, wrap the extraction call to handle the new validation logic.

Before (Passive)

# In the original loop
data = extract_data(driver, url)
pipeline.add_data(data) # Data added even if empty

After (Active Validation)

# Modified loop in the scraper
try:
    data = extract_data(driver, url)
    if data:
        # validate() is called inside add_data or via __post_init__
        pipeline.add_data(data) 
except DataValidationError as e:
    logger.warning(f"Skipping {url} due to DOM change: {e}")
except Exception as e:
    logger.error(f"Hard failure on {url}: {e}")

This change ensures that your output.jsonl file contains only validated data, while your logs provide an early warning system for site updates.

Performance and Trade-offs

Using dataclasses for validation is lightweight and built into the Python standard library. However, as projects grow, consider these trade-offs:

Complexity: For complex nested structures, manual validation in __post_init__ can become verbose.
Maintenance: You must update validation logic whenever business requirements change, such as if the site starts listing "Price on Request" items.

For larger projects, Pydantic is a strong alternative. It offers a more powerful version of this pattern with automatic type coercion and advanced validation, though it does add a third-party dependency.

To Wrap Up

Detecting DOM changes should happen during extraction, not after the data is already in your database. By moving validation into the dataclass layer, you catch issues immediately.

Key Takeaways:

Transform Passive Containers: Use __post_init__ to turn dataclasses into active validators.
Enforce Critical Fields: Identify fields that must exist, such as ID or Name, and raise explicit errors if they are missing.
Apply Business Logic: Use relationships between data, like Price vs. Availability, to detect stale selectors.
Monitor the Failure Rate: Use an "Error Budget" to distinguish between a single bad page and a site-wide layout change.

To see these patterns in a production context, explore the implementations in the Crateandbarrel Scrapers repository. If you prefer not to hunt for selectors manually, the ScrapeOps AI Scraper Generator can generate Python code for you based on a URL, following the same structure used in the Crate & Barrel examples.

Share of Shelf Analysis: How to Scrape Zappos Search Results

Jerry A. Henley — Thu, 05 Mar 2026 19:18:00 +0000

In e-commerce, price isn't the only factor that determines success. If a customer searches for "winter boots" on Zappos and your product appears on the third page, the sale is likely lost before the customer even sees your price tag. This visibility is known as Share of Shelf (SoS).

Measuring your digital Share of Shelf manually is impossible at scale. Zappos is a highly protected site that frequently updates its layout and employs sophisticated bot detection. To gain a competitive edge, you need to automate the collection of search rankings.

This guide covers how to build a Python script using Selenium and undetected-chromedriver to track product positions for specific keywords. By the end, you’ll be able to calculate exactly what percentage of the "digital shelf" your brand owns on Zappos.

Prerequisites

To follow along, you'll need:

Python 3.7+
A ScrapeOps API Key: Zappos uses aggressive anti-botting measures. ScrapeOps proxies rotate IPs to bypass these blocks. You can get a free API key here.
Chrome Browser (required for Selenium).

Environment Setup

We'll use undetected-chromedriver to make the browser instance look like a real user and selenium-wire to handle proxy authentication.

pip install selenium-wire undetected-chromedriver pandas

Mapping the Zappos Search Page

Before writing code, examine how Zappos structures its search results. Open Chrome DevTools (F12) on a search page to find these patterns:

Product Containers: Each product is wrapped in an <article> tag, typically using classes like yW-z or attributes like data-style-id.
Data Points: Inside these articles, you'll find the brand name, product name, and price.
Pagination: Zappos uses a URL parameter (usually p) for pagination. For example, &p=1 loads page two.
JSON-LD: Zappos often embeds JSON-LD (Linked Data) in <script type="application/ld+json"> tags. This is a goldmine for scrapers because structured data is much more stable than HTML classes.

Implementing the Search Scraper

The core scraper needs to use ScrapeOps proxies to ensure Zappos doesn't rate-limit your IP address after a few requests.

Proxy and Driver Configuration

We use selenium-wire to inject ScrapeOps credentials into the request headers.

import undetected_chromedriver as uc
from seleniumwire import webdriver
import os

API_KEY = "YOUR_SCRAPEOPS_API_KEY"

# ScrapeOps Residential Proxy Configuration
PROXY_CONFIG = {
    'proxy': {
        'http': f'http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181',
        'https': f'http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181',
        'no_proxy': 'localhost:127.0.0.1'
    }
}

def get_driver():
    options = uc.ChromeOptions()
    options.add_argument("--headless") # Run without a visible window

    driver = webdriver.Chrome(
        options=options,
        seleniumwire_options=PROXY_CONFIG
    )
    return driver

The Extraction Logic

Next, we need a function to loop through the search results. We'll focus on grabbing the brand and product name to identify which items belong to you versus your competitors.

from selenium.webdriver.common.by import By

def extract_search_results(driver, page_number):
    products = []
    # Locate all product articles
    elements = driver.find_elements(By.CSS_SELECTOR, "article.yW-z")

    for index, el in enumerate(elements):
        try:
            # Calculate organic rank: (Page - 1) * ItemsPerPage + CurrentIndex + 1
            rank = ((page_number - 1) * 100) + index + 1

            brand = el.find_element(By.CSS_SELECTOR, ".dC-z").text
            name = el.find_element(By.CSS_SELECTOR, ".eC-z").text
            price = el.find_element(By.CSS_SELECTOR, "._Y-z").text

            products.append({
                "rank": rank,
                "brand": brand.strip(),
                "product_name": name.strip(),
                "price": price.strip()
            })
        except Exception:
            continue

    return products

Calculating Share of Shelf

To calculate SoS, aggregate results across multiple pages and group them by brand. If you are monitoring "running shoes," you want to know what percentage of the top 50 or 100 results your brand occupies.

def run_sos_analysis(keyword, target_pages=2):
    driver = get_driver()
    search_url = f"https://www.zappos.com/search?term={keyword.replace(' ', '+')}"
    all_data = []

    for page in range(1, target_pages + 1):
        url = f"{search_url}&p={page-1}"
        driver.get(url)

        # Wait for elements to load
        driver.implicitly_wait(10)

        page_results = extract_search_results(driver, page)
        all_data.extend(page_results)
        print(f"Scraped {len(page_results)} items from page {page}")

    driver.quit()
    return all_data

# Example Usage
results = run_sos_analysis("running shoes", target_pages=1)

Analyzing the Data with Pandas

With the data collected, use Pandas to calculate the visibility metrics.

import pandas as pd

df = pd.DataFrame(results)

# Calculate Share of Shelf (Percentage of total visible slots)
sos = df['brand'].value_counts(normalize=True) * 100

print("--- Share of Shelf Analysis ---")
print(sos.head(10)) # Top 10 brands by visibility

Overcoming Anti-Bot Challenges

Zappos uses several layers of protection, including TLS fingerprinting and behavioral analysis. This script addresses these issues in three ways:

Undetected Chromedriver: Standard Selenium is easily detected by the navigator.webdriver property. undetected-chromedriver patches the binary to hide these signals.
Proxy Rotation: Using the ScrapeOps Residential Proxy network makes every request appear to come from a different home, preventing IP-based rate limiting.
Modern Headless Mode: Running in headless=new mode ensures the page renders exactly as a user would see it without the performance overhead of a GUI.

Exporting for Business Intelligence

While JSON is useful for development, e-commerce managers usually need Excel or CSV files.

# Save to CSV for reporting
output_file = "zappos_sos_report.csv"
df.to_csv(output_file, index=False)

print(f"Analysis complete. Data saved to {output_file}")

This CSV provides a ranked list of every product on the search page. Running this script daily allows you to visualize whether SEO efforts or ad spends are actually improving your shelf position.

To Wrap Up

Measuring Share of Shelf on Zappos provides a clear picture of your organic visibility compared to competitors. Automating this process moves your strategy away from guesswork and toward data-driven decision making.

Key Takeaways:

Rank is relative: Your position on page one is often more valuable than your price.
Use structured data: Check for JSON-LD scripts first, as they are more reliable than CSS selectors.
Rotate proxies: E-commerce sites block data center IPs quickly. Residential proxies are necessary for consistent data collection.

For more advanced implementations, including Node.js and Playwright versions, check out the Zappos Scraper Bank on GitHub. Your next step is to schedule this script to run automatically to track visibility fluctuations in real-time.

Hardening Python Scrapers: Managing Anti-Bot Triggers on Wayfair

Jerry A. Henley — Mon, 02 Mar 2026 04:30:00 +0000

Scraping e-commerce giants like Wayfair is a constant game of cat and mouse. As one of the largest home goods retailers in the world, Wayfair invests heavily in sophisticated anti-bot protections, including Cloudflare and PerimeterX, to shield product data and pricing. If you have tried scraping Wayfair with a standard Selenium setup, you have likely encountered a persistent "Verify you are human" CAPTCHA or a flat 403 Forbidden error.

Modern bot detection doesn't just look at what you do; it looks at who you are at the browser, network, and TLS levels. To extract data at scale, you need to harden your scrapers.

This guide breaks down a production-ready Python script from the Wayfair.com-scrapers repository that combines undetected_chromedriver with residential proxies to bypass these triggers and maintain high success rates.

The Anatomy of a Block: Why Standard Selenium Fails

Wayfair's defense mechanisms are multi-layered. When a standard Selenium instance connects to their servers, it leaves a trail of "leakage" that signals automation.

The WebDriver Flag: By default, Selenium sets the navigator.webdriver property to true. Anti-bot scripts check this immediately.
TLS Fingerprinting: Standard HTTP libraries and basic Selenium setups have a distinct Transport Layer Security (TLS) handshake. If your handshake doesn't match a real Chrome browser, you are flagged before the first byte of HTML even loads.
IP Reputation: Most scrapers run from data centers like AWS or GCP. Wayfair knows that real customers don't browse for couches from a Northern Virginia server farm, leading to instant IP blacklisting.

Simply rotating a User-Agent string is no longer enough. You must patch the browser binary and route traffic through high-reputation IPs.

Prerequisites

To follow along, ensure your environment is set up with the following:

Python 3.8+
Libraries: undetected-chromedriver, selenium-wire, and re.
ScrapeOps API Key: For residential proxy rotation. You can get a free API key here.

pip install undetected-chromedriver selenium-wire

1. Implementing Undetected ChromeDriver

The first step in hardening a scraper is hiding the automation signature. undetected_chromedriver (UC) is a specialized version of the Chrome driver that automatically patches the browser's executable to remove bot detection triggers.

Here is how the get_driver() function is implemented in the Wayfair product search scraper:

import undetected_chromedriver as uc
from seleniumwire import webdriver

def get_driver():
    options = uc.ChromeOptions()
    options.add_argument("--headless=new") # Use the 'new' headless mode
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-blink-features=AutomationControlled")
    options.add_argument("--window-size=1920,1080")

    # Initialize the driver with selenium-wire for proxy support
    driver = webdriver.Chrome(
        options=options,
        seleniumwire_options=PROXY_CONFIG
    )

    # Execute CDP command to manually overwrite the webdriver property
    driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
        "source": """
            Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
        """
    })

    return driver

Why this works

--disable-blink-features=AutomationControlled: This flag prevents the browser from sending the EnableAutomation flag to the rendering engine.
--headless=new: Older headless modes were easily detectable because they lacked certain browser features. The "new" mode is a full Chrome browser running without a UI, making it much harder to distinguish from a real user.
CDP Script Injection: Using the Chrome DevTools Protocol (CDP) to inject a script on every new document ensures that even if an anti-bot script runs early, it sees undefined instead of true for the navigator.webdriver property.

2. Proxy Integration and the Network Layer

Even with a perfect browser fingerprint, a data center IP will eventually trigger a block. Wayfair requires residential proxies. These are IP addresses assigned by Internet Service Providers (ISPs) to real homes, giving them the highest possible trust score.

Using selenium-wire instead of standard Selenium allows you to handle proxy authentication (username and password) directly in the configuration, which standard Selenium lacks.

API_KEY = "YOUR-SCRAPEOPS-API-KEY"

# ScrapeOps Residential Proxy Configuration
PROXY_CONFIG = {
    'proxy': {
        'http': f'http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181',
        'https': f'http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181',
        'no_proxy': 'localhost:127.0.0.1'
    }
}

By routing undetected_chromedriver through ScrapeOps, every request originates from a different residential IP, bypassing rate limits and IP-based blocks.

3. Managing Concurrency with Thread Locals

When scraping Wayfair at scale, you will want to run multiple browsers in parallel. However, WebDriver instances are not thread-safe. If two threads try to control the same browser, the script will crash.

The repository uses a pattern involving threading.local() to manage unique driver instances per thread:

import threading

# Thread-local storage for WebDriver instances
thread_local = threading.local()

def get_driver():
    """Get thread-local undetected ChromeDriver instance."""
    if not hasattr(thread_local, "driver"):
        # ... [Driver Initialization Code from above] ...
        thread_local.driver = webdriver.Chrome(
            options=options,
            seleniumwire_options=PROXY_CONFIG
        )
    return thread_local.driver

This pattern allows a ThreadPoolExecutor to scrape multiple search pages concurrently. Each thread checks its own local storage; if a driver doesn't exist, it creates one. This prevents session cross-contamination and ensures that a CAPTCHA block in one thread doesn't stop the entire process.

4. Handling Soft Blocks and Retries

Wayfair often uses "soft blocks." Instead of a 403 error, they might serve the page successfully with a 200 status but replace the content with a CAPTCHA. In Selenium, this usually manifests as a TimeoutException because the element you're looking for, such as the product grid, never appears.

Wrap your extraction logic in error handling:

from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def safe_extract(driver, url):
    try:
        driver.get(url)
        # Wait up to 10 seconds for the product grid to load
        WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, "[data-test-id='Browse-Grid']"))
        )
        # Proceed with extraction...
    except TimeoutException:
        logger.warning(f"Soft block or timeout on {url}. Rotating proxy and retrying...")
        # Close current driver and remove the thread-local reference
        if hasattr(thread_local, "driver"):
            thread_local.driver.quit()
            del thread_local.driver
        return None

If you hit a block, do not simply retry with the same browser session. Close the driver instance, delete the thread-local reference, and let the next attempt initialize a fresh browser with a new proxy session.

5. Cleaning the Data

Once you bypass the bots, the data can still be messy. Prices often include currency symbols, "Sale" tags, or "Was" prices. The repository includes a utility function to sanitize this data using Regular Expressions:

import re

def clean_price(price_str: str) -> float:
    """Extracts numeric float from strings like '$1,299.99 Sale'"""
    if not price_str:
        return 0.0
    # Match digits, commas, and dots
    match = re.search(r'[\d,]+\.?\d*', price_str)
    if not match:
        return 0.0
    try:
        return float(match.group().replace(",", ""))
    except ValueError:
        return 0.0

This ensures your final JSONL output is ready for analysis or database insertion without further post-processing.

To Wrap Up

Bypassing Wayfair's anti-bot triggers requires a multi-layered strategy. No single tool is a total solution. Success comes from combining browser hardening with a strong network reputation.

Hide Automation: Use undetected_chromedriver and CDP commands to mask the navigator.webdriver flag.
Residential IPs: Use proxies like ScrapeOps to ensure your network signature matches a real consumer.
Thread Safety: Use threading.local() to manage multiple browser instances for concurrent scraping safely.
Graceful Recovery: Treat timeouts as potential blocks. Rotate your session and IP immediately rather than continuing with a blocked connection.

For a complete, runnable version of this code, see the Wayfair Selenium Scraper in the Scraper Bank repository. It provides a solid foundation for building a reliable e-commerce data pipeline.

How to Track Competitor Pricing on Wayfair with Python (For Non-Coders)

Jerry A. Henley — Fri, 27 Feb 2026 03:30:00 +0000

Pricing visibility is the lifeblood of any modern e-commerce strategy. In a marketplace as dynamic as Wayfair, where discounts shift daily and "Flash Deals" can undercut your margins in hours, staying informed is a requirement for survival. Most growth teams face a frustrating bottleneck: manual price checking is agonizingly slow, and waiting for engineering resources to build a custom internal tool can take months.

You can bypass the developer queue by using pre-built open-source tools to extract this data yourself.

This guide demonstrates how to use a production-ready Python script to scrape Wayfair product data, including prices, ratings, and availability. You can then export this information into a clean format like Excel. You don't need to be a software engineer; you just need to know how to run a few commands and edit a single line of text.

Why Python Over Browser Extensions?

You might wonder why we aren't using a simple "No-Code" browser extension. Wayfair uses sophisticated anti-bot measures that easily block most browser plugins. To get reliable data at scale, you need a more resilient approach.

We will use the Wayfair.com-scrapers repository, specifically the Playwright version. Playwright is a tool that controls a real browser, allowing the script to interact with the page exactly like a human would. This is essential for handling Wayfair's dynamic content.

To handle the technical headaches of proxy rotation and bot detection, we’ll integrate ScrapeOps. This acts as a middleman that automatically optimizes your requests so Wayfair doesn't block your connection.

Phase 1: Environment Setup

Before running the scraper, you need to prepare your "workbench." This is the most technical part of the process, but you only have to do it once.

1. Install Python

Visit Python.org and download the latest version for your operating system.
Important: During installation, check the box that says "Add Python to PATH." This allows you to run Python from your terminal or command prompt.

2. Download the Scraper Code

Go to the Wayfair Scrapers Repository. Click the green "Code" button and select "Download ZIP." Extract this folder to your desktop or another easy-to-find location.

3. Get Your ScrapeOps API Key

Sign up for a free account at ScrapeOps. Once logged in, copy your API Key from the dashboard. The free credits provided are plenty for tracking a significant list of competitors.

Phase 2: Preparing the Scraper

Now that you have the files, you need to install the "libraries," which are the pre-written code packages that tell Python how to talk to a browser.

Open your Terminal (Mac/Linux) or Command Prompt (Windows).
Navigate to the downloaded folder. Use the cd command (Change Directory): cd Desktop/Wayfair.com-scrapers-main/python/playwright/product_data
Run the following command to install the necessary tools:

pip install playwright playwright-stealth
playwright install

The playwright-stealth plugin makes the automated browser look like a normal user, which reduces the chance of being flagged by Wayfair.

Phase 3: Targeting Your Competitors

Now you need to tell the script which products to monitor. Open the file named wayfair_scraper_product_data_v1.py in a text editor like Notepad, TextEdit, or VS Code.

1. Insert Your API Key

Look for the line near the top that says API_KEY = "YOUR-API_KEY". Replace the placeholder with the key you copied from ScrapeOps.

# Before
API_KEY = "YOUR-API_KEY"

# After
API_KEY = "5b32-your-real-key-here"

2. Define Your Product List

Scroll to the bottom of the file to the if __name__ == "__main__": section. Modify this to loop through a list of your competitor's URLs. Replace the single URL logic with a list like this:

if __name__ == "__main__":
    # The list of competitor products you want to track
    urls_to_track = [
        "https://www.wayfair.com/furniture/pdp/allmodern-george-77-upholstered-sofa-w004245645.html",
        "https://www.wayfair.com/furniture/pdp/mercury-row-perdue-815-velvet-square-arm-convertible-sofa-w001831846.html"
    ]

    async def run_tracker():
        async with async_playwright() as playwright:
            browser = await playwright.chromium.launch(headless=True)
            for url in urls_to_track:
                page = await browser.new_page()
                data = await extract_data(page, url)
                if data:
                    pipeline.add_data(data)
                await page.close()
            await browser.close()

    asyncio.run(run_tracker())

Pro Tip: When copying Wayfair URLs, remove everything after the .html (such as ?piid=...). This ensures you are targeting the clean product page.

Phase 4: Running the Script

Go back to your terminal. Ensure you are still in the product_data folder and run:

python wayfair_scraper_product_data_v1.py

Logs will begin appearing in your terminal:
INFO:root:Saved item to wayfair_com_product_page_scraper_data_20231027...jsonl

The script is now visiting each page, extracting the price, brand, and reviews, and saving them to a file in the same folder.

Phase 5: From JSONL to Excel

The script outputs data in a JSONL format. While this looks like a wall of text to a human, it's easy to import into Excel for analysis.

How to open the data in Excel:

Open a blank Excel workbook.
Go to the Data tab.
Click Get Data > From File > From JSON.
Select the .jsonl file created by the script.
Excel will open the Power Query Editor. Click "Into Table" and then use the small icon at the top of the "Column1" header to expand all fields (Price, Brand, Name, etc.).
Click Close & Load.

You now have a formatted spreadsheet containing live competitor pricing, star ratings, and review counts.

Troubleshooting & Best Practices

Anti-Bot Blocks: If the script returns empty data, check your ScrapeOps dashboard. If the error rate is high, you may need to enable "Residential Proxies" in the PROXY_CONFIG section of the script.
Frequency: Avoid running the script every few minutes. For e-commerce pricing, once every 24 hours is standard and less likely to trigger security blocks.
Currency: The script uses a detect_currency function. If you are scraping wayfair.co.uk or wayfair.ca, it will automatically identify GBP or CAD. Just ensure you aren't comparing different currencies directly in your final report.

To Wrap Up

By following this process, you've moved from manual data entry to automated market intelligence. You now have a repeatable way to monitor your market without writing original code from scratch.

To take this further, explore the product_search folder in the repository. You can use those scripts to track which products are ranking #1 for keywords like "mid-century sofa," giving you a complete view of your competitor's SEO and pricing strategy.

Production-Grade Costco Scraping: Error Budgets, Retries, and Pipelines

Jerry A. Henley — Wed, 25 Feb 2026 04:30:00 +0000

Building a basic web scraper for a handful of pages is a straightforward task. However, when you scale that script to scrape thousands of products from a major retailer like Costco, the complexity shifts from parsing HTML to keeping the process alive.

Costco employs aggressive anti-bot measures, strict rate limits, and a massive catalog that can easily overwhelm a naive script. If your scraper stores data in memory and crashes at product #999, you lose everything. If it gets blocked by a 403 Forbidden error, the entire run stops.

To build a reliable Costco scraper in Node.js, you need to focus on fault tolerance, stream writing, and proxy management. We can look at the architectural patterns found in the Costco.com-Scrapers repository to see how this works in practice.

Prerequisites

To follow along, you should have:

Node.js installed (v16+ recommended).
A basic understanding of Puppeteer or Playwright.
A ScrapeOps API Key for proxy rotation.

The Configuration Strategy: Error Budgets

Production scraping requires a shift in mindset. You must assume that network requests will fail. Instead of using hardcoded values, define a CONFIG object that establishes an "error budget."

The configuration pattern used in the repository's costco_scraper_product_data_v1.js looks like this:

const CONFIG = {
    maxRetries: 3,
    maxConcurrency: 1,
    timeout: 180000, // 3 minutes
    outputFile: generateOutputFilename()
};

Why these numbers matter:

Timeout Math: Standard 30-second timeouts are often too aggressive for Costco. When using residential proxies or waiting for heavy JavaScript to render, pages can take over a minute to stabilize. Setting the timeout to 3 minutes ensures you don't kill a request that was actually succeeding slowly.
Concurrency Control: While it's tempting to set concurrency to 10 or 20, Costco's anti-bot system is sensitive to request patterns. Starting with a single concurrent request prioritizes session stability over raw speed.
The Error Budget: The maxRetries setting defines the budget. If a page fails three times, the script logs the failure and moves on rather than hanging indefinitely.

Proxy Rotation with ScrapeOps

Costco quickly flags and bans datacenter IP addresses. To scrape at scale, you need residential proxies that rotate with every request. The repository integrates ScrapeOps to handle this automatically.

Configure the proxy server within the Puppeteer or Playwright setup:

const PROXY_SERVER = 'residential-proxy.scrapeops.io';
const PROXY_PORT = '8181';
const PROXY_USERNAME = 'scrapeops';
const PROXY_PASSWORD = API_KEY;

const browser = await puppeteer.launch({
    args: [`--proxy-server=${PROXY_SERVER}:${PROXY_PORT}`]
});

By routing requests through a residential proxy, the scraper appears to Costco as a legitimate home user. When combined with the puppeteer-extra-plugin-stealth, you can bypass common fingerprinting techniques that trigger CAPTCHAs or 403 errors.

The DataPipeline Class: Stream Writing

A common mistake in web scraping is storing all results in a large array and saving it to a file at the very end.

This is risky. If the script crashes after four hours of work, the variable is cleared and your data is gone. Furthermore, storing 50,000 product objects in memory can lead to memory exhaustion.

The repository solves this using a DataPipeline class that implements JSONL (JSON Lines) stream writing:

class DataPipeline {
    constructor(outputFile = CONFIG.outputFile) {
        this.itemsSeen = new Set();
        this.outputFile = outputFile;
        this.writeFile = promisify(fs.appendFile);
    }

    async addData(scrapedData) {
        if (!this.isDuplicate(scrapedData)) {
            try {
                // Convert object to string and append a newline
                const jsonLine = JSON.stringify(scrapedData) + '\n';
                // Write immediately to disk
                await this.writeFile(this.outputFile, jsonLine, 'utf8');
                console.log('Saved item to', this.outputFile);
            } catch (error) {
                console.error('Error saving data:', error);
            }
        }
    }
}

The Benefits of JSONL:

Persistence: Every time addData is called, the item is physically written to the disk. A crash only loses the current item being processed.
Memory Efficiency: You only hold one product object in memory at a time.
Database Friendly: Most modern data warehouses like BigQuery or Snowflake prefer JSONL for bulk imports.

Deduplication Logic

When running a scraper with retries, there is a risk of saving the same data twice if a network hiccup occurs after the data is fetched but before the request is marked complete.

The DataPipeline uses a Set called itemsSeen to prevent this. Before writing to the file, the isDuplicate method checks if the item is already recorded.

isDuplicate(data) {
    const itemKey = data.productId || JSON.stringify(data);
    if (this.itemsSeen.has(itemKey)) {
        console.warn('Duplicate item found, skipping');
        return true;
    }
    this.itemsSeen.add(itemKey);
    return false;
}

Using the Costco productId as a key ensures the final dataset is clean, saving hours of post-processing work.

Implementing Checkpointing: Resuming Scrapes

Production scrapers often need to resume after a fatal error, such as a power outage or system update. Currently, if you restart the script, the itemsSeen Set is empty, meaning the scraper might re-process thousands of items it already saved.

You can upgrade the DataPipeline to support checkpointing by reading the existing file on startup.

The Upgrade: Adding Checkpointing

Add this logic to the DataPipeline constructor to make the scraper resume-aware:

class DataPipeline {
    constructor(outputFile = CONFIG.outputFile) {
        this.outputFile = outputFile;
        this.itemsSeen = new Set();
        this.writeFile = promisify(fs.appendFile);

        if (fs.existsSync(this.outputFile)) {
            console.log(`Loading existing data from ${this.outputFile}...`);
            const fileContent = fs.readFileSync(this.outputFile, 'utf8');
            fileContent.split('\n').forEach(line => {
                if (line.trim()) {
                    try {
                        const item = JSON.parse(line);
                        this.itemsSeen.add(item.productId || JSON.stringify(item));
                    } catch (e) {
                        // Skip malformed lines
                    }
                }
            });
            console.log(`Resuming with ${this.itemsSeen.size} items already indexed.`);
        }
    }
}

With this modification, the scraper becomes much more resilient. You can stop and start the process at will, and it will only scrape what is missing from the output file.

To Wrap Up

Building a production-grade Costco scraper requires moving beyond simple DOM parsing and focusing on data integrity and network resilience.

Use Error Budgets: Set long timeouts and reasonable retries to handle network variance.
Rotate Residential Proxies: Use a service like ScrapeOps to bypass IP-based blocking.
Stream to JSONL: Write data to disk line-by-line to prevent data loss.
Implement Checkpointing: Populate your deduplication Set from existing files to allow the scraper to resume after a crash.

To see these patterns in action, you can clone the official repository and try implementing the checkpointing logic yourself.

From Scrape to Feed: Generating Google Shopping XML from Costco Product Data

Jerry A. Henley — Sat, 21 Feb 2026 05:43:05 +0000

In e-commerce arbitrage and market research, raw data is only the starting point. Simply having a list of products isn't enough to drive sales or marketing campaigns. If you are scraping Costco to build a price comparison engine or an automated inventory system, the raw JSONL data your scrapers produce is incompatible with platforms like Google Shopping.

Google Shopping requires a specific, strictly formatted XML feed. If a single price is missing a currency code or an availability status uses the wrong terminology, the Merchant Center will reject your entire feed.

This guide walks through the process of taking raw Costco product data and transforming it into a production-ready Google Shopping XML feed using Python.

Prerequisites

To follow along, you will need:

Python 3.8+ installed.
Basic familiarity with JSONL and XML structures.
A ScrapeOps API key for data extraction, which you can get here.

Step 1: The Data Source (Scraping Costco)

Before generating a feed, we need data. We’ll use the Costco.com-Scrapers repository, which provides reliable implementations for extracting Costco product information.

First, clone the repository:

git clone https://github.com/scraper-bank/Costco.com-Scrapers.git
cd Costco.com-Scrapers/python/selenium/product_data/scraper

The Selenium implementation is effective for Costco because it handles the site's dynamic content. When you run costco_scraper_product_data_v1.py, it produces a JSONL (JSON Lines) file. This format is ideal for scraping because it writes one product per line, preventing data loss if the script is interrupted.

The raw output for a single product looks like this:

{
  "productId": "1234567",
  "name": "Kirkland Signature Coffee, 2 lb",
  "price": 19.99,
  "currency": "USD",
  "availability": "in_stock",
  "images": [{"url": "https://bfasset.costco-static.com/.../1234567-847__1.jpg"}]
}

Step 2: Google Merchant Center Feed Requirements

Google Shopping feeds are RSS 2.0 files with a custom namespace (g:). To be accepted, your XML must map Costco's data to Google's specific attribute names.

Costco Field	Google XML Tag	Requirement
`productId`	`g:id`	Unique alphanumeric ID
`name`	`g:title`	Max 150 characters
`description`	`g:description`	Clean text (no HTML)
`price`	`g:price`	Must include currency (e.g., "19.99 USD")
`availability`	`g:availability`	Must be: `in stock`, `out of stock`, or `preorder`
`images[0]['url']`	`g:image_link`	Absolute URL to the main image

Step 3: Setting Up the Transformation Script

Python's built-in xml.etree.ElementTree is a lightweight way to generate XML and handle namespaces. We'll start by defining the skeleton of the RSS feed.

import json
import xml.etree.ElementTree as ET

def create_feed_skeleton():
    # Define the namespace for Google attributes
    ET.register_namespace('g', "http://base.google.com/ns/1.0")

    rss = ET.Element("rss", version="2.0")
    channel = ET.SubElement(rss, "channel")

    # Basic channel metadata
    ET.SubElement(channel, "title").text = "Costco Product Feed"
    ET.SubElement(channel, "link").text = "https://www.costco.com"
    ET.SubElement(channel, "description").text = "Daily automated product feed from Costco"

    return rss, channel

Step 4: Data Normalization and Mapping

The logic inside the scraper repository handles the extraction, but Google's requirements are strict. We need helper functions to ensure the data matches Google's expected formats.

For example, the Costco scraper might return in_stock, but Google requires in stock.

def format_availability(raw_status):
    """Maps scraper status to Google's valid values."""
    mapping = {
        "in_stock": "in stock",
        "out_of_stock": "out of stock",
        "preorder": "preorder"
    }
    return mapping.get(raw_status, "out of stock")

def format_price(amount, currency="USD"):
    """Google requires the currency code inside the price string."""
    if amount is None:
        return "0.00 USD"
    return f"{float(amount):.2f} {currency}"

Step 5: Generating the XML File

Now we combine these elements. We will stream the JSONL file line-by-line. This is a recommended approach for data engineering because it ensures that even with 50,000 products, the script won't exhaust your system's memory.

def generate_google_feed(input_jsonl, output_xml):
    rss, channel = create_feed_skeleton()
    g_ns = "{http://base.google.com/ns/1.0}"

    with open(input_jsonl, 'r', encoding='utf-8') as f:
        for line in f:
            product = json.loads(line)

            # Create the item element
            item = ET.SubElement(channel, "item")

            # Map standard fields
            ET.SubElement(item, f"{g_ns}id").text = str(product.get('productId'))
            ET.SubElement(item, "title").text = product.get('name')
            ET.SubElement(item, "description").text = product.get('description', 'No description available')
            ET.SubElement(item, "link").text = product.get('url')

            # Process formatted fields
            ET.SubElement(item, f"{g_ns}price").text = format_price(
                product.get('price'), 
                product.get('currency', 'USD')
            )
            ET.SubElement(item, f"{g_ns}availability").text = format_availability(
                product.get('availability')
            )

            # Handle images
            images = product.get('images', [])
            if images:
                ET.SubElement(item, f"{g_ns}image_link").text = images[0].get('url')

    # Write the XML file
    tree = ET.ElementTree(rss)
    tree.write(output_xml, encoding="utf-8", xml_declaration=True)
    print(f"Successfully generated {output_xml}")

if __name__ == "__main__":
    generate_google_feed('costco_data.jsonl', 'google_shopping_feed.xml')

Step 6: Validation and Automation

Once you have your google_shopping_feed.xml, validate it using the Feed Debugger in Google Merchant Center. Watch out for these common issues:

Missing Brand: Google often requires a g:brand tag. You can pull this from the brand field in the Costco scraper output.
GTIN/MPN: For many categories, Google requires a Global Trade Item Number.

To keep the feed current, you can automate the process with a shell script and a cron job:

#!/bin/bash
# 1. Run the scraper
python3 costco_scraper_product_data_v1.py
# 2. Run the converter
python3 convert_to_xml.py
# 3. Optional: Upload to S3
# aws s3 cp google_shopping_feed.xml s3://my-bucket/feeds/

Running this daily ensures your Google Shopping ads reflect current Costco pricing and stock levels.

To Wrap Up

Converting raw scraped data into a platform-ready format is a vital step in any data pipeline. By using open-source scrapers and a custom Python transformation script, you can turn a basic JSONL file into a structured marketing asset.

Key Takeaways:

Use JSONL during scraping to maintain data integrity.
Normalize Price and Availability to meet Google's specific requirements.
Use streaming (line-by-line) processing to handle large datasets.

For more information, check out our guide on Handling Anti-Bot Measures to keep your scrapers running smoothly as sites update their security.

Monitoring Share of Search: Automating IKEA Product Visibility Tracking

Jerry A. Henley — Fri, 20 Feb 2026 07:55:59 +0000

In e-commerce, being the "best" product doesn't matter if no one can find you. For brands selling on large marketplaces like IKEA, visibility is the primary currency. If your ergonomic office chair is buried on page five of the search results, your sales will reflect that invisibility, regardless of your price or build quality.

This concept is known as Share of Search or Digital Shelf Analytics. It involves measuring how often and how prominently your products appear for specific keywords compared to your competitors. While standard web scraping focuses on what a product is (price or description), rank tracking focuses on where a product is.

This guide walks through building a Python-based IKEA scraper specifically to track search rankings. You will learn how to monitor a product's position for target keywords and calculate visibility over time.

Prerequisites & Setup

You’ll need a basic understanding of Python and JSON data. We will use a few key libraries to handle requests and data organization.

Install the necessary dependencies:

pip install requests pandas

IKEA employs sophisticated anti-bot measures and dynamic content loading. A standard requests.get() call often results in a 403 Forbidden error or an empty page. We will use the ScrapeOps Proxy API to handle proxy rotation, browser headers, and JavaScript rendering.

If you don't have one, you can get a free ScrapeOps API key here.

The Strategy: Rank vs. Data

Most developers approach scraping as a data extraction task, such as fetching the price of a specific SKU. Tracking Share of Search requires a different logic. We aren't just looking for a product; we are looking for its Rank Position.

The challenge with IKEA is that results are paginated or use infinite scroll. A product at the top of Page 2 is effectively at Rank 25 if the site displays 24 items per page. To get an accurate Rank Position, we must:

Scrape search results in the order they appear to the user.
Maintain a global counter across multiple pages.
Identify target SKUs within that ordered list.

Step 1: Configuring the Target List

Before writing the scraper, we need to define our targets. We’ll create a configuration object that maps generic category keywords to the specific Product IDs (SKUs) we want to monitor.

# config.py

# Keywords to track
TARGET_KEYWORDS = ["standing desk", "velvet sofa", "gaming chair"]

# The SKUs (Product IDs) we are tracking (ours or competitors)
MY_PRODUCTS = {
    "standing desk": ["S49440193", "S19222267"],
    "velvet sofa": ["80430535"],
    "gaming chair": ["70483818"]
}

Step 2: The Search Scraper

IKEA’s search page is dynamic. To get the same results a human sees, the request must mimic a real browser. We will route requests through ScrapeOps to handle JavaScript rendering and bypass anti-bot protections.

The following function takes a keyword and a page number, then returns the HTML content.

import requests
from urllib.parse import urlencode

SCRAPEOPS_API_KEY = 'YOUR_API_KEY'

def get_ikea_search_page(keyword, page_number=1):
    base_url = f"https://www.ikea.com/us/en/search/?q={keyword}&p={page_number}"

    payload = {
        'api_key': SCRAPEOPS_API_KEY,
        'url': base_url,
        'render_js': 'true', # IKEA requires JS to load product grids
    }

    proxy_url = 'https://proxy.scrapeops.io/v1/?' + urlencode(payload)

    try:
        response = requests.get(proxy_url)
        if response.status_code == 200:
            return response.text
        return None
    except Exception as e:
        print(f"Error fetching page {page_number}: {e}")
        return None

Step 3: Parsing Ranks and Handling Pagination

Next, we parse the HTML, find product containers, and calculate their absolute rank. IKEA typically uses data-product-number attributes in their HTML tags, which makes identification straightforward.

We use a start_rank variable to ensure positions are calculated correctly across pages. If we are on Page 2 and there are 24 items per page, the first item on that page is Rank 25.

from bs4 import BeautifulSoup

def parse_search_results(html, current_page, items_per_page=24):
    soup = BeautifulSoup(html, 'html.parser')
    products = soup.select('.plp-fragment-wrapper') 

    results = []
    start_rank = (current_page - 1) * items_per_page

    for index, item in enumerate(products):
        product_id = item.get('data-product-number') or ""
        product_name = item.select_one('.plp-price-module__product-name')

        if product_id:
            results.append({
                'product_id': product_id.strip(),
                'name': product_name.text.strip() if product_name else "Unknown",
                'rank': start_rank + index + 1
            })

    return results

Step 4: Structuring Data for Analysis

To make this data useful, we need to store it for time-series analysis. We'll combine the previous steps into a loop that checks target SKUs against the scraped ranks and saves the results to a CSV.

import pandas as pd
from datetime import datetime

all_rankings = []
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M")

for keyword in TARGET_KEYWORDS:
    print(f"Tracking rank for: {keyword}")

    # Check the first 2 pages (top 48 results)
    for page in range(1, 3):
        html = get_ikea_search_page(keyword, page)
        if not html:
            continue

        page_results = parse_search_results(html, page)

        for item in page_results:
            if item['product_id'] in MY_PRODUCTS.get(keyword, []):
                all_rankings.append({
                    'date': timestamp,
                    'keyword': keyword,
                    'product_id': item['product_id'],
                    'product_name': item['name'],
                    'rank_position': item['rank']
                })

# Save to CSV
df = pd.DataFrame(all_rankings)
df.to_csv('ikea_rankings.csv', mode='a', index=False, header=not False)

Step 5: Analyzing Volatility

After collecting data for a few days, you can analyze Rank Volatility. If a product consistently appears in the top 10 for "standing desk," you have a high share of search. A sudden drop to position 30 might indicate a competitor has optimized their listing or IKEA has updated its algorithm.

Use pandas to calculate average positions and identify how often products hit the "Top 10."

data = pd.read_csv('ikea_rankings.csv')

# Calculate average rank per product
avg_ranks = data.groupby(['keyword', 'product_id'])['rank_position'].mean()
print(avg_ranks)

# Count Top 10 appearances
top_10_count = data[data['rank_position'] <= 10].shape[0]
print(f"Total Top 10 Placements: {top_10_count}")

Practical Tips

When building a rank tracker for IKEA, keep these factors in mind:

Regional Differences: Search results vary by zip code based on local stock. If you are tracking a specific market, ensure your requests use the correct localized IKEA URLs or parameters.
Sponsored Results: IKEA mixes organic results with "Sponsored" or "New" labels. Most analysts count these as they occupy physical space on the digital shelf, but you can filter them out if you only care about organic performance.
Frequency: Search rankings rarely fluctuate by the hour. Running this script once every 24 hours is usually enough to identify trends without consuming unnecessary API credits.

To wrap up, monitoring Share of Search on IKEA.com-scraping reprository turns web scraping into a competitive intelligence tool. By focusing on rank position rather than just product details, you get a clear view of your brand's actual performance on the digital shelf.

From Script to Spreadsheet: Building a Self-Serve Etsy Competitor Tracker

Jerry A. Henley — Tue, 17 Feb 2026 06:05:07 +0000

Data is only valuable if people can actually use it. For many developers, a web scraping project ends once they have a clean JSON file or a database entry. However, for marketing and growth teams, these formats often create a bottleneck. These teams live in spreadsheets. Every time they need updated pricing or product details, they have to ask an engineer for a fresh export.

This creates a cycle of manual exports and imports that wastes everyone's time. This guide breaks that cycle. We will build a self-serve Etsy competitor tracker that bridges the gap between Python and Google Sheets.

By the end of this tutorial, you’ll have a pipeline where a non-technical user can paste Etsy URLs into a Google Sheet, run a script, and watch as the spreadsheet populates with product titles, prices, and shop details using Python, Playwright, and the Google Sheets API.

Prerequisites

To follow along, you'll need:

Python 3.8+ installed on your machine.
A Google Cloud Account (the free tier works fine).
Basic familiarity with Python functions and CSS selectors.
The following libraries: playwright, gspread, and oauth2client.

Phase 1: Setup Google Cloud and API Credentials

Before writing any Python, we need to give the script permission to communicate with Google Sheets. This is the most common point of failure, so follow these steps carefully.

1. Create a Google Cloud Project

Go to the Google Cloud Console and create a new project named "Etsy-Tracker."

2. Enable APIs

Search for and enable these two APIs:

Google Sheets API
Google Drive API

3. Create a Service Account

Navigate to APIs & Services > Credentials. Click Create Credentials and select Service Account. Name it "sheets-scraper" and click Create and Continue. You can skip the optional role assignments.

4. Generate the JSON Key

Once the account is created, click on it, go to the Keys tab, and select Add Key > Create New Key. Choose JSON. This downloads a file to your computer. Rename this file to credentials.json and move it to your project folder.

5. Share the Sheet

Open the Google Sheet you want to use. Open your credentials.json file and find the client_email field. It will look like sheets-scraper@your-project.iam.gserviceaccount.com. Share your Google Sheet with this email address, giving it Editor access. Without this step, your script will return a "404 Sheet Not Found" error.

Phase 2: Building the Etsy Scraper with Playwright

Playwright is ideal for this project because Etsy uses various anti-bot measures and dynamic content. Since Playwright runs a real browser, the script is harder to distinguish from a real shopper.

First, install the dependencies and the browser binaries:

pip install playwright gspread oauth2client
playwright install chromium

Create a file named scraper.py. This function takes a URL and returns a dictionary of clean data.

from playwright.sync_api import sync_playwright
import re

def scrape_etsy_product(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
        )
        page = context.new_page()

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=60000)

            # Etsy uses specific utility classes for product data
            title = page.locator("h1.wt-text-body-01").inner_text(timeout=5000).strip()

            # Clean the price string (e.g., "$25.00+" becomes 25.00)
            raw_price = page.locator(".wt-p-xs-1 .wt-text-title-03").first.inner_text().strip()
            price = re.sub(r'[^\d.]', '', raw_price) 

            shop_name = page.locator("a.wt-text-link-no-underline span").first.inner_text().strip()

            return {
                "Title": title,
                "Price": float(price) if price else 0.0,
                "Shop": shop_name,
                "Status": "Success"
            }
        except Exception as e:
            return {"Title": "N/A", "Price": 0, "Shop": "N/A", "Status": f"Error: {str(e)}"}
        finally:
            browser.close()

Etsy’s DOM structure is generally stable, but they use utility classes like wt-text-body-01. Using .strip() and regex ensures the price becomes a clean float, making it easier to analyze in a spreadsheet.

Phase 3: Connecting Python to Google Sheets

We’ll use gspread to handle the spreadsheet communication. The script reads URLs from the first column and writes results starting from the second column.

Create a file named sheets_config.py:

import gspread
from oauth2client.service_account import ServiceAccountCredentials

def get_sheet(sheet_name):
    scope = ["https://spreadsheets.google.com/feeds", "https://www.googleapis.com/auth/drive"]
    creds = ServiceAccountCredentials.from_json_keyfile_name("credentials.json", scope)
    client = gspread.authorize(creds)

    return client.open(sheet_name).sheet1

This setup uses your credentials.json file. The scope defines the permissions, allowing the script to read and write files in your Google Drive.

Phase 4: The Integration Logic

The real power of this tool comes from the integration logic. Instead of just dumping data, the script:

Checks the spreadsheet for URLs.
Skips URLs that have already been scraped.
Scrapes only the missing data.
Updates the sheet using a batch update.

Batch updates are essential. If you update the sheet cell-by-cell in a loop, you’ll hit Google's API rate limits, which are usually 60 requests per minute.

Here is the complete main.py:

from scraper import scrape_etsy_product
from sheets_config import get_sheet
import time

SHEET_NAME = "Etsy Competitor Tracker"

def main():
    sheet = get_sheet(SHEET_NAME)

    # Column A: URLs, B: Title, C: Price, D: Shop, E: Status
    all_rows = sheet.get_all_values()

    if not all_rows:
        print("Sheet is empty.")
        return

    data_rows = all_rows[1:] # Skip header
    updates = []

    for index, row in enumerate(data_rows, start=2): # Sheets is 1-indexed
        url = row[0]

        # Only scrape if the URL exists and Title (Col B) is empty
        if url and (len(row) < 2 or not row[1]):
            print(f"Scraping: {url}")
            result = scrape_etsy_product(url)

            row_update = [result['Title'], result['Price'], result['Shop'], result['Status']]
            range_label = f"B{index}:E{index}"
            updates.append({'range': range_label, 'values': [row_update]})

            # Brief pause to respect Etsy's rate limits
            time.sleep(2)

    if updates:
        print(f"Updating {len(updates)} rows in Google Sheets...")
        sheet.batch_update(updates)
        print("Done!")
    else:
        print("No new URLs to scrape.")

if __name__ == "__main__":
    main()

How it works

Optimization: By checking if the "Title" column is filled, the script avoids re-scraping the same data. Users can add new URLs to the bottom of the list and only the new entries will be processed.
Error Handling: If a product is sold out or a URL is dead, the script catches the error and writes it to the "Status" column, providing immediate feedback to the user.

Phase 5: Making it Accessible

To make this a true self-serve tool, the team needs an easy way to run it.

Local Execution

Create a simple batch file (Windows) or shell script (Mac/Linux) on the desktop. A user can double-click it to run the scraper and update the sheet.

Windows (run_tracker.bat):

@echo off
python main.py
pause

GitHub Actions

For a more professional setup, host the code on GitHub and use GitHub Actions. You can set the script to run on a schedule (like every morning at 8 AM) or via a manual trigger. This way, users don't need Python installed on their computers at all.

To Wrap Up

This tool transforms a simple scraping script into a functional internal product. You've empowered your team to gather their own data without needing to understand HTML selectors or API authentication.

Key Takeaways:

Service Accounts bridge Python and Google Cloud; always share the sheet with the service account email.
Playwright provides the reliability needed for high-traffic e-commerce sites.
Batch Updates are required for performance and to stay within API limits.
Deduplication logic prevents redundant work and saves resources.

Try adding historical price tracking by creating a new tab for each day, or use smtplib to send email alerts when a competitor's price drops.

To speed up development and generate reliable scraping logic with AI, you can also use the AI Web Scraping Assistant.

How to Generate a Google Shopping Feed from Any E-commerce Site Using Python

Jerry A. Henley — Sun, 15 Feb 2026 23:30:00 +0000

For many e-commerce businesses, Google Shopping drives the majority of their revenue. However, getting product data into Google Merchant Center isn't always straightforward. If you are working with a legacy CMS, a custom-built platform, or a client who lacks direct database access, you likely won't have a native "Export to Google Shopping" plugin at your disposal.

When traditional tools fail, Python can bridge the gap. By scraping a website's front-end and transforming that data into a Google-compliant XML format, you can build a reliable product feed from scratch.

This guide walks through building a custom pipeline that crawls an e-commerce site, cleans the product data, and generates a valid feed.xml file ready for high-performance ad campaigns.

Understanding Google’s Feed Requirements

Before writing code, you need to understand the target format. Google Merchant Center is strict. If a currency is formatted incorrectly or an availability status uses the wrong keyword, Google will reject the entire feed.

Google Shopping feeds are XML-based and rely on a specific namespace (g:). While there are dozens of optional attributes, every product must include these mandatory fields:

Attribute	Description	Requirement Example
`id`	A unique, persistent identifier	`sku_123` or a stable hash
`title`	The name of your product	Max 150 characters
`description`	Detailed product info	No marketing jargon
`link`	The product landing page	Must be a verified domain
`image_link`	URL of the main image	High res, no watermarks
`availability`	Current stock status	`in_stock`, `out_of_stock`, `preorder`
`price`	Numerical price + currency	`1299.00 USD`

For a full list of attributes, refer to the official Google Product Data specification.

Prerequisites

To follow this tutorial, you will need:

Python 3.7+ installed
Familiarity with requests and BeautifulSoup
The feedgen library for XML generation

Install the necessary packages via pip:

pip install requests beautifulsoup4 feedgen

Step 1: Acquiring the Raw Data

You need a way to gather product information. In a production scenario, you might use a spider to crawl the entire site, but for this example, we’ll create a function that extracts data from a single product page.

If you need a real-world example of structured e-commerce extraction, the Flipkart.com Scrapers Repository demonstrates scalable patterns that can be adapted into this feed generation pipeline.

The following code intentionally grabs "dirty" data, such as raw strings with currency symbols and extra whitespace, to simulate the reality of web scraping.

import requests
from bs4 import BeautifulSoup

def scrape_product_page(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Extracting raw data points
    product_data = {
        'url': url,
        'title': soup.find('h1').text.strip(),
        'description': soup.find('div', class_='description').text.strip(),
        'price_raw': soup.find('span', class_='price').text.strip(), # "$1,299.00"
        'stock_raw': soup.find('div', class_='stock-status').text.strip(), # "In Stock"
        'image_url': soup.find('img', id='main-image')['src']
    }

    return product_data

Step 2: Data Transformation and Cleaning

Google will reject a price listed as "$1,299.00". It expects a numeric value followed by the ISO 4217 currency code, such as 1299.00 USD.

The product id must also be stable. If an ID changes, Google treats it as a new product, which resets the performance history and quality scores your ads have accumulated. If the site doesn't expose a SKU, you can create a deterministic ID by hashing the product URL.

import re
import hashlib

def clean_data(raw_item):
    # 1. Normalize Price: Remove symbols/commas and add currency
    price_numeric = re.sub(r'[^\d.]', '', raw_item['price_raw'])
    formatted_price = f"{float(price_numeric):.2f} USD"

    # 2. Map Availability: Convert "In Stock" or "3 Left" to "in_stock"
    stock_text = raw_item['stock_raw'].lower()
    availability = "in_stock" if "in stock" in stock_text or "left" in stock_text else "out_of_stock"

    # 3. Generate a Stable ID: Hash the URL for consistency
    product_id = hashlib.md5(raw_item['url'].encode()).hexdigest()[:10]

    return {
        'id': product_id,
        'title': raw_item['title'][:150], 
        'description': raw_item['description'],
        'link': raw_item['url'],
        'image_link': raw_item['image_url'],
        'availability': availability,
        'price': formatted_price
    }

Step 3: Generating the XML Feed

The feedgen library is more efficient than Python’s built-in xml.etree for this task because it handles RSS/Atom standards and custom namespaces smoothly.

Google Shopping feeds require the http://base.google.com/ns/1.0 namespace. We use the feedgen Google Merchant extension to append the g: tags correctly.

from feedgen.feed import FeedGenerator

def generate_xml_feed(cleaned_products, output_file="google_feed.xml"):
    fg = FeedGenerator()

    fg.title('My Custom E-commerce Store')
    fg.link(href='https://example-shop.com', rel='alternate')
    fg.description('Daily updated product feed for Google Shopping')

    # Load the Google Merchant extension
    fg.load_extension('google_re_base', atom=True)

    for item in cleaned_products:
        fe = fg.add_entry()
        fe.id(item['id'])
        fe.title(item['title'])
        fe.description(item['description'])
        fe.link(href=item['link'])

        # Add Google-specific attributes
        fe.google_re_base.image_link(item['image_link'])
        fe.google_re_base.availability(item['availability'])
        fe.google_re_base.price(item['price'])

    fg.rss_file(output_file, encoding='UTF-8')
    print(f"Success! Feed generated at {output_file}")

Step 4: Validation and Upload

Once the google_feed.xml is ready, use the Google Merchant Center "Test" feature. This allows you to upload a file and see errors without affecting live ads.

Host the file: Google needs a static URL to fetch the feed. You can host this on an AWS S3 bucket, a VPS, or a private GitHub Pages site.
Add Feed in Merchant Center: Navigate to Feeds > Primary Feeds > Add New.
Select "Scheduled Fetch": Enter the URL where you hosted your XML file.
Check for Errors: Look for encoding issues or missing mandatory attributes. A common pitfall is unescaped HTML characters in the description.

Automating the Update Cycle

A product feed is a living document. If you scrape the site once and never update it, your ads will eventually show incorrect prices. If Google detects a price mismatch between your feed and your landing page, they may disapprove the product or suspend your account.

Automate the script to run at least once every 24 hours. On a Linux server, a Cron job is the most reliable method. To run the script every night at 2:00 AM, use crontab -e and add:

0 2 * * * /usr/bin/python3 /path/to/your/script.py

In Google Merchant Center, set the "Fetch Schedule" to 3:00 AM to ensure Google picks up the fresh data shortly after it is generated.

To Wrap Up

Building your own Google Shopping feed pipeline gives you total control over your data. Instead of being limited by a plugin, you can inject custom labels, optimize titles for SEO, and include products from platforms that were previously difficult to advertise.

By following this workflow, you can extract raw web data and transform it into a professional, compliant XML feed. To speed up development, you can explore the ScrapeOps AI Web Scraping Assistant to automatically generate scraper logic for your pipeline.

Future-Proofing Your Scrapers: Strategies for Resilient DOM Selection

Jerry A. Henley — Thu, 12 Feb 2026 10:14:02 +0000

Imagine waking up to a Slack notification: your primary data pipeline has crashed. You check the logs and find that a site you've been scraping for months suddenly changed its layout. A developer at the target company renamed a single div class from product-price to price-wrapper__value, and now your entire extraction logic is useless.

This "maintenance hell" is common for data engineers. Modern websites are moving targets. They are frequently updated with frameworks like React, Vue, or Tailwind CSS that often generate random, obfuscated class names. Furthermore, large e-commerce platforms constantly run A/B tests, meaning two users might see entirely different HTML structures for the same product.

To survive, we need to move away from rigid, fragile selectors and embrace resilient selection logic. We can build more reliable scrapers by shifting our focus from "where an element is" to "what an element represents."

The Anatomy of a Brittle Selector

The fastest way to break a scraper is to rely on the exact path of an element. Many beginners start by right-clicking an element in Chrome DevTools and selecting "Copy XPath." This usually results in an absolute path like this:

# A brittle, absolute XPath
price = tree.xpath('/html/body/div[2]/div[5]/section/div[2]/span[1]/text()')

This selector is fragile because it assumes the website's structure is static. If the site adds a promotional banner at the top of the page, div[2] might become div[3], and your scraper will return None.

Similarly, relying on auto-generated class names leads to failure. Modern frontend tools often produce hashes like css-1q2w3e. These hashes change every time the site is redeployed. If your selector targets .css-1q2w3e, your scraper might only work for a few days.

Strategy 1: Anchoring to Semantic Attributes

The most effective way to future-proof a scraper is to find attributes that developers are unlikely to change. These are often semantic attributes or metadata used for SEO and accessibility.

Prioritize Data Attributes and Microdata

Developers often use data-* attributes for testing or tracking, and these are much more stable than CSS classes. Additionally, many sites use Schema.org microdata to help search engines understand their content.

Instead of selecting by a class, look for itemprop or data-testid:

# Using CSS selectors to target stable attributes
# Instead of: response.css('.price-val-392')
price = response.css('[itemprop="price"]::text').get()
product_id = response.css('[data-testid="product-id"]::attr(value)').get()

Use ARIA Labels

Accessibility is a legal requirement for many companies. Developers rarely change aria-label or role attributes because doing so could break screen readers. This makes them excellent anchors for scraping.

# Finding a search button via its ARIA label
search_button = page.locator('button[aria-label="Submit search"]')

Strategy 2: Text-Based and Relative Selection

When HTML attributes are unreliable, turn to what the human user sees. Labels like "Price:", "SKU:", or "Availability" are highly stable because they are essential for the user experience.

We use relative selection to find a stable "anchor" (the label) and then navigate to the "target" (the data).

Use XPath Axes

XPath is powerful here because it allows us to move sideways or upwards in the DOM tree. For example, if we want to find the price next to the text "Our Price:", we can use the following-sibling axis.

# Find the span containing 'Our Price:', then get the next span sibling
xpath_selector = "//span[contains(text(), 'Our Price:')]/following-sibling::span[1]/text()"
price = tree.xpath(xpath_selector)

In Playwright, this is even more intuitive with modern locators:

# Playwright relative positioning
price = page.get_by_text("Our Price:").locator("..").locator("span.value")

Strategy 3: The Multi-Selector Fallback Chain

No single selector is 100% bulletproof. The gold standard for resilient scraping is graceful degradation. Instead of relying on one path, define a list of potential selectors and try them in order of trustworthiness.

This Python utility function implements a fallback chain:

import logging

def extract_with_fallback(response, selector_list):
    """
    Attempts to extract data using a list of selectors.
    Returns the first successful match.
    """
    for i, selector in enumerate(selector_list):
        data = response.css(selector).get()

        if data:
            if i > 0:
                logging.warning(f"Primary selector failed. Fallback {i} succeeded: {selector}")
            return data.strip()

    return None

# Usage
price_selectors = [
    '[itemprop="price"]',          # Strategy 1: Semantic (Best)
    '.product-page-price',         # Strategy 2: Descriptive Class (Good)
    '#main-content div span.bold'  # Strategy 3: Structural (Last resort)
]

price = extract_with_fallback(response, price_selectors)

With this pattern, your scraper can survive a site update that removes the itemprop attribute as long as the CSS class remains intact.

Real-World Example: Refactoring an Alibaba Scraper

Alibaba and AliExpress are known for heavy A/B testing and complex DOM structures. A selector that works on a "Flash Deal" page might fail on a "Standard" product page.

We can refactor a brittle scraper into a resilient one using a configuration-driven approach:

from parsel import Selector

def get_product_price(html_content):
    sel = Selector(text=html_content)

    # Define strategies for different layouts
    strategies = {
        "schema": 'span[itemprop="price"]::text',
        "meta": 'meta[property="og:price:amount"]::attr(content)',
        "ab_test_variant": '.price-format-mod__price::text',
        "text_anchor": '//div[contains(text(), "Price:")]/following-sibling::div//text()'
    }

    for name, query in strategies.items():
        if query.startswith('//'):
            result = sel.xpath(query).get()
        else:
            result = sel.css(query).get()

        if result:
            # Clean the data (remove currency symbols)
            return result.replace('$', '').strip()

    return None

This logic handles multiple layouts automatically. If Alibaba changes their price UI for an A/B test, the ab_test_variant or text_anchor will likely catch it.

Monitoring and "Canary" Checks

Resilience is not just about preventing crashes; it is about knowing when your logic is starting to decay. If your scraper consistently relies on its third fallback selector, it is a "canary in the coal mine" signaling a major site change.

Implement Sanity Checks

Always validate the type and range of your extracted data. If you are scraping a price and your fallback returns "In Stock," your selector logic has failed silently.

def validate_price(price_str):
    try:
        value = float(price_str)
        return 0 < value < 1000000 # Sanity range
    except (ValueError, TypeError):
        return False

price = get_product_price(html)
if not validate_price(price):
    logging.error("Data Validation Failed: Price is non-numeric or out of range.")
    # Trigger an alert or save the HTML for manual review

By logging a warning whenever a fallback is used, you turn emergency midnight fixes into scheduled maintenance. You will know the site has changed long before the scraper actually fails.

To Wrap Up

Building resilient scrapers requires more effort upfront, but it pays dividends by reducing maintenance time. By moving away from rigid absolute paths and using a multi-layered selection strategy, you can create bots that navigate the shifting sands of the modern web.

Key Takeaways:

Avoid Absolute Paths: Never use selectors generated by "Copy XPath" in your browser.
Anchor to Semantics: Use data-*, itemprop, and aria-label attributes, which are less likely to change than CSS classes.
Use Fallback Chains: Implement a list of selectors ranging from precise to fuzzy to handle A/B tests and layout updates.
Monitor Decay: Log when primary selectors fail and validate your data to catch silent failures early.

For your next project, try implementing a fallback_extract utility. You will spend less time fixing broken code and more time analyzing the data.