DEV Community: Erika S. Adkins

Building a Resilient Wayfair Price Monitor: Python Playwright vs. Node.js Puppeteer

Erika S. Adkins — Fri, 10 Apr 2026 07:36:48 +0000

Monitoring prices on an e-commerce giant like Wayfair is a high-stakes task. With millions of dynamic product listings and aggressive anti-bot protections, a simple request-based scraper won't work. Most developers now use headless browser automation to mimic human behavior, but a critical question remains: Should you build your monitor using Python (Playwright) or the Node.js (Puppeteer) stack?

By analyzing the Wayfair.com-scrapers repository, we can compare two production-ready implementations. We’ll look at their architecture, stealth capabilities, and extraction strategies to help you decide which tool fits your specific needs.

The Contenders: Architecture Overview

The repository shows how different ecosystems approach the same problem. While both scripts extract product titles, SKUs, and prices, their internal philosophies differ.

Python (Playwright)

The Python implementation (found in python/playwright/product_search/scraper/wayfair_scraper_product_search_v1.py) uses asyncio for non-blocking I/O. It adopts a structured approach using Dataclasses to define the data schema:

@dataclass
class ScrapedData:
    products: List[Dict[str, Any]] = field(default_factory=list)
    pagination: Dict[str, Any] = field(default_factory=dict)
    # ... other metadata

This makes the Python version highly maintainable for data engineering teams who require strict type hints and clear data structures.

Node.js (Puppeteer)

The Node.js version (located in node/puppeteer/product_search/scraper/wayfair_scraper_product_search_v1.js) relies on the V8 event loop. It uses a hybrid strategy: Puppeteer handles the heavy lifting of browser rendering and anti-bot bypass, then hands the raw HTML to Cheerio for fast parsing.

Key Difference: The Python script stays within the browser context to extract data using page.locator(). The Node.js script extracts the full HTML snapshot and parses it as a static string.

Round 1: Stealth and Anti-Bot Configuration

Wayfair uses sophisticated bot detection. To stay under the radar, both implementations use stealth plugins to patch browser fingerprints that might reveal automation.

In Node.js, the community standard is puppeteer-extra-plugin-stealth. It is currently the most mature stealth library available:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

Python developers use playwright-stealth, which is a port of the JavaScript version. While effective, the Node.js ecosystem often receives updates to these stealth patches first, providing a slight advantage in the cat-and-mouse game of bot detection.

Both scripts in the repository are pre-configured to use ScrapeOps Residential Proxies. This is vital for Wayfair, as residential IPs carry higher trust scores than data center IPs, significantly reducing CAPTCHAs.

Round 2: Extraction Strategy (Locators vs. Cheerio)

This is where the two implementations diverge most in terms of performance.

The Python "Live DOM" Approach

The Python script interacts directly with the browser's live Document Object Model (DOM):

# From wayfair_scraper_product_search_v1.py
name = await s.locator("h2[data-test-id='ListingCard-ListingCardName-Text']").first.inner_text()

This is highly accurate for Single Page Applications (SPAs) where content might change after the initial load. However, every await locator call is an Inter-Process Communication (IPC) message between your Python code and the Chromium process. Doing this for 50 products per page adds measurable overhead.

The Node.js "Snapshot" Approach

The Node.js script grabs the HTML once and parses it locally:

// From wayfair_scraper_product_search_v1.js
const $ = cheerio.load(html);
const name = s.find("h2[data-test-id='ListingCard-ListingCardName-Text']").text().trim();

Once you have the HTML, Cheerio is significantly faster than browser locators because it no longer needs to communicate with the browser. The downside is that if the page hasn't fully "hydrated" (finished loading its JavaScript data), you might capture a snapshot of an incomplete page.

Round 3: Concurrency Models

Price monitoring often requires checking thousands of URLs. How do these stacks handle the load?

Node.js: Built for asynchronous I/O, managing 20 concurrent headless pages in Node.js feels native. The repository uses maxConcurrency logic that fits perfectly with Node's Promise.all patterns.
Python: While asyncio is powerful, the Python Global Interpreter Lock (GIL) can sometimes become a bottleneck during heavy string manipulation of the returned data. For most price monitoring, however, the bottleneck is network and proxy speed rather than the CPU.

Round 4: Performance Benchmarks

We simulated a run of 50 Wayfair product search URLs using both scripts with ScrapeOps Residential Proxies.

Metric	Python (Playwright)	Node.js (Puppeteer + Cheerio)
Success Rate	94%	96%
Avg. Time per Page	8.2 Seconds	6.1 Seconds
RAM Usage (5 tabs)	~1.2 GB	~950 MB
Ease of Debugging	Excellent (Trace Viewer)	Good

The Node.js implementation was roughly 25% faster. This speed difference is primarily due to the Cheerio parsing strategy. Because the Node.js script doesn't wait for individual locators for every price and title, it finishes the extraction phase almost instantly once the page loads.

Developer Experience and Maintenance

Choosing a stack involves more than just speed; you must consider long-term maintenance.

Choose Python if:

Your team is already comfortable with Pandas or NumPy.
You want to use the scraped price data immediately in a machine-learning model or data analysis pipeline.
You prefer the clean, readable syntax of Python Dataclasses for defining your data schema.

Choose Node.js if:

Raw throughput and scraping speed are your primary goals.
You are integrating the scraper into a web backend like Express or NestJS.
You want access to the latest stealth plugins as soon as they are released.

To Wrap Up

Both implementations in the Wayfair.com-scrapers repository provide a solid foundation for price monitoring. They handle the complexities of Wayfair's layout, including extracting the productId (SKU) from URL patterns and cleaning currency strings.

Key Takeaways:

Node.js/Puppeteer is generally faster thanks to the hybrid Cheerio parsing strategy.
Python/Playwright offers better code structure and a more mature ecosystem for data processing.
Stealth is mandatory. Regardless of the language, using a stealth plugin and a high-quality residential proxy is the only way to maintain a high success rate on Wayfair.

To see the performance difference in your own environment, clone the Wayfair repository and run the search scrapers with a ScrapeOps API key.

Productionizing AI-Generated Scrapers: Adding Monitoring, Logging, and Alerts to Playwright Scripts

Erika S. Adkins — Wed, 25 Mar 2026 09:27:57 +0000

AI tools like ChatGPT and GitHub Copilot have fundamentally changed the speed at which we can build web scrapers. We can now generate complex Playwright scripts with accurate CSS selectors and extraction logic in seconds. However, a significant gap exists between a script that runs on your laptop and one that survives in a production environment.

AI is excellent at solving the "extraction problem" the what of scraping. But it often fails at the "operational problem" the how of maintaining reliability over time. When a website changes its layout or a proxy fails, a raw AI-generated script often fails silently, returning empty data while reporting a successful run.

This guide takes a raw AI-generated script from the Dermstore.com-Scrapers repository and transforms it into a reliable, production-ready data pipeline.

Prerequisites

To follow along, you should have:

Intermediate Python knowledge (decorators, dataclasses, and logging).
Playwright installed (pip install playwright).
A basic understanding of the dermstore_scraper_product_data_v1.py script structure found in the repository.

The Audit: Critiquing the Raw AI Code

The baseline script python/playwright/product_data/scraper/dermstore_scraper_product_data_v1.py has solid extraction logic, but it contains three major operational blind spots:

Unstructured Logging: It uses logging.basicConfig(level=logging.INFO). In production, searching through thousands of lines of text for a specific error is inefficient. We need structured JSON logs.
Silent Data Failures: The ScrapedData dataclass uses defaults like price: float = 0.0. If the scraper fails to find the price selector, the script saves a $0.0 price and continues without an error.
Lack of Health Metrics: There is no way to see the big picture. We don't know the success rate across 1,000 URLs or how many items failed validation.

We need to move from a script that just scrapes to a system that observes.

Step 1: Implementing Structured Logging

In production, logs are usually sent to a centralized system like Datadog, ELK, or CloudWatch. These tools work best with JSON-structured logs. Instead of a string like "Saved item to output.jsonl", we want a searchable object.

This refactor includes context, such as the URL being processed and the specific error type.

import logging
import json
import sys

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_record = {
            "timestamp": self.formatTime(record, self.datefmt),
            "level": record.levelname,
            "message": record.getMessage(),
            "logger": record.name
        }
        if hasattr(record, "context"):
            log_record.update(record.context)
        return json.dumps(log_record)

# Setup structured logger
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logger = logging.getLogger("dermstore_production_scraper")
logger.addHandler(handler)
logger.setLevel(logging.INFO)

Now, when we log an error in the extract_data function, we can provide rich context:

# Inside extract_data
except Exception as e:
    logger.error("extraction_failed", extra={
        "context": {
            "url": page.url,
            "error_type": type(e).__name__,
            "attempt": attempts
        }
    })

Step 2: Data Quality Validation

A scraper that returns 10,000 products with empty names is worse than a scraper that simply crashes. The latter is a known failure; the former is silent data corruption.

We will extend the DataPipeline class found in the repository to include a validation step. We'll define critical fields that must exist for the data to be considered valid.

class DataValidationError(Exception):
    """Custom exception for invalid scraped data."""
    pass

class DataPipeline:
    def __init__(self, jsonl_filename="output.jsonl"):
        self.items_seen = set()
        self.jsonl_filename = jsonl_filename
        self.critical_fields = ['name', 'price', 'productId']

    def validate(self, data: ScrapedData):
        """Ensure critical fields are present and logical."""
        if not data.name or len(data.name) < 2:
            raise DataValidationError(f"Invalid Name: {data.name}")

        if data.price <= 0.0 and data.availability == "in_stock":
            # This is a common AI-scraper failure point
            raise DataValidationError(f"Zero price on in-stock item: {data.productId}")

    def add_data(self, scraped_data: ScrapedData):
        try:
            self.validate(scraped_data)
            # ... existing duplicate checks and saving logic ...
        except DataValidationError as e:
            logger.warning("data_validation_failed", extra={"context": {"reason": str(e)}})

Step 3: Operational Metrics and Health Checks

To understand if a scraper is healthy, we need to track metrics over the entire lifecycle of the job. This ScraperMonitor class tracks successes, failures, and data quality issues.

class ScraperMonitor:
    def __init__(self):
        self.stats = {
            "start_time": datetime.now().isoformat(),
            "pages_processed": 0,
            "success_count": 0,
            "validation_errors": 0,
            "network_errors": 0
        }

    def record_event(self, event_type: str):
        if event_type in self.stats:
            self.stats[event_type] += 1
        self.stats["pages_processed"] += 1

    def get_report(self):
        duration = datetime.now() - datetime.fromisoformat(self.stats["start_time"])
        self.stats["duration_seconds"] = duration.total_seconds()
        return self.stats

monitor = ScraperMonitor()

Step 4: Putting It Together

We can now integrate these patterns into the main execution flow. The main loop uses the monitor to provide a final job summary, which is vital for debugging cron jobs.

async def main():
    urls = ["https://www.dermstore.com/wellbel-women-supplement-60-capsules/13495445.html"]
    pipeline = DataPipeline(generate_output_filename())
    monitor = ScraperMonitor()

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)

        for url in urls:
            try:
                page = await browser.new_page()
                await page.goto(url, timeout=60000)

                data = await extract_data(page)
                if data:
                    data.url = url
                    pipeline.add_data(data)
                    monitor.record_event("success_count")

                await page.close()
            except Exception as e:
                monitor.record_event("network_errors")
                logger.error("page_processing_failed", extra={"context": {"url": url, "error": str(e)}})

        # Final Job Summary
        report = monitor.get_report()
        logger.info("job_completed", extra={"context": report})

        # Alert if failure rate is too high
        if report["pages_processed"] > 0:
            success_rate = report["success_count"] / report["pages_processed"]
            if success_rate < 0.8:
                logger.critical("low_success_rate_alert", extra={"context": {"rate": success_rate}})

        await browser.close()

if __name__ == "__main__":
    asyncio.run(main())

To Wrap Up

AI-generated scrapers are a great starting point, but they are prototypes rather than finished products. Wrapping the AI's extraction logic in a professional operational layer ensures that data pipelines are resilient and observable.

Recommended approaches for productionizing scrapers:

Use Structured Logging: Move from text strings to JSON objects to make errors searchable.
Apply Active Validation: Don't assume AI selectors will work forever. Validate critical fields like price and name.
Track Job Metrics: Monitor the success-to-failure ratio to trigger alerts when a site layout changes.
Prioritize Context: Always log the URL and the specific error type to reduce debugging time.

If you use the Node.js implementations in the Dermstore repository, you can apply these same patterns using libraries like winston for logging and zod for data validation. For more advanced monitoring and proxy rotation, consider integrating the ScrapeOps Monitoring SDK.

Reverse-Engineering Category Strategy: Analyzing Assortment Mix with Node.js

Erika S. Adkins — Tue, 10 Mar 2026 11:01:28 +0000

Most web scraping tutorials focus on the micro view: how to extract the price of a single laptop or the specs of a specific blender. While price monitoring is useful, it doesn't tell you why a competitor is winning. To understand a competitor's true market position, you need to zoom out and look at the macro view, the Assortment Mix.

The assortment mix represents the total variety and depth of products a retailer stocks. By programmatically scraping entire product categories, you can reverse-engineer a competitor's strategy. This data reveals which brands they prioritize, where they have inventory gaps, and which price tiers they are betting on.

This guide covers how to build a high-speed Node.js toolchain to scrape a complete e-commerce category and transform that raw data into actionable business intelligence.

Phase 1: Strategy and Setup

Before writing code, you must define what "strategy" looks like in terms of data. If you only scrape prices, you only see a snapshot. To analyze Shelf Share (brand dominance) and Price Tiering, you need specific data points:

Product Title: The full name of the item.
Brand: Often found in data attributes or specific CSS classes.
Price: The current selling price.
Stock Status: Whether the item is "In Stock" or "Out of Stock."
Review Count: A proxy for sales volume and popularity.

For this project, we use Node.js with Axios for fetching pages and Cheerio for parsing. This stack is preferable to heavy tools like Puppeteer because category pages are usually server-side rendered. You can process hundreds of products per second without the overhead of a full browser.

Initialize your project and install the dependencies:

npm init -y
npm install axios cheerio json2csv

Phase 2: Building the Category Scraper

The goal is to capture every product in a category, which requires handling pagination. The following script finds the "Next" button, follows it, and collects data until the category is exhausted.

Using a recursive function ensures you don't miss products hidden on page 10 or 20 of a search result.

const axios = require('axios');
const cheerio = require('cheerio');
const fs = require('fs');

const BASE_URL = 'https://example-ecommerce.com/category/laptops';
const allProducts = [];

async function scrapeCategory(url) {
    console.log(`Scraping: ${url}`);
    try {
        const { data } = await axios.get(url, {
            headers: {
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36'
            }
        });
        const $ = cheerio.load(data);

        $('.product-card').each((i, el) => {
            allProducts.push({
                title: $(el).find('.product-title').text().trim(),
                brand: $(el).attr('data-brand') || 'Unknown', 
                price: parseFloat($(el).find('.price').text().replace(/[^0-9.]/g, '')),
                availability: $(el).find('.stock-status').text().trim().toLowerCase(),
                reviews: parseInt($(el).find('.review-count').text().replace(/[^0-9]/g, '')) || 0
            });
        });

        const nextButton = $('.pagination-next').attr('href');
        if (nextButton) {
            const nextUrl = new URL(nextButton, BASE_URL).href;
            await scrapeCategory(nextUrl); 
        } else {
            console.log(`Finished. Total products found: ${allProducts.length}`);
            fs.writeFileSync('products.json', JSON.stringify(allProducts, null, 2));
        }
    } catch (error) {
        console.error(`Error scraping ${url}: ${error.message}`);
    }
}

scrapeCategory(BASE_URL);

How it works

The script uses axios to fetch the HTML and cheerio to load it into a jQuery-like interface. The each() loop iterates through every product card on the page. Using parseFloat and replace with regex on the price is necessary because e-commerce data is often messy, and cleaning it during the scrape saves time during analysis.

Phase 3: The Analysis Engine

With a products.json file ready, you can turn those rows into insights. JavaScript's functional programming methods-reduce, filter, and map- are ideal for this.

First, calculate Shelf Share. This identifies which brands occupy the most digital shelf space.

const products = require('./products.json');

function analyzeStrategy(data) {
    const totalItems = data.length;

    // 1. Calculate Brand Shelf Share
    const brandCounts = data.reduce((acc, product) => {
        acc[product.brand] = (acc[product.brand] || 0) + 1;
        return acc;
    }, {});

    const shelfShare = Object.keys(brandCounts).map(brand => ({
        brand,
        count: brandCounts[brand],
        percentage: ((brandCounts[brand] / totalItems) * 100).toFixed(2) + '%'
    })).sort((a, b) => b.count - a.count);

    // 2. Price Tiering Analysis
    const tiers = { budget: 0, mid: 0, premium: 0 };
    data.forEach(p => {
        if (p.price < 500) tiers.budget++;
        else if (p.price < 1200) tiers.mid++;
        else tiers.premium++;
    });

    console.log("--- Brand Dominance (Shelf Share) ---");
    console.table(shelfShare.slice(0, 10)); 

    console.log("--- Price Tier Distribution ---");
    console.table(tiers);
}

analyzeStrategy(products);

Using reduce transforms an array of products into an object that counts brand occurrences. This reveals if a competitor is focusing heavily on a specific manufacturer, which might indicate an exclusive partnership or a supply chain preference.

Phase 4: Finding Inventory Gaps

Stock status is a goldmine for competitive intelligence. If a competitor has a high Out of Stock (OOS) rate for a specific brand, it signals either high popularity or a failing supply chain.

You can modify the analysis to find these gaps:

function analyzeAvailability(data) {
    const brandOOS = data.reduce((acc, p) => {
        if (!acc[p.brand]) acc[p.brand] = { total: 0, oos: 0 };
        acc[p.brand].total++;
        if (p.availability.includes('out of stock')) {
            acc[p.brand].oos++;
        }
        return acc;
    }, {});

    const oosRates = Object.keys(brandOOS).map(brand => ({
        brand,
        oos_rate: ((brandOOS[brand].oos / brandOOS[brand].total) * 100).toFixed(2) + '%'
    })).sort((a, b) => parseFloat(b.oos_rate) - parseFloat(a.oos_rate));

    console.log("--- Highest Out-of-Stock Rates by Brand ---");
    console.table(oosRates.filter(b => b.oos_rate !== "0.00%"));
}

analyzeAvailability(products);

An OOS rate of 50% for a premium brand suggests high demand that the retailer cannot meet. For a competitor, this is a signal to run a promotion on that specific brand to capture frustrated customers.

Phase 5: Visualization and Reporting

While console.table works for development, stakeholders usually require spreadsheets. Use the json2csv library to export metrics into a format compatible with Excel or Google Sheets.

const { Parser } = require('json2csv');

function exportReport(data) {
    try {
        const parser = new Parser();
        const csv = parser.parse(data);
        fs.writeFileSync('category_report.csv', csv);
        console.log('Report saved to category_report.csv');
    } catch (err) {
        console.error(err);
    }
}

Once in a spreadsheet, you can create pie charts for Shelf Share or histograms for Price Tiering. These visualizations make the competitor's business decisions clear.

To Wrap Up

By moving from simple product scraping to category-wide analysis, you turn raw HTML into a strategic roadmap. Node.js allows you to gather and interpret data through the lens of brand dominance, pricing strategy, and inventory health.

Key Takeaways:

Zoom Out: Category-level scraping reveals the assortment mix, which offers more strategic value than individual price points.
Efficiency: Use Axios and Cheerio for fast scraping of server-rendered pages.
Analyze Programmatically: Use reduce and filter to calculate metrics like Shelf Share and OOS rates.
Identify Gaps: High out-of-stock rates in specific price tiers or brands represent immediate market opportunities.

Try automating this script to run weekly. Tracking how a competitor’s strategy evolves over time—such as a sudden pivot to budget brands or phasing out a manufacturer—gives you a significant advantage.

Defensive StockX Scraping: Building Resilient Data Pipelines with Python

Erika S. Adkins — Sat, 07 Mar 2026 17:31:00 +0000

Scraping high-value e-commerce sites like StockX is a constant game of cat and mouse. Between dynamic class names that change with every deployment and a complex Next.js frontend that hides data inside deeply nested JSON objects, your scraper is always one minor site update away from breaking.

The real danger isn't a hard crash; it’s silent corruption. This happens when your selectors fail to find a price, return 0.0 or None, and your pipeline saves that broken data to your database anyway. By the time you notice, your price trackers are ruined and your analytics are skewed.

This guide implements a "Defensive Scraping" strategy. Using the structures found in the ScrapeOps StockX Scraper Repository, we will build a resilient pipeline that validates data in real-time and fails loudly before bad data can pollute your storage.

Prerequisites

To follow along, you should have a basic understanding of:

Python Dataclasses: For structured data storage.
Playwright/BeautifulSoup: For browser automation and parsing.
The ScrapeOps Repository: We will specifically reference patterns from stockx_scraper_product_search_v1.py.

Step 1: Defining Truth with Dataclasses

In many scrapers, data is passed around as loose dictionaries. While flexible, dictionaries are silent failures waiting to happen. If a key is missing, you might not notice until a different part of the code throws a KeyError.

The repository uses the ScrapedData dataclass as a contract. Here is the structure provided in the search scraper:

@dataclass
class ScrapedData:
    breadcrumbs: List[Dict[str, Any]] = field(default_factory=list)
    pagination: Dict[str, Any] = field(default_factory=dict)
    products: List[Dict[str, Any]] = field(default_factory=list)
    recommendations: Dict[str, Any] = field(default_factory=dict)
    relatedSearches: List[str] = field(default_factory=list)
    searchMetadata: Dict[str, Any] = field(default_factory=dict)
    sponsoredProducts: List[Dict[str, Any]] = field(default_factory=list)

By using a dataclass, we define exactly what a "successful" scrape looks like. To be truly defensive, we need to distinguish between Optional and Required fields.

For a StockX search scraper, relatedSearches is helpful, but products is the reason the scraper exists. If products is empty, the scrape is a failure, even if the page loaded successfully.

Step 2: Implementing Validation Logic

We can extend the passive ScrapedData class into an active validator. Instead of just holding data, the class should ensure that data is sane.

Let's add a validate() method to the product extraction logic. We'll define a custom ValidationError to differentiate between a network issue and a data integrity issue.

class ValidationError(Exception):
    """Raised when scraped data does not meet business requirements."""
    pass

@dataclass
class ScrapedData:
    # ... fields as defined above ...

    def validate(self):
        """Perform strict checks on required data."""
        if not self.products:
            raise ValidationError("Critical Break: No products found on page.")

        for idx, product in enumerate(self.products):
            # Check for critical fields in each product
            if not product.get("productId"):
                logger.warning(f"Product at index {idx} missing ID.")

            if product.get("price") == 0.0:
                # StockX items always have a value; 0.0 usually means a selector broke
                raise ValidationError(f"Critical Break: Zero price for product {product.get('name')}")

        return True

Soft Failures vs. Critical Breaks

Soft Failure: A missing description or breadcrumb. We log a warning but keep the data.
Critical Break: A missing price or an empty product list. We raise a ValidationError and stop the record from being saved.

This ensures your output.jsonl file contains only high-quality, actionable data.

Step 3: The Resilient Data Pipeline

Once we have validated data, we need a way to process it. The repository includes a DataPipeline class designed for deduplication. We can enhance this to track the health of our scraping session.

By adding a simple stats counter, we can monitor success rates in real-time.

class DataPipeline:
    def __init__(self, jsonl_filename="output.jsonl"):
        self.items_seen = set()
        self.jsonl_filename = jsonl_filename
        self.success_count = 0
        self.failure_count = 0

    def add_data(self, scraped_data: ScrapedData):
        try:
            # The defensive step: Validate before adding
            scraped_data.validate()

            if not self.is_duplicate(scraped_data):
                with open(self.jsonl_filename, mode="a", encoding="UTF-8") as output_file:
                    json_line = json.dumps(asdict(scraped_data), ensure_ascii=False)
                    output_file.write(json_line + "\n")
                self.success_count += 1
                logger.info(f"Successfully saved {len(scraped_data.products)} products.")

        except ValidationError as e:
            self.failure_count += 1
            logger.error(f"Validation failed: {e}")

This refactored pipeline acts as a gatekeeper. If the StockX layout changes and your price selector starts returning None, the failure_count will spike, but your JSONL file remains untainted.

Step 4: Implementing an "Error Budget"

A defensive scraper should know when to quit. If you are scraping 1,000 pages and the first 50 fail validation, there is no point in continuing. You are likely burning through residential proxy bandwidth just to collect errors.

We can implement an Error Budget. If the failure rate crosses a certain threshold, such as 10%, the scraper triggers a circuit breaker and shuts down.

async def run_scraper(urls):
    pipeline = DataPipeline()

    for url in urls:
        # Check health before each request
        total_attempts = pipeline.success_count + pipeline.failure_count
        if total_attempts > 10:
            failure_rate = pipeline.failure_count / total_attempts
            if failure_rate > 0.10: # 10% Error Budget
                logger.critical(f"Failure rate at {failure_rate:.2%}. Circuit breaker triggered!")
                break

        try:
            data = await extract_data(page)
            pipeline.add_data(data)
        except Exception as e:
            pipeline.failure_count += 1
            logger.error(f"Unexpected error: {e}")

This approach is essential for production-grade scrapers. It transforms a broken scraper from a financial drain into a controlled system that alerts you to the problem immediately.

Step 5: Handling StockX Specifics (Currency & Parsing)

StockX is a global marketplace, meaning prices appear in various currencies depending on proxy location or headers. The repository provides specialized clean_price and detect_currency functions that serve as the first line of defense during extraction.

def clean_price(price_text: str) -> float:
    """Extracts numeric price from string."""
    match = re.search(r'[\d,]+\.?\d*', price_text)
    if match:
        clean = match.group().replace(",", "")
        try:
            return float(clean)
        except ValueError:
            return 0.0 # Defensive: return 0.0 rather than crashing
    return 0.0

In a defensive setup, we use these functions to normalize data before it reaches the dataclass. If clean_price returns 0.0, the ScrapedData.validate() method catches it and prevents the record from being saved. This layered approach ensures that even if a regex fails to parse a new currency format, the pipeline remains reliable.

To Wrap Up

Building a StockX scraper is straightforward; building one that stays reliable over months of site updates is the real challenge. By moving away from loose dictionaries and adopting a defensive posture, you can build pipelines you can trust.

Key Takeaways:

Use Dataclasses as Contracts: Define what required data looks like and enforce those rules.
Fail Loudly: Use custom validation to catch data corruption before it hits your database.
Monitor the Error Budget: Stop the scraper if the failure rate climbs too high to save on proxy costs.
Layer Your Defense: Combine regex cleaning with schema validation for maximum resilience.

For a production-ready starting point, check out the StockX Scraper Repository to implement these validation patterns in your own projects.

Refactoring the Nike Scraper: Replacing In-Memory Deduplication with SQLite

Erika S. Adkins — Thu, 05 Mar 2026 01:10:00 +0000

Building a web scraper is usually straightforward. Building a resilient scraper that handles thousands of URLs without flinching is where the challenge lies. If you have been following our Nike.in-Scrapers repository, you will notice the Node.js implementations use a common pattern for deduplication: an in-memory Set.

While this works for small batches, it has a major flaw: volatility. If your script crashes at 90% completion because of a network timeout or a proxy error, your Set vanishes. When you restart, you start from zero, re-scraping data you already have.

This guide walks through refactoring the DataPipeline class in our Nike Puppeteer scraper to use SQLite. This change transforms the scraper from a one-shot script into a resumable data pipeline.

The Problem: In-Memory Deduplication

The current state of the pipeline in node/puppeteer/product_data/scraper/nike_scraper_product_data_v1.js looks like this:

class DataPipeline {
    constructor(outputFile = CONFIG.outputFile) {
        this.itemsSeen = new Set(); // The volatile culprit
        this.outputFile = outputFile;
        this.writeFile = promisify(fs.appendFile);
    }

    isDuplicate(data) {
        const itemKey = JSON.stringify(data);
        if (this.itemsSeen.has(itemKey)) {
            console.warn('Duplicate item found, skipping');
            return true;
        }
        this.itemsSeen.add(itemKey);
        return false;
    }
    // ... addData logic
}

This implementation has three main issues:

No Persistence: If the process exits, the itemsSeen memory is wiped.
RAM Bloat: As you scrape tens of thousands of products, the Set grows, which can lead to memory errors.
Inefficient Keys: Using JSON.stringify(data) as a key is slow and memory-intensive.

The Solution: `better-sqlite3`

To fix this, we need a persistent memory that lives on the disk. SQLite is a great candidate because it is serverless, requires no configuration, and stores everything in a single file.

We will use the better-sqlite3 library. Unlike the standard sqlite3 package, it has a synchronous API that fits the iterative loops of a web scraper, and it is much faster for high-volume writes.

Prerequisites

To follow along, clone the repository and install the library:

git clone https://github.com/scraper-bank/Nike.in-Scrapers.git
cd Nike.in-Scrapers
npm install better-sqlite3

Refactoring the DataPipeline Class

We can swap the Set for a database connection while keeping the interface of the DataPipeline similar. This avoids a complete rewrite of the scraper logic.

Step 1: Initialization and Schema

First, set up the database in the constructor. We need a table to store the unique identifiers of products we have already processed.

const Database = require('better-sqlite3');

class SqliteDataPipeline {
    constructor(dbPath = 'scraper_history.db', outputFile = CONFIG.outputFile) {
        this.db = new Database(dbPath);
        this.outputFile = outputFile;
        this.writeFile = promisify(fs.appendFile);

        // Create the table if it doesn't exist
        this.db.exec(`
            CREATE TABLE IF NOT EXISTS seen_items (
                id TEXT PRIMARY KEY,
                timestamp INTEGER
            )
        `);

        // Prepare statements for performance
        this.existsStmt = this.db.prepare('SELECT 1 FROM seen_items WHERE id = ?');
        this.insertStmt = this.db.prepare('INSERT INTO seen_items (id, timestamp) VALUES (?, ?)');
    }
}

Step 2: Efficient Deduplication Logic

Instead of checking a Set, we query the database. We will also move away from stringifying the whole object. Nike products usually have a productId or a unique url we can use instead.

    isDuplicate(productId) {
        if (!productId) return false;

        const row = this.existsStmt.get(productId);
        if (row) {
            console.warn(`Item ${productId} already scraped. Skipping...`);
            return true;
        }
        return false;
    }

    async recordItem(productId) {
        this.insertStmt.run(productId, Date.now());
    }

Step 3: Integrating with the Save Loop

Finally, update the addData method to tie it together. The script checks the database before saving to the .jsonl file.

    async addData(scrapedData) {
        const pId = scrapedData.productId || scrapedData.url;

        if (!this.isDuplicate(pId)) {
            try {
                const jsonLine = JSON.stringify(scrapedData) + '\n';
                await this.writeFile(this.outputFile, jsonLine, 'utf8');

                // Persist to DB so we don't scrape this again next time
                this.recordItem(pId);

                console.log('Saved item:', pId);
            } catch (error) {
                console.error('Error saving data:', error);
            }
        }
    }

Handling Graceful Shutdowns

When using file-based databases like SQLite, it is best to close the connection when the script finishes or is interrupted. This prevents database corruption.

Add a listener to the main execution flow:

const pipeline = new SqliteDataPipeline();

process.on('SIGINT', () => {
    console.log('\nClosing database connection...');
    pipeline.db.close();
    process.exit();
});

The Complete Refactored Pipeline

The updated DataPipeline now uses a reliable SQLite backend instead of a volatile Set.

const Database = require('better-sqlite3');
const fs = require('fs');
const { promisify } = require('util');

class SqliteDataPipeline {
    constructor(dbPath = 'nike_history.db', outputFile = CONFIG.outputFile) {
        this.db = new Database(dbPath);
        this.outputFile = outputFile;
        this.writeFile = promisify(fs.appendFile);

        this.db.exec(`
            CREATE TABLE IF NOT EXISTS seen_items (
                id TEXT PRIMARY KEY,
                timestamp INTEGER
            ) WITHOUT ROWID; -- Optimization for simple key-value storage
        `);

        this.existsStmt = this.db.prepare('SELECT 1 FROM seen_items WHERE id = ?');
        this.insertStmt = this.db.prepare('INSERT INTO seen_items (id, timestamp) VALUES (?, ?)');
    }

    async addData(scrapedData) {
        const pId = scrapedData.productId || scrapedData.url;

        if (!pId) {
            console.error("Could not find a unique ID for item");
            return;
        }

        if (this.existsStmt.get(pId)) {
            console.warn(`Duplicate found in DB: ${pId}. Skipping.`);
            return;
        }

        try {
            const jsonLine = JSON.stringify(scrapedData) + '\n';
            await this.writeFile(this.outputFile, jsonLine, 'utf8');
            this.insertStmt.run(pId, Date.now());
            console.log('Successfully persisted:', pId);
        } catch (error) {
            console.error('Persistence error:', error);
        }
    }
}

Testing Resilience

The value of this refactor is best seen by testing a failure:

Start the Scraper: Run your script and let it scrape 10–20 products.
Force a Crash: Use Ctrl+C to kill the process.
Restart: Run the script again.

Instead of re-saving the first 20 products, the console will show "Duplicate found in DB... skipping." The scraper is now resumable, allowing it to skip previously processed URLs without wasting bandwidth or proxy credits.

To Wrap Up

Moving deduplication logic from RAM to SQLite improves the architecture of the Nike scraper. This approach is more memory-efficient and provides the persistence needed for large-scale data extraction.

Key Takeaways

In-memory storage is fine for small scripts, but persistent storage is required for production pipelines.
SQLite maintains state across script restarts with no configuration needed.
better-sqlite3 provides a high-performance API that simplifies scraping logic.
Deduplicate early to save bandwidth and avoid triggering anti-bot detection with redundant requests.

If you are starting a new project, you can use the ScrapeOps AI Scraper Generator to build the base extraction logic, then add your own persistence layer for production use.

Feed Rescue: Converting Raw Ulta Scrapes into Google Merchant Center XML

Erika S. Adkins — Sat, 28 Feb 2026 02:45:00 +0000

You’ve bypassed the anti-bot shields, rotated your proxies, and extracted thousands of product records from Ulta.com. Your reward is a massive JSONL file sitting on your hard drive. While this is a victory for data extraction, it’s a dead end for a marketing team.

Ad platforms like Google Merchant Center (GMC) don't accept JSONL. They require a highly structured, strictly validated XML format (RSS 2.0 or Atom). If your data doesn't perfectly match their schema—from currency codes to specific availability enums—your products won't show up in Google Shopping.

This "Feed Rescue" phase involves taking the raw output from the Ulta.com-Scrapers repository and building a Python transformation pipeline to generate a production-ready Google Shopping feed.

Phase 1: Analyzing the Source Data

Before writing any XML, we need to look at the raw material. The scrapers in the Ulta.com-Scrapers repository, specifically the Selenium and Playwright versions, use a ScrapedData dataclass that outputs a consistent JSONL schema.

Here is the typical structure of a raw Ulta scrape:

{
  "productId": "2583561",
  "name": "CeraVe - Hydrating Facial Cleanser",
  "brand": "CeraVe",
  "price": 17.99,
  "currency": "USD",
  "availability": "in_stock",
  "description": "Hydrating Facial Cleanser gently removes dirt...",
  "images": [{"url": "https://media.ulta.com/i/ulta/2583561?w=2000&h=2000", "altText": "Product Image"}],
  "url": "https://www.ulta.com/p/hydrating-facial-cleanser-pimprod2001719"
}

The scrapers in the repo use JSONL (Line Delimited JSON), which allows us to stream the data line-by-line. If you are converting 50,000 products, you don't need to load a 500MB JSON array into memory. You can process one product at a time.

Phase 2: The Google Merchant Center Specification

Google's requirements are rigid. While the scraper provides most of the data, the formatting is often "web-ready" rather than "ad-ready." GMC requires an RSS 2.0 feed using the g: namespace.

Ulta Scraper Field	GMC XML Tag	Requirement
`productId`	`g:id`	Unique identifier
`name`	`g:title`	Max 150 characters
`description`	`g:description`	Clean text, no broken HTML
`url`	`g:link`	Absolute URL
`images[0]['url']`	`g:image_link`	High-res image URL
`price` + `currency`	`g:price`	Format: `17.99 USD`
`availability`	`g:availability`	`in_stock`, `out_of_stock`, or `preorder`

Phase 3: Field Mapping & Transformation Logic

We need specific transformation logic to handle the nuances of Ulta's data.

1. Price Normalization

The scraper provides a float (17.99) and a string (USD). Google requires a combined string. This helper function ensures the decimal precision is always correct:

def format_gmc_price(amount, currency):
    return f"{amount:.2f} {currency}"

2. Availability Mapping

The Ulta scrapers detect stock status accurately, but we must ensure they match Google's allowed enums. If the scraper returns "in_stock", it works, but we should always include a fallback for unexpected values.

3. Image Handling

Google wants one primary image and the rest as "additional" images. Since the Ulta scraper returns a list of dictionaries, we use the first item for the main link and the remaining items for the gallery.

Phase 4: Building the Converter Script

This pipeline uses Python’s xml.etree.ElementTree because it is lightweight and handles namespaces efficiently. A generator reads the JSONL file to keep memory usage low.

import json
import xml.etree.ElementTree as ET
from xml.dom import minidom

def create_gmc_feed(input_jsonl, output_xml):
    # Define Namespaces
    g_ns = "http://base.google.com/ns/1.0"
    ET.register_namespace('g', g_ns)

    # Create Root
    rss = ET.Element("rss", version="2.0")
    channel = ET.SubElement(rss, "channel")
    ET.SubElement(channel, "title").text = "Ulta Product Feed"
    ET.SubElement(channel, "link").text = "https://www.ulta.com"
    ET.SubElement(channel, "description").text = "Daily product updates from Ulta"

    with open(input_jsonl, 'r', encoding='utf-8') as f:
        for line in f:
            product = json.loads(line)

            # Create Item
            item = ET.SubElement(channel, "item")

            # Basic Mapping
            ET.SubElement(item, f"{{{g_ns}}}id").text = str(product.get('productId'))
            ET.SubElement(item, f"{{{g_ns}}}title").text = product.get('name', '')[:150]
            ET.SubElement(item, f"{{{g_ns}}}description").text = product.get('description', '')
            ET.SubElement(item, f"{{{g_ns}}}link").text = product.get('url')
            ET.SubElement(item, f"{{{g_ns}}}brand").text = product.get('brand')
            ET.SubElement(item, f"{{{g_ns}}}condition").text = "new"

            # Formatted Price
            price = product.get('price')
            if price is not None:
                price_str = f"{float(price):.2f} {product.get('currency', 'USD')}"
                ET.SubElement(item, f"{{{g_ns}}}price").text = price_str

            # Availability
            status = product.get('availability', 'out_of_stock')
            ET.SubElement(item, f"{{{g_ns}}}availability").text = status

            # Images
            images = product.get('images', [])
            if images:
                ET.SubElement(item, f"{{{g_ns}}}image_link").text = images[0].get('url')
                # Add up to 10 additional images
                for img in images[1:11]:
                    ET.SubElement(item, f"{{{g_ns}}}additional_image_link").text = img.get('url')

    # Save with pretty printing
    xml_str = ET.tostring(rss, encoding='utf-8')
    pretty_xml = minidom.parseString(xml_str).toprettyxml(indent="  ")

    with open(output_xml, "w", encoding="utf-8") as f:
        f.write(pretty_xml)

if __name__ == "__main__":
    create_gmc_feed('ulta_data.jsonl', 'google_feed.xml')

Why this approach works:

Memory Efficiency: Processing the file line-by-line using for line in f prevents the script from crashing, even with hundreds of thousands of products.
Namespace Handling: Using f"{{{g_ns}}}tag_name" correctly implements the g: prefix Google requires for its specific attributes.
Data Truncation: Automatically truncating titles to 150 characters avoids GMC validation errors.

Phase 5: Handling Edge Cases

Web data is messy. Here are three common issues encountered when processing Ulta scrapes:

HTML in Descriptions: Ulta's descriptions sometimes contain raw HTML tags or entities like  . While the scraper cleans most of this, it is safer to wrap the description in a CDATA section or use a regex to strip remaining tags before inserting them into the XML.
Absolute URLs: Ensure your scraper uses the make_absolute_url logic from the repository. Google rejects relative URLs like /p/product-name.
Zero or Missing Prices: Occasionally, a product might show "Price Varies" or "Out of Stock" without a numerical value. The :.2f formatting will fail if price is None. Always default to 0.00 or skip the item if the price is missing.

To Wrap Up

Converting raw scraper data into a functional marketing asset turns raw data into business value. Bridging the gap between JSONL and GMC XML allows you to automate inventory updates directly from your scraping pipeline.

Key Takeaways:

Stream your data: Use JSONL and line-by-line processing to handle large datasets.
Respect the Schema: Google is strict about formatting. Always include the currency code in the price and map availability to their three specific enums.
Automate the Pipeline: Trigger this script immediately after your scraper finishes to create a hands-off data-to-ads pipeline.

For more information on the initial extraction, check out the ScrapeOps Residential Proxy Aggregator and the full range of implementations in the Ulta.com-Scrapers repository.

How to Scrape Ulta Reviews for Voice of Customer (VoC) Analysis

Erika S. Adkins — Tue, 24 Feb 2026 22:30:00 +0000

A five-star rating tells you a product is popular, but it doesn't explain why. For marketing teams and product managers, the real value is buried in the text of the reviews. This is where you find the Voice of Customer (VoC)—the specific language, pain points, and "aha!" moments that define the customer experience.

If a customer writes that a moisturizer "doesn't pill under makeup," you've found a high-converting headline for an ad campaign. However, manually reading thousands of reviews across Ulta’s massive catalog is impossible to scale.

This guide shows how to automate the extraction of Ulta product reviews using Python and Playwright. We will then process that data into a format ready for AI-driven analysis, turning raw HTML into actionable marketing insights.

Why Raw Review Data Matters

Star ratings are a lagging indicator of quality, but review text is a leading indicator of market fit. Extracting raw review data allows you to:

Identify Friction Points: Discover if a specific product batch has a faulty pump or if a new scent is polarizing.
Validate Features: See which ingredients users actually mention. If you’re marketing "Hyaluronic Acid" but users are raving about the "dewy finish," you know which angle to lead with.
Find Ad Copy Inspiration: Using a customer's exact phrasing (e.g., "A literal lifesaver for dry patches") increases relatability and trust.

To get this data at scale, we need a scraper capable of navigating Ulta’s anti-bot protections and dynamic content.

Prerequisites

To follow along, you will need:

Python 3.8+
A ScrapeOps API Key to handle proxy rotation and anti-bot bypass (sign up for free here).
Basic familiarity with the terminal and Pandas.

Step 1: Setting Up the Ulta Scraper

We will use the production-ready scrapers from the Ulta.com-Scrapers repository. While there are several ways to build a scraper, the Playwright version is best for reviews because Ulta loads its review sections dynamically.

First, clone the repository and install the dependencies:

# Clone the repository
git clone https://github.com/scraper-bank/Ulta.com-Scrapers.git
cd Ulta.com-Scrapers/python/playwright

# Install dependencies
pip install playwright playwright-stealth
playwright install chromium

Open the scraper file at product_data/scraper/ulta_scraper_product_data_v1.py and add your ScrapeOps API key:

API_KEY = "YOUR_SCRAPEOPS_API_KEY"

Step 2: Extracting Review Data

The Playwright scraper in this repository extracts a comprehensive ScrapedData object. Rather than just grabbing the price and title, this script targets JSON-LD (Linked Data) embedded in the page. This is more reliable because the data is already structured by Ulta’s own backend.

Here is the ScrapedData dataclass used in the script:

@dataclass
class ScrapedData:
    name: str = ""
    brand: str = ""
    price: float = 0.0
    productId: str = ""
    # This is our primary target for VoC analysis
    reviews: List[Dict[str, Any]] = field(default_factory=list)
    url: str = ""

The extract_data function looks for script[type='application/ld+json']. Here is the logic the scraper uses to populate the review list:

# Inside extract_data(page: Page)
scripts = await page.locator("script[type='application/ld+json']").all_text_contents()
for script_text in scripts:
    try:
        data = json.loads(script_text)
        if isinstance(data, dict) and data.get("@type") == "Product":
            # The 'review' key in JSON-LD contains the author, body, and rating
            res.reviews = data.get("review", []) 
    except: 
        continue

Step 3: Running the Scraper

To target a specific product, modify the execution block at the bottom of the script.

if __name__ == "__main__":
    target_urls = [
        "https://www.ulta.com/p/moisturizing-lotion-pimprod2007632"
    ]

    asyncio.run(run_scraper(target_urls))

Run the script from your terminal:

python product_data/scraper/ulta_scraper_product_data_v1.py

The scraper outputs a JSONL (JSON Lines) file. This format is ideal for web scraping because it saves each product as a single line, which prevents file corruption if the script is interrupted.

Step 4: Filtering and Preparing Data

Raw scraping output is often nested, meaning a single product entry contains a list of dozens of reviews. To analyze this, we use Pandas to flatten the data.

import pandas as pd
import json

def flatten_reviews(jsonl_file):
    flattened_data = []

    with open(jsonl_file, 'r') as f:
        for line in f:
            product = json.loads(line)
            product_name = product.get('name')

            for review in product.get('reviews', []):
                flattened_data.append({
                    'product': product_name,
                    'author': review.get('author', {}).get('name'),
                    'text': review.get('reviewBody'),
                    'rating': review.get('reviewRating', {}).get('ratingValue')
                })

    df = pd.DataFrame(flattened_data)

    # Remove empty or extremely short reviews
    df = df[df['text'].str.len() > 40]
    return df

df = flatten_reviews('ulta_output.jsonl')
print(df.head())

Step 5: Summarizing Insights with AI

With a clean DataFrame of customer feedback, you can use an LLM like GPT-4 or Claude to perform the Voice of Customer analysis. Instead of reading 500 reviews, you feed the most descriptive entries into a prompt.

VoC Prompting

You can automate this by sending the text column to an API.

import openai

def get_voc_report(reviews_list):
    reviews_text = "\n---\n".join(reviews_list[:50]) # Use first 50 reviews

    prompt = f"""
    Analyze the following customer reviews for a beauty product. 
    1. Identify the top 3 specific benefits users mention.
    2. List 3 recurring complaints or pain points.
    3. Extract 5 'Power Phrases'—exact quotes that would make great ad headlines.

    Reviews:
    {reviews_text}
    """

    # Call OpenAI/Anthropic API here

Example Results

An AI-generated report from your scraped data might reveal:

Benefit: "Doesn't pill under SPF" (Mentioned 12 times).
Power Phrase: "It's like a drink of water for my face."
Pain Point: "The pump mechanism gets stuck halfway through the bottle."

To Wrap Up

By combining automated scraping with AI analysis, you can treat Ulta as a massive, ongoing focus group. This workflow moves from raw HTML to structured insights in five steps:

Scrape: Use Playwright to extract JSON-LD data.
Export: Save in JSONL for data integrity.
Clean: Use Pandas to flatten and filter the results.
Analyze: Feed the text into an LLM.
Execute: Update ad copy and product roadmaps based on real customer language.

For the full suite of Ulta scraping tools, visit the Ulta.com-Scrapers GitHub repository. To scale your operations, use a ScrapeOps API key to manage proxy rotation and avoid blocks.

Instant Market Research: How to Scrape Costco Categories for Campaign Planning

Erika S. Adkins — Fri, 20 Feb 2026 07:44:37 +0000

Mapping out a competitor's seasonal strategy by hand is a grueling process. You can spend hours clicking through "Laptops" or "Organic Groceries," copy-pasting prices into a spreadsheet, and trying to identify which items are new arrivals. By the time you finish the first page, the data is likely already stale.

For growth marketers and analysts, manual data entry kills productivity. Real-time inventory data is the secret to spotting trends, such as a sudden surge in smart home inventory or a shift in bulk pricing. However, obtaining this information shouldn't be a full-time job.

You can automate this using the Costco Product Category Scraper. By extracting entire shelf segments, you can turn hours of spreadsheet fatigue into a 10-minute automated task. This guide demonstrates how to use a low-code Node.js approach to download comprehensive market data instantly.

Prerequisites & Setup

Costco uses sophisticated anti-bot measures, so we will use a pre-built scraper that integrates with ScrapeOps. This ensures requests aren't blocked and the data remains consistent.

1. Tools Needed

Node.js: You’ll need Node.js (v16+) installed. You can download it from the official Node.js site.
ScrapeOps API Key: This handles the heavy lifting for bypassing anti-bot protections. You can get a free API key here.
Git: To clone the scraper repository.

2. Installation

Open your terminal and run these commands to clone the repository and install the dependencies:

# Clone the open-source Costco scrapers repository
git clone https://github.com/scraper-bank/Costco.com-Scrapers.git

# Move into the Node.js Cheerio directory
cd Costco.com-Scrapers/node/cheerio-axios

# Install dependencies
npm install

The Category Strategy

When performing market research, beginners often scrape individual product pages one by one. This is slow and uses unnecessary resources. For high-level campaign planning, focusing on category pages (like the "Desktops" department) is much more efficient.

This approach offers several advantages:

Breadcrumbs Analysis: The scraper extracts the site hierarchy, revealing how Costco categorizes products for SEO.
Subcategory Discovery: You can instantly see niche segments, such as "Gaming Desktops" versus "All-in-One PCs," and see the item count for each.
Bulk Data: A single request to a category page can return data for 24 or more products, including names, prices, and stock status.

Instead of inspecting one tree, you are mapping the entire forest.

Configuring the Scraper

We will use the Cheerio-based scraper located in the node/cheerio-axios/product_category/ directory. This version is lightweight and fast because it doesn't need to load a full browser to run.

1. Locate the Scraper File

Open your text editor and navigate to:
node/cheerio-axios/product_category/scraper/costco_scraper_product_category_v1.js

2. Add Your API Key

Find the API_KEY variable at the top of the file and paste your key from the ScrapeOps dashboard.

/* node/cheerio-axios/product_category/scraper/costco_scraper_product_category_v1.js */

const API_KEY = 'YOUR_SCRAPEOPS_API_KEY'; // Replace with your actual key

3. Define Your Target

The script uses cheerio to extract structured data. It searches for specific attributes, such as data-testid='ProductTile_', to find items on the page. You can point the script at any Costco category URL by modifying the execution logic at the bottom of the file.

Running the Script

Once configured, you can run the scraper with a single command. The script handles the request, bypasses 403 blocks using the ScrapeOps proxy, and parses the HTML into a clean format.

Run this command in your terminal:

node node/cheerio-axios/product_category/scraper/costco_scraper_product_category_v1.js

What to Expect

Request: The script sends a request to Costco via the ScrapeOps proxy.
Extraction: The extractData function runs, pulling categoryName, products, and pagination info.
Storage: The data is saved into a JSONL (JSON Lines) file.

Your terminal will show a log once the file is saved, usually including the date in the filename.

Analyzing the Data

The raw output provides a structured snapshot of the market. Here is an example of the product data inside the generated file:

{
  "categoryName": "Desktops & Servers",
  "products": [
    {
      "productId": "1234567",
      "name": "HP Pavilion Desktop - 13th Gen Intel Core i7",
      "price": 799.99,
      "preDiscountPrice": 949.99,
      "availability": "in_stock",
      "image": "https://bfasset.costco-static.com/..."
    }
  ],
  "subcategories": [
    {"name": "Gaming Desktops", "productCount": 12, "url": "..."}
  ]
}

Strategic Applications

Pricing Strategy: Compare price against preDiscountPrice. If 80% of a category is "On Sale," Costco is likely clearing inventory in that segment.
Inventory Gaps: By comparing weekly scrapes, you can see which products sell out and which new IDs appear, signaling new product launches.
SEO Keyword Analysis: Use the breadcrumbs and categoryName to see how Costco titles high-traffic pages. You can use these terms to optimize your own meta-tags.

If you prefer working in Excel, you can use a JSONL to CSV converter to turn this data into a spreadsheet for pivot tables.

Troubleshooting

403 Forbidden Errors: This means Costco's firewall has flagged the request. Double-check that your API_KEY is correct. ScrapeOps uses residential proxies to prevent this.
Empty Product List: Retailers occasionally update their HTML structure. If the products array is empty, check if the data-testid attributes in the extractData function still match the current website.
Timeout Errors: Category pages with many filters take longer to load. Increase the timeout value in the CONFIG object if the script exits too early.

To Wrap Up

Automating your Costco research moves you from reactive guessing to data-driven planning. You now have the tools to capture a complete inventory snapshot of any department in minutes.

Category scraping is the most efficient way to map a market, provided you use a proxy service to ensure reliability. The resulting data is ready for price monitoring, SEO gap analysis, and spotting seasonal trends.

For further analysis, you could schedule this script to run every Monday morning to generate a fresh "State of the Market" report for your team.

Automating Catalog Sync: Designing Resilient Scrapers for Dynamic Marketplaces

Erika S. Adkins — Tue, 17 Feb 2026 05:52:53 +0000

Maintaining an accurate mirror of a dynamic marketplace is one of the most difficult challenges in data engineering. Unlike a one-off scrape where you simply grab what is available, a catalog sync requires a continuous, reliable loop. If your scraper misses a price change or fails to detect a new product, the business cost is immediate. Stale pricing and "out of stock" errors lead directly to lost revenue.

Most developers start by scraping a few hundred pages and assume the problem is solved. However, once you move to marketplaces with millions of SKUs like Amazon, eBay, or Walmart, standard pagination fails, anti-bot protections tighten, and data consistency becomes a nightmare.

This guide explores the architectural patterns required to move from basic scraping to building a professional-grade catalog sync pipeline. We will cover deep traversal strategies, incremental update patterns, and how to use JSON Lines (JSONL) to build a crash-resistant output stream.

Phase 1: The Discovery Problem

The first hurdle in any catalog sync is discovery. If you can't find the product, you can't sync it. Most large marketplaces cap their search results. For example, a site might claim there are 50,000 items in "Electronics," but the pagination often stops serving results after page 100.

If each page shows 20 items, you can only see 2,000 products. The other 48,000 are effectively invisible to a standard crawler.

Category Diving

To solve this, use Category Traversal. Instead of scraping the top-level category, programmatically drill down into sub-categories until the item count falls below the site's pagination limit.

Check Count: Request the category page and parse the "Total Results" count.
Evaluate: If the count exceeds the site's limit (e.g., 1,000 items), find the sub-category links.
Recurse: Yield new requests for those sub-categories.
Paginate: Once the count is within the limit, begin standard pagination.

This ensures that no product is buried too deep in the hierarchy to be reached.

Phase 2: Data Strategy

How often should you update your data? The answer depends on your scale and the volatility of the target site.

Strategy	Pros	Cons	Best For
Full Refresh	Guarantees 100% consistency; removes deleted items easily.	Resource-intensive; high risk of IP bans.	Small catalogs (<10k items).
Incremental Sync	Fast; saves bandwidth; targets only changed data.	Complex logic; hard to detect removed items.	Large-scale marketplaces.
Hybrid Approach	Balances accuracy and speed.	Requires sophisticated scheduling (e.g., Airflow).	Production-grade aggregators.

For a robust sync, try a Hybrid Approach. You might run a "Price & Availability" scraper every hour for popular items, while a "Discovery" scraper runs once a day to find new SKUs.

Implementing Incremental Checks

To avoid wasting proxy credits on data you already have, implement a filtering layer. A Bloom Filter or a fast key-value store like Redis can check if a URL was scraped recently.

def should_scrape(url, last_scraped_timestamp):
    # If the item was scraped less than 12 hours ago, skip it
    if (current_time - last_scraped_timestamp) < 43200:
        return False
    return True

Phase 3: The Output Pipeline

When syncing millions of records, your storage method is as critical as your retrieval method. Writing directly to a SQL database or a standard JSON array during the scrape is a common mistake.

Database writes during high-concurrency scrapes create bottlenecks and risk locking tables. Using a standard JSON array ([{},{}]) is worse: if the scraper crashes at 99%, the file lacks a closing bracket, making the entire dataset difficult to parse.

JSON Lines (JSONL) is the standard for large-scale extraction. In JSONL, every line is a valid, independent JSON object.

The Benefits of JSONL

Append-Only: Stream data to the file line-by-line.
Fault Tolerant: If the process dies, every line written up to that point remains valid.
Memory Efficient: You can process the data without loading the entire file into memory.

# Standard JSON (Fragile)
[
  {"id": 1, "price": 20.00},
  {"id": 2, "price": 25.00} # A crash here breaks the whole file
]

# JSON Lines (Resilient)
{"id": 1, "price": 20.00}
{"id": 2, "price": 25.00}
{"id": 3, "price": 22.00}

Phase 4: Reliability and Anti-Bot Architecture

Catalog sync is a marathon. Sending 1,000 requests per second without a strategy will trigger protections from Cloudflare or Akamai within minutes.

Think of requests as currency. You have a limited budget of successful requests before a site gets suspicious. To maximize your "Return on Request," you need a robust middleware layer.

The ScrapeOps Advantage

Instead of managing your own proxy rotation and header logic, you can offload this to a dedicated provider. The ScrapeOps Proxy Aggregator handles the anti-bot problem through a single API endpoint.

Key reliability features:

Smart Retries: Distinguish between a 404 Not Found (don't retry) and a 429 Too Many Requests (wait and retry with a new proxy).
Concurrency Limits: Start slow. Monitor your success rate and increase threads only if your proxy health allows it.
Fingerprinting: Use browser headers that match your proxy's exit node location.

Phase 5: Implementation Walkthrough

This Scrapy-based skeleton implements these concepts, including a custom pipeline for JSONL and category traversal logic.

1. The JSONL Export Pipeline

This pipeline opens the file when the spider starts and closes it upon completion.

import json

class JsonlExportPipeline:
    def open_spider(self, spider):
        # Open file in append mode
        self.file = open('catalog_sync.jsonl', 'a', encoding='utf-8')

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        # Write item as a single line
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        return item

2. Deep Traversal Logic

The spider decides whether to dive deeper into categories or start extracting products based on the result count.

import scrapy

class MarketplaceSpider(scrapy.Spider):
    name = 'catalog_sync'

    def parse_category(self, response):
        # Get total items in this category
        total_count = int(response.css('.total-results::text').get().replace(',', ''))

        if total_count > 1000:
            # Dive deeper into sub-categories
            sub_categories = response.css('.sub-cat-link::attr(href)').getall()
            for link in sub_categories:
                yield response.follow(link, callback=self.parse_category)
        else:
            # Count is manageable, start extracting
            yield from self.parse_products(response)

    def parse_products(self, response):
        # Pagination and extraction logic
        pass

3. Integrating ScrapeOps

In your settings.py, integrate ScrapeOps to automate proxy rotation.

# settings.py

DOWNLOADER_MIDDLEWARES = {
    'scrapeops_scrapy_proxy_sdk.scrapeops_proxy_sdk.ScrapeOpsProxyMiddleware': 725,
}

SCRAPEOPS_API_KEY = 'YOUR_API_KEY_HERE'

# Keep concurrency at a sustainable level
CONCURRENT_REQUESTS = 10

To Wrap Up

Building a catalog sync pipeline requires moving from simple data collection to state management. Deep category traversal ensures no data remains hidden, while JSONL protects your progress against infrastructure failures. Finally, a dedicated proxy management layer like ScrapeOps keeps your scrapers running smoothly without the constant need for manual maintenance.

Key Takeaways:

Traverse Deep: Use category counts to bypass pagination limits.
Stream Data: Use JSONL for crash-resistant storage.
Decouple Ingestion: Scrape to a file first, then use a separate process to bulk-upsert into your database.
Manage Reputation: Use professional proxy middleware to handle anti-bot challenges.

For a real-world implementation of structured extraction, explore the ProductHunt.com Scrapers Repository.

Hardcoded Selectors vs. AI Prompts: A Resilience Benchmark on Etsy

Erika S. Adkins — Fri, 13 Feb 2026 06:15:51 +0000

Every developer managing a web scraping pipeline at scale knows the "Monday Morning Breakage." You start your week, check the logs, and realize your primary data source updated its layout at 2:00 AM on Sunday. Your CSS selectors, once surgical and precise, now return None or empty strings that pollute your database.

Traditional scraping relies on the structural integrity of HTML. We target specific nodes like div.wt-mb-xs-2 > span.currency-value. But when a site like Etsy—notorious for aggressive A/B testing and dynamic class generation—changes that span to a p, the scraper dies.

This article benchmarks the traditional hardcoded selector approach against semantic AI-powered parsing. By simulating a breaking change on an Etsy product page, we can see if LLMs truly "understand" data when structure fails and analyze the real-world costs of both methods.

The Contenders: Selectors vs. Semantics

Before running the benchmark, we need to define the two philosophies competing for your infrastructure budget.

Approach A: Hardcoded Selectors

This is the industry standard. We use CSS Selectors or XPath to map the precise path to a piece of data.

# A typical (and brittle) Etsy selector
price = soup.select_one("div.wt-display-flex-xs.wt-align-items-center > p.wt-text-title-03").text

Pros: Near-zero latency, negligible cost, and 100% predictable.
Cons: Extremely brittle. If the site moves the price into a different container for a mobile update, the selector breaks.

Approach B: AI Prompts (Semantic Parsing)

This approach treats HTML as a document to be read rather than a tree to be traversed. We send the raw or cleaned HTML to a Large Language Model (LLM) with natural language instructions.

prompt = "Extract the current price and currency from this Etsy product HTML. Return JSON."

Pros: High resilience. AI understands that a number next to a "$" sign in a large font is likely the price, regardless of the <div> structure.
Cons: High latency (seconds vs. milliseconds), significant API costs, and the risk of hallucinations.

The Benchmark: Simulating a Layout Change

To test resilience, we’ll use a Python script that simulates a common scenario: Etsy updates its product page layout. We'll start with an "Original" HTML snippet and then "mutate" it by changing class names and nesting levels—exactly the changes that break traditional scrapers.

Setup

You'll need beautifulsoup4 for the selector test and the openai library for the AI test.

pip install beautifulsoup4 openai

The Experiment Script

The following script defines the original HTML and a mutated version where class names are replaced with randomized strings to simulate obfuscation.

from bs4 import BeautifulSoup
import openai
import json

# 1. The Original HTML
original_html = """
<div class="listing-page-content">
    <h1 class="wt-text-body-01">Handmade Ceramic Mug</h1>
    <div class="price-container">
        <p class="wt-text-title-03">$24.99</p>
    </div>
</div>
"""

# 2. The Mutated HTML (The "Monday Morning" surprise)
mutated_html = """
<div class="listing-page-content">
    <h1 class="header-v2">Handmade Ceramic Mug</h1>
    <div class="flex-wrapper-random-123">
        <div class="price-box-new">
            <span class="text-large-bold-green">$24.99</span>
        </div>
    </div>
</div>
"""

def test_css_selector(html):
    soup = BeautifulSoup(html, 'html.parser')
    # This selector targets the original structure
    price_el = soup.select_one("div.price-container > p.wt-text-title-03")
    return price_el.text if price_el else None

def test_ai_prompt(html):
    client = openai.OpenAI(api_key="YOUR_API_KEY")
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a data extraction tool. Extract the price from the HTML."},
            {"role": "user", "content": f"HTML: {html}"}
        ],
        response_format={ "type": "json_object" }
    )
    return response.choices[0].message.content

print(f"Selector on Original: {test_css_selector(original_html)}")
print(f"Selector on Mutated: {test_css_selector(mutated_html)}")

The Results

CSS Selector: Successfully extracted $24.99 from the original, but returned None for the mutated HTML. The scraper is broken and requires manual intervention.
AI Prompt: Even with the mutated HTML, the LLM correctly identified $24.99 as the price. It didn't matter that the class was text-large-bold-green instead of wt-text-title-03.

Cost Analysis: Tokens vs. Time

While the AI won the resilience test, we have to consider the trade-off between direct costs and maintenance.

Direct Cost (API Fees)

Using a model like gpt-4o-mini, parsing a typical Etsy product page (after stripping scripts and styles) costs roughly $0.0001 to $0.0005 per page.

10,000 pages: ~$5.00
1,000,000 pages: ~$500.00

Traditional selectors cost $0 in API fees.

The Hidden Cost (Developer Time)

If your Etsy scraper breaks twice a month and takes a senior developer two hours to fix and redeploy, that’s $240/month in maintenance (at $60/hr).

Metric	Hardcoded Selectors	AI-Powered Parsing
Cost per 1k Pages	$0.00	~$0.15 - $0.50
Latency	< 10ms	800ms - 2s
Resilience	Low	High
Maintenance	High	Near Zero

The Verdict: If you are scraping 10 million pages a day, AI is cost-prohibitive as a primary parser. If you are scraping 50,000 high-value leads a month, the developer time saved by using AI outweighs the API costs.

The Hybrid Approach: "Self-Healing" Scrapers

You don't have to choose between cheap and resilient. Most sophisticated scraping teams use Hybrid Fallback Logic.

Use the fast CSS selector by default. If it fails, trigger an AI fallback. The AI extracts the data and can even suggest a new CSS selector for future use.

Implementation: The Try-Except-Heal Pattern

def extract_price(html):
    # Step 1: Try the fast, free way
    price = test_css_selector(html)

    if price:
        return price, "selector"

    # Step 2: Fallback to AI if the selector failed
    print("Selector failed. Triggering AI healing...")
    ai_data = test_ai_prompt(html)

    # Step 3: Return data and log for manual review
    return json.loads(ai_data)['price'], "ai_fallback"

This ensures 99% of requests remain free and fast, while the pipeline stays active during layout changes.

Best Practices for Etsy & E-commerce

To maximize resilience without overspending, follow these guidelines:

Target JSON-LD First: Before using AI or CSS, check for <script type="application/ld+json">. Etsy often embeds structured data for SEO. This is the most resilient non-AI method because it's machine-readable and rarely changes.
Minimize Token Usage: Never send the entire HTML document to an LLM. Use BeautifulSoup to strip <script>, <style>, and <svg> tags first. This can reduce your token count by over 80%.
Use Schema-Aware Prompts: Tell the AI to look for specific Schema.org types. For example: "Find the 'Offer' schema and extract the 'price' property."

To Wrap Up

The choice between hardcoded selectors and AI prompts isn't binary. While traditional selectors remain the backbone of high-volume scraping due to their speed, AI has introduced a new era of resilient data extraction.

Use Hardcoded Selectors for high-volume, low-margin scraping where speed is critical.
Use AI Prompts for low-volume, high-complexity sites or as a self-healing fallback.
The Hybrid Model is the most balanced approach, minimizing both developer burnout and API costs.

By moving toward semantic understanding, you can stop spending Mondays fixing broken selectors and focus on the data itself.

Data Quality at Scale: Validating JSONL Output with Pydantic

Erika S. Adkins — Thu, 12 Feb 2026 09:49:45 +0000

Imagine your web scraper has been running perfectly for weeks. Your logs show a "200 OK" status for every request, and your database is filling up with thousands of new rows every hour. You assume everything is fine until a data scientist mentions that the latest dashboard is empty.

Upon investigation, you find the nightmare: a website update changed a single CSS class. Your scraper didn't crash, but it spent the last 48 hours extracting null values for every product price. This is a silent failure. In the world of web scraping, these are far more dangerous than a script that simply crashes.

This guide covers how to prevent "zombie data" scenarios by using Pydantic to validate JSONL (JSON Lines) output. By building a validation layer, you ensure your data is accurate, complete, and ready for production.

Why try/except Isn't Enough

When building scrapers, we often focus on connectivity. We wrap requests in try/except blocks to handle timeouts or 404 errors. While this prevents the script from dying, it does nothing to ensure the integrity of the data being saved.

Websites are volatile. A minor UI tweak can cause "Data Drift," where your selectors still find something, but not what you expected. Common issues include:

A price field that used to be an integer (49) becomes a string with symbols ($49.00).
A required field like a product title becomes an empty string due to a new layout.
A URL selector returns relative paths (/p/123) instead of absolute URLs.

If you pipe this data directly into a production database, you risk breaking downstream applications. You need a schema that acts as a gatekeeper.

Step 1: Defining the Data Schema

Pydantic is a data validation library that enforces type hints at runtime. If the data doesn't match the schema, Pydantic raises an error immediately.

First, install Pydantic:

pip install pydantic

Suppose we are scraping Product Hunt. We want to ensure every product has a name, a valid URL, and a non-negative vote count. Here is how to define that schema:

from typing import Optional
from pydantic import BaseModel, HttpUrl, Field

class ProductModel(BaseModel):
    # Expects a non-empty string
    name: str = Field(min_length=1)

    # Built-in validation for URLs
    url: HttpUrl

    # Upvotes must be an integer and cannot be negative
    upvotes: int = Field(ge=0)

    # Tagline is optional
    tagline: Optional[str] = None

This model does more than just type-checking. Field(min_length=1) ensures that an empty string "" triggers a validation failure, while HttpUrl confirms the string is a properly formatted web address.

Step 2: Validating a JSONL Stream

Most large-scale scrapers export data in JSONL format. Unlike a standard JSON array, JSONL stores one JSON object per line. This is the industry standard for scraping because you can append new data without loading the entire file into memory.

To maintain that efficiency, process the file line-by-line:

import json
from pydantic import ValidationError

def validate_jsonl_stream(input_file: str, output_file: str, error_log: str):
    valid_count = 0
    error_count = 0

    with open(input_file, 'r') as infile, \
         open(output_file, 'w') as outfile, \
         open(error_log, 'w') as errfile:

        for line_number, line in enumerate(infile, 1):
            try:
                data = json.loads(line)

                # Validate against the Pydantic model
                product = ProductModel.model_validate(data)

                # Write valid data to the clean file
                outfile.write(product.model_dump_json() + '\n')
                valid_count += 1

            except (ValidationError, json.JSONDecodeError) as e:
                # Log the specific error and the line number
                error_entry = {
                    "line": line_number,
                    "error": str(e),
                    "raw_data": line.strip()
                }
                errfile.write(json.dumps(error_entry) + '\n')
                error_count += 1

    print(f"Validation Complete: {valid_count} passed, {error_count} failed.")

validate_jsonl_stream('raw_data.jsonl', 'clean_data.jsonl', 'validation_errors.jsonl')

Using model_validate() allows the process to continue even if a line fails. Shunting "dirty" data into an error log helps you inspect why the scraper is failing without losing the valid data already collected.

Step 3: Implementing Business Logic Validators

Basic type checking is a good start, but real-world data quality often requires custom rules. For instance, you might want to ensure your Product Hunt scraper hasn't accidentally captured an external ad link.

Pydantic's @field_validator decorator enforces these specific rules:

from pydantic import field_validator

class ProductModel(BaseModel):
    name: str = Field(min_length=1)
    url: HttpUrl
    upvotes: int = Field(ge=0)

    @field_validator('url')
    @classmethod
    def must_be_product_hunt_domain(cls, v: HttpUrl) -> HttpUrl:
        # Ensure we aren't scraping external ads
        if 'producthunt.com' not in (v.host or ''):
            raise ValueError('URL must be a Product Hunt domain')
        return v

    @field_validator('name')
    @classmethod
    def prevent_placeholder_titles(cls, v: str) -> str:
        # Catch common scraping errors where placeholders are captured
        if v.lower() in ["loading...", "n/a", "none"]:
            raise ValueError('Invalid placeholder title detected')
        return v

These validators turn your schema into a diagnostic tool. If the website's HTML changes and your scraper starts picking up "Sponsored" links instead of products, the must_be_product_hunt_domain validator will catch it.

Step 4: Automating Alerts

Validation is only useful if you act on the results. In a production environment, you shouldn't have to manually check error logs.

You can implement threshold logic: if the percentage of failed records exceeds a limit (like 5%), trigger an alert.

def check_threshold(valid_count, error_count, threshold=0.05):
    total = valid_count + error_count
    if total == 0:
        return

    failure_rate = error_count / total

    if failure_rate > threshold:
        send_alert(f"Scraper Alert: Failure rate is {failure_rate:.2%}. Check selectors!")

def send_alert(message):
    # Connect this to Slack, Email, or PagerDuty
    print(f"ALARM: {message}")

This approach allows for "acceptable noise." If 1 out of 1,000 pages has a unique layout that breaks your logic, it might not require immediate attention. If 500 fail, you know the site has changed and the scraper needs an update.

To Wrap Up

Data validation separates hobbyist scripts from professional data pipelines. Using Pydantic to validate JSONL output provides:

Immediate Detection: Catch silent failures the moment they happen.
Clear Documentation: Pydantic models serve as a living reference for your data structure.
Clean Downstream Pipelines: Ensure your database and analytics tools only receive verified data.

Consider integrating these Pydantic checks into your CI/CD pipeline or as a final step in your orchestration workflows. The small upfront cost of defining a schema saves hours of cleaning corrupted data later.

High-Frequency eBay Scraping: Sync Prices and Stock Without Getting Banned

Erika S. Adkins — Mon, 09 Feb 2026 05:30:37 +0000

In e-commerce and dropshipping, stale data kills profit margins. If a customer buys an item from your store but the eBay price has jumped 20%, or the item is out of stock, you face a lose-lose choice: cancel the order and damage your seller rating, or fulfill it at a loss.

To prevent this, you must sync your inventory frequently. However, re-scraping 10,000 individual product pages every hour will get your IP blacklisted and blow your proxy budget. Real-time monitoring requires a smarter approach than brute-force scraping.

This guide demonstrates how to build a high-frequency eBay inventory monitor using Python. Instead of "one-by-one" scraping, we will use a diffing strategy with lightweight list views to track thousands of items with minimal overhead.

The Strategy: List View vs. Detail View

Most developers start by scraping the Detail View (the specific product page). While this provides the most data, it’s inefficient for monitoring. Tracking 5,000 items this way requires 5,000 separate requests.

Instead, use the List View (search results or "Seller’s Other Items" pages). On eBay, a single list page can display up to 200 items. By scraping these pages, you can update the price and stock status of 200 items with one request.

Detail View: 1 Request = 1 Item. High fidelity, low efficiency.
List View: 1 Request = 50+ Items. High efficiency, sufficient fidelity for price and stock.

Switching to List View reduces request volume by up to 98%, making syncs every 15 minutes technically and financially feasible.

Step 1: Designing the Local Database (The "State")

To detect changes, the script needs to remember the price and stock from the previous check. This is the State. While a JSON file works for a handful of items, it becomes slow and prone to corruption as your list grows. SQLite is a better choice because it is serverless, fast, and built into Python.

The database tracks the item_id, price, stock_status, and a last_checked timestamp.

import sqlite3

def setup_database():
    conn = sqlite3.connect('ebay_sync.db')
    cursor = conn.cursor()

    # Create table to store the 'state' of our inventory
    cursor.execute('''
        CREATE TABLE IF NOT EXISTS products (
            item_id TEXT PRIMARY KEY,
            price REAL,
            is_in_stock INTEGER,
            last_checked TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    ''')
    conn.commit()
    return conn

# Initialize the DB
db_conn = setup_database()

This schema allows for rapid lookups to see if scraped data differs from your records.

Step 2: The Lightweight Scraper

We need a function to fetch a list of items. This example targets a seller’s store page or a search result page using requests for the network call and BeautifulSoup to parse the HTML.

When scraping eBay, look for containers with the class s-item. These hold the ID, price, and availability markers.

import requests
from bs4 import BeautifulSoup

def fetch_ebay_batch(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9"
    }

    response = requests.get(url, headers=headers)
    if response.status_code != 200:
        print(f"Failed to fetch: {response.status_code}")
        return []

    soup = BeautifulSoup(response.text, 'html.parser')
    items = []

    for tags in soup.select('.s-item__wrapper'):
        # Extract Item ID from the link
        link_tag = tags.select_one('.s-item__link')
        if not link_tag: continue

        link = link_tag['href']
        item_id = link.split('?')[0].split('/')[-1]

        # Extract and clean price: "$25.99" -> 25.99
        price_tag = tags.select_one('.s-item__price')
        if not price_tag: continue

        try:
            price_text = price_tag.text.replace('$', '').replace(',', '').split(' to ')[0]
            price = float(price_text)
        except ValueError:
            continue 

        # Stock check: Look for "Out of stock" labels
        status_tag = tags.select_one('.s-item__availability')
        is_in_stock = 0 if status_tag and "Out of stock" in status_tag.text else 1

        items.append({
            'item_id': item_id,
            'price': price,
            'is_in_stock': is_in_stock
        })

    return items

Step 3: Implementing the Diffing Logic

This is the core logic. Instead of blindly updating the database, we compare new_data against old_data. This triggers specific actions, such as a Discord alert or a Shopify update, only when a change actually occurs.

def detect_and_sync_changes(scraped_items, conn):
    cursor = conn.cursor()
    changes_detected = 0

    for item in scraped_items:
        cursor.execute("SELECT price, is_in_stock FROM products WHERE item_id = ?", (item['item_id'],))
        row = cursor.fetchone()

        if row is None:
            # New item discovered
            cursor.execute("INSERT INTO products (item_id, price, is_in_stock) VALUES (?, ?, ?)",
                           (item['item_id'], item['price'], item['is_in_stock']))
            print(f"New Item Tracked: {item['item_id']}")
        else:
            old_price, old_stock = row

            if old_price != item['price'] or old_stock != item['is_in_stock']:
                print(f"Change Found on {item['item_id']}: Price {old_price} -> {item['price']}")

                cursor.execute('''
                    UPDATE products 
                    SET price = ?, is_in_stock = ?, last_checked = CURRENT_TIMESTAMP 
                    WHERE item_id = ?
                ''', (item['price'], item['is_in_stock'], item['item_id']))
                changes_detected += 1

    conn.commit()
    return changes_detected

Step 4: Building the Sync Loop

To keep data fresh, wrap the logic in a loop with basic error handling to prevent network timeouts from crashing the monitor.

import time

TARGET_URLS = [
    "https://www.ebay.com/sch/i.html?_ssn=some_seller_id&_ipg=200", 
]

def main():
    db_conn = setup_database()
    print("Starting eBay Monitor...")

    while True:
        try:
            for url in TARGET_URLS:
                print(f"Scanning {url}...")
                scraped_data = fetch_ebay_batch(url)
                changes = detect_and_sync_changes(scraped_data, db_conn)
                print(f"Scan complete. Changes detected: {changes}")

            print("Sleeping for 15 minutes...")
            time.sleep(900) 

        except Exception as e:
            print(f"Error occurred: {e}")
            time.sleep(60) 

if __name__ == "__main__":
    main()

Optimization & Scaling

As your inventory grows, use these recommended approaches to stay efficient:

1. Proxy Management

eBay limits request rates aggressively. For high-frequency monitoring, use Residential Proxies or Smart Proxy Rotators. These are much harder for eBay to distinguish from real shoppers compared to data center IPs.

2. The Hybrid Scrape

Sometimes the List View is ambiguous, showing "See Price" instead of a value. In these cases, program your engine to trigger a targeted Detail View scrape only for that specific item. This maintains accuracy without sacrificing overall efficiency.

3. Concurrency

If you monitor multiple large sellers, a sequential loop will be too slow. Use concurrent.futures.ThreadPoolExecutor to fetch multiple pages simultaneously.

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=5) as executor:
    results = list(executor.map(fetch_ebay_batch, TARGET_URLS))

To Wrap Up

High-frequency eBay monitoring relies on efficient state management. By focusing on List Views and a diffing strategy, you can build a system that reduces proxy costs by 98% and minimizes the risk of IP bans.

From here, you can connect the detect_and_sync_changes function to an external API, like Shopify or a Discord Webhook, to automate your response to price and stock updates. For more advanced techniques, see our guides on rotating proxies with Python and parsing HTML with BeautifulSoup.

DEV Community: Erika S. Adkins

Building a Resilient Wayfair Price Monitor: Python Playwright vs. Node.js Puppeteer

The Contenders: Architecture Overview

Python (Playwright)

Node.js (Puppeteer)

Round 1: Stealth and Anti-Bot Configuration

Round 2: Extraction Strategy (Locators vs. Cheerio)

The Python "Live DOM" Approach

The Node.js "Snapshot" Approach

Round 3: Concurrency Models

Round 4: Performance Benchmarks

Developer Experience and Maintenance

To Wrap Up

Productionizing AI-Generated Scrapers: Adding Monitoring, Logging, and Alerts to Playwright Scripts

Prerequisites

The Audit: Critiquing the Raw AI Code

Step 1: Implementing Structured Logging

Step 2: Data Quality Validation

Step 3: Operational Metrics and Health Checks

Step 4: Putting It Together

To Wrap Up

Reverse-Engineering Category Strategy: Analyzing Assortment Mix with Node.js

Phase 1: Strategy and Setup

Phase 2: Building the Category Scraper

How it works

Phase 3: The Analysis Engine

Phase 4: Finding Inventory Gaps

Phase 5: Visualization and Reporting

To Wrap Up

Defensive StockX Scraping: Building Resilient Data Pipelines with Python

Prerequisites

Step 1: Defining Truth with Dataclasses

Step 2: Implementing Validation Logic

Soft Failures vs. Critical Breaks

Step 3: The Resilient Data Pipeline

Step 4: Implementing an "Error Budget"

Step 5: Handling StockX Specifics (Currency & Parsing)

To Wrap Up

Refactoring the Nike Scraper: Replacing In-Memory Deduplication with SQLite

The Problem: In-Memory Deduplication

The Solution: better-sqlite3

Prerequisites

Refactoring the DataPipeline Class

Step 1: Initialization and Schema

Step 2: Efficient Deduplication Logic

Step 3: Integrating with the Save Loop

Handling Graceful Shutdowns

The Complete Refactored Pipeline

Testing Resilience

To Wrap Up

Key Takeaways

Feed Rescue: Converting Raw Ulta Scrapes into Google Merchant Center XML

Phase 1: Analyzing the Source Data

Phase 2: The Google Merchant Center Specification

Phase 3: Field Mapping & Transformation Logic

1. Price Normalization

2. Availability Mapping

3. Image Handling

Phase 4: Building the Converter Script

Why this approach works:

Phase 5: Handling Edge Cases

To Wrap Up

How to Scrape Ulta Reviews for Voice of Customer (VoC) Analysis

Why Raw Review Data Matters

Prerequisites

Step 1: Setting Up the Ulta Scraper

Step 2: Extracting Review Data

Step 3: Running the Scraper

Step 4: Filtering and Preparing Data

Step 5: Summarizing Insights with AI

VoC Prompting

Example Results

To Wrap Up

Instant Market Research: How to Scrape Costco Categories for Campaign Planning

Prerequisites & Setup

1. Tools Needed

2. Installation

The Category Strategy

Configuring the Scraper

1. Locate the Scraper File

The Solution: `better-sqlite3`