DEV Community

coreclaw
coreclaw

Posted on

eBay Scraper GitHub: How to Build a Product and Price Monitoring Pipeline with Python

eBay Scraper GitHub: How to Build a Product and Price Monitoring Pipeline with Python

The most practical way to monitor public eBay listings over time is to pair an open-source scraper repository with a small Python pipeline that snapshots results daily, normalizes them into a stable schema, and diffs each run against the last one to surface price changes. This article is for e-commerce analysts, resellers, and developers who want a repeatable local workflow they own end to end — not a managed service with a per-request meter.

The data-scrape/ebay-scraper repository covers the eBay collection step, and the broader data-scrape profile hosts sibling repositories for other marketplaces such as Etsy and AliExpress, which matters if you want to compare the same product category across platforms. Your job is to wrap collection in a pipeline that deduplicates records, enforces a schema, and produces a daily change report.

TL;DR

  • Use the open-source data-scrape/ebay-scraper repository to collect public eBay listing data as JSON.
  • Normalize records into a fixed schema: source URL, listing ID, title, price, currency, condition, shipping, seller, and capture timestamp.
  • Store each run as a dated JSONL snapshot so the history is auditable and replayable.
  • Diff the newest snapshot against the previous one to generate a price-change report (CSV plus console summary).
  • Keep request rates conservative, respect eBay's terms and robots directives, and never touch private or login-gated data.
  • For the broader research context behind this workflow, see this Chinese-language public-web-data guide for market research.

Why eBay Price Monitoring Is Harder Than It Looks

eBay is one of the largest resale marketplaces, and its public pages hide several traps that break naive scraping attempts:

  • Dynamic listing cards. Search-result pages mix server-rendered HTML with JavaScript-injected fragments. A plain requests.get() against a search URL often returns placeholders instead of prices and shipping details.
  • Mixed listing formats. Auctions, Buy It Now, and Best Offer listings coexist in the same search results, each with different price semantics — a current bid is not a buy price.
  • Locale and currency variation. The same query can render in different currencies and formats depending on region settings, so $1,299.00, 1.299,00 €, and US $1,299.00 all need locale-aware parsing.
  • Schema drift. eBay experiments with card layouts and field names, so selectors that worked last month can silently return empty values.
  • Rate limiting and anti-bot measures. Aggressive polling from a single IP invites throttling or interstitial challenges. Slowing down is not optional; it is the price of staying unblocked.
  • Volume discipline. A single search can return thousands of listings. Without a narrow query and a page cap, one run can balloon into hours of requests.

The obvious approach — one ad-hoc script per question — fails for the boring reason that monitoring is a recurring job. You need stable schemas, timestamps, and history comparison, or the second run tells you nothing.

What the Verified Repositories Provide

The workflow in this article uses three repositories from the data-scrape profile:

  1. data-scrape/ebay-scraper — the eBay-focused collection repository for public product and price data. Check its README for the current quick-start command and supported output formats before you run it; repository instructions change over time.
  2. data-scrape/etsy-scraper — a sibling repository for public Etsy products, shops, reviews, and listing data. Its documented CLI pattern (scraper.py --query "..." --output results.json --max-results 100) produces JSON or CSV records with timestamps and source URLs.
  3. data-scrape/aliexpress-scraper — the AliExpress counterpart for cross-marketplace comparison using the same query-driven pattern.

The sibling repositories publish a consistent interface: a query in, structured records out, with explicit source_url and captured_at fields for traceability. That consistency is what makes a shared normalization and monitoring layer useful across marketplaces. Do not assume the eBay repository exposes an identical CLI until you have read its README — treat each repository's own documentation as the source of truth for its current options.

Setup

Create a project directory and clone the eBay repository:

mkdir ebay-monitor && cd ebay-monitor
git clone https://github.com/data-scrape/ebay-scraper.git
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install requests
Enter fullscreen mode Exit fullscreen mode

The monitoring layer below only needs the standard library plus requests. If the eBay repository's requirements.txt lists additional packages, install them the same way.

Environment configuration comes from variables, never from hard-coded values:

# Where snapshots are stored (created automatically)
export EBAY_MONITOR_DIR="./snapshots"

# Optional: override the default request delay in seconds
export EBAY_REQUEST_DELAY="2.0"
Enter fullscreen mode Exit fullscreen mode

The Monitoring Pipeline

The script below is the part you own. It has three stages:

  1. Collect — run the cloned scraper (or any collector you wire in) and write raw records to a dated JSONL snapshot.
  2. Normalize — flatten each record into a fixed schema, parse prices defensively, and drop records without a price.
  3. Diff — compare against the previous snapshot by listing key and emit a change report.
# ebay_monitor.py
import csv
import json
import os
import re
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

SNAPSHOT_DIR = Path(os.environ.get("EBAY_MONITOR_DIR", "./snapshots"))
REQUEST_DELAY = float(os.environ.get("EBAY_REQUEST_DELAY", "2.0"))
SCRAPER_DIR = SNAPSHOT_DIR.parent / "ebay-scraper"

PRICE_RE = re.compile(r"([\d.,]+)")


def parse_price(value):
    """Best-effort price parse. Returns float or None.

    Handles 'US $1,299.00' and '1.299,00 €' styles; adjust for your locale mix.
    """
    if value is None:
        return None
    if isinstance(value, (int, float)):
        return float(value)
    match = PRICE_RE.search(str(value))
    if not match:
        return None
    raw = match.group(1)
    # If both separators exist, the last one is the decimal separator.
    if "." in raw and "," in raw:
        if raw.rfind(",") > raw.rfind("."):
            raw = raw.replace(".", "").replace(",", ".")
        else:
            raw = raw.replace(",", "")
    else:
        raw = raw.replace(",", ".") if raw.count(",") == 1 and len(raw.split(",")[1]) != 3 else raw.replace(",", "")
    try:
        return float(raw)
    except ValueError:
        return None


def normalize(raw: dict) -> dict | None:
    """Map a raw scraper record onto the fixed monitoring schema."""
    url = raw.get("source_url") or raw.get("url") or ""
    if not url:
        return None
    price = parse_price(raw.get("price"))
    if price is None:
        return None  # monitoring requires a comparable price
    # Stable listing key: prefer an explicit id, else the URL path.
    key = raw.get("item_id") or raw.get("id") or url.rstrip("/").split("/")[-1]
    return {
        "key": str(key),
        "source_url": url,
        "title": (raw.get("title") or "").strip(),
        "price": price,
        "currency": raw.get("currency") or "USD",
        "condition": raw.get("condition") or "",
        "shipping": raw.get("shipping") or "",
        "seller": raw.get("seller") or raw.get("seller_name") or "",
        "captured_at": raw.get("captured_at")
        or datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }


def collect(query: str) -> Path:
    """Run the cloned scraper and return the snapshot path.

    Adapt this to the eBay repository's current CLI once you have read its
    README. The pattern below matches the query-in/JSON-out style used across
    the data-scrape marketplace repositories.
    """
    stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
    out_path = SNAPSHOT_DIR / f"raw-{stamp}.json"
    SNAPSHOT_DIR.mkdir(parents=True, exist_ok=True)
    cmd = [
        sys.executable, "scraper.py",
        "--query", query,
        "--output", str(out_path),
    ]
    subprocess.run(cmd, cwd=SCRAPER_DIR, check=True, timeout=600)
    return out_path


def load_snapshot(raw_path: Path) -> list[dict]:
    records = json.loads(raw_path.read_text(encoding="utf-8"))
    if isinstance(records, dict):  # some exporters wrap results
        records = records.get("results") or records.get("data") or []
    normalized = [normalize(r) for r in records]
    return [r for r in normalized if r]


def previous_snapshot(current: Path) -> Path | None:
    raws = sorted(SNAPSHOT_DIR.glob("raw-*.json"))
    older = [p for p in raws if p != current]
    return older[-1] if older else None


def diff(current: list[dict], previous: list[dict]) -> list[dict]:
    prev_by_key = {p["key"]: p for p in previous}
    changes = []
    for rec in current:
        old = prev_by_key.get(rec["key"])
        if old is None:
            changes.append({**rec, "change": "new", "old_price": None,
                            "price_delta": None, "pct_change": None})
            continue
        if rec["price"] != old["price"]:
            delta = round(rec["price"] - old["price"], 2)
            pct = round(delta / old["price"] * 100, 2) if old["price"] else None
            changes.append({**rec, "change": "price_change",
                            "old_price": old["price"], "price_delta": delta,
                            "pct_change": pct})
    return changes


def write_report(changes: list[dict], stamp: str) -> Path:
    report = SNAPSHOT_DIR / f"changes-{stamp}.csv"
    fields = ["change", "key", "title", "old_price", "price", "price_delta",
              "pct_change", "currency", "condition", "shipping", "seller",
              "source_url", "captured_at"]
    with open(report, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
        writer.writeheader()
        writer.writerows(changes)
    return report


def main():
    query = os.environ.get("EBAY_QUERY", "thinkpad x1 carbon")
    stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")

    raw_path = collect(query)
    current = load_snapshot(raw_path)
    normalized_path = SNAPSHOT_DIR / f"normalized-{stamp}.jsonl"
    with open(normalized_path, "w", encoding="utf-8") as f:
        for rec in current:
            f.write(json.dumps(rec, ensure_ascii=False) + "\n")

    prev = previous_snapshot(raw_path)
    if prev is None:
        print(f"Baseline snapshot: {len(current)} records. Run again tomorrow to diff.")
        return

    previous = load_snapshot(prev)
    changes = diff(current, previous)
    report = write_report(changes, stamp)
    drops = sum(1 for c in changes if (c.get("price_delta") or 0) < 0)
    print(f"Compared {len(current)} records against {prev.name}")
    print(f"Changes: {len(changes)} ({drops} price drops)")
    print(f"Report: {report}")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it once to establish a baseline, then on a schedule:

EBAY_QUERY="used thinkpad x1 carbon" python ebay_monitor.py
Enter fullscreen mode Exit fullscreen mode

Schedule the second run with cron (or Task Scheduler on Windows), spaced at least a day apart:

0 7 * * * cd /opt/ebay-monitor && ./venv/bin/python ebay_monitor.py >> monitor.log 2>&1
Enter fullscreen mode Exit fullscreen mode

Representative Output

The exact shape depends on the collector and the pages it parses, so treat this as an example of the normalized schema, not a promise of specific fields:

{
  "key": "296789123456",
  "source_url": "https://www.ebay.com/itm/296789123456",
  "title": "Lenovo ThinkPad X1 Carbon Gen 11 i7 16GB 512GB",
  "price": 749.99,
  "currency": "USD",
  "condition": "Used",
  "shipping": "Free shipping",
  "seller": "example_reseller",
  "captured_at": "2026-11-17T07:00:00+00:00"
}
Enter fullscreen mode Exit fullscreen mode

After the second daily run, the change report summarizes what moved:

change,key,title,old_price,price,price_delta,pct_change,currency
price_change,296789123456,Lenovo ThinkPad X1 Carbon Gen 11 i7 16GB 512GB,799.00,749.99,-49.01,-6.13,USD
new,296789654321,ThinkPad X1 Carbon Gen 10 i5 16GB,,429.00,,,USD
Enter fullscreen mode Exit fullscreen mode

Console output looks like:

Compared 118 records against raw-20261116-070001.json
Changes: 9 (6 price drops)
Report: snapshots/changes-20261117-070002.csv
Enter fullscreen mode Exit fullscreen mode

Choosing Between Build and Buy

Dimension Repository-based pipeline Managed web data API
Best for Learning, small daily research runs, full schema control Production volume, uptime, broad coverage
Setup model Clone, install, adapt the collector API key and endpoint from the provider's console
Output format Your JSON/CSV schema, you decide Provider-defined JSON; verify fields in their docs
Maintenance burden You own selectors, pacing, and breakage Provider owns infrastructure; you own integration
Cost model Your time plus optional proxies Per the provider's current published pricing — verify on their official pricing page
Integration path Direct into Python, pandas, SQLite REST calls from any stack

The honest trade-off: the repository path is free of per-request fees but transfers maintenance to you. When eBay changes its layout, your pipeline breaks until you update it. If monitoring becomes business-critical, a managed option — or a hybrid where the repository covers exploratory work and a production service covers the daily feed — is worth evaluating.

Use Cases

  • Reseller sourcing. Track a narrow category (for example, used business laptops) and get a morning report of new listings and price drops before other buyers.
  • Competitor monitoring. Watch a set of seller names in the seller field and diff their inventory and pricing daily.
  • Cross-marketplace comparison. Run the same normalization layer over data-scrape/etsy-scraper and data-scrape/aliexpress-scraper snapshots to compare price bands for equivalent product categories.
  • Demand estimation. Accumulate weeks of snapshots to see listing churn — how fast items appear and sell — as a rough demand signal.
  • AI agent context. The JSONL snapshots are ready to embed into a RAG pipeline or feed as structured context to an agent that answers pricing questions.

Compliance and Maintenance Checklist

Before the first scheduled run:

  • [ ] Read eBay's current User Agreement and robots.txt, and confirm your use case fits them.
  • [ ] Collect only public, non-login pages. Never target private data or gated content.
  • [ ] Keep the request delay conservative (the script defaults to 2 seconds via EBAY_REQUEST_DELAY) and cap pages per run.
  • [ ] Do not bypass CAPTCHAs, access controls, or rate limits. If you hit a wall, stop and reassess.
  • [ ] Record purpose, source URL, and timestamp for every record — the schema above does this by design.
  • [ ] Store snapshots securely and respect data-protection obligations in your jurisdiction.

Ongoing maintenance:

  • [ ] Run a small smoke query weekly; empty price fields usually mean the page layout changed.
  • [ ] Keep the collector repository updated (git pull) and re-read its README when it changes.
  • [ ] Archive raw snapshots so you can re-normalize historical data after schema fixes.
  • [ ] Alert on record-count anomalies — a sudden drop to zero often signals blocking, not a quiet market.

Limitations

  • The pipeline is only as good as its collector. If the eBay repository's parser returns partial fields, normalization cannot invent them.
  • Auction prices move continuously; a daily snapshot measures one moment, not the full bidding curve.
  • Currency and locale rendering can shift between runs, so keep the currency field and compare within it.
  • Public search pages may cache or personalize results; two runs may not see identical slices of the market.
  • This workflow observes prices; it does not predict them, and it does not cover eBay's non-public APIs.

FAQ

Do I need a paid service to monitor eBay prices?
Not for low-volume daily research. The repository-plus-pipeline approach runs on your machine. Managed services become relevant when you need higher volume, guaranteed uptime, or coverage you cannot maintain yourself.

What fields does the eBay repository return?
Check the repository's README and example output for the current field list — repositories in this profile publish an example record alongside their quick start. Do not assume fields beyond what the README documents.

How often should the pipeline run?
Once or twice a day is plenty for price monitoring. More frequent runs increase the chance of throttling without adding much information, since most listing prices change slowly.

What happens when eBay changes its page layout?
The collector's parser will start returning empty or malformed fields. The weekly smoke test catches this early; then update the collector (or pull its latest version) and re-run.

Can I monitor several queries at once?
Yes — loop over a query list and snapshot each separately, keeping the request delay between queries. Store each query in its own snapshot file so diffs stay comparable.

Is this compliant with eBay's terms?
Only your specific use can answer that. Public, low-volume research with conservative pacing is a very different risk profile from bulk commercial redistribution. Read the current agreement, and get legal advice for commercial use.

Can I extend this to other marketplaces?
Yes. Because data-scrape/etsy-scraper and data-scrape/aliexpress-scraper use the same query-driven, timestamped-output pattern, the normalize-and-diff layer in this article applies with minimal changes.

Next Steps

  1. Clone data-scrape/ebay-scraper, read its README, and run its documented quick start once.
  2. Copy ebay_monitor.py, point SCRAPER_DIR at your clone, and take a baseline snapshot for one narrow query.
  3. Schedule a daily run and inspect the first change report.
  4. Browse the data-scrape profile for sibling marketplace repositories if you want cross-platform comparisons.

Keep it public, keep it slow, and keep the history — the value of a monitoring pipeline compounds with every snapshot you keep.

Top comments (0)