DEV Community

coreclaw
coreclaw

Posted on Originally published at data-scrape.github.io

Amazon Price Monitoring at Scale: Build a Product Tracker Without Managing Proxies

Amazon Price Monitoring at Scale: Build a Product Tracker Without Managing Proxies

The fastest way to build a real Amazon price monitor is to skip the proxy infrastructure entirely and call a production-ready scraper API that already handles CAPTCHAs, IP rotation, and regional catalog differences. Everything else — the Python loop, the SQLite history, the alert logic — is the easy part. The reason most Amazon monitoring projects die in week two isn't bad code; it's the operational tax of running your own scraper fleet.

This walkthrough shows you a working 30-line Python script that polls 100 ASINs every 15 minutes, stores price history in SQLite, and fires alerts on configurable thresholds. No AWS Signature Version 4. No headless browser cluster. No proxy vendor invoices. You bring Python; the API brings the rest.

Why Most Amazon Price Monitors Stall

If you search "Amazon price monitoring Python" the top results all funnel you toward one of three paths: Amazon's PA-API 5.0, the SP-API, or roll-your-own with Scrapy + residential proxies. Each has a real problem that breaks most side projects before they ship:

  • PA-API 5.0 requires you to generate sales within 180 days or your access key gets revoked. If you're building a monitor for a niche product line that doesn't convert, you lose API access mid-project.
  • SP-API demands AWS SigV4 signatures, IAM roles, and a Python SDK that breaks every time Amazon updates their token endpoint. Setup is two days minimum.
  • Scrapy + proxies means you're now running a proxy rotation service, a headless browser pool, a CAPTCHA solver budget, and a job that fails the moment Amazon changes a single CSS class. Engineering time explodes.

There's a fourth path that nobody talks about in the tutorials: a managed Amazon scraper that already solved all of this and returns clean JSON. You send an ASIN, you get back price, Buy Box, seller, rating, stock, and a timestamp. You focus on the business logic — when to alert, what threshold matters, how to plug the data into Slack or your CRM. The scraper is someone else's problem.

The 30-Line Monitor

Here's the entire pipeline. It assumes you have a CoreClaw API key and the endpoint URL shown in the product documentation. Start with the plan and quota that match your test volume; the endpoint returns structured product data with no HTML parsing on your side.

import os
import time
import sqlite3
import requests
from datetime import datetime

# Get your API key and the exact endpoint from the CoreClaw console.
# Worker reference: https://coreclaw.com/coreclaw/amazon-global-product-scraper
API_KEY = os.environ["CORECLAW_API_KEY"]
ENDPOINT = os.environ["CORECLAW_AMAZON_ENDPOINT"]

ASINS = [
    "B08N5WRWNW",  # Echo Dot
    "B09G9FPHY6",  # AirPods
    "B07XJ8C8F5",  # Kindle
    # ... add up to 100 ASINs
]

ALERT_DROP_PCT = 5.0  # alert on 5% price drop

def init_db(path="prices.db"):
    conn = sqlite3.connect(path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS history (
            asin TEXT, ts TEXT, list_price REAL,
            buybox_price REAL, seller TEXT, in_stock INTEGER
        )""")
    conn.commit()
    return conn

def fetch_price(asin):
    r = requests.post(ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"asin": asin, "marketplace": "US"},
        timeout=20)
    r.raise_for_status()
    d = r.json()
    return {
        "asin": asin,
        "ts": datetime.utcnow().isoformat(),
        "list_price": d.get("list_price"),
        "buybox_price": d.get("buybox_price"),
        "seller": d.get("buybox_seller"),
        "in_stock": int(d.get("in_stock", False)),
    }

def maybe_alert(conn, snap):
    row = conn.execute(
        "SELECT buybox_price FROM history WHERE asin=? ORDER BY ts DESC LIMIT 1",
        (snap["asin"],)).fetchone()
    if not row or not row[0] or not snap["buybox_price"]:
        return
    prev, cur = row[0], snap["buybox_price"]
    change = (cur - prev) / prev * 100
    if change <= -ALERT_DROP_PCT:
        print(f"ALERT {snap['asin']}: ${prev:.2f} -> ${cur:.2f} ({change:.1f}%)")

def run():
    conn = init_db()
    while True:
        for asin in ASINS:
            try:
                snap = fetch_price(asin)
                maybe_alert(conn, snap)
                conn.execute(
                    "INSERT INTO history VALUES (?,?,?,?,?,?)",
                    (snap["asin"], snap["ts"], snap["list_price"],
                     snap["buybox_price"], snap["seller"], snap["in_stock"]))
                conn.commit()
            except Exception as e:
                print(f"err {asin}: {e}")
            time.sleep(2)  # gentle pacing
        time.sleep(15 * 60)  # 15-minute cycle

if __name__ == "__main__":
    run()
Enter fullscreen mode Exit fullscreen mode

That's the whole thing. No proxy config, no browser pool, no AWS signature helper. The endpoint already knows how to talk to Amazon's regional catalogs (US, UK, DE, JP, etc.) and returns whatever the live page shows for that ASIN — including Buy Box price, current seller, and stock status.

What You Get Back Per ASIN

Most price monitors only track the listed price. That's a mistake — on Amazon the Buy Box price is what customers actually pay, and it can be 10-20% lower than the list price when a third-party seller wins the box. The scraper returns both, plus the seller name and stock state. Here's the kind of payload you'll see:

  • asin — the product ID you queried
  • list_price — the official MSRP shown on the product page
  • buybox_price — the real price customers pay right now
  • buybox_seller — who currently holds the Buy Box
  • in_stock — boolean for availability
  • rating, review_count — useful for tracking seller competitiveness
  • fetched_at — ISO 8601 timestamp

The buybox_seller field is the one most homegrown monitors miss, and it's the most useful for competitive analysis. When you see a new seller name pop into your Buy Box column, you know someone is undercutting you or your competitors. That's a signal worth alerting on, not just price changes.

Polling Cadence and the Hidden Cost of Going Too Fast

A common mistake: poll every minute "to catch the fastest competitors." Two problems with that:

  1. You'll burn your API quota without gaining signal. Most Amazon price changes happen in 15-30 minute windows when sellers adjust via repricer tools. Polling faster than that just costs more money for the same data.
  2. You'll get rate-limited or blocked. Even managed scrapers have fair-use limits. Hammering them gets you throttled or, worse, your queries start returning cached data because you're hitting a shared cache tier.

The sweet spot for most use cases is 15 minutes. For high-velocity categories (electronics during Black Friday, ticket resellers) you can drop to 5 minutes. For slow-mover catalogs, once an hour is plenty. The script above defaults to 15 minutes via the time.sleep(15 * 60) line.

Comparing the Three Approaches

  • PA-API 5.0: setup depends on eligibility and access requirements; it may return cached data and is tied to the affiliate program.
  • SP-API + self-hosted: gives you more control but requires AWS authentication, infrastructure, and ongoing maintenance.
  • Managed scraper API: reduces setup and maintenance work; freshness, coverage, quotas, and price depend on the provider and plan.

For a real comparison, confirm current access requirements, endpoint behavior, regional coverage, quota rules, and pricing on each vendor's official documentation before committing.

The managed scraper column is the model offered by CoreClaw's Amazon Product Scraper. Check the current CoreClaw pricing and match the plan to your request volume before estimating total cost of ownership.

When the DIY Path Still Makes Sense

A managed scraper is the right answer for 90% of price monitoring projects. The 10% where you still want to roll your own:

  • You're scraping 10,000+ ASINs continuously and the per-call economics of any API start to dominate your infrastructure cost.
  • You need the raw HTML response for some custom field extraction the API doesn't expose (specific bullet point text, A+ content blocks, image URLs).
  • You're a marketplace analytics company where the scraper itself is a product, not just an input.

For everything else — competitive intelligence, repricing triggers, content marketing data, drop-catching — the managed path saves you weeks of engineering and ongoing maintenance pain.

Scaling Past 100 ASINs

Once the script above is running cleanly, three changes unlock much larger catalogs:

  1. Move the loop to a job queue. Replace the for asin in ASINS with a Celery worker that processes ASINs from a Redis queue. Now you can scale horizontally by adding workers.
  2. Switch the storage to Postgres. SQLite handles 100 ASINs fine; at 10,000 ASINs with 15-minute polling you're writing 40M rows per month. Postgres with proper indexes on (asin, ts DESC) keeps query latency sane.
  3. Use a hosted version of the scraper. Instead of running the Python script on your laptop 24/7, deploy the worker to CoreClaw's Workers platform — same infrastructure you'd build on AWS Lambda, but with the scraper API already wired in. The 100+ Workers Store on coreclaw.com/product/store has a pre-built Amazon price monitor you can fork and customize in minutes.

The Part Nobody Tells You About Amazon Monitoring

Price data alone is not the product. Every team I've seen build an Amazon monitor thinks the hard part is collecting the data. It's not. The hard part is what you do with it:

  • Alerts need context. A 5% price drop is uninteresting; a 5% drop on a product where you also lost the Buy Box at the same time is urgent. Combine signals.
  • History needs aggregation. Raw rows are useless. Build a daily summary: open price, close price, min, max, average Buy Box seller. That's what your pricing team actually reads.
  • The catalog drifts. ASINs get delisted, redirected, or merged. Build a janitor that validates each ASIN weekly and flags dead ones.

The scraper gets you the first 1% of the work. The other 99% is product thinking. Skip the infrastructure tax and spend that energy on the part that actually differentiates your business.

Get Started in an Afternoon

If you want to ship a real Amazon price monitor this weekend:

  1. Review the current CoreClaw pricing page and obtain an API key for your test volume.
  2. Copy the script above, swap in 5-10 ASINs you actually care about.
  3. Run it for a week and watch the data flow into your local prices.db file.
  4. Add Slack or email alerts using your existing notification stack.
  5. Once it's earning its keep, scale up the ASIN list and move it off your laptop.

The full code is below. Replace the API key, drop in your ASINs, and you have a production-ready monitor before dinner. The 5% price drop alert is just the starting point — once you see the data, you'll think of ten things to do with it.

Top comments (0)