Amazon API vs Web Scraping: Stop Choosing. Build Both Routes.

Every team integrating Amazon data eventually hits the same debate: use the official SP-API, or scrape?

The debate never resolves cleanly, and it never will, because the question is wrong. It assumes the two are alternative implementations of the same thing, so the comparison collapses into which is more compliant, cheaper or more stable. That framing guarantees a bad decision.

Here is the actual situation: they retrieve two different classes of data. The official SP-API gives you your account. Public pages give you the market. Once you internalise that, the route debate disappears and what remains is a concrete architectural division of labour.

This post covers the authorization boundary, the public-versus-private line, what SP-API throttling actually costs in engineering time, and a hybrid architecture with runnable code.

The two domains

Category	Nature	Route
Product facts (title, price, rating, variants, BSR)	Publicly visible	Public collection
Search and ads (keyword, rank, ad type, creative)	Publicly visible	Public collection
Review content (rating, body, date, variant)	Publicly visible	Public collection
Ranks and categories	Publicly visible	Public collection
Orders and fulfillment	Account-private	SP-API (authorized)
Buyer information	Account-private + personal	SP-API (restricted PII role)
Inventory and settlement	Account-private	SP-API (authorized)
Own ad performance	Account-private	SP-API (authorized)

The route follows from the nature of the data, not from preference. This is also why "can we just use the official API" is answered no in most teams — you need market data, and it sits outside the official interface's authorization scope.

You would not use the SP-API to check a competitor's price, because it is not designed to expose that. You would not scrape your own orders, because they sit behind login and are account-private.

So the useful question is not "which one". It is: which of my requirements are account-domain and which are market-domain? For most teams the answer is both.

Where each route gets its legitimacy

Compliance discussions slide into an unanswerable "which is more legal". The two routes draw legitimacy from entirely different places, and what determines compliance is what you collect, not which method you use.

The SP-API is legitimate through an agreement chain: developer registration, seller authorization via Login with Amazon, role-based access granting minimum necessary scope. Operations touching personally identifiable information require a separate restricted role and review. What you can violate is the developer agreement and data protection policy, and the consequence is revoked authorization. The boundary is clear but narrow — only what a seller has actively authorized your application to access.

Public page collection has no contract. It sits under the platform's conditions of use, robots directives and rate limits. That is not the same as prohibition; publicly visible, non-personal page information can generally be collected, which is long-standing practice.

Risk rises with three specific things:

• Login-gated content — anything visible only after signing in has clearly crossed the public line.
• Personal data — buyer names, addresses and contact details are governed by data protection law.
• Aggressive request rates — traffic heavy enough to affect site operation shifts the activity from reading public information to interfering with a service.

The one-line test: is the data publicly visible and non-personal? The method is irrelevant. Misusing PII through the official interface is still a violation; collecting a fully public page can still be compliant.

Here is what structured public data looks like. Every field is public and non-personal:

{
  "asin": "B0CXYZ1234",
  "marketplace": "amazon.com",
  "title": "Stainless Steel Insulated Water Bottle, 32 oz",
  "price": { "current": 34.99, "currency": "USD", "listPrice": 44.99 },
  "rating": { "average": 4.6, "count": 12847 },
  "bsr": [ { "category": "Sports & Outdoors", "rank": 128 } ],
  "availability": "In Stock",
  "sponsored": false,
  "fetchedAt": "2026-08-30T10:22:41Z"
}

No buyer information, nothing requiring seller authorization. That is the compliance basis. By contrast, orders, buyer addresses and settlement detail never appear on public pages — they are only reachable through the SP-API after explicit authorization.

What SP-API throttling actually costs

This is the part teams consistently underestimate: even though the official interface is free, throttling converts into engineering complexity.

The SP-API uses a token bucket model. Each operation has its own request rate and quota, and quota refills continuously at that rate. For some operations, initial quota and refill rate also scale with seller business volume. Three consequences:

Quotas are independent per operation. Estimating on "total call volume" is always wrong. getOrders and getInventorySummaries are separate buckets and need separate models. This is the single most common design mistake.

Being throttled is normal, not exceptional. At high polling frequency it is close to inevitable. The correct response is queueing with backoff, not letting it throw and kill the job. Treating 429 as an error also destroys your alerting signal — the team learns to ignore it, and then the real failure goes unnoticed.

Values change. Amazon adjusts rates and quotas. Verify current figures in the official documentation rather than reusing numbers from older docs. I have watched a team design quotas from a year-old document and get throttled into the ground on day one.

The code

A per-operation token bucket

import threading
import time


class TokenBucket:
    """Per-operation token bucket.

    rate:      tokens refilled per second (SP-API restore rate)
    capacity:  bucket size (SP-API maximum quota / burst)
    """

    def __init__(self, rate: float, capacity: float):
        self.rate = rate
        self.capacity = capacity
        self._tokens = capacity
        self._updated = time.monotonic()
        self._lock = threading.Lock()

    def acquire(self, tokens: float = 1.0) -> float:
        """Block until tokens are available. Returns seconds waited."""
        waited = 0.0
        while True:
            with self._lock:
                now = time.monotonic()
                self._tokens = min(
                    self.capacity,
                    self._tokens + (now - self._updated) * self.rate,
                )
                self._updated = now
                if self._tokens >= tokens:
                    self._tokens -= tokens
                    return waited
                deficit = tokens - self._tokens
                sleep_for = deficit / self.rate
            # sleep outside the lock so other threads are not blocked
            time.sleep(sleep_for)
            waited += sleep_for


class OperationLimiter:
    """Maintains an independent bucket per SP-API operation."""

    def __init__(self, config: dict):
        self._buckets = {
            op: TokenBucket(cfg["rate"], cfg["capacity"])
            for op, cfg in config.items()
        }

    def acquire(self, operation: str) -> float:
        bucket = self._buckets.get(operation)
        return bucket.acquire() if bucket else 0.0

Calls with backoff

import random
import requests

RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def call_sp_api(limiter, operation, url, headers, params, max_retries=3):
    """SP-API call with token bucket queueing and exponential backoff."""
    last_error = None

    for attempt in range(max_retries + 1):
        limiter.acquire(operation)  # queue for a token first
        try:
            resp = requests.get(url, headers=headers, params=params, timeout=30)
        except requests.RequestException as exc:
            last_error = f"network: {exc}"
            time.sleep(2 ** attempt + random.uniform(0, 0.5))
            continue

        if resp.status_code == 200:
            return resp.json(), None

        if resp.status_code in RETRYABLE_STATUS:
            retry_after = resp.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay + random.uniform(0, 0.3))
            last_error = f"http {resp.status_code}"
            continue

        # 4xx other than 429: parameter or auth problem, retrying is pointless
        return None, f"http {resp.status_code} (not retryable): {resp.text[:200]}"

    return None, f"exhausted retries: {last_error}"

Two details that matter. Prefer the server's Retry-After header over your own backoff — it is more accurate. And only retry 429 and 5xx; a 400, 401 or 403 is a parameter or authorization problem that will fail identically a hundred times while burning quota and hiding the real error.

The unified ingestion layer

from dataclasses import dataclass, field
from typing import Callable


@dataclass
class IngestResult:
    ok: bool = False
    data: dict = field(default_factory=dict)
    failure_type: str = ""   # throttled | blocked | incomplete | request_error
    missing: list = field(default_factory=list)


class UnifiedIngestion:
    """Converges SP-API and public collection into one internal model."""

    def __init__(self, field_map: dict, required_fields: list):
        self.field_map = field_map      # external field -> internal field
        self.required = required_fields

    def normalize(self, payload: dict) -> dict:
        """Allowlist normalization: keep only declared internal fields."""
        out = {}
        for src_key, dst_key in self.field_map.items():
            if src_key in payload and payload[src_key] not in (None, "", [], {}):
                out[dst_key] = payload[src_key]
        return out

    def validate(self, record: dict) -> list:
        return [f for f in self.required if f not in record]

    def ingest(self, fetcher: Callable, **kwargs) -> IngestResult:
        """fetcher returns (payload, error); non-empty error means request failure."""
        result = IngestResult()
        payload, error = fetcher(**kwargs)

        if error:
            result.failure_type = "throttled" if "429" in str(error) else "request_error"
            return result

        record = self.normalize(payload)
        missing = self.validate(record)

        if missing:
            # parsed fine but fields absent: a coverage problem, not a request problem
            result.failure_type = "incomplete"
            result.missing = missing
            return result

        result.ok = True
        result.data = record
        return result

The critical design choice is three separate failure classes:

• throttled / request_error — request layer; look at quota and backoff
• blocked — a 200 that is not a real page (public collection side)
• incomplete — parsed fine, required field missing (a coverage problem)

These have completely different fixes. Merge them into one error log and you cannot tell whether to adjust quotas, lower concurrency or chase coverage.

Detecting blocked responses at the content layer

This is the most expensive gap on the public collection side. A blocked page returns HTTP 200. Captcha interstitials, bot-detection pages and pages that never finished rendering all come back green. If your check is the status code, that bad data enters your warehouse wearing a success badge.

BLOCK_MARKERS = [
    "captcha", "robot check", "enter the characters you see",
    "automated access", "to discuss automated access",
]


def looks_blocked(html: str) -> bool:
    """Content-layer block detection. Status code alone is not enough."""
    lowered = html[:20000].lower()
    hits = sum(1 for m in BLOCK_MARKERS if m in lowered)
    # markers present, or page implausibly short for a real product page
    return hits > 0 or len(html) < 2000

Wire this into UnifiedIngestion and blocked responses get classified as blocked instead of silently becoming data.

Compliance as code, not as a wiki page

Most teams keep compliance requirements in a document and enforce nothing in code. The result is "the doc says don't collect that, the code has been collecting it for months" — which is the worst position to be in during an audit.

Four constraints worth hardcoding:

A field allowlist. Keep only fields your internal model declares; drop everything else. Note that normalize() above is already allowlist-based rather than denylist-based, so even if an upstream response starts carrying personal data, it never lands in your store.

Path constraints. Collection jobs may only request public paths. Account pages and login redirects should be rejected and alerted on.

A hard rate ceiling at the job layer, so one runaway job cannot push overall frequency into "interfering with a service" territory.

Audit logging of source, time and field list per collection, plus a retained fetchedAt on every record. Without a capture timestamp a record cannot enter a time series and cannot be audited later.

Freshness is a budget decision

Having argued the official API is not free, the opposite error deserves equal attention: running every collection job at maximum frequency.

Field change rates vary enormously. Brand, category and variant dimensions barely move. Price, stock, Buy Box and sponsored placements move constantly. Treating them identically is the most common form of budget waste in this category.

Three tiers work in practice:

High frequency (minutes to hours) — price, stock, Buy Box holder, sponsored placements, BSR. For these, a stale value does not weaken the insight, it invalidates it. A competitor's price from yesterday cannot support a pricing decision today.

Daily — new reviews, rating changes, seller lists, best seller ranks. Trend signals that matter over weeks, not minutes.

Weekly or on demand — brand, category, variant dimensions, images, A+ content.

The top 20% of ASINs drive roughly 80% of decisions. Run those hot and the long tail cold and cost typically halves with almost no business impact. No vendor negotiation required — it is a scheduling decision in your own ingestion layer.

Implement it there rather than hoping a provider does it, and revisit it periodically. A tiering decision made eighteen months ago is usually wrong today.

When comparing prices on the collection side, use cost per thousand usable records, not per thousand requests. Failed, blocked and field-incomplete responses all bill. A plan that is 25% cheaper on the rate card is often only about 10% cheaper in reality.

And for the build decision, convert engineering hours to money first. A maintenance load of 0.3 engineer-days a month sounds trivial; priced at real internal cost over a year it frequently exceeds the API bill for a mid-sized volume — and unlike the bill, it recurs every quarter and grows with every page redesign.

What good looks like

Median latency under five seconds for on-demand fetches. p95 no more than roughly twice the median. Success rate above 98% measured at the content layer, not the status code. Field completeness above 95% against your own list.

Those are not laws of nature. They are the range where teams stop maintaining the data layer and start building on it. Below that range, the data layer becomes the most expensive dependency you have — not because of the invoice, but because of everything your engineers stop building while they babysit it.

Two fields worth treating as non-negotiable:

fetchedAt. Without a capture timestamp, a record cannot enter a time series and cannot be audited. Three months later, when a price looks wrong, you cannot tell whether it was wrong then or got mangled in processing.

isSponsored. Without it, organic rank and ad placement are indistinguishable. You think you rank third; the top two slots are ads; your true organic position is first. Every decision built on that number optimises for the wrong target.

Sponsored is also the hardest object to collect reliably, which is precisely why it needs its own evaluation line rather than an assumption that every provider has it.

The decision table

Skip the route debate and read your task off this:

Your task	Domain	Route
Sync your own orders and fulfillment	Account-private	SP-API
Manage inventory and settlements	Account-private	SP-API
Monitor competitor price and stock	Public market	Public collection / specialized API
Track keyword organic rank	Public market	Public collection / specialized API
Analyse competitor ad placements	Public market	Public collection / specialized API
Run review insights	Public market	Public collection / specialized API
Need both orders and competitor monitoring	Both	Hybrid architecture
Only your own operating data	Account-private	SP-API only

That last row deserves emphasis: if your requirements stop at your own operations, do not add public collection. Architectural complexity should match problem complexity. Converging two channels costs real money, and it is only worth paying when requirements genuinely span both domains.

One operational trap

A specific misdiagnosis shows up constantly in hybrid pipelines.

Symptom: volume for a field drops, or a window of data has holes.

First instinct: data quality degraded. So the team goes to the vendor, checks parsers, digs through logs. Frequently the real cause is quota — the token bucket saturated, jobs queued, and some timed out and were dropped.

Distinguishing them is easy if you classified failures: if throttled is climbing while blocked and incomplete are flat, it is a quota problem, not a data problem. The fixes are entirely different — tiering and scheduling for the former, coverage and parsing for the latter.

And do not alert on 429 directly. Record it as a throttled counter and watch the trend; alert only when the share crosses a sustained threshold. Otherwise the team learns to ignore the alert, and then the real failure goes unseen.

When to build nothing

Worth stating the inverse, since architecture posts have a bias toward building.

Requirements stop at your own operations — orders, inventory, your listings, your ads. Use the SP-API and stop there. Public collection buys coverage you will never use.

You need a one-off answer — a category survey, a competitor list for a deck. Do not build a pipeline. Pull the data and move on. Infrastructure built for a single question tends to outlive the question.

Volume is tiny and fields are narrow — thirty ASINs, four fields, weekly. Write a script. The break-even arithmetic only crosses once maintenance becomes recurring work, and below that line simpler is genuinely better.

Hybrid is correct when requirements span both domains. Verify that condition before committing engineering time. A decision framework is only half useful if it cannot also tell you not to build.

Wrapping up

The ordering matters more than any individual decision:

1. Split requirements into account-domain and market-domain.
2. Build the unified ingestion layer before either pipeline.
3. Model SP-API throttling per operation; retry only 429 and 5xx.
4. Detect blocked at the content layer, never on status code alone.
5. Classify failures three ways and put the ratios on a dashboard.

For the market side, Amazon Scraper API and Amazon Review API cover the public objects; Amazon Data MCP is the path when an agent calls data directly. Keys come from the console, and the MCP docs cover the agent path.

The full comparison, including the authorization boundary and decision table: Amazon API vs Web Scraping.

One more note on why the classification approach is worth adopting: it ends arguments that otherwise recur. Framed as compliance versus pragmatism, this debate never resolves, because nobody changes values in a meeting. Framed as "is this data public and non-personal", it resolves in half an hour.

If you are running a hybrid pipeline already, I would be curious which of the three failure classes dominates for you. In my experience incomplete is the one nobody budgets for.