DEV Community: Pangolinfo

From API Key to Production: a 5-Gate Acceptance Checklist for an Amazon Data API

Pangolinfo — Mon, 21 Sep 2026 06:48:34 +0000

A team signs up for an Amazon data API, runs a POC against fifty ASINs, sees latency that "feels fine," and cuts straight to production. The first traffic spike takes it down, and the month-end bill lands three times over estimate. The API was not the problem. The team treated a proof of concept as a launch plan.

A POC proves the pipeline exists. Production demands that the pipeline holds under load. Between the two sit five acceptance gates. This post turns them into a checklist you can execute in two weeks.

The five gates

Trial targets. Turn "feels fast" into numbers before you build: success rate ≥ 99%, median latency ≤ 3s, P95 ≤ 8s, field fill rate ≥ 95%. Cut the numbers to your business and lock them in writing.
Sample acceptance. Fire a batch of real ASINs — include variants, out-of-stock items, zero-review items, cross-category items — and check every field against the docs. Every field gets three columns: expected, actual, verdict. Nothing stays "need_confirm."
Load testing. Test concurrency, bursts, and rate-limit behavior. Watch the latency decay curve before the success rate; P95 climbs while the median holds steady. Measure data freshness under load too, since providers often fall back to cache under pressure.
Budget guardrails. Nail the pricing model (per request vs per record differs by an order of magnitude), then wire usage alerts, a circuit breaker, and unit-cost monitoring.
Launch gate. A pass-or-fail checklist with no "almost pass": threshold re-check, rate-limit handling, degradation switch, budget circuit, monitoring, and a rollback plan. Launch as canary → observe → widen → roll back.

Why field verification is its own gate

Gate 2 exists because docs describe what a provider promises, and the response shows what you get. The gap is the top source of rework. Two variants of one product can disagree on several of ten fields — and the scary part is that nothing errors. We measured that side in a separate piece on field contracts and variant drift. The lesson there applies here: pin the meaning of every field before launch, because a field your team reads right today may not survive a new teammate three months from now.

A two-week calendar

Days 1–2: set trial targets, build the sample-acceptance table.
Days 3–5: load testing — concurrency, burst, rate limit.
Days 6–7: budget — pricing model, unit cost, alerts and circuit breaker.
Days 8–10: pass the launch gate, build the canary, write the rollback plan.
Days 11–14: canary ramp at 10% → 50% → 100%, with an observation window at each step.

Two weeks is not hard law; it scales with team size. What matters is the order: targets, samples, load, budget, then launch. Reverse it and you get most integration disasters.

The API is one variable

Pick a provider that holds up under load, fields, and pricing, and most gates clear without a wall of fallback code. Ours — the Amazon Scraper API — serves over 30 million calls a day with a median latency around 3 seconds and a 99% success rate, and publishes sponsored-placement coverage across 13 marketplaces (91.4% overall). Those numbers are a baseline to check against, not a reason to skip acceptance. Run the five gates on it. If it holds, then launch.

The full checklist, threshold tables, and the launch-gate template live in the Amazon Data API integration checklist.

Amazon Product Data JSON API: Field Contracts, Variant Drift, and Contract Tests

Pangolinfo — Wed, 16 Sep 2026 06:05:52 +0000

The failure mode of an Amazon product data JSON API is not "no data came back." It is data that looks correct and is not. A parser that runs without an exception can still write two incomparable quantities into the same column, and nothing in your monitoring will notice for weeks.

This article works through a real case, using payloads captured on 2026-09-14 from the US marketplace. Two child ASINs share one parent, and six of their fields disagree. One page of reviews returns ten rows, and none of them belong to the ASIN that was requested. Both facts have direct consequences for how you model the data.

Everything below is runnable Python and TypeScript, and every sample value is quoted verbatim.

The case that motivates a field contract

Take one iPhone 15 Pro Renewed, 512GB, in white and in black. Both are children of parentAsin = B0GP8D698X. Their strikethroughPrice objects share a structure, share a type, and are non-empty:

// B0CMZFCQ6D (White 512GB)
"strikethroughPrice": { "key": "List Price", "value": "$649.00" }

// B0CMZ5KBNS (Black 512GB)
"strikethroughPrice": { "key": "Typical price", "value": "$629.95" }

List Price is the manufacturer's suggested retail price. Typical price is the 90-day median price paid by customers on Amazon. Two different baselines.

A parser that reads strikethroughPrice.value and labels it "original price" has just merged a suggested retail price with a median transaction price. The insert succeeds. The dashboard renders. The discount comparison is wrong.

Field-level defects share one property: they are silent. No parse exception, no null, no failed job. That rules out error-based detection. The only viable test is whether your assumption about a field holds across every variant and marketplace.

Divide the fields into four objects first

Before writing a dictionary, group the payload by business object. The fields are not a flat bag — they belong to four objects with different cadences and different nullability rules.

Object	Representative fields	Cadence	Nullability policy
Identity	`asin` `parentAsin` `title` `itemName` `brand`	Static	Required, non-empty
Transaction	`price` `strikethroughPrice` `inStock` `shipper` `seller`	Minutes	Required, nullable, every null needs semantics
Reputation	`star` `rating` `ratingDistribution` `reviews`	Daily	Model aggregates and details on separate tables
Spec	`attributes` `productOverview` `variantDetails` `images`	Weekly	Use if present, degrade if absent

That last row is a hard rule. Never mark spec fields required. Information completeness on Amazon varies by a wide margin, so strict validation on spec fields makes your pipeline throw on any product with incomplete attributes — and incomplete attributes are the norm, not the exception.

Grouping by business object rather than by page module also pays off in the architecture: identity maps to a key table, transaction to a fact table, spec to a sparse extension table. Group by DOM region instead and the dictionary has no clean mapping to storage.

Model three nullability states, not two

Key presence and business nullability are independent dimensions. Collapsing them produces two opposite modeling errors. The TypeScript below separates them, and the key move is using | null rather than ? to say "the key is always there, the value may be blank."

// Field contract: three nullability states, modeled as three types
type NonEmpty<T> = T;                 // always present, never blank
type Nullable<T> = T | null;          // always present, value may be blank
type Optional<T> = T | undefined;     // the key itself may be absent

interface ProductContract {
  // --- Identity: required, non-empty ---
  asin: NonEmpty<string>;
  parentAsin: NonEmpty<string>;
  title: NonEmpty<string>;

  // --- Split title: key always present, value may be empty string ---
  // Legacy listings always return an empty itemHighlights; do not use ?
  itemName: Nullable<string>;
  itemHighlights: Nullable<string>;

  // --- Transaction: key always present, value may be blank ---
  price: Nullable<string>;             // e.g. "$628.95", currency symbol included
  inStock: Nullable<string>;           // free text, not an enum
  shipper: Nullable<string>;           // may be ""; "" != "no shipper"
  savingsPercentage: Nullable<string>; // e.g. "6%", percent sign included

  // --- Structured price: nested, and key is NOT a stable enum ---
  strikethroughPrice: Nullable<{
    key: string;   // observed: "List Price" and "Typical price"
    value: string;
    tip: string;
  }>;

  // --- Spec fields: always optional, absence means degrade ---
  attributes?: Array<{ key: string; value: string }>;
  productOverview?: Array<{ key: string; value: string }>;
  size?: string;           // note: this is RAM, not storage capacity
}

Three entries in that interface map to measured traps.

Trap 1: the split title has a boundary you cannot see

As of 2026-07-27 Amazon splits the product title into itemName (the body) and itemHighlights (material, use case, selling points). The product sampled here is a legacy listing that has not been migrated. On such listings itemName equals the full title and itemHighlights is an empty string.

Assume that concatenating the two reconstructs the full title and the second half drops out on legacy listings. Assume itemHighlights always has a value and every legacy listing fails. Correct handling: use itemName, and fall back to title when itemHighlights is empty.

Trap 2: `size` holds RAM, not storage

This product returns size: "8 GB". It reads like storage. It is RAM. In the same payload, the attribute Memory Storage Capacity is "512 GB" and RAM Memory Installed is "8 GB".

Write size into a capacity column and every storage filter is off by a factor of 64. The problem compounds because the field's meaning shifts by category — for apparel size is a garment size, for phones it has been repurposed as a memory spec. Identical field name, different semantics.

Trap 3: the word "size" appears twice with two meanings

Top-level size is "8 GB" (RAM). variantDetails[].size is " 512GB " (storage, with surrounding whitespace). Same word, same JSON, two concepts. A contract that records only names misses this.

Diff two variants of one parent

If you do only one thing from this article, do this. Take at least two variants of the same parent and flatten every field for comparison. Here is what two children of B0GP8D698X produce:

Field	White 512GB	Black 512GB	Risk
`strikethroughPrice.key`	List Price	Typical price	Different semantics
`strikethroughPrice.value`	$649.00	$629.95	Different baselines
`inStock`	Only 13 left in stock - order soon.	In Stock	Free text
`shipper`	(empty string)	Amazon	Null semantics undefined
`attributes` length	48	49	Key set drifts
`Display Resolution Maximum`	2556 × 1179 pixels	2556x1179 pixels	Fullwidth sign vs lowercase x
`product_dims`	6 x 4 x 2 inches	5.77 x 2.78 x 0.33 inches	Different precision
`price`	$628.95	$628.95	Matches
`parentAsin`	B0GP8D698X	B0GP8D698X	Usable as grouping key
`rating`	(5258)	(5258)	Shared at parent level

Field-level diff across two variants of parentAsin B0GP8D698X: discount baseline, attributes length, and resolution spelling

Six of ten rows disagree. Testing a single ASIN will never surface any of them.

Two rows deserve detail.

attributes is a sparse map, not a fixed schema. 48 entries versus 49; the extra key is Model Series. You cannot premise anything on "every variant has attribute X," and you cannot expand the array into fixed columns. Index by key, apply defaults on absence.

Spec text does not follow one spelling. The resolution field uses a fullwidth multiplication sign × (U+00D7) on one variant and a lowercase x on the other. Same physical spec, unequal strings. Exact-match on spec text registers them as different products — which breaks deduplication and clustering.

This is not a vendor defect. The Amazon page itself is inconsistent, and every scraper reproduces it as written. Normalize before storage:

import re

def normalize_resolution(value: str) -> str:
    """'2556 x 1179 pixels' and '2556x1179 pixels' must normalize equal."""
    v = value.replace("\u00d7", "x").replace("\u00d7", "x")
    v = re.sub(r"\s+", "", v.lower())
    return v.replace("pixels", "")

The review payload's `asin` is not the ASIN you requested

This is the most consequential finding, and it is the reason a contract must be verified field by field rather than read off documentation.

We requested reviews for B0CMZFCQ6D. The API returned 10 reviews. Here is the asin distribution:

B0CMYXFK3R  x2
B0CMZL2TJ9  x3
B0CMZBXYWX  x1
B0CMZ7L14T  x1
B0CRJRNTNS  x1
B0CMZ9KS3G  x1
B0CMZCGQDK  x1
-----------------------------
distinct ASINs = 7
reviews matching the requested ASIN = 0

Requested ASIN B0CMZFCQ6D: 10 reviews across 7 distinct variants, 0 matching the requested ASIN

Not one of the ten carries the ASIN we asked for.

This is not an API defect. It reflects how Amazon attributes reviews: reviews attach to specific variants, and the variants of a product family share a review pool that the page aggregates for display. Each review's asin identifies which variant it came from.

Two consequences follow.

First, never write reviews[].asin back onto the main product record. Reviews end up attached to the wrong product and variant-level reputation analysis drifts.

Second, filter on asin when you need variant-level reviews. You cannot assume the payload was scoped to your request. Requesting the 512GB variant returned reviews whose attributes show Size: 256GB, Size: 1TB, and other variants — a distribution that affects whether that sample represents the variant you care about.

There is a finer format divergence hiding in the same field name. The reviews endpoint returns star as "1.0 out of 5 stars" (one decimal), while the reviews array inside the product payload returns "5 out of 5 stars" (integer). One field name, two endpoints, two formats. Share a parser between them and one side mis-parses or truncates without an error.

Null rules: three shapes, three behaviors

Shape	Observed example	Semantics	Downstream behavior
Key present, empty string	`shipper: ""`, `itemHighlights: ""`	Not captured this run / not applicable	Keep raw value, flag unknown
Key present, null	`reviews: null`, `importantInfo: null`	That block is absent from the page	Skip; not a failure
Key absent	Spec fields on some products	Merchant left the attribute blank	Apply default; count toward fill rate

Do not normalize empty strings into null. They carry different information: an empty shipper means the shipper was not captured this run, while a null reviews means this product page has no review module. Merge them and you lose both fill-rate measurement and the ability to tell collection problems from page changes.

Fill rate answers "is the field populated." It does not answer "how old is the value," and those are separate failures. We covered the second one in Verifying "Real-Time" Amazon Data APIs: 3 Clocks, 5 Cache Signatures, and a 48-Hour Protocol — a field can be populated on every call and still carry a two-day-old snapshot. Monitor both, or one hides the other.

`inStock` is free text, and unknown must not default to available

The two observed values are " Only 13 left in stock - order soon. " and " In Stock " — note the surrounding whitespace. The field carries Amazon's raw page copy.

import re

def normalize_stock(raw: str | None) -> tuple[bool | None, int | None]:
    """Normalize inStock free text into (available, units_left).

    (None, None) means undeterminable; the caller must treat it as
    unknown and must not default to available.
    """
    if not raw or not raw.strip():
        return None, None
    text = raw.strip().lower()
    if "left in stock" in text:
        m = re.search(r"(\d+)\s+left in stock", text)
        return True, int(m.group(1)) if m else None
    if "in stock" in text:
        return True, None
    if "unavailable" in text or "out of stock" in text:
        return False, 0
    return None, None   # unrecognized form; keep raw text for review

The last line is the design point. Faced with an unrecognized stock string, mark it unknown and keep the raw text — do not default to available. Reading out-of-stock as available breaks restock alerts; reading available as out-of-stock triggers pointless emergency repricing.

Schema diff for version drift

Fields change. Amazon splits title fields, adjusts attribute key sets, ships page redesigns. Vendors change response structures. A contract needs a versioning strategy to absorb this.

The script below flattens a payload into {path: type} and diffs it against a baseline.

import json
from collections import defaultdict

def flatten(obj, prefix: str = "") -> dict:
    """Flatten nested JSON into {path: type}; arrays are suffixed with []."""
    out = {}
    if isinstance(obj, dict):
        for k, v in obj.items():
            out.update(flatten(v, f"{prefix}.{k}" if prefix else k))
    elif isinstance(obj, list):
        out[prefix + "[]"] = "array"
        for item in obj[:5]:          # first 5 samples only; bounded
            out.update(flatten(item, prefix + "[]"))
    else:
        out[prefix] = type(obj).__name__
    return out

def schema_diff(baseline: dict, current: dict) -> dict:
    added   = sorted(set(current) - set(baseline))
    removed = sorted(set(baseline) - set(current))
    retyped = sorted(
        p for p in set(baseline) & set(current)
        if baseline[p] != current[p]
    )
    return {"added": added, "removed": removed, "retyped": retyped}

def load(path):
    with open(path, encoding="utf-8") as f:
        return flatten(json.load(f))

if __name__ == "__main__":
    diff = schema_diff(load("baseline.json"), load("current.json"))
    for label, items in diff.items():
        print(f"[{label}] {len(items)}")
        for item in items:
            print("   ", item)
    # exit code for CI: any field change marks the run for review
    raise SystemExit(1 if any(diff.values()) else 0)

The exit code is deliberate: any difference returns 1 so the pipeline marks the run "needs review" rather than failing outright. Added fields are harmless in most cases and can be accepted without review. Removed fields and type changes can break parsing. A contract test is a warning system, not a gate — over-blocking is how teams learn to route around it.

Contract tests: assert equivalence, not constants

import re
import pytest

# ---------- Trap 1: strikethroughPrice.key semantics are unstable ----------
@pytest.mark.parametrize("key,expected", [
    ("List Price",    "list_price"),
    ("Typical price", "typical_price"),
])
def test_strikethrough_key_is_classified(key, expected):
    assert classify_strikethrough(key) == expected

def classify_strikethrough(key: str) -> str:
    k = key.strip().lower()
    if "list price" in k:
        return "list_price"
    if "typical" in k:
        return "typical_price"
    return "unknown"          # unmapped forms must surface for review

# ---------- Trap 2: spec text does not follow one spelling ----------
def test_resolution_variants_are_equal():
    assert normalize_resolution("2556 \u00d7 1179 pixels") == \
           normalize_resolution("2556x1179 pixels")

# ---------- Trap 3: review asin != requested asin ----------
def test_reviews_are_not_implicitly_filtered(reviews, requested_asin):
    own    = [r for r in reviews if r["asin"] == requested_asin]
    others = [r for r in reviews if r["asin"] != requested_asin]
    assert len(own) + len(others) == len(reviews)
    # the point: writing back without filtering corrupts the product record
    if others:
        assert all(r["asin"] != requested_asin for r in others)

# ---------- Fill-rate monitor: earliest signal of an upstream change ----------
def test_fill_rate_does_not_regress(products, baseline: dict):
    for field, floor in baseline.items():
        filled = sum(1 for p in products if p.get(field) not in (None, "", []))
        rate = filled / len(products)
        assert rate >= floor, f"{field} fill rate {rate:.2%} below baseline {floor:.2%}"

The shared principle: assert normalized equivalence and explicit unknowns, not constant values. Upstream spellings for specs, stock copy, and discount baselines change. Hard-coded constants produce constant false alarms, and a suite that cries wolf gets ignored. Asserting equivalence after normalization survives spelling changes and fails only when semantics shift.

Cost tiering by tolerated delay

Field choices end at a cost question: which fields do you collect at high frequency, and which on a slow cycle.

Field class	Movement	Suggested cadence	Orders per product per day
Transaction (price, stock, Buy Box, coupons)	Minutes	Minute-level polling	Thousands
Reputation aggregates (rating, count)	Daily	Daily	1
Review content, specs	Weekly	Weekly	~0.1
Identity	Static	Collect once, reuse	~0

Per product, minute-level price polling means thousands of calls a day while weekly spec collection means dozens. Set the tolerated delay per field first, then choose the cadence, and only then compare vendors. Reverse that order and the price comparison is meaningless.

Field acquisition cost can also be optimized by endpoint choice. Product detail endpoints return dozens of fields per call at low unit cost; review and category endpoints often bill per page or per object at a higher unit cost. Carry identity, transaction, and spec fields on the product endpoint; call reviews only when review detail is needed, and prefer a critical-star filter to raise signal density per record.

Landing this on Pangolinfo

Every sample in this article came from running this contract process against real payloads. The traps — the title split boundary, the size semantics drift, the unstable discount baseline, the review asin not matching the request — were measured, not read off a spec sheet.

Pangolinfo's Amazon Scraper API returns structured JSON covering all four field classes above, and supports ZIP-code-specific collection so prices and delivery estimates match the region you care about. We hold a 99% success rate and 3-second median latency at over 30 million calls a day, with field fill rate monitored as its own metric alongside success rate — a green success rate can hide a payload missing a whole block of fields.

For review detail, the Amazon Review API filters by star rating, sort order, and media type, so you can pull critical reviews only and raise signal density per record.

On architecture, this walkthrough of the layers you still own in an Amazon data pipeline covers where the field contract belongs. Exact endpoint field definitions and response samples live in the Pangolinfo developer docs.

Write the contract before the parser. Reversed, the rework cost is an order of magnitude higher.

Verifying "Real-Time" Amazon Data APIs: 3 Clocks, 5 Cache Signatures, and a 48-Hour Protocol

Pangolinfo — Mon, 14 Sep 2026 06:26:44 +0000

Every vendor on the first page of a search for "real-time Amazon data API" uses the same word. Real-time. The claim is free to write and expensive to prove. A response that arrives in 100 milliseconds can carry data that a cache wrote nine hours earlier. A response that takes four seconds can carry a page that Amazon served 400 milliseconds before your request. The two numbers describe different things, and conflating them is the root of most bad procurement decisions in this category.

This article treats "real-time" as a measurement problem, not a marketing label. The unit of measurement is a single delta: the moment your program issued the request, minus the moment the source page was captured. That one subtraction is the only honest test. Everything else below is machinery to compute it, to prove it, and to defend against a vendor who quotes the first number while shipping the second.

The code samples are Python and runnable in shape. They use a placeholder vendor_fetch so you can drop in any provider and run the same checks against your own account.

The one delta that decides everything

A data API is real-time when the gap between your request clock and the capture clock is small enough for your use case. Small is case-specific, so the framing has to be precise:

Seconds of gap: real-time. A price-war bot that reprices on a competitor move needs this band.
Minutes to hours of gap: near real-time. Most pricing dashboards, stock alerts, and rank trackers live here without trouble.
Day-scale gap: snapshot. Fine for market structure research, category maps, and long-horizon price curves.

The trap is that a vendor can serve all three from one cache and call the whole thing real-time. The gap is what you pay for, and the gap is what you must measure before signing. If you cannot name the gap in your own words for the field you care about, you do not yet know what you are buying.

Three clocks, and why exposure matters more than speed

Three timestamps exist in every fetch, whether or not the vendor tells you about them:

Request clock. The moment your program calls the API. You always own this one.
Capture clock. The moment the vendor's server fetched the source page from Amazon. This lives on the vendor side.
As-of clock. The moment encoded in the data itself — a delivery date, a coupon expiry, a promo countdown. This lives on the Amazon page.

The vendor's documentation and response shape decide how far down this list you can see. If the response carries a capture timestamp, you can compute stale time with one subtraction. If it carries only an as-of field, you can still probe liveness by checking that the as-of value tracks your request day. If it carries neither, you are blind and must build your own page-side control to learn anything.

The practical rule: ask for the capture clock first. A vendor who exposes capturedAt is handing you the one number that ends arguments. A vendor who hides it is asking you to trust a word.

The structured log below records all three clocks on every call. Where the capture clock is absent, stale_ms stays None and the audit falls back to the as-of probe described later.

import time, json, logging

logger = logging.getLogger("freshness")

def fetch_product(asin, marketplace, do_fetch):
    # request clock: the moment our program issues the call
    request_time = time.time_ns()
    payload = do_fetch(asin, marketplace)  # vendor call
    received_at = time.time_ns()           # client wall clock at response

    # capture clock: server-side capture time, if the vendor exposes it
    captured_at = payload.get("capturedAt") or payload.get("meta", {}).get("capturedAt")

    # as-of clock: the moment encoded in the page data itself
    as_of = payload.get("asOf") or payload.get("delivery", {}).get("deliveryTime")

    stale_ms = None
    if captured_at is not None:
        stale_ms = received_at - int(captured_at)

    logger.info(json.dumps({
        "asin": asin,
        "marketplace": marketplace,
        "request_time": request_time,
        "received_at": received_at,
        "captured_at": captured_at,
        "as_of": as_of,
        "stale_ms": stale_ms,
    }))
    return payload

Freshness tiers by object, not by vendor promise

Amazon data is not one thing with one freshness need. A price can move dozens of times in a single day during a price war, while a product description sits unchanged for a year. Writing "real-time" across the whole catalog is either lazy or misleading. The honest design is layered, with a refresh period you set per object class:

Object class	Examples	Freshness tier	Why
Price, stock, Buy Box, coupon	`price`, `inStock`, `buyBox`, `coupon`	Minute	Price wars churn dozens of times per day; "Only 1 left" can flip between two polls.
Rank and placement	`bestSellersRank`, ad slot spread, keyword rank	Hour	Moves within the day, stable enough for hourly reads.
Reviews and structure	review count, rating, listing date, variant tree	Day	Slow to change; daily capture covers it.
Long-form content	review text, description, images	Week	Near-static; weekly snapshot suffices.

A correct plan reads like configuration, not a contract sentence. "Price every 10 minutes, BSR every hour, reviews once a day" is executable. "The vendor provides real-time data" is not. The second sentence hides the tier split that the first one makes visible, and the split is where cost and value live.

Segment your ASINs into hot, warm, and cold tiers by how often their fields move. Hot ASINs — the ones in an active price war — get the shortest interval. Cold ASINs — category anchors you watch for structure — get the longest. The official push stream sits at a 5-minute aggregation floor, so do not set your own interval more aggressive than that; you would be measuring noise, not winning a race.

The official ceiling: Amazon's own APIs are not magic

Before demanding a number from a vendor, anchor to what Amazon itself exposes. The ceiling defines a fair ask.

The Amazon Notifications API delivers price-change events through aggregation windows. Only two window sizes exist: 5 minutes and 10 minutes. Inside a window, intermediate events are dropped and only the first and last state are sent. That means the official push channel is, on price events, near real-time with lossy sampling. A vendor cannot beat that ceiling on the same signal without a different mechanism, and most do not have one.

The Selling Partner API (SP-API) generates most reports on request and returns them on an hours-to-days timeline. Advertising data refreshes daily. Public subscription tools state a 24-to-72-hour refresh window. Each of these is an Amazon-owned number, not a vendor weakness.

The takeaway for procurement: size your demand to the official data model. If you ask a vendor for sub-minute price freshness sourced from SP-API reports, you are asking for a contradiction. If you ask for it from a request-triggered scrape, you are asking for something the source page can support. Match the ask to the mechanism.

A real sample: B0CMZFCQ6D, captured twice

On 2026-09-09 we pulled one ASIN twice, about 20 minutes apart. The item is an iPhone 15 Pro 512GB Renewed. Both pulls returned the same values:

price: $618.90
inStock: "Only 1 left in stock - order soon."
bestSellersRank: #24 / #15 / #21 (the three category ranks)
delivery.deliveryTime: Friday, September 11

The request day was Wednesday, September 9. Three conclusions follow, and none of them are opinions.

Conclusion one: the delivery date is a live probe. Amazon computes deliveryTime server-side from the request context. A pull on Wednesday returning a Friday delivery is a computed answer, not a stored string. A cached record would either drift against the request day or freeze on a fixed value that looks too neat. Delivery dates, coupon validity windows, and promo countdowns are free liveness probes. Watch them. If the delivery date stops tracking your request day, the feed went stale without a single field turning empty.

Conclusion two: the single-unit stock reading is stable, and that is the point. "Only 1 left" came back identical across both pulls. That is a conclusion, not a defect: the listing did not change between the two captures, so the monitor reported no change. A monitoring system earns its keep by catching the moment stock moves from 1 to 0. A stable reading inside a stable interval is the system doing its job.

Conclusion three: the payload carried no capture timestamp. We could read every field value. We could not read when the server captured those values. The gap — how long the data has been alive — was not exposed. That single omission is what turns a confident "real-time" claim into an unverifiable one. Without a capture clock you have two paths: ask the vendor for it, or build your own page-side probe and measure divergence yourself.

Four pitfalls that make latency measurements lie

Teams who measure "speed" often measure the wrong speed. Four mistakes repeat:

Measuring total time without segments. A single end-to-end number hides where the seconds went. Split DNS, TCP, TLS, first byte, and total. A cache hit shows a tiny total with no TLS cost. A real scrape shows a TLS cost followed by a parse cost. The shape tells the story the sum hides.
Ignoring the cold-versus-warm gap. A reused session handshake costs about 15 ms. A fresh handshake costs about 53 ms. Quote the warm number to a prospect and the cold number to your on-call engineer, and you have described two systems. Measure both.
Reporting the average, not the distribution. An average of 3 seconds with a p95 of 19 seconds is a different product from an average of 3 seconds with a p95 of 4 seconds. Ask for median and p95, never for a mean alone.
Testing from one region. Network segments and server placement differ by geography. Run the same probe from two regions and you separate your local network from the vendor's server time. A single-region number conflates the two.

The warning that ties these together: a fast response is not a fresh response. A cache returns last night's data in 100 ms. A real scrape crosses Amazon and parses in seconds. If you optimize for the first number, you buy the second problem.

The sampler below reports cold and warm medians with p95, the shape you need before trusting any vendor latency claim.

import statistics, time

def sample_latency(open_session, n=20):
    cold, warm = [], []
    for i in range(n):
        if i == 0:
            t0 = time.perf_counter()
            _ = open_session()           # brand-new handshake
            cold.append((time.perf_counter() - t0) * 1000)
        else:
            t0 = time.perf_counter()
            _ = open_session()           # reuse warm session
            warm.append((time.perf_counter() - t0) * 1000)

    def p95(xs):
        if not xs:
            return None
        s = sorted(xs)
        k = max(0, int(round(0.95 * (len(s) - 1))))
        return s[k]

    return {
        "cold_median_ms": statistics.median(cold),
        "cold_p95_ms": p95(cold),
        "warm_median_ms": statistics.median(warm),
        "warm_p95_ms": p95(warm),
    }

Five signatures that a feed is a cache wearing a real-time badge

A cached feed can pass a demo and fail a month. These five signatures are each a reason to stop trusting the label. Any one hit is enough to open a conversation with the vendor.

Timestamps stall or vanish. A capture clock that never moves, or a response with no capture field at all, is the loudest signal.
Volatile fields refuse to track the real page. Capture the same ASIN three times and compare against a browser page in the same postal zone. If price or stock never moves while the page shows movement, the feed is not live.
Responses reproduce in single-digit milliseconds with no jitter. A page behind a strong bot defense that returns ten times at a few milliseconds each looks like a memory read, not a network fetch. Real scrapes carry variance.
Time-bound fields contradict the request day. A delivery date that ignores your request day, or a countdown frozen across pulls, breaks the as-of clock.
Freshness varies by endpoint without disclosure. Product detail arrives fresh while reviews land in a weekly batch, and the vendor does not say so. The inconsistency is the tell.

Write these as a repeatable script with timestamped records. Run it against every candidate. The output is one table per vendor, and the tables are your negotiation document.

The probe below encodes signatures two and four as a loop you can schedule.

import time

def probe_cache_signature(asin, vendor_fetch, browser_fetch, zone="02110"):
    samples = []
    for _ in range(3):
        samples.append(vendor_fetch(asin))
        time.sleep(7)  # wait past a likely cache TTL boundary

    page = browser_fetch(asin, zone)  # control: real page, same postal zone

    # signature 2: volatile fields must move when the page moves
    volatile = "price"
    stable = all(s[volatile] == samples[0][volatile] for s in samples)

    # signature 4: time-bound fields must track the request day
    delivery = samples[0].get("delivery", {}).get("deliveryTime")
    return {
        "three_captures_identical": stable,
        "delivery_field": delivery,
        "page_control_match": (page["price"] == samples[0]["price"]),
    }

A scenario SLA table, read backwards

Pick your row before you pick a vendor. The table reads from use case to tolerance to frequency. The tolerance column is the gap you measured in the first section; the frequency column is the config you write.

Scenario	Key fields	Tolerable data age	Suggested frequency
Dynamic pricing / price war	price, Buy Box, coupon	5–15 min	hot ASIN 5–10 min, normal 30–60 min
Stock-out / replenish alert	stock, listing status	minute to hour	core rival 15–30 min, long tail hourly
Ad slot / keyword rank	SP ad slot, organic rank	hour	every 1–6 hours
BSR / category rank	BSR, leaderboard	hour to day	every 4–24 hours
Review rating monitor	review count, rating, new content	day	1–2 times daily
Selection / market structure	category, variant, long price curve	day to week	weekly snapshot

The discipline is to tier by activity, not by vendor. Hot ASINs get the shortest interval; cold ASINs stretch out. No tier should run more aggressive than the official 5-minute push floor. Past that line you measure jitter, not signal.

Seven questions every vendor must answer in writing

Send this as a procurement questionnaire. A vendor who answers in one-liners with the word "real-time" repeated is telling you they cannot answer.

Data model. Does each request trigger one live capture, or does it read the last batch result? At what frequency? One sentence, no "real-time."
Capture timestamp. Does the response header or payload carry a server-side capture time, and what is the field name? If not, how does a customer prove data age?
Latency distribution. Median and p95, which region, at what concurrency, over the last 30 days — not a screenshot from a good day.
Success definition. Does a 200 with missing fields count as success? Does an intercept page count? Give field fill rate, not just status codes.
Probe access. Will you allow a script during the contract window to compare price and stock fields against the live page in real time?
Limits and fallback. What is the concurrency cap, and on overflow do you queue, drop, or serve a cache? If cache, what is its age?
SLA granularity. Measured monthly, weekly, or daily, and what is the compensation formula?

The first two questions decide how new the data is. The middle three decide whether it arrives on time and whether bad data slips in. The last two decide who pays when it breaks. A vendor who answers all seven with numbers and field names is rare; that rarity is the shortlist.

A 48-hour verification protocol

A demo account shows you the best case. The protocol below shows you the real one. Run it before any annual commit. The baseline script at the end seeds the first window.

Hours 0–6, baseline. Twenty ASINs spanning price-active electronics, daily consumables, and your own category, across at least two marketplaces. One capture per object type. Record success, fill rate, and whether a capture timestamp appeared.
Hours 6–24, repeat and compare. Five price-active samples every 30 minutes, plus a browser page comparison every hour. Record every divergence: the moment and the direction.
Hours 24–30, concurrency and latency. Ten cold and ten warm connections, median and p95. Press at 5, 10, and 20 concurrent for one minute each. Plot how latency rises with load.
Hours 30–48, change-capture drill. Watch one real changing event — a promo start, a Buy Box change, a stock hit zero. Measure the gap from event to API reflection. If no event occurs, extend the window. A verification that never captured a change verified nothing.

The deliverable is three tables: latency distribution, field fill rate, and the longest event-to-visible interval. Those three tables are the contract's evidence folder.

from datetime import datetime, timezone

ASINS = ["B0CMZFCQ6D"] + ["<electronics>", "<consumable>", "<own_category>"]
MARKETS = ["US", "DE"]

def run_baseline(window="0-6h"):
    rows = []
    for market in MARKETS:
        for asin in ASINS:
            payload = vendor_fetch(asin, market)
            rows.append({
                "asin": asin,
                "market": market,
                "window": window,
                "ok": payload is not None,
                "fill_rate": field_fill_rate(payload),
                "has_capture_ts": "capturedAt" in (payload or {}),
                "captured_at": (payload or {}).get("capturedAt"),
                "measured_at": datetime.now(timezone.utc).isoformat(),
            })
    return rows  # feed into the 6-48h windows of the protocol

Poll versus push, and the webhook queue you forgot

Two delivery models fit two event shapes. Sparse events that demand an immediate reaction — a stock recovery, a Buy Box change — belong on push. Continuous state you need as a series — a price curve, a BSR trail — belongs on poll. Choosing wrong costs you either missed reactions or wasted calls.

Push is not free of delay. Two sources add latency. Upstream, the aggregation window drops intermediate events; the official 5- and 10-minute windows throw away the middle of a burst. Downstream, the webhook queue backs up under load. Your receiver must stay resident, acknowledge delivery, and replay on failure. Add a timeout and an out-of-order handler to the webhook intake, and record two clocks on the consumer side: the event time the vendor claims, and the arrival time you observed. The gap between them is the push latency you run.

Poll gives you the series but spends calls. Size the poll to the tier table above and stop there. Polling a static description every minute is a bill with no signal.

Runtime staleness audit

The job is not done at procurement. Every return in production should write the three clocks — request_time, received_at, and captured_at when present — as one structured log line. Compute stale_ms and alert when it crosses the tolerance for that scenario. No captured_at means the fallback: sample price-active ASINs against the live page daily and watch for a systematic drift toward older values.

A single threshold per scenario beats a global one. Price alerts fire at fifteen minutes; review monitors can wait a day. Tune the alarm to the row in the SLA table, or you will page on noise and miss the real stale feed.

A measured claim beats a printed one

The point of this article is narrow: "real-time" is a number you can compute, and a vendor who welcomes the computation is different from a vendor who only prints the word. Run the three-clock log in production, run the cache signatures during evaluation, and run the 48-hour protocol before you commit. The tables those runs produce are the only honest answer to the question "is this feed real-time for my case?"

The companion piece on empty fields in a daily job covers the next failure mode: a feed that is fresh yet arrives with gaps. For the Node.js side, why a job dies at block 17 and a rerun doubles your rows traces the three failures a compiler never sees. And if you want the cost view after the scrapers are gone, we wrote a piece on what an Amazon data pipeline costs you.

A request-triggered scrape with no cache layer on the request path is the model that makes the three-clock log honest, because the capture clock then matches the request clock within one fetch. The Amazon Scraper API runs on that model: one request triggers one live capture of the source page and returns structured JSON, with residential IP, fingerprint, render, and parse bundled into the single call. Public anchors put median latency near 3 seconds including one full capture and parse, success near 99 percent, and more than 30 million calls per day. Take the protocol in section ten and test it; decide after the numbers are on the table. The pricing shows what one call costs across markets.

What does your current vendor's capture clock look like, and how would you prove it to your own team this week?

Amazon Data API Node.js: Why a Job Dies at Block 17 and a Rerun Doubles Your Rows

Pangolinfo — Thu, 10 Sep 2026 02:29:48 +0000

A job that pulls Amazon data in Node.js died at block 17. The logs showed HTTP 200 for every request that ran. Nobody panicked; we reran the job from the start. The next morning the report showed double the rows for every ASIN in blocks 1 through 17, and the price column held stale numbers for the new rows. The rerun did not resume; it overwrote nothing and appended everything.

That failure is not a bug in your loop. It is three gaps that TypeScript types cannot see: the network fingerprint, the HTTP/2 concurrency model, and the missing idempotency key. This article walks through each gap with runnable TypeScript and explains where a managed Amazon Data API Node.js call removes the work.

What you will learn

Why Node.js has no default answer that equals Python curl_cffi for TLS fingerprints, and what that costs in production.
Why turning on HTTP/2 reshapes your concurrency model instead of just speeding it up.
How to make a rerun idempotent so block 17 never writes a second row.
How to assert data shape at the edge with Zod, split timeouts per phase, name Node errors, and shut down without losing rows.

Thread one: Node has no curl_cffi equivalent for fingerprints

In Python you reach for curl_cffi when a target inspects the TLS handshake. The library ships a BoringSSL build and replays the exact ClientHello a real Chrome sends, including the extension order and the HTTP/2 SETTINGS frame. Node.js has no default answer that does the same.

For a while the community pointed to got-scraping. The project is now unmaintained. Worse, it only rewrote request headers. It never touched the TLS handshake. Any checkpoint that keys on JA3 or JA4 sat upstream of what the library could change, so the gaps it claimed to close were never closed.

This is the part most teams miss: a fingerprint is not the User-Agent string. It is the bytes of the TLS ClientHello and the HTTP/2 frame settings. A header swap leaves those bytes untouched.

impit claims Chrome and speaks Rust

impit gets close and then stops. Under the hood it uses a Rust HTTP client built on rustls. Its HTTP/2 SETTINGS frame uses the underlying Rust library defaults. That frame omits HEADER_TABLE_SIZE, and it sends a MAX_FRAME_SIZE value that Chrome never puts on the wire. The result is a client that claims Chrome in its docs yet speaks Rust in its bytes. A fingerprint check that sorts and hashes the SETTINGS frame will read the request as a Rust client, not a browser.

For Amazon, the cost shows up as a block of ASINs that return a challenge page instead of a product object. Your code sees a 200, parses nothing, and writes a row with empty fields. The fingerprint is the first gate, and Node leaves you to build it by hand. A wrong fingerprint is invisible until the fill rate drops, at which point the rows are already in the table.

The 2026-08 benchmark, read as a shape not a verdict

Numbers shift with versions, so treat the table below as a snapshot from a public benchmark measured on 2026-08-06 on an M-series Mac, 300 serial requests against a local service. The point is the shape, not the exact figures, and the figures will move as the libraries ship updates.

Library	Engine	Latest Chrome	HTTP/2 fingerprint	req/s	Cold start
wreq-js	Rust wreq + BoringSSL	149	Correct	12842	7 ms
impers	curl-impersonate	146	Correct	8439	16 ms
node-wreq	Rust	149	Correct	6500	10 ms
impit	Rust reqwest + rustls	124	Not correct	6710	37 ms
CycleTLS	Go subprocess IPC	Not in run	Not in run	Not in run	IPC overhead

wreq-js leads on throughput and gets the HTTP/2 fingerprint right because it rides BoringSSL and copies Chrome's frame values. impit posts a high req/s but its fingerprint is wrong. For an Amazon data job, a wrong fingerprint costs more than a slow one, because the slow request still returns data while the wrong one returns a checkpoint page.

JA3 moved, so JA4 took over

JA3 is a hash of five ClientHello fields concatenated and run through MD5. After Chrome 110 the browser permutes the extension order on each connection, so the JA3 value moves between runs. Most risk systems now use JA4, which sorts the fields before hashing. The lesson for a Node shop is to pick a client whose ClientHello is stable and matches a real browser, or the values you send will never line up with what a browser sends.

Reuse the session or pay the handshake twice

A single TLS handshake costs about 15 ms when you reuse a session, against about 53 ms for a fresh one. The gap is the handshake, not the request. Reuse the session across the loop:

import { createSession } from "pangolinfo";

const session = await createSession({ browser: "chrome_149" });
try {
  for (const asin of asins) {
    await session.fetch(buildUrl(asin));
  }
} finally {
  await session.close();
}

That block runs on every exit path, so the warm handshake you paid for is not wasted on a thrown error.

Thread two: HTTP/2 changes the concurrency model

Flipping on HTTP/2 sounds like a free speed win. In Node.js it changes the rules of the game, and the two entry points disagree on the default.

Node's built-in fetch rides on undici. By default it does not negotiate HTTP/2. The team flags it as experimental and leaves it off. You must set it by hand:

import { Agent, setGlobalDispatcher } from "undici";

const dispatcher = new Agent({
  allowH2: true,
  connections: 8,
  pipelining: 0,
  maxConcurrentStreams: 100,
  bodyTimeout: 30_000,
  headersTimeout: 15_000,
  connect: { timeout: 5_000 },
});
setGlobalDispatcher(dispatcher);

Note the split: fetch keeps allowH2 off by default, while the undici Client keeps it on by default. The same library, two defaults. Miss the one for fetch and you stay on HTTP/1.1 without an error. Your code runs, your throughput does not move.

Streams replace pipelining

When HTTP/2 is on, a single connection opens many streams. The ceiling is maxConcurrentStreams, which defaults to 100. This replaces HTTP/1.1 pipelining. The flow control window initialWindowSize defaults to 262144 bytes. You are no longer racing one request per socket; you are filling a window across streams.

Most teams discover this the hard way. They turn on HTTP/2, watch the connection count drop, and expect throughput to climb. It does not, because the old per-socket mental model no longer applies. You now reason about streams per connection and the window size, not sockets. A pool of eight connections with 100 streams each is not eight workers; it is eight windows you must keep full without tripping the rate limit at the other end. Get the window wrong and you stall with the socket idle, which is the opposite of the problem you set out to fix.

Concurrency is not rate

p-limit(8) caps how many calls run at once. It does not cap the rate. At a 10 ms response you push about 800 requests per second. At a 5 second response the same cap yields about 1.6 requests per second. The concurrency number stayed the same; the rate swung by a factor of 500. If your target throttles by rate, a flat concurrency cap will not save you.

import PQueue from "p-queue";

const queue = new PQueue({
  concurrency: 8,
  interval: 1_000,
  intervalCap: 4,
});

for (const asin of asins) {
  queue.add(() => fetchProduct(asin));
}
await queue.onIdle();

Add intervalCap with interval and you cap both the in-flight count and the rate per window. That is the knob HTTP/2 work needs.

Thread three: idempotent reruns

Back to block 17. The rerun appended because nothing told the database that row 17 for an ASIN was the same row as the one from the first run. The fix is an idempotency key, not a WHERE NOT EXISTS guess.

Build the key from four fields: asin, marketplace, capturedAt, and contractVersion. The first three say which item and when. The fourth says which shape of the response you stored. When the contract changes, you want a new row, not an overwrite of old data with a new schema.

import { Pool } from "pg";

const pool = new Pool();
const sql = `
  INSERT INTO product_snapshot
    (asin, marketplace, captured_at, contract_version, title, price, currency, captured_at_ts)
  VALUES ($1, $2, $3, $4, $5, $6, $7, now())
  ON CONFLICT (asin, marketplace, captured_at, contract_version)
  DO UPDATE SET
    title = EXCLUDED.title,
    price = EXCLUDED.price,
    currency = EXCLUDED.currency;
`;

await pool.query(sql, [asin, marketplace, capturedAt, contractVersion, title, price, currency]);

Chunk the work and write a checkpoint after each chunk. A rerun that starts at the last good checkpoint never touches blocks 1 through 16 again. The 200 you got for those blocks stays as one row, not two.

The checkpoint is a small table of its own: block id, row count, and a timestamp. A rerun reads the last finished block and starts there. If the job dies at block 17, blocks 1 through 16 are marked done and never fetched again. The double-row bug dies at the root because the second run never touches the rows the first run wrote. Without the checkpoint, every rerun is a full replay, and a full replay is where duplicate rows come from.

Coverage is not fill rate

Two numbers get confused: coverage and fill rate. Coverage is the share of ASINs you attempted that returned a response. Fill rate is the share of those responses that held the fields your job needs. A job can hit 100 percent coverage and still ship a report where the price field is empty for half the rows. The gap between coverage and fill rate is where a 200 lies to you. We cover that split in a companion piece on empty fields in a daily job.

TypeScript types do not hold runtime data

TypeScript types describe the data you expect. They do not describe the data you receive at 03:14. The bytes from the wire hold whatever the source sent. Validate at the edge with Zod and use safeParse, not parse, so a bad row returns a result you can route instead of a thrown error that kills the chunk.

import { z } from "zod";

const ProductSnapshot = z.object({
  asin: z.string().regex(/^[A-Z0-9]{10}$/),
  marketplace: z.enum(["US", "DE", "JP", "UK"]),
  capturedAt: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
  contractVersion: z.string().default("2026-09-01"),
  title: z.string().min(1),
  price: z.number().positive().nullish(),
  currency: z.string().length(3).nullish(),
});

const P0_FIELDS = ["title", "price", "currency"] as const;

function assertP0(row: unknown) {
  const parsed = ProductSnapshot.safeParse(row);
  if (!parsed.success) {
    throw new Error(`contract violation: ${parsed.error.message}`);
  }
  for (const field of P0_FIELDS) {
    const value = parsed.data[field];
    if (value === null || value === undefined || value === "") {
      throw new Error(`P0 field ${field} missing on ${parsed.data.asin}`);
    }
  }
  return parsed.data;
}

Mark title, price, and currency as P0. A row that passes the shape check but drops a P0 field is still a failure for the report. Catch it at the edge, log it, and skip the row. Do not let it reach the upsert as a silent null.

This pattern moves the contract from a comment in your type file to a check that runs on every row. When the source changes a field from a number to a string, the safeParse result tells you which block broke and which ASIN, instead of a cryptic cast error two modules away. The type gave you no warning; the runtime check does.

Per-phase timeouts and the Node error names

A single fetch fails in stages: connect, then headers, then body. Set a timeout per stage so a stalled body does not hold a socket forever while connect and headers were fine.

import { Agent, setGlobalDispatcher, TimeoutError } from "undici";

const dispatcher = new Agent({
  connect: { timeout: 5_000 },
  headersTimeout: 15_000,
  bodyTimeout: 30_000,
});
setGlobalDispatcher(dispatcher);

async function fetchProduct(url: string, asin: string) {
  try {
    const res = await fetch(url, { signal: AbortSignal.timeout(45_000) });
    return await res.json();
  } catch (err) {
    if (err instanceof TimeoutError) {
      log.warn({ asin, reason: "timeout" });
    } else if ((err as any).code === "UND_ERR_CONNECT_TIMEOUT") {
      log.warn({ asin, reason: "connect_timeout" });
    } else if ((err as any).code === "UND_ERR_HEADERS_TIMEOUT") {
      log.warn({ asin, reason: "headers_timeout" });
    } else if ((err as any).code === "UND_ERR_BODY_TIMEOUT") {
      log.warn({ asin, reason: "body_timeout" });
    } else if ((err as any).code === "ENOTFOUND" || (err as any).code === "EAI_AGAIN") {
      log.warn({ asin, reason: "dns" });
    } else {
      log.error({ asin, err });
    }
    throw err;
  }
}

Know the error names: UND_ERR_CONNECT_TIMEOUT, UND_ERR_HEADERS_TIMEOUT, UND_ERR_BODY_TIMEOUT from undici; TimeoutError (a DOMException) from AbortSignal.timeout; and ENOTFOUND or EAI_AGAIN for DNS. Routing on the name lets you retry the right failures and drop the rest. AbortSignal.any is available from Node 22 if you need to combine two signals.

Graceful shutdown keeps the rows you pulled

A kill signal should stop new work and flush what you already pulled. Catch SIGTERM, set a flag, and let in-flight requests finish before you close the pool.

let shuttingDown = false;
const pending: Promise<void>[] = [];

process.on("SIGTERM", () => {
  shuttingDown = true;
  log.info({ event: "sigterm", pending: pending.length });
});

async function run(asin: string) {
  if (shuttingDown) return;
  const row = await fetchProduct(buildUrl(asin), asin);
  if (shuttingDown) {
    await flush([row]); // write the rows you already pulled
    return;
  }
  await store(row);
}

async function flush(rows: unknown[]) {
  for (const row of rows) await store(row);
}

Add structured logs as JSON, not prose strings. A log line that reads { "event": "body_timeout", "asin": "B0X…", "stage": "body" } is queryable; a line that reads timeout on B0X is not. When the scheduled run fails at block 17, you want to count failures by stage in seconds, not read a wall of text.

A hard exit that skips the flush loses every row in flight. On a 30 million call per day scale, even a small loss rate is thousands of missing rows by morning. The flag plus the flush turns a crash into a clean pause you can resume from the checkpoint. The rows already pulled are safe; the rows still queued wait for the next run.

What one managed request already includes

All of the above is work you do because the raw call returns a 200 with gaps. A single Amazon Data API Node.js request wraps the parts that break:

Residential and mobile exit IPs with rotation.
TLS and HTTP/2 fingerprints that match a real browser.
Browser fingerprint coherence across the handshake, the headers, and the frames.
JavaScript rendering when the page needs it.
Challenge and intercept page handling with server-side retries.
Geo alignment so the exit region matches the marketplace you query.

One request, one price. None of these are add-ons. You send a request, you get a structured real-time JSON back. The fingerprint work, the HTTP/2 tuning, and the idempotency key stay on your side of the line, in the data contract rather than in your retry loop.

In production the service holds a median latency around 3 seconds, a success rate near 99 percent, and more than 30 million calls each day, with sponsored slot capture at 91.4 percent across 13 markets.

See the pricing for what one call costs across those markets. If you want the cost view after the scrapers are gone, we wrote a piece on what an Amazon data pipeline costs you.

Your Amazon Data API Returns 200 With Empty Fields: Eight Gaps to a Daily Job

Pangolinfo — Wed, 09 Sep 2026 02:09:32 +0000

A ten-line Python script fetches your first Amazon record. Getting that script to deliver clean data thirty days in a row needs eight more gaps closed. Most tutorials stop at the 200 response and four printed fields. The hard part starts there. The script breaks not from anti-bot code but from a field that vanishes while your code reports success, duplicate rows in batch two, and a time.sleep(3) that hides a 429 instead of surfacing it.

The 200-with-empty-fields case is the quiet one. Your dashboard renders, the row count climbs, and three weeks later someone notices the price column is blank for half the catalog. The status code never told you. A quality gate before the write is the only thing that catches it, and the vendor's invoice counts that empty row as a success.

Production asks a different set of questions that the request cannot answer. Did this batch cover the catalog you asked for yesterday. Do the fields for one product line up across two marketplaces. If this run fails, will a rerun write two different prices. When the invoice lands, can you say how many usable records each dollar bought. Those answers sit in the layer around the request, and the eight sections below follow that boundary: the first three cover whether you got the thing you asked for, the next three cover getting it on schedule, the last two cover whether the spend pays off and who finds out first. Hours saved on structure come back with interest during a data incident.

This post starts with the minimal version that breaks, then patches it one gap at a time. The code is runnable. If you want a vendor that returns structured JSON with rendering and IP cost inside one credit multiplier, the Amazon Scraper API notes cover field coverage.

The version that breaks in week three

Most tutorials end with this script:

import requests, time, random

def fetch(keyword):
    for page in range(1, 21):
        r = requests.get(
            "https://api.example.com/v1/amazon/search",
            params={"q": keyword, "page": page},
            headers={"Authorization": "Bearer KEY"},
        )
        rows = r.json().get("results", [])
        save(rows)                       # append to CSV
        time.sleep(random.uniform(1, 3)) # the "rate limit" fix

This version carries five silent failure modes. A page redesign returns 200 with None fields, and no alert fires. Search paging hits a ceiling and data just grows slower. A retry after failure keeps success rate at 99% while upstream health drops. The CSV appends forever and the business side complains about duplicates. None of these throws an error.

Each of those five modes costs real money. The duplicate rows mean you pay twice for one product and then pay again to clean the join. The missing field means a downstream model trains on silence. The retry loop means you burn quota on a request that will never succeed. The sleep that hides the 429 means the failure shows up only on the invoice. One time.sleep(random.uniform(1, 3)) makes the job slow. It does not hold a rate. With random delays, real QPS moves with worker count, and hitting the limit is a matter of time, not a controlled boundary.

Patch 1: timeouts, credentials, and retry in one class

First, collect the three things that scatter: timeout source, key location, retry owner. Scatter them and one night a job runs on a default timeout and hangs the scheduler.

import os, time, random, httpx

BASE_URL = os.environ["AMZ_API_BASE"]      # your vendor domain
API_KEY  = os.environ["AMZ_API_KEY"]       # credentials from env only

ENDPOINTS = {                               # endpoints in one place
    "product": "/v1/amazon/product",
    "search":  "/v1/amazon/search",
    "review":  "/v1/amazon/reviews",
}

TIMEOUT = httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=10.0)

class AmazonDataClient:
    def __init__(self, base_url: str = BASE_URL, key: str = API_KEY) -> None:
        self._http = httpx.Client(
            base_url=base_url,
            timeout=TIMEOUT,
            headers={
                "Authorization": f"Bearer {key}",
                "User-Agent": "amz-pipeline/1.0 (+ops@example.com)",
            },
        )

    def get_json(self, endpoint: str, params: dict, attempts: int = 3) -> dict:
        path = ENDPOINTS[endpoint]
        last = None
        for attempt in range(1, attempts + 1):
            try:
                resp = self._http.get(path, params=params)
            except httpx.TransportError as exc:      # transport layer, retryable
                last = exc
            else:
                if resp.status_code < 400:
                    return resp.json()
                if resp.status_code not in (429, 500, 502, 503, 504):
                    raise ValueError(f"{endpoint} rejected: {resp.status_code}")
                last = ValueError(f"HTTP {resp.status_code}")
            time.sleep(min(2 ** attempt, 30) + random.uniform(0, 1))   # backoff + jitter
        raise RuntimeError(f"{endpoint} failed after {attempts} attempts") from last

Timeouts split into connect, read, write, and pool; read gets 30 seconds because rendering endpoints wait on async loads. Jitter follows the backoff so concurrent tasks do not retry in the same second. The User-Agent carries a contact address so the vendor can reach a human before a ban.

A default timeout of zero or None means the request can hang until the OS cuts it, which can be minutes. One hung request inside a sequential loop blocks every later keyword. Putting the timeout in one place keeps that risk in one reviewable spot. The same logic applies to the key: reading it from an environment variable means a leaked log line never exposes the secret, and rotation needs no code change. The backoff caps at 30 seconds because a longer wait only delays the failure signal.

Patch 2 and 3: field contract, P0 fields, and the quality gate

Do not write the raw dict straight into storage. Put a model between them that maps vendor names to your internal names, asserts non-empty P0 fields, and invalidates old data on version moves.

from dataclasses import dataclass
from datetime import date
from typing import Optional

CONTRACT_VERSION = "2026-09-01"      # any field change bumps this

@dataclass(frozen=True)
class ProductRow:
    contract_version: str
    asin: str
    marketplace: str
    captured_at: date
    title: Optional[str]
    price: Optional[float]
    currency: Optional[str]
    rating: Optional[float]
    review_count: Optional[int]
    is_sponsored_slot: bool

P0_FIELDS = ("title", "price", "currency")    # missing any one, the row is unusable

def quality_gate(rows: list[ProductRow], min_fill: float = 0.95) -> dict:
    n = len(rows) or 1
    report = {}
    for field in P0_FIELDS:
        filled = sum(1 for r in rows if getattr(r, field) is not None)
        report[field] = round(filled / n, 4)
    report["usable"] = all(report[f] >= min_fill for f in P0_FIELDS)
    return report

# gate = quality_gate(rows)
# {'title': 1.0, 'price': 0.61, 'currency': 1.0, 'usable': False}
# price at 0.61 means stop everything and investigate, not write more rows

P0_FIELDS turns "is this data good" from a judgement call into a CI assertion. Once contract_version enters the primary key, a contract bump triggers a history rerun on its own, with no risk of two contracts mixing inside one table. Your vendor does not set the P0 list; your business does. For a field-by-field method of picking a vendor, the Amazon Scraper API notes walk the comparison.

Contract version deserves its own sentence. When the vendor adds a field or renames one, your old rows keep the old version string. A query that asks for version 2026-09-01 never mixes with 2026-10-01, so a schema change becomes a rerun of the affected date range, not a table-wide migration. The P0 list is yours because only you know which fields your pricing model refuses to accept null.

Run this gate before the write, not inside the dashboard. Reverse the order and dirty rows are committed, which multiplies cleanup cost. The vendor's billing system books a 200 as revenue; your report books an empty field as a gap. Those two ledgers never reconcile, so the third definition lives in your pipeline. The credit multipliers on the pricing page give you part of the numerator. The denominator exists only in your logs.

Picture a batch where price fills at 0.61. The products are there, the titles are there, the currency is there, and the report looks alive. But 39% of rows have no price, and your repricer will treat null as zero or skip the row. The gate returns usable: False, and that False should stop the write. Treat the threshold as a hard wall, not a warning you scroll past.

Patch 4: paging, dedupe key, and four failure classes

Set the key first: (asin, marketplace, captured_at, contract_version). That tuple singles out one row, so a rerun replaces rather than appends. Discussing paging before the key wastes the conversation.

def collect_search(client, keyword, marketplace, max_pages=20):
    seen, rows = set(), []
    for page in range(1, max_pages + 1):
        payload = client.get_json("search", {"q": keyword, "page": page, "marketplace": marketplace})
        items = payload.get("results", [])
        if not items:
            break
        new = 0
        for raw in items:
            item = to_row(raw, marketplace, date.today())
            key = (item.asin, item.marketplace, item.captured_at, item.contract_version)
            if key in seen:
                continue
            seen.add(key)
            rows.append(item)
            new += 1
        if new == 0:                 # a page of nothing but repeats means you are done
            break
    return rows

Paging carries two constraints. Depth has a ceiling: one query does not page past about twenty, so reaching further means splitting by category, price band, or brand into parallel subtasks. Cross-page repetition is the norm: the same product surfaces on adjacent pages, above all among organic results outside the ad blocks.

The dedupe ratio is also your first signal that paging broke. A healthy run sits near 3%. If it jumps to 40% in one day, suspect your paging parameter before you suspect Amazon. The same key that drives dedupe drives the cost math two sections down, because records_per_call feeds the denominator.

Split failures into four classes: transient (connect timeout, 502/503/504, retry with backoff), rate limited (429 with Retry-After, wait the header's seconds), contract error (400, missing parameter, expired key, fail and alert, no retry), content missing (200 with empty P0 fields, block before write, dead letter). Class three should stop the whole batch because retrying spends quota and changes nothing. Class four belongs in the quality gate, never in a status code. Log the attempts count because failed attempts are billable at most vendors and show up on the invoice.

Patch 5: semaphore, snapshot storage, and the cost meter

Replace random sleep with a semaphore. Concurrency starts at four; raise it while watching the 429 ratio against end-to-end latency. Async buys the ability to do other work while waiting; it does not create capacity.

import asyncio, httpx

async def fetch_all(client, jobs, concurrency=8):
    sem = asyncio.Semaphore(concurrency)
    results = []
    async def one(job):
        async with sem:
            for attempt in range(1, 4):
                try:
                    r = await client.get(ENDPOINTS[job["endpoint"]], params=job["params"])
                except httpx.TransportError:
                    await asyncio.sleep(2 ** attempt)
                    continue
                if r.status_code == 429:
                    await asyncio.sleep(int(r.headers.get("Retry-After", 5)))
                    continue
                if r.status_code < 400:
                    results.append((job, r.json()))
                    return
                if r.status_code not in (500, 502, 503, 504):
                    raise RuntimeError(f"job {job} rejected: {r.status_code}")
            raise RuntimeError(f"job {job} exhausted retries")
    await asyncio.gather(*[one(j) for j in jobs])
    return results

class MeteredClient(AmazonDataClient):
    def __init__(self, *a, **kw):
        super().__init__(*a, **kw)
        self.calls = self.usable_rows = self.credits = 0

    def get_json(self, endpoint, params, attempts=3):
        self.calls += 1
        data = super().get_json(endpoint, params, attempts)
        self.credits += data.get("_meta", {}).get("credits", 1)
        return data

    def cost_per_1k_usable(self, usable_rows: int) -> float:
        if not usable_rows:
            return float("inf")
        return self.credits * 0.0015375 * 1000 / usable_rows   # per-credit price on the pricing page

A semaphore caps concurrent in-flight requests at a number you choose. Random sleep caps nothing; it only adds latency. With a semaphore at eight and an upstream grant of ten per second, you stay under the line and finish faster than a sleep-based loop that wastes the gap between retries. Store snapshots, not overwrites: INSERT OR REPLACE with the four-key primary key means a same-day rerun replaces the same rows and leaves no half-old state. Change detection reduces to one window function over adjacent snapshots.

The snapshot table carries the same four columns as the key, plus the fields you track: title, price, currency, rating, review count, and the sponsored flag. Primary key on all four key columns means a same-day rerun is an upsert, never an append. History accumulates as new capture dates arrive, so a price-change report is a self-join on asin and marketplace across two capture dates. The captured_at column is what turns a flat price dump into a time series you can plot months later.

The numerator is credits consumed. The denominator is records that cleared the gate, not rows that arrived. The distance between those two numbers is duplicates, empty fields, and retried requests. The cost meter closes the loop: 0.0015375 per credit times credits consumed, divided by usable rows, gives cost per thousand usable records. If that number climbs, the cause is in your call shape, not the vendor's price.

Four metrics, two alerts, one pre-launch check

Four metrics suffice, all pulled from the code above: P0 field fill rate, average attempts, cross-page repeat rate, cost per 1K usable records. Two alerts carry the load. One fires when any P0 field drops below its threshold. The other fires when cost per thousand usable records rises more than 30% month over month. The first is a data incident; the second is a budget incident. Both reach your logs ahead of any complaint.

P0 fill rate comes straight from the gate. Average attempts comes from the retry loop and shows upstream health: a rise means the vendor's success rate fell and your bill rose with it. Cross-page repeat rate comes from the seen set and points at a broken paging parameter. Cost per 1K usable comes from the meter and flags a usage shift. One alert per axis keeps the noise down; two alerts cover the two failure classes that cost you the most.

Before shipping all of it, run one five-line check: are timeouts and retries inside a single class; does every P0 field have a hard assertion; does the key carry capture date and contract version; are failures split into four classes rather than one except; does each call count requests and credits. Miss one line and the reports drift from week three onward.

Outsourcing collection does not remove your ownership of the contract, dedupe, and snapshot layers. For what you still own after outsourcing collection, the breakdown shows the parts that stay on your side.

What stays on your side after you pick a vendor

Picking a route is picking how much code stays yours. Official SP-API serves only your own catalog. A raw requests-plus-BeautifulSoup stack leaves anti-bot, rendering, and retries to you. A dedicated Amazon data API moves parsing and anti-bot upstream but leaves contract, dedupe, storage, and telemetry on your side. That split is why this post spends its length on the parts no vendor writes for you.

Pangolinfo carries the part that ends before the request class: REST endpoints return structured JSON for products, search, reviews, Sponsored slots, and Alexa modules, with rendering and IP cost inside one per-endpoint credit multiplier instead of a separate line for rendering or rotating residential proxies. Published numbers: median latency near three seconds, 99% success, over 30 million calls per day, Sponsored placement coverage at 91.4% across 13 marketplaces. Those figures feed your numerator. They replace nothing in the denominator.

A team tracking forty ASINs still needs three parts of this structure: request-layer timeouts and retries, P0 checks, and a date-keyed snapshot table. That is about sixty lines and it buys rerun safety plus a three-month price curve. Skip the concurrency layer until volume forces it.

The gap between a quote and the invoice is worth a separate read: the multiple between a quoted rate and the final invoice walks the math. And the cost items that remain on your side once collection moves upstream lists what you keep paying for after collection moves to a vendor. Run the whole structure on the free tier first: 60 requests after signup, no card required.

Appendix: the four gates, and what one request includes

Everything above covers what happens after data arrives. One question sits before it: whether the request counts as normal traffic at all. Four gates decide that, in this order: TLS and HTTP/2 fingerprints, browser fingerprint coherence, IP type and reputation, and obfuscation of cadence and sessions.

Gate one catches most teams. The decision happens during the TLS handshake, not at the HTTP layer. requests builds its ClientHello through urllib3 and OpenSSL, and the resulting cipher and extension order matches no Chrome release, so defenses match it against known client fingerprints and hit. That is why a new User-Agent, a new proxy, and a lower rate change nothing. Change the client:

from curl_cffi import requests
r = requests.get("https://www.amazon.com/dp/B08N5WRWNW",
                 impersonate="chrome",                     # matching UA and header order included
                 headers={"Accept-Language": "en-US,en;q=0.9"},
                 timeout=25)

Do not override the User-Agent, since the profile supplies a matching one along with the Sec-Fetch-* family and header order, and match Accept-Language to the storefront, de-DE for amazon.de. Verify against a public fingerprint endpoint instead of treating a 200 response as proof.

Gate two exists only if you run a browser. The tells are navigator.webdriver, Canvas and WebGL, the font list, screen and pixel ratio, and timezone against language. Keep those five from one profile. A Windows User-Agent with a UTC timezone reads worse than an older build.

Gate three is the IP. Datacenter ranges are classified as server traffic at the ASN level, with success rates of 10–40% on protected surfaces. Residential comes from ISPs and mobile from carrier CGNAT, both far higher. Use residential or mobile for product, search, and review pages. Datacenter is fine for unprotected APIs you own and for storefronts with light defenses.

Gate four is cadence. A fixed interval and a uniform random draw look the same in a time series. Human gaps are long-tailed and arrive in clusters, so requests per IP per minute, requests per session, and job start times matter more than the interval itself.

Four gates plus eight gaps is the whole self-hosted workload. The Pangolinfo Amazon Scraper API closes the gates server-side: residential and mobile exits with rotation, TLS and HTTP/2 fingerprints, browser fingerprint coherence, JavaScript rendering where an endpoint needs it, challenge handling and retries, and geo and postal-code alignment all sit inside the price of one request, with no residential surcharge and no render multiple. You send one request and receive structured JSON. Rates and plans are on the pricing page.

The 4.9x Gap Between Amazon Data API Quotes and Invoices

Pangolinfo — Tue, 08 Sep 2026 04:39:24 +0000

The unit on an Amazon data API pricing page is almost never the unit you buy. The quote shows a price per call, per thousand results, or per credit. What your budget needs is the record that lands in your warehouse, passes your quality gate, and shows up in a report. Five multipliers sit between those two numbers: attempt count, the usable share of successful responses, records per call, polling redundancy, and the allowance you buy and cannot spend. This article turns those five multipliers into one formula you can recompute, breaks them down across three volume scenarios, shows three runnable Python blocks you can paste into your own stack, and gives you seven questions to get in writing before you sign.

You have lived this gap. The pricing page says three hundred dollars a month. The first invoice reads seven hundred. You walk back to the vendor and every line on the invoice matches a request you sent. The vendor made no error. The error is in the unit: you estimated a cost per usable record from a price per call, and no equal sign has ever connected those two numbers.

Fixing the unit needs no new vendor and no better negotiator. It needs five multipliers in your budget table. This article gives you those five multipliers, one formula, a calculator you can run, a logging harness that measures each input from your own traffic, a retry policy that will not inflate the bill, and an idempotency design that keeps retries from billing twice. You supply the parameters from your own system and reach your own conclusion.

What Sits Between the Quote and the Record You Pay For?

An Amazon data API pricing page gives you a unit price. The unit is a call, a result, a credit, a gigabyte, or a rate allowance. Your budget needs a total, and the unit is a record you can use. The distance between them is not one division. It is five multipliers:

Attempt count — how many requests you send to get one success. This depends on the endpoint failure rate, your retry policy, and whether the failed attempt is billed at all.
The usable share of successful responses — the vendor defines success on the invoice. Your quality gate defines success in your warehouse. A 200 response with empty business fields is revenue to the billing system and a gap in your report.
Usable rate is not a number you estimate. Your quality gate has to run as code before it produces one, and the gate, the dedupe rules, and the field mapping stay on your side after you move to a data API. That maintenance load, broken down layer by layer.
Records per call — a product detail endpoint returns one record per call. Search and review endpoints return pages, and the rows per page that survive into your tables depend on pagination parameters, cross-page deduplication, and ad placements mixed into results.
Polling redundancy — six polls a day against 1.2 changes a day means 80% of your calls buy a row that did not change.
Allowance you buy and cannot spend — the monthly total works out on paper. The per-minute refill rate does not. The overnight batch does not finish. The remainder expires at the end of the month.

All five multipliers consume budget. One appears on the pricing page. Of the rest, two hide in the contract (failure billing rules and how a result page is counted) and two hide in your own architecture (retry policy and polling cadence). That is why two companies on the same vendor and the same unit price can land on monthly bills a factor of two apart.

Stop asking what a thousand calls costs. Ask what you pay for a thousand records that pass your quality gate, and how much of that number is engineering hours nobody invoices you for.

Who Lands on This Page, and What Do They Need?

The query reads like a price check. Three groups land on the same page looking for different things. Their situations differ, and the missing piece is the same: a way to convert a quote into their own invoice.

Group one: quotes in hand, budget to build. They hold two quotes. One bills per call, one bills per result, and the prices look three times apart with no common denominator. They do not need a verdict on which vendor is cheaper. They need a formula they can fill in and get a comparable number. Give this group the formula and they leave satisfied.

Group two: invoice over forecast, cause unknown. Three months in, the bill moved from a projected two thousand to five thousand, and request volume did not move. The cause sits in retry amplification and polling cadence: the client retried three times, the server retried once, and the failed attempts were billable. This group needs a triage checklist that walks the multipliers one at a time.

Group three: choosing a vendor for next year. No data yet, writing the evaluation doc. This group is the most exposed to entry pricing, because the entry price corresponds to the simplest endpoint, the lowest multiplier, one marketplace, no rendering. They need the list of contract terms a pricing page leaves out.

All three groups share the same gap. None lacks a price comparison table. All lack the ability to turn a cell in that table into their own number. So this article does not compare vendor prices. Those tables go stale in three months, and without your failure rate and usable rate nobody can fill in the cell you need. This article gives you the conversion. You supply the parameters and reach your own conclusion.

How Many Ways Can an Amazon Data API Bill You?

Amazon data services bill in about six ways. These are not six prices. They are six definitions of what you are paying for. Normalize them to one denominator before you compare, or you are comparing six unrelated things.

Per attempt. Every request you send is billable, whatever comes back. This is the least favorable structure for you, because failed, timed-out, and blocked attempts are yours to absorb. The unit price sits at the low end, and that is the trade: the vendor moved the risk to you and compensated in price. To judge this kind of quote you need your own attempt success rate, then the amplification factor 1 / success rate. A 95% success rate is a 1.05x factor. At 80% it is 1.25x. At 60% it is 1.67x. The curve is convex: each point you lose costs more than the point before it.

Per successful request. Billing applies to responses that succeed. That reads as a good deal for the buyer until you ask who defines success. Most billing systems record HTTP 200 as success, and a 200 carrying empty fields, a placeholder value, or a challenge page is a failure for you and revenue for them. Get two answers in writing: does a 200 with empty business fields count as success, and does a valid empty response, such as a delisted product, count as success. Those two answers move your invoice by 10% to 20%.

Per thousand results (CPM). Billed on the number of results returned. Public ranges for e-commerce targets run from about $0.45 to $1.60 per thousand, depending on target difficulty and output format. The distortion lives in the word "result." Is a result one page or one product row inside that page? Search endpoints bill by page, and rows per page follow your pagination parameters. If the docs say "up to N results per page" rather than "N results per page," you convert using your measured average, not the ceiling.

Per credit. Each endpoint carries a multiplier, and harder work costs more: one credit for a static page, 5 to 25 credits for rendered pages, residential proxies, postal-code targeting, or domains with heavy protection. This model is flexible, and its transparency depends on documentation quality. Do three things: pull the multiplier table for every endpoint you will use; confirm whether multipliers can be switched off per endpoint, so rendering applies to search while product detail stays static; and compute a weighted multiplier across your real endpoint mix rather than extrapolating from the endpoint behind the entry price.

Per gigabyte. Billed on bytes transferred, common among proxy products. The variable that matters is not the unit price but page weight. Public proxy cost analyses converge on two ranges: about 100 to 300 KB per page for optimized collection, and 1 to 3 MB per page for collection that runs a full browser render. Same data, an order of magnitude apart. If your vendor bills by traffic and your method depends on rendering, shrinking the payload beats negotiating the rate: block image and font loading, reuse connections, enable compression. Every one of those lands on the invoice.

Per rate allowance. You buy tokens per minute or concurrent connections. Tokens refill at a fixed rate and expire after a window. The trap is the gap between the monthly total and the instantaneous rate. One tier generates 892,800 tokens a month, which looks like enough, and refills at 20 tokens per minute with a bucket ceiling of 1,200. A nightly batch that needs 500,000 rows inside a two-hour window will not fit through that pipe, and you paid for the month.

The table below lists what each unit still requires before it becomes the same denominator. Those parameters have no default. You measure them.

Billing unit	What you pay for	Parameters still missing	Where it distorts
Per attempt	Requests sent	Attempt success rate, records per call, usable rate	Failures and retries are billed
Per successful request	Success as the vendor defines it	Definition of success, records per call, usable rate	200 with empty fields counts as success
Per thousand results	Result rows returned	Whether a result is a page or a row, net rows after dedup	Rows per page is a ceiling, not a guarantee
Per credit	Difficulty-weighted work	Multiplier per endpoint, whether it can be disabled per endpoint	Entry price reflects the lowest-multiplier endpoint
Per gigabyte	Bytes transferred	Average page weight, rendering method, compression	Rendered weight is 5 to 10 times bare HTML
Rate allowance	Quota per minute	Refill rate, bucket ceiling, token expiry rule	Monthly total is enough, the per-minute rate is not

Sources: unit definitions and public ranges come from vendor pricing documentation and 2026 industry benchmark roundups. Page weight ranges come from published proxy cost analyses. This article names no vendor and quotes no vendor's price.

Why Does the Pricing Page Leave These Out?

Not to hide anything. Those four parameters cannot be measured on the vendor side. Attempt success rate depends on your target set, your concurrency pattern, and your retry policy. Usable rate depends on your quality gate and your field list. Records per call depend on your pagination parameters and dedup rules. Wasted allowance depends on your scheduling window. All four live on your side of the line. What a vendor can give you is a recomputable unit price and a multiplier table, and that is what you should require on the pricing page.

So the order of operations is: confirm that unit price and multipliers are published and recomputable, measure the four parameters on your own system, then run the formula. Skip the second step and the ranking you get from the price table means nothing.

What Are the Five Multipliers From One Call to One Usable Record?

This section takes the five multipliers apart, one at a time, with arithmetic you can apply and the order of magnitude each one moves.

Does a Failed Attempt Cost You Money?

Let s be the success rate of a single attempt and assume you retry until success. The average number of attempts behind one success is 1/s. Whether the failed attempts reach your invoice is the coefficient β: β=0 means failures are not billed or are refunded, β=1 means failures bill at full price. The surcharge on your invoice is β·(1−s)/s.

The table below holds every other parameter constant (usable rate 0.9, wasted allowance 0.1, unit price $0.0015375 per credit, $1.5375 per thousand) and varies the failure billing rule:

Attempt success rate s	Failures not billed (β=0)	Failures billed in full (β=1)	Difference
99%	$1.898 / 1K	$1.917 / 1K	+1%
97%	$1.898 / 1K	$1.957 / 1K	+3%
95%	$1.898 / 1K	$1.998 / 1K	+5%
90%	$1.898 / 1K	$2.109 / 1K	+11%
80%	$1.898 / 1K	$2.373 / 1K	+25%

The finding: failure billing is a footnote at high success rates and a decision-changer at low ones. Amazon sits among the most protected targets on the web, so your success rate is not in the 99% row. Count a week of your own logs. That is faster than arguing the clause.

Why Does a 200 Response Fail Your Quality Gate?

This is the most expensive of the five multipliers and the one least present in a budget. The vendor's billing system judges success by HTTP status and response shape. Your quality gate judges success by your business rules. The distance between the two is your usable rate, v.

Here is one concrete form. A product page returns a price field, and the value is a placeholder: the parent variant's price rather than the selected variant's. The type is valid, the format is correct, schema validation waves it through, and your warehouse gains a record with the wrong price. Or the review endpoint returns a 200 and a well-formed page of JSON whose reviews array is empty, because that ASIN was not collected that day. The billing system books a success.

The amplification is 1/v, and it is convex:

Usable rate v	Cost per 1K usable records	Multiple vs v = 1.0
1.00	$1.798	1.00x
0.95	$1.893	1.05x
0.90	$1.998	1.11x
0.85	$2.116	1.18x
0.80	$2.248	1.25x
0.70	$2.569	1.43x
0.60	$2.997	1.67x
0.50	$3.597	2.00x

A solution with a usable rate under 0.70 does not win on cost, whatever its unit price. And the usable rate is a number that will never appear on a pricing page, because it is not the vendor's obligation. It lives in your quality gate. How to measure it, and how to build the four quality gate numbers that sit behind it, is the subject of our write-up on data pipelines, where collection can be outsourced but data quality cannot.

Is One Call One Record?

Product detail endpoints return one record per call. Search, bestseller, and review endpoints return pages, and three things erode the count. Pagination ceilings: the docs say "up to N per page" and the real number is often lower. Cross-page duplication: the same ASIN appears on two adjacent pages, and deduplication drops your net row count. Ad placements: paid slots occupy part of a search page, so if your target is organic results, fewer rows on that page are usable.

There is one conversion: run 200 real queries, count net rows written to your tables divided by calls made, and you have your measured r. Extrapolating from the documented ceiling is the most common error in a budget table and can be off by 1.5x.

Are You Paying for Changes or for Polls?

You poll to detect change, and you pay for calls, not changes. The ratio between them is the redundancy factor ρ = daily polls ÷ daily changes.

Daily polls	Daily changes	Redundancy	Calls that carried no new information
6	1.2	5.0x	80%
4	1.2	3.3x	70%
2	1.2	1.7x	40%
12	3.0	4.0x	75%
24	3.0	8.0x	88%

This redundancy is not waste to be eliminated: if you stop polling, you stop knowing. What you can optimize is its distribution. Tiered polling puts P0 fields on a four-hour cycle, P1 on a daily cycle, and P2 on a weekly cycle. Change-driven collection replaces full rescans with endpoints that surface price drops and rank movement. The first concentrates high-frequency budget on the fields that are sensitive to time; the second moves the cost of detecting change from full volume to incremental volume. Together they remove more than half of the calls that carry no new information, without losing detection latency.

Why Did You Buy Allowance You Cannot Spend?

The waste coefficient w has three sources. Rate shaping: the monthly allowance is sufficient and the per-minute refill rate is not, so the batch window closes before the work is done. Expiry: tokens expire 60 minutes after generation, so allowance a nightly batch leaves on the table does not roll into daytime work. Concurrency ceilings: you bought a high-throughput tier and never raised client concurrency, so half the allowance goes unused.

All three produce the same signal: the invoice is billed on allowance, rows land on consumption, and the ratio stays above 1. Track this number once a month. It is the one multiplier where no party made an error and you lost money anyway, and the one most open to recovery through scheduling.

What Is the Formula From Unit Price to Cost per Usable Record?

Combine the five multipliers and you get one expression from quoted unit price to cost per thousand usable records.

Which Variables Feed the Formula?

Symbol	Meaning	Where it comes from
`p`	Quoted unit price	Pricing page, mind the tier
`m`	Billing multiplier (rendering, proxy, postal code, endpoint difficulty)	Endpoint multiplier table, weighted across your mix
`s`	Success rate of one attempt	Your own call logs, one week
`β`	Billing share of failed attempts (0 to 1)	The contract, not the pricing page
`r`	Net records per successful call	200 measured queries
`v`	Share of successes passing your quality gate	Quality gate telemetry
`w`	Share of allowance bought and unspent	Billed allowance ÷ rows landed
`E`	Monthly engineering cost (parsing, QA, maintenance hours × loaded rate)	Time tracking

Of the seven parameters, p and m come from the pricing page. The other five come from your system. That is the reason a pricing page cannot answer "what will this cost": it supplies two of seven.

How Is the Formula Written?

Data cost per thousand usable records

    C = 1000 · p · m · [ 1 + β·(1 − s)/s ]
        ───────────────────────────────────
              r · v · (1 − w)

True unit cost including engineering

    C_tco = C + 1000 · E / U

    U = usable records needed per month

Can You Run the Calculator Yourself?

Save this as unit_cost.py and fill in your own parameters. It prints the data cost, the engineering cost, and the engineering share. The numbers in the if __name__ == "__main__" block use one published tier so you can reproduce the next section's scenario B line by line; swap in your own values to model your traffic.

def cost_per_1k(p, m=1.0, s=1.0, v=1.0, r=1.0, beta=0.0, w=0.0):
    """Data cost per thousand usable records.

    p     quoted price per billing unit
    m     weighted endpoint multiplier
    s     success rate of one attempt
    v     usable share of successful responses
    r     net records per successful call
    beta  billed share of failed attempts, 0 means not billed
    w     share of allowance bought and unspent
    """
    # attempts: how many sends stand behind one usable success.
    # beta=0 -> failures free; beta=1 -> every failed send is billed.
    attempts = 1 + beta * (1 - s) / s
    # usable_per_call: rows that clear the gate and are not wasted allowance.
    usable_per_call = r * v * (1 - w)
    return 1000.0 * p * m * attempts / usable_per_call


def tco_per_1k(p, monthly_records, engineering_cost, **kw):
    """True unit cost including engineering hours."""
    c = cost_per_1k(p, **kw)
    return c + 1000.0 * engineering_cost / monthly_records


if __name__ == "__main__":
    # One published tier: Expert $369 / 240,000 credits.
    P = 369 / 240_000                     # $0.0015375 per credit
    U = 500_000                           # 500K usable records per month
    E = 30 * 90                           # 30 engineering hours, $90 loaded

    # Three views of the same vendor, same unit price, same month.
    ideal = cost_per_1k(P)
    real  = cost_per_1k(P, m=1.0, s=0.95, v=0.88, r=1.0, beta=1.0, w=0.12)
    tco   = tco_per_1k(P, U, E, m=1.0, s=0.95, v=0.88, r=1.0, beta=1.0, w=0.12)

    print("pricing page      $%.4f / 1K   monthly $%.2f" % (ideal, ideal * U / 1000))
    print("five multipliers  $%.4f / 1K   monthly $%.2f" % (real, real * U / 1000))
    print("with engineering  $%.4f / 1K   monthly $%.2f" % (tco, tco * U / 1000))
    print("engineering share %.0f%%" % (100 * E / (tco * U / 1000)))

Output:

pricing page      $1.5375 / 1K   monthly $768.75
five multipliers  $2.0899 / 1K   monthly $1044.95
with engineering  $7.4899 / 1K   monthly $3744.95
engineering share 72%

A factor of 4.9 separates the first and last line, and all three describe the same vendor, the same unit price, and the same month. Everything between them is unit conversion.

What Is Each of the Three Lines For?

The first line, pricing page terms, serves shortlisting. Before you have measured your parameters, use it to eliminate options in the wrong order of magnitude. Do not put it in a budget. The second line, after the five multipliers, is for budgeting and reconciliation. It is the expected value of next month's invoice, and it should land within 15%. A wider gap means a parameter is wrong, most often v or w. The third line, with engineering, is for decisions. It alone answers build versus buy or stay versus switch, because the first two compare invoices and a decision compares total cost.

Putting all three lines side by side in an evaluation doc has a second effect. It moves the review meeting from "is this vendor expensive" to "why is our usable rate 0.85," and the second question has an answer.

Can You Instrument Your Logs to Measure s, v, r, and w?

The formula is useless until you feed it numbers from your own traffic. Most teams guess s, v, r, and w from memory and then wonder why the forecast misses. The fix is to log every call once, at the boundary where the response meets your system, and derive the four inputs from that ledger. The block below is a drop-in CallLedger you can call from your API client. It tallies attempts, successful HTTP responses, gate passes, net rows written, and credits spent, then returns the four inputs the formula needs.

class CallLedger:
    """Record the four formula inputs from live call logs.

    Call .log() once per request at the point where the response
    returns to your code, before any retry loop hides the attempt.
    """

    def __init__(self):
        self.attempts = 0        # every request sent, retries included
        self.oks = 0             # responses with a usable HTTP status
        self.gate_pass = 0       # of the oks, those your quality gate kept
        self.rows = 0            # net records written from gate_pass calls
        self.credits_bought = 0  # allowance you paid for this period
        self.credits_used = 0    # allowance consumed in this period

    def log(self, http_ok, passed_gate, net_rows, credits):
        self.attempts += 1
        self.credits_used += credits
        if http_ok:
            self.oks += 1
            if passed_gate:
                self.gate_pass += 1
                self.rows += net_rows

    def metrics(self):
        s = self.oks / self.attempts if self.attempts else 0.0        # success rate
        v = self.gate_pass / self.oks if self.oks else 0.0            # usable rate
        r = self.rows / self.gate_pass if self.gate_pass else 0.0     # records per call
        w = 1 - (self.credits_used / self.credits_bought) if self.credits_bought else 0.0
        return {"s": s, "v": v, "r": r, "w": w}


# Usage in a fetch loop:
#   ledger = CallLedger()
#   ledger.credits_bought = 240_000
#   for resp in responses:
#       ledger.log(resp.http_ok, resp.passed_gate, resp.net_rows, resp.credits)
#   m = ledger.metrics()   # hand m["s"], m["v"], m["r"], m["w"] to cost_per_1k

Wire http_ok to the status check your vendor uses for billing, not to your own gate. Wire passed_gate to the same function that decides whether a row reaches your warehouse. The split between those two is the usable-rate gap the second multiplier measures. Run this for one week and you replace four guesses with four measured numbers.

How Do You Build a Retry Policy That Does Not Inflate the Bill?

Retries are where the attempt-count multiplier turns from a rounding error into a budget line. A naive retry loop sends until it succeeds and never caps the wait, so a struggling endpoint produces both a storm of billable attempts and a long tail of latency. The block below uses exponential backoff with full jitter and a hard ceiling, and it stops after max_attempts so a dead endpoint cannot bill you forever.

import random
import time


class TransientError(Exception):
    """Raised for errors worth retrying (timeout, 429, 5xx)."""


def call_with_retry(fn, max_attempts=5, base=0.5, ceiling=30.0):
    """Call fn with exponential backoff, full jitter, and a hard ceiling.

    max_attempts: stop after this many tries; the last try is not retried.
    base:         first backoff in seconds.
    ceiling:      longest single wait, in seconds.
    """
    last_err = None
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()                      # success: return the result
        except TransientError as err:
            last_err = err
            if attempt == max_attempts:
                break                        # no point waiting after the last try
            # exponential growth, capped at the ceiling...
            wait = min(ceiling, base * (2 ** (attempt - 1)))
            # ...then jittered across [0, wait] so peers do not sync up.
            wait = random.uniform(0, wait)
            time.sleep(wait)
    raise last_err

Two details matter for cost. First, the ceiling bounds the worst-case latency, which bounds how many retries a single logical fetch can trigger inside one poll window. Second, jitter spreads retries across time so you do not hammer a rate-limited endpoint in lockstep with every other client, which would push your success rate s down and raise the billing multiplier 1/s. Pair this with the idempotency design below so the retries that do happen never bill twice.

How Do You Stop Retries From Double-Billing?

A retry is the same logical fetch sent twice. If the server bills per attempt and your client retries, the second send can land on the invoice even though it produced the same row. The clean fix is an idempotency key: a stable fingerprint of the work, derived from its business identity rather than from request time, attached to every retry of that work. The server records the key and, on a repeat, returns the cached outcome without a second charge.

import hashlib


def idempotency_key(marketplace, asin, field_set, day_bucket):
    """Stable key so a retry counts once on the invoice, never twice.

    Build it from the business identity of the work, not from request
    time, so the same logical fetch yields the same key across retries.
    """
    raw = f"{marketplace}:{asin}:{sorted(field_set)}:{day_bucket}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]


# Client side: attach the key to every retry of the same logical call.
key = idempotency_key("US", "B0XXXXXX", {"price", "bsr"}, "2026-09-07")

# Server side: if the key was seen, return the cached outcome and do
# not bill a second time. Your retry policy and the invoice then agree.

This design keeps the attempt-count multiplier honest. Your client can retry for resilience without the billing system counting each retry as a fresh purchase. Note the key is scoped to a day bucket: a row you fetch today and the same row you fetch tomorrow are different work and should bill twice, while two sends of today's fetch are one purchase. Scoping the key to the right grain is the whole trick.

What Does This Look Like at 50K, 500K, and 5M Records?

The parameters below follow common Amazon data project shapes, not any vendor's quote. The unit price is one published Expert tier ($369 for 240,000 credits, $0.0015375 per credit), so you can recompute every figure. The three scenarios show where the leverage moves as volume grows, and that shift is the point of the whole exercise.

Scenario A: 50,000 Records a Month, One Marketplace, One Category

Shape: price and BSR monitoring for one category in one marketplace, one full pass per day. Parameters: r=1 (product detail, one record per call), m=1 (structured JSON), s=0.97, v=0.95, β=1, w=0.05, 8 engineering hours per month.

Basis	Per 1K usable records	Monthly data cost	Monthly engineering	Monthly total
Pricing page	$1.5375	$76.88	$720	$796.88
After five multipliers	$1.7563	$87.81	$720	$807.81

At this volume the data invoice is 11% of total cost and engineering is 89%. Spending three weeks negotiating the unit price has an upside of eleven dollars a month. Writing the retry policy and automating the quality gate is worth several hundred. At low volume, optimize engineering hours, not unit price.

Scenario B: 500,000 Records a Month, Five Marketplaces

Shape: price and BSR monitoring across five marketplaces, twice a day. Parameters: r=1, m=1, s=0.95, v=0.88, β=1, w=0.12, 30 engineering hours per month.

Basis	Per 1K usable records	Monthly data cost	Monthly engineering	Monthly total
Pricing page	$1.5375	$768.75	$2,700	$3,468.75
After five multipliers	$2.0899	$1,044.95	$2,700	$3,744.95

The five multipliers raise the data cost by 36%, and engineering is still 72% of the total. Two actions pay best here: push w from 0.12 to 0.05 through scheduling and concurrency, worth $73 a month, and lift v from 0.88 to 0.95 by moving the quality gate forward, worth $78 a month. That is $151 a month, more than the spread between tiers at this volume.

Scenario C: 5,000,000 Records a Month, 13 Marketplaces With Reviews

Shape: 13 marketplaces, 60% product detail and 40% reviews, near-real-time monitoring. Weighted multiplier m = 0.6×1.0 + 0.4×(5/8) = 0.85, where a review page costs 5 credits and returns 8 rows. Parameters: s=0.94, v=0.85, β=1, w=0.18, 90 engineering hours per month.

Basis	Per 1K usable records	Monthly data cost	Monthly engineering	Monthly total
Pricing page	$1.3069	$6,534.38	$8,100	$14,634.38
After five multipliers	$1.9947	$9,973.40	$8,100	$18,073.40
With w at 0.05	$1.7217	$8,608.62	$8,100	$16,708.62

Here the data cost overtakes engineering for the first time, so negotiating the unit price starts to matter. The order still holds: push w from 0.18 to 0.05 first, worth $1,365 a month, then negotiate. A discount applied to an optimized denominator is a discount you keep.

Where Does the Leverage Move Across the Three?

Scenario	Monthly data invoice	Monthly engineering	Engineering share	What to optimize
A: 50K records	$87.81	$720	89%	Hours: retry policy, quality gate automation
B: 500K records	$1,044.95	$2,700	72%	Parameters: w and v, then hours
C: 5M records	$9,973.40	$8,100	45%	Push w down first, then tier and multiplier

The difference across the three is not the amount. It is where the leverage sits. In scenario A, negotiating the unit price caps out at eleven dollars a month while cutting engineering hours is worth hundreds. In scenario C, ten percent off the unit price and multipliers is close to a thousand dollars a month. Applying the scenario A playbook at scenario C scale, or the reverse, is pulling the wrong lever.

The engineering volumes (8, 30, and 90 hours per month) follow common project shapes, at a loaded rate of $90 an hour. Put in your own numbers and your conclusions may land in a different band, which is the reason to take the formula rather than the answers.

What Cost Never Appears on the Invoice?

Across the three scenarios, engineering is 45% to 89% of monthly total cost. No invoice carries it, and it decides whether the project survives. Before you argue a unit price, compute this number, because in two of the three scenarios it is the larger half of your bill. The build-versus-buy question lives here too: we wrote up how API access and page scraping compare as two routes you run in parallel rather than choose between, and the engineering line is where that decision is paid for.

Which Three Engineering Costs Should You Estimate?

Initial integration and field mapping: mapping response fields onto your data model, handling nested structures, unit conversion, multi-byte characters, and variant relationships. This cost is one-time and scales with field count and schema stability. More fields and a looser contract mean a larger bill. Estimate in person-days per field, with a separate line for every field needing special handling: price, rank, review timestamps, variant dimensions.

Weekly maintenance and triage: target site changes, broken parsers, fields going null with no error, false alarms, and the operations team asking why a number moved. Published operations surveys put maintenance work for a mid-size collection footprint at 5 to 10 hours per week. To estimate, add up the hours spent on this over the past eight weeks, divide by eight for a weekly average, and multiply by 4.3 for the month. This is the number to compute before any cost conversation.

Quality gate and monitoring build: automated tracking of the four numbers, coverage, fill rate, freshness, and distribution drift, plus the runbook for what happens when one of them fires. The four numbers and threshold guidance sit in the pipeline write-up linked above.

How Do You Set a Loaded Hourly Rate?

Do not divide base salary by 2,080. That understates the number by 25% to 40%. A loaded rate carries benefits and payroll tax, tooling and cloud allocation, management overhead, and amortized recruiting and onboarding. The common practice is a multiplier of 1.25 to 1.4 on direct salary. For internal comparison, one consistent factor is enough; precision past that adds nothing.

When Do You Stop Optimizing Unit Price?

Multiply monthly engineering hours by the loaded rate to get E, and call the data invoice S. When E > S, your optimization target this quarter is hours, not price: taking maintenance from ten hours a week to three is worth more than any tier discount. When S > 2E, you have reached the volume band where negotiation and architecture work on unit price start to pay. The three scenarios trace that transition: in A, E is far larger than S; in B, E is 2.6 times S; in C, S overtakes E.

Four layers of add-ons behind the cheap quote

Two quotes for the same job can sit a factor of two apart. The expensive one ships no extra field and no better service commitment. The gap is not the unit price. It is what the price includes: mandatory pieces get moved into add-ons, the number on the pricing page drops, and the invoice still carries every line.

JavaScript rendering. A static request takes the bytes the server returns. Rendering runs the page in a headless browser and waits for the async calls to fill content before extraction. The cost behind it is CPU time and memory, not bandwidth. You can price this layer in public: the Oxylabs pricing page carries two numbers inside one tier, Amazon results at $0.50 per thousand without JavaScript rendering and $1.35 per thousand with it, a multiplier of 2.7 (readings from September 2026; prices move with tier and contract).

IP grade. The gap between datacenter IPs and rotating residential IPs shows up in published proxy price lists, and it is measured in multiples rather than percentages. Amazon scores automated requests on IP quality, browser fingerprint, request cadence, and session continuity. When the score trips, one of two things happens: you get blocked, or you get a page with parts missing. The second costs more. It returns 200, the fields look populated, and the top ad placements plus a couple of modules never loaded. This layer settles on a separate product line, not on the request price.

Login sessions and accounts. Parts of the Alexa modules and the full body of some review content are not served outside a session. Keeping account pools alive, absorbing risk controls, solving CAPTCHAs, and replacing dead accounts all cost money. That figure lands on a maintenance fee or on the endpoint multiplier, and never in a column labeled price per call.

Multipliers you cannot switch off per endpoint. One question stays open after the first three: can you enable rendering for the endpoints that need it. A global toggle turns this line into a constant instead of a variable you tune.

Data class	Login needed	Rendering needed	IP grade	What goes wrong in comparison
Product static fields (title, price, BSR)	No	No	Low	Taking this unit price as the price of everything
Search results with Sponsored placements	No	Partial	High	Missing placements counted as neither output nor loss
Alexa traffic modules	Yes	Yes	High	Absent from the price list, so nothing to compare

def quoted_vs_real(base_price, k_js=1.0, k_ip=1.0, k_login=1.0):
    """Fold mandatory add-ons back into the base price.
    base_price   published unit price (per call or per thousand results)
    k_js         rendering multiplier, visible on public price lists, usually 2 to 3
    k_ip         IP grade multiplier, computed from the proxy product line quote
    k_login      account maintenance multiplier for endpoints behind a session
    """
    return base_price * k_js * k_ip * k_login

print(quoted_vs_real(0.50, k_js=2.7))            # 1.35
print(quoted_vs_real(0.50, k_js=2.7, k_ip=3.0))  # 4.05, replace with your own quote

One line of multiplication, and its value is that it forces three parameters onto the page. If you cannot fill them, the vendor has not given you enough visibility.

One number stays off every price list: with rendering on and the same IP grade, do two vendors bring back the same thing. Sponsored placement coverage is the cleanest test, because it is the hardest part of a search result to capture and close to the only part comparable as a number. A result missing its placements still returns 200. When your quality gate misses it, the row lands in your warehouse as a usable record and pulls your apparent unit cost down. When the gate drops it, you paid for a call with no output. The cheap quote and the expensive quote can trade places.

The bundled design keeps all four in one price: rendering on as default, the IP grade set where Sponsored coverage holds, no separate product line for rendering or proxies. In quote form that reads as a high unit price, and there is no rendering toggle on our pricing page. Whether that certainty earns a premium depends on how much of your data sits in rows two and three of the table above.

Which Five Clauses Set Next Year's Bill?

The pricing page sets month one. The contract sets year two. None of the five clauses below appears on a pricing page, and each one moves year two by more than the spread between tiers.

How Is Failure Defined, and by Whom?

Ask for a written definition of one successful call, covering three response types: a 200 with empty business fields, a 200 whose body is a placeholder or a challenge page, and a valid empty result such as a delisted product. The first is the most frequent and the easiest to leave vague. You need a sentence that fits in an acceptance test, not a sentence that sounds reassuring.

Are Failures Billed, and How Do Retries Count?

Ask three separate questions: are failed attempts billed; are the retries the vendor's server performs billed; are client-side retries billed. Some vendors retry server-side and bill the request once, others bill each pass. At a 90% success rate this clause is worth 11%, and at 80% it is worth 25% (see the table in the failed-attempt section).

What Is the Overage Unit Price, and Do Tiers Fall Back?

The price per unit above your bundle runs above the bundle's implied rate, by a factor between 1.5 and 3 in most contracts. Confirm three things: the per-unit price past the bundle, whether the tier falls back on its own or requires a new agreement, and whether service degrades or stops at the limit. The third is the one that hurts: some plans throttle at the ceiling, and your month-end batch does not finish.

What About Downgrades, Refunds, and Commitment?

The common restrictions: one downgrade per month, with the difference returned as account credit rather than cash; no mid-term refund on annual plans; a 12-month commitment on enterprise tiers. Before signing, ask what happens if you need to halve your volume in three months. That answer is worth a month's invoice the day you pick the wrong tier.

What Are the Multipliers for Marketplace, Postal Code, and Rendering?

Multiple marketplaces are not a flat multiple of one marketplace, because target difficulty and therefore multipliers differ by market. Postal-code-level targeting, used to read local price and inventory for a given ZIP, carries its own multiplier. Rendering is the same: if the search endpoint alone needs rendering and product detail works over static HTTP, can you enable it for search alone? Can you configure it per endpoint? The answer decides whether the multiplier term in your formula is a constant or a variable you can tune.

Where Did the Budget Go Wrong?

Three real failure patterns show the invoice was right and the unit was wrong.

"Billed per Successful Request" Is Not "Billed per Usable Record"

One team ran a plan billed on successful requests. The success dashboard held at 99% for months while cost stayed above budget. The cause: responses returned 200 with a valid shape and an empty price field on 18% of rows. The vendor recorded success, the billing system charged, and the team's quality gate caught it three days later. Fix: split usable rate out of success rate as its own metric and put usable rate into the reconciliation.

Billed per Result Page, Budgeted per Product Row

One team budgeted on 20 rows per call. Measurement returned 12 net rows: pagination delivered 16, cross-page duplicates removed 3, and an ad slot removed 1. The budget was off by 1.67x. Fix: measure r over 200 real queries. Do not extrapolate from the documented ceiling.

The Monthly Allowance Worked, the Per-Minute Rate Did Not

One team bought a two-million-allowance tier and spent 1.2 million, forfeiting the rest each month. The refill rate worked out to 50,000 per hour, and their overnight batch window was four hours, so 200,000 would not fit. Fix: move the batch to a rolling window, or split the allowance into a low-frequency baseline plus on-demand burst. That line item was $1,200 a month for this team.

What Seven Questions Do You Send Before You Sign?

Send these seven to the vendor and ask for written answers. A plan that answers all seven has a predictable cost. For a plan that does not, treat the missing answer as a negative signal in itself.

What is the definition of one successful call? For a 200 with empty business fields, a 200 whose value is a placeholder, and a product that is delisted: success or failure, in each case?
Are failed attempts billed? Are the retries your server performs counted? Are client-side retries counted? Can you supply a billing breakdown by response type?
Is the billing unit the request, the successful request, the result page, or the record? If it is a result page, is rows-per-page a guarantee or a ceiling, and is a row repeated across two pages billed once?
What does a cache hit cost, and what is the TTL? Can a request specify skip-cache? Can it require data newer than the TTL?
What is the per-unit price past the bundle? Does the tier fall back on its own? Does service slow or stop at the limit?
What are the multipliers for additional marketplaces, postal-code targeting, browser rendering, and residential proxies? Can each be switched on per endpoint, and what are the defaults?
How do downgrades, refunds, and commitment work? How often can you downgrade, in what form does the difference come back, how long are historical data and raw responses retained after cancellation, and can you export them?

Question 4 is the one teams skip, and its effect on cost rivals the unit price. If you poll every four hours and the vendor's cache TTL is six hours, half your calls buy data someone already fetched, at full price. Turn that around: if you can require data newer than the TTL, you are paying for an explicit freshness commitment rather than for a probability.

How Do You Reach Your Own True Unit Cost in Ninety Minutes?

0–20 minutes: from your call logs, compute the past week's attempt success rate s. Keep two counts: requests sent and responses usable.
20–45 minutes: run your quality gate over the most recent batch of landed records for the usable rate v. With no gate in place, write a minimal one: required fields non-null, types correct, timestamps inside the expected window.
45–65 minutes: run 200 real queries and compute net rows landed divided by calls made for r. Measure search and review endpoints on their own.
65–80 minutes: take the allowance you paid for from the monthly invoice, divide by rows landed, and you have the waste coefficient w.
80–90 minutes: put the five parameters into the calculator from the formula section, read the cost per thousand usable records, and add monthly engineering hours. Write that one number into your evaluation doc. It will decide more than any price comparison table.

If you reach a step where a parameter cannot be measured, that is itself the finding: your current vendor is not giving you the visibility to run the arithmetic.

What Does One Published Price Look Like, and How Do You Verify It?

The sections above are vendor-neutral and apply to any plan. This section covers one concrete published unit so you can test the formula against numbers a vendor has put on paper. This is the sole vendor named here, and the point is not to sell it but to show a worked example you can reproduce.

The Billing Unit and Published Numbers

One credit unit is used, and the per-endpoint multipliers are public and adjustable in the calculator on the pricing page, which also shows credit multipliers per endpoint and the price per tier: product detail at 1 credit per page, reviews at 5, niche data at 2, AI Overview at 2, with raw HTML output at 75% of structured JSON because parsing is skipped. Tiers are Starter at $19 for 9,600 credits, Professional at $99 for 60,000, and Expert at $369 for 240,000, with tiered usage billing past Expert and annual billing 20% below monthly. At the Expert tier that works out to $1.54 per thousand product records in structured JSON, $1.15 in HTML, and $1.23 on annual billing.

Substitute those numbers into the formula in the formula section and you can compute your own cost. That is the reason they are published: unit price is the easiest of the seven parameters to compare, hiding it gains nothing, and recomputability is what makes trust possible.

Why Is Field Coverage Not Field Availability?

A field on the contract and a field with a value are two different states. One vendor may present field coverage as field availability, which is the gap this whole article warns about in another form: a checkmark in a feature table is not a value in your warehouse. We would rather tell you the fill rate for a field in a given marketplace than put a checkmark in a feature table. How to verify that field by field is in the field-by-field comparison we ran, where the checkboxes were lying, and the same point about coverage versus availability is made in our write-up on comparing Amazon data APIs field by field.

When Should You Not Use This Vendor?

You need historical price curves, nothing else. Multi-year price and BSR time series are a database query, not live collection, and products built for that have a cost structure an order of magnitude below a live collection tier.

You need data from your own seller account, nothing else. If your scope sits inside your own account and the official selling partner API quota covers it, that is the cheapest route available to you.

A few hundred rows, one time. At that volume, manual work plus a spreadsheet is faster, and integration cost exceeds data cost.

What Does This Vendor Leave Out?

It does not report success rates on a best-efforts basis. The published numbers are a median latency of about three seconds, a 99% success rate, more than 30 million calls a day, and a 91.4% sponsored-placement detection rate across 13 marketplaces in published comparisons. Those are checkable figures, not sales language.

It does not promise zero maintenance. After you connect an API, the quality gate, the monitoring, and the contract tests still sit on your side. What it can do is compress what you maintain from parsers, proxy pools, and anti-bot work down to a quality gate and business rules. That remaining part does not compress away.

It does not present field coverage as field availability, covered above.

What Questions Do Teams Ask?

How is an Amazon data API billed? Six units: per attempt, per successful request, per thousand results, per credit, per gigabyte, and per rate allowance. Convert them to one number before comparing: what you pay per thousand records that pass your quality gate. Four parameters are needed, and no pricing page carries them: net records per call, attempt success rate, usable share of successes, and whether failures bill.

Are failed requests billed? That is in the contract, not on the pricing page. Some vendors bill successes alone, some bill every attempt, and some charge a reduced rate for failures. The more important question is who defines failure: a 200 response with empty business fields counts as success in most billing systems. Get it in writing, with a billing breakdown by response type.

Is one call one record? No. A product detail endpoint returns one record per call, and search and review endpoints return pages where rows per page depend on pagination parameters, cross-page deduplication, and ad placements. If the vendor bills by result page, convert using measured net rows; using the documented ceiling puts your budget off by about 1.5x.

How do I tell whether a quote is cheap? Run the same ASIN set and the same field list for seven days and capture four numbers: attempt success rate, usable share of successes, net records per call, and the invoice. Put them into the formula for cost per thousand usable records, then add monthly engineering hours. The lowest unit price is often not the cheapest option on this basis.

What costs sit outside the invoice? Three: initial parsing and field mapping, weekly maintenance and triage, and allowance you buy but cannot spend behind a rate limit. The first two never appear on an invoice. The third appears with no output attached. Across the three volume scenarios in this article, engineering is 45% to 89% of monthly total cost.

What an Amazon Data Pipeline Costs You After the Scrapers Are Gone

Pangolinfo — Mon, 07 Sep 2026 06:23:46 +0000

Every team that runs Amazon collection in-house knows the loop by heart.

Write the parser. Ship it. Amazon changes the page. The parser throws. You get paged. You fix the parser. Two weeks later, repeat.

Run that loop enough times and somebody says out loud what everyone is thinking: can we stop maintaining scrapers?

Yes. You can. But here is the part that does not show up in any vendor pitch deck, and it took us a while to internalise: outsourcing collection removes the parsers and the proxy pool. It does not remove data quality governance. Plenty of teams migrate, look up six months later, and realise their engineers did not get free time — they got a different thing to maintain, one that is harder to notice.

The difference is loud versus quiet.

A broken parser is loud. It throws, it returns nothing, the logs fill up, you know within minutes. Data quality degradation is quiet. HTTP 200, valid JSON, schema validation passes, twelve fields are null, and your dashboard looks healthy while drifting into a lie.

This post is the architecture we converged on for that quiet half of the problem: six layers, what each one owns, the retry bugs that triple your bill with no alert, the quality gate you have to build yourself, and a 90-minute checklist to get a first version running. All the code runs as written.

Scope note: this is not the "should I use an API or scrape at all" post. If you have not settled that question, read Amazon API vs Web Scraping: Stop Choosing, Build Both Routes first — it answers the prior question, and it will save you reading the wrong half of this one.

Step zero: compute your baseline before you compare anything

This is the step everybody skips, and it is the reason most cost conversations about data collection degenerate into vibes.

Add up the engineering hours your team spent on collection maintenance over the last eight weeks and divide by eight. That weekly number is the denominator for every decision that follows, because it puts a monthly API invoice on the same scale as the headcount cost of keeping the scrapers alive.

# baseline.py — how much of your week does collection already eat?
import csv
from collections import defaultdict
from datetime import date, timedelta

WINDOW_WEEKS = 8

def weekly_maintenance_hours(csv_path, since):
    """Expects rows: date, hours, category (bugfix/false_alarm/ops_q/deps)."""
    by_week = defaultdict(float)
    for row in csv.DictReader(open(csv_path)):
        d = date.fromisoformat(row["date"])
        if d < since:
            continue
        by_week[d.isocalendar()[1]] += float(row["hours"])
    weeks = list(by_week.values()) or [0.0]
    return sum(weeks) / WINDOW_WEEKS, weeks

def break_even(monthly_fee_usd, loaded_hourly_usd, weeks_per_month=4.33):
    """How many hours/week does the fee have to buy back to be worth it?"""
    return round(monthly_fee_usd / (weeks_per_month * loaded_hourly_usd), 2)

Most teams are surprised the first time they run it, and the reason is instructive. The total is not just bug-fixing time. It includes chasing false alarms, answering questions from operations, updating dependencies, re-learning the scraper's quirks each time you touch it, and the context-switch cost of every one of those interruptions. Those add up to far more than the actual fixes.

Once you have the number, break_even() gives you the honest question. If a managed service costs $3,000 a month and your loaded engineering rate is $90/hour, it needs to buy back about 7.7 hours a week. If your measured baseline is 14 hours a week, the maths is not close. If it is 3, it is not close either — in the other direction.

Skip this step and you will argue about price without ever establishing what you are paying for.

Four routes out of parsing, and what each one costs

No vendor names and no second-hand prices here. Prices move; cost structure does not.

Route	Dominant cost	Who owns maintenance	Typical failure mode	Best fit
Self-built scraper stack	Engineer debugging time, far above servers	All of it, in-house	Layout change breaks parsing	Very long-tail pages, dedicated team
Managed platform / self-hosted framework	Compute time plus parser upkeep	Runtime outsourced, logic in-house	Harder anti-bot means a bigger bill	Existing expertise, custom logic
Official SP-API	Authorisation ops, per-operation rate planning	In-house, but low failure rate	Throttling and expired grants	Your own operational data
Vertical Amazon data API	Per-call billing plus self-built quality governance	Collection outsourced, judgement in-house	Fields going null, data going stale	Market and competitor data

Two columns deserve your attention: maintenance ownership and typical failure mode. Wherever maintenance lives is where your engineers will spend their staring time. Whatever the failure mode is, that is what your alerts need to be built around. Most teams compare only the cost column and then discover after go-live that they have zero monitoring for the failure mode they just signed up for.

Self-built: the cost lands in the wrong bucket

Headless browser cluster, proxy pool, parsers. Maximum control, no per-record billing, marginal cost flattens as you scale. But you think you are spending money on servers and you are spending it on engineering time — in a live scraper stack, servers are almost always the smaller line item. And anti-bot evasion is a permanent arms race: the trick that works today is not promised to work in six months.

Managed platforms and self-hosted frameworks

Scrapy, Playwright clusters, hosted actor marketplaces. Fast to first result, mature ecosystem, much of the parsing logic already written. But maintenance did not transfer. The platform took over browsers and runtime; you still own parsing logic, selectors, retry policy, and anti-bot strategy. Worse, most of these platforms bill by compute duration, which means the harder the anti-bot, the higher your bill. Cost correlating with difficulty is a miserable property for a budget model.

Official SP-API

Stable, compliant, explicit schema, no anti-bot fight. But coverage is bounded by its authorisation model. It gives you orders, inventory, fulfilment, and your own advertising data for accounts a seller has authorised by name in the grant. It does not give you competitor prices, market-wide search rankings, or other sellers' reviews — not "not yet supported," but not offered by design. SP-API is one component of a pipeline, never the whole thing.

Vertical Amazon data API

You hand parsing and anti-bot to a specialist and receive structured JSON. Here maintenance transfers: page redesign becomes the vendor's problem, the proxy pool becomes the vendor's problem, your engineers stop watching selectors. What does not transfer is quality governance — and field coverage varies from vendor to vendor in ways that never appear on a feature grid.

The realistic end state is a mix. Your own operational data through SP-API, market and competitor data through a data API, maybe a thin self-built layer for the longest-tail pages. There is a simple test for drawing the boundary: if the question is "did I get authorisation," use the official channel; if it is "what is happening in the marketplace right now," use a data service. One is an authorisation problem, the other a collection problem, and their failure modes, cost structures, and compliance obligations are different enough that forcing both onto one technical path raises complexity on both sides.

What moved: three shifts, all of them quieter

This is the premise for the architecture below. If you do not accept it, everything after this will look over-engineered.

Parse failure → fields going null with no error

When a scraper fails it is loud. When an API fails without raising an error, you get HTTP 200, valid JSON, schema validation passes, and twelve of the fields are null. The failure signal moved from the transport layer to the business-semantics layer — and nobody writes business-semantics validation for you.

{
  "asin": "B0C7V9K2XQ",
  "title": "…",
  "price": null,
  "availability": { "status": null },
  "rating": { "value": 4.3, "count": null }
}

Every check you have in place passes. Row count correct. Status 200. Dashboard green. Your price column is empty.

Getting blocked → data going stale with no alert

Anti-bot blocking is an unambiguous failure. The API-side equivalent is freshness decay: the vendor adjusts a cache policy, collection frequency drops in one marketplace, an upstream job backs up — and the value you receive describes the world as it was three days ago. Format valid. Content no longer true. The only defence is a freshness SLO plus monitoring the staleness distribution.

Proxy cost → retry-amplified cost

In the self-built era your costs were proxies and machines, and most of that spend was fixed. In the API era your cost is unit price × attempts, and attempts get amplified by failure rate, backoff policy, and idempotency design together. A pipeline with a 70% usable record rate and 1.8 average attempts is paying 2.57× list price. Unmonitored, that multiplier drifts — nobody changed any code — and one day the invoice is 40% higher.

The story that made this concrete for us

A six-person data team replaced their in-house scraper with a third-party API and reclaimed around two engineering days a week. Everyone was happy. In month three, operations started saying the competitor prices in the report looked odd.

Investigation showed the fill rate for the price field in two marketplaces had fallen from 96% to 61%. Over five weeks. With no alert. Because every request returned 200 and the success-rate metric never dropped below 99%.

Fixing it took two weeks — a week and a half of which was archaeology. With no raw response archive, nobody could establish which day it started, so they had to bisect the problem by re-running history and eyeballing output.

The lesson is not that they should not have migrated. It is that quality monitoring has to be built in the same project as the migration. Ship raw archiving and field-level fill-rate monitoring alongside the switch, and that decay surfaces in week one as a parameter change instead of two weeks of forensics.

The six-layer architecture

This is not theory. It is the shape teams running stable pipelines converge on, and the rule is one responsibility per layer so a failure can be localised.

Business goal
   │  field contract · freshness SLO · cost budget
   ▼
┌──────────────────────────────────────────┐
│ L1 Collect      fetch raw, no judgement   │
├──────────────────────────────────────────┤
│ L2 Raw archive  verbatim, only replay     │
├──────────────────────────────────────────┤
│ L3 Queue+retry  idempotency · backoff·DLQ │
├──────────────────────────────────────────┤
│ L4 Normalise    contract check · upsert   │
├──────────────────────────────────────────┤
│ L5 Quality gate four numbers decide       │
├──────────────────────────────────────────┤
│ L6 Observe      panels · alerts · weekly  │
└──────────────────────────────────────────┘
   │
   ▼
Downstream: reports / models / agents / alerts

Layer zero: the inputs are contracts, not requirements

Before any code, write three things down, each decidable by a machine:

A field contract — the fields the business depends on, graded P0/P1/P2.
A freshness SLO — the P95 staleness ceiling for P0 fields, say six hours.
A cost budget — the ceiling per thousand usable records.

CONTRACT = {
    "P0": ["asin", "title", "price.amount", "availability.status"],
    "P1": ["brand", "rating.value", "review_count", "bsr.rank_main"],
}
FRESHNESS_FIELD, SLO_P95_HOURS = "collected_at", 6.0

These three are the basis for every automated judgement downstream. Without them the gate has to guess its thresholds, and a guessed threshold gets switched off after the first false alarm.

The most common mistake in writing a field contract is wanting everything. A contract listing 80 fields all marked P0 is equivalent to no contract: the usable record rate collapses toward zero, the gate screams on every run, somebody turns it off. P0 means a record missing this field cannot go into the downstream report — no more than 15 fields in most contracts. P1 means nice to have. P2 means useful in a few reports, may be null. This grading is not bureaucracy; it determines retry policy, degradation tiers, and gate thresholds.

L1: the collector fetches, it does not judge

Draw this boundary hard. The collector sends the request, takes the response, hands it onward. No field validation, no business judgement, no cleaning. The reason is attributability: if business logic lives inside the collector, you cannot tell whether bad data was fetched wrong or judged wrong.

import json, time, uuid
from datetime import datetime, timezone

def collect(client, item, timeout):
    """L1 contract: return raw body + call metadata. Never inspect the payload."""
    request_id = str(uuid.uuid4())
    started = time.perf_counter()
    status, body = client.get(item.endpoint, params=item.params, timeout=timeout)
    meta = {
        "request_id":   request_id,
        "object_type":  item.object_type,
        "marketplace":  item.marketplace,
        "status":       status,
        "duration_ms":  int((time.perf_counter() - started) * 1000),
        "attempt":      item.attempt,
        "collected_at": datetime.now(timezone.utc).isoformat(),
    }
    return {"raw": body, "meta": meta}

The collector should emit two things: the raw response body, and metadata about the call — duration, status code, timestamp, attempt number. That last one is what makes cost attribution possible later.

L2: raw archive is the only basis for reproducibility

Write responses verbatim, partitioned by object, marketplace, and date, retained 30 to 90 days.

import gzip, os

ARCHIVE_ROOT = "/var/data/raw"

def archive(res):
    meta = res["meta"]
    day = meta["collected_at"][:10]
    path = os.path.join(ARCHIVE_ROOT, meta["object_type"], meta["marketplace"], day)
    os.makedirs(path, exist_ok=True)
    name = f"{meta['request_id']}-{meta['status']}.json.gz"
    with gzip.open(os.path.join(path, name), "wt", encoding="utf-8") as fh:
        json.dump({"meta": meta, "body": res["raw"]}, fh)
    return os.path.join(path, name)

The value shows up only when something breaks: if you suspect a record is wrong, can you go back to that exact response body from three months ago? If not, every root-cause analysis you run is guesswork. Compressed JSON is cheap.

There is a practical rule for retention: it has to be longer than your detection cycle. If your team takes three weeks on average to notice an anomaly, 30 days lets you look back once and 90 days lets you compare across two incidents. Compressed raw JSON almost always costs less than recomputing aggregate metrics over the same window, so err on the generous side.

L3: queue and retry — route by error class

This layer owns scheduling, concurrency control, throttling, and retries. The key design is routing by error class. Mixing the three classes below into one retry loop is the single biggest cause of runaway retry cost.

class Retryable(Exception):    pass   # 429 / 5xx / timeout
class NonRetryable(Exception): pass   # 401 / 422 / bad params
class SoftFailure(Exception):  pass   # 200 but every P0 field is empty

def classify(status, body, p0_fields):
    if status in (408, 425, 429) or 500 <= status < 600:
        return Retryable(f"http_{status}")
    if status in (400, 401, 403, 404, 422):
        return NonRetryable(f"http_{status}")
    if status == 200:
        if not body:
            return SoftFailure("empty_body")
        if not any(field_filled(body, f) for f in p0_fields):
            return SoftFailure("p0_empty")
        return None
    return NonRetryable(f"unexpected_{status}")

import random, time

BASE_SECONDS, CAP_SECONDS, MAX_ATTEMPTS = 0.5, 30.0, 5

def backoff(attempt):
    # CAP stops runaway tail latency; random() breaks up retry storms
    return min(CAP_SECONDS, BASE_SECONDS * 2 ** (attempt - 1)) * (0.5 + random.random())

def run_item(client, item, p0_fields, dlq, softq):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        item.attempt = attempt
        try:
            res = collect(client, item, timeout=item.timeout)
            err = classify(res["meta"]["status"], res["raw"], p0_fields)
            if err:
                raise err
            archive(res)                      # archive before parsing, always
            return res
        except Retryable:
            if attempt == MAX_ATTEMPTS:
                dlq.push(item, reason="attempts_exhausted")
                return None
            time.sleep(backoff(attempt))
        except NonRetryable as e:
            dlq.push(item, reason=str(e))     # retrying 401 ten times buys nothing
            return None
        except SoftFailure as e:
            softq.push(item, reason=str(e), raw=res)
            return None

Two details people get wrong here.

Backoff needs jitter. Exponential backoff without it produces retry storms: a batch fails together, waits together, retries together, and saturates the rate limit on the same millisecond. Jitter spreads the retries out. You also need the ceiling, or tail latency runs away.

Timeouts matter more than retry counts. Teams tune retry counts down to the last attempt and set timeouts to whatever number came to hand, but timeouts move cost and throughput more. Too short, and requests that would have succeeded get marked failed and retried — inflating cost and duplicating work. Too long, and failing requests hold concurrency slots, the queue backs up, and total throughput drops.

Derive the value from your own latency distribution, per operation. Do not inherit a default.

def percentiles(values, ps=(50, 90, 95, 99)):
    s = sorted(values)
    out = {}
    for p in ps:
        idx = max(0, min(len(s) - 1, int(round(p / 100 * len(s))) - 1))
        out[f"p{p}"] = s[idx]
    return out

# from your own call logs, per operation
lat = percentiles([m["duration_ms"] for m in call_log if m["op"] == "product"])

CONNECT_TIMEOUT_S = 3.0                                  # TCP + TLS: a few seconds, always
READ_TIMEOUT_S    = max(10.0, lat["p99"] / 1000 * 1.5)   # follows the response distribution

Take P99 as the baseline and add headroom. And separate connect timeout from read timeout: the former is a small constant, the latter follows the response distribution. In data services some complex objects are slow, so a blanket short timeout raises your failure rate on every one of them.

Idempotency keys must include the contract version

Many teams use object ID + marketplace + date, which works right up until the contract changes. Add a new P0 field and historical records no longer satisfy the new contract — but without a version component they look already processed and get skipped.

import hashlib

CONTRACT_VERSION = "2026-08-01"

def idempotency_key(asin, marketplace, data_date, contract_version=CONTRACT_VERSION):
    return hashlib.sha1(
        f"{asin}|{marketplace}|{data_date}|{contract_version}".encode()
    ).hexdigest()

Include the version and a contract upgrade triggers a full re-run without a human in the loop — which is the correct behaviour, and the one you want at 3am without having to think about it.

A dead-letter queue is not a bin

Dead-letter contents have to be consumed, or the queue is just slow deletion.

from collections import defaultdict

def dlq_report(items):
    counts = defaultdict(int)
    for it in items:
        counts[it["reason"]] += 1
    total = sum(counts.values()) or 1
    return {k: (v, round(v / total, 3)) for k, v in
            sorted(counts.items(), key=lambda kv: -kv[1])}

Minimum bar: aggregate by failure reason, produce a daily list, alert on shifts in the distribution. The distribution reveals problems earlier than the failure rate does on most teams — a sudden rise in attempts_exhausted points to degrading upstream success while your headline success metric is still above threshold.

L4: normalise and store — strict types, idempotent upsert

FIELD_TYPES = {
    "asin":                str,
    "title":               str,
    "price.amount":        (int, float),
    "availability.status": str,
    "rating.value":        (int, float),
    "review_count":        int,
    "bsr.rank_main":       int,
}

def dig(rec, path):
    cur = rec
    for part in path.split("."):
        if not isinstance(cur, dict) or part not in cur:
            return None
        cur = cur[part]
    return cur

def assert_contract(record):
    """Strict: presence AND type. A str→dict rename must not slip through."""
    missing, wrong_type = [], []
    for field, expected in FIELD_TYPES.items():
        value = dig(record, field)
        if value is None:
            missing.append(field)
        elif isinstance(value, bool) or not isinstance(value, expected):
            # bool check matters: isinstance(True, int) is True in Python
            wrong_type.append((field, type(value).__name__))
    return missing, wrong_type

That isinstance(value, bool) guard is not paranoia. isinstance(True, int) returns True in Python, so a field that flips from review_count: 1284 to review_count: true passes a naive type check and lands in your warehouse as 1.

A field changing from string to object is the most common breaking change, and implicit coercion in a dynamically typed language lets it pass unnoticed. Route validation failures to the soft-fail queue; never skip them without a log entry.

Then write with idempotency:

INSERT INTO product_daily
  (asin, marketplace, data_date, contract_version, payload, p0_missing)
VALUES (%(asin)s, %(marketplace)s, %(data_date)s, %(contract_version)s,
        %(payload)s, %(p0_missing)s)
ON CONFLICT (asin, marketplace, data_date, contract_version)
DO UPDATE SET payload     = EXCLUDED.payload,
              p0_missing  = EXCLUDED.p0_missing,
              updated_at  = now();

Processing the same object for the same day twice must produce the same result. This layer also records which P0 fields were missing, for the next layer to consume.

The quality gate: four numbers plus a freshness SLO

This is the layer you must build yourself and cannot buy. Its job is to convert "is this data usable" from a judgement call into quantities a machine can compute.

Coverage and fill rate are different numbers

Presence is not value. Coverage asks whether the path exists at the schema level; fill rate asks whether a value arrived. Both low means the field is not supported. High coverage with low fill rate is the dangerous combination — the field is on the contract, the vendor populates it on a small share of calls, and schema validation waves it through.

Related: the field-level comparison of two Amazon data APIs runs the same ASIN against both vendors and counts fields. One returned twelve, the other fifty-eight, and both pricing pages carried the same checkmark.

def field_present(rec, path):
    cur = rec
    for part in path.split("."):
        if not isinstance(cur, dict) or part not in cur:
            return False
        cur = cur[part]
    return True

def field_filled(rec, path):
    if not field_present(rec, path):
        return False
    cur = rec
    for part in path.split("."):
        cur = cur[part]
    if cur is None:
        return False
    if isinstance(cur, str) and cur.strip() == "":
        return False
    if isinstance(cur, list) and len(cur) == 0:
        return False
    return True

def coverage(records, fields):
    total = len(records) * len(fields)
    return sum(1 for r in records for f in fields if field_present(r, f)) / total if total else 0.0

def fill_rate(records, fields):
    total = len(records) * len(fields)
    return sum(1 for r in records for f in fields if field_filled(r, f)) / total if total else 0.0

Usable record rate judges the whole record

The two metrics above are field-level, but downstream consumes records. Define usable record rate as the share of records where every P0 field is filled. It is the only metric that maps onto "can this row go into the report," and it should be the gate's primary threshold.

from datetime import datetime, timezone

def usable_rate(records, p0_fields):
    if not records:
        return 0.0
    ok = sum(1 for r in records if all(field_filled(r, f) for f in p0_fields))
    return ok / len(records)

def staleness_p95(records, ts_field, now=None):
    """Return P95 age in hours. None means no usable timestamps at all."""
    now = now or datetime.now(timezone.utc)
    deltas = []
    for r in records:
        ts = r.get(ts_field)
        if not ts:
            continue
        t = datetime.fromisoformat(str(ts).replace("Z", "+00:00"))
        deltas.append((now - t).total_seconds() / 3600)
    if not deltas:
        return None
    deltas.sort()
    return deltas[max(0, int(round(0.95 * len(deltas))) - 1)]

Related: I stopped trusting "real-time" claims after one bad quarter covers the freshness side in much more depth, including the 100-call test I use to establish whether a "real-time" claim means anything.

Write the freshness SLO as a distribution, not a boolean

"Is the data real time" cannot be alerted on, because real time has no quantified definition. The executable version is: P95 staleness of P0 fields is at most X hours.

Use P95 rather than the mean, because the mean hides the tail. A distribution with a 20-minute P50 and a 31-hour P95 produces an excellent average while a meaningful slice of your report is a day old. That is the failure the 96%→61% team walked into — they were watching the mean of a success metric, not the tail of a freshness one.

Five quantities, one function

P0 = ["asin", "title", "price.amount", "availability.status"]
P1 = ["brand", "rating.value", "review_count", "bsr.rank_main"]

def gate(records):
    m = {
        "n":             len(records),
        "coverage_p0":   coverage(records, P0),
        "fill_p0":       fill_rate(records, P0),
        "fill_p1":       fill_rate(records, P1),
        "usable":        usable_rate(records, P0),
        "staleness_p95": staleness_p95(records, FRESHNESS_FIELD),
    }
    m["pass"] = (
        m["fill_p0"] >= 0.95
        and m["usable"] >= 0.90
        and m["staleness_p95"] is not None
        and m["staleness_p95"] <= SLO_P95_HOURS
    )
    return m

Note the explicit is not None. When the timestamp field is absent, staleness_p95 returns None. Skip that check and None <= 6.0 raises a TypeError in Python 3 — but in an implementation with light validation it evaluates to False and fails the batch with no error. That second outcome is worse: it makes data with no timestamp at all look like it failed a freshness check it never took, which sends you debugging the wrong layer.

Record level marks, batch level decides

One design choice changes whether the gate survives contact with production: judge per record or per batch.

Per record suits pre-write cleaning — a record failing P0 is not written. Clean, but it loses data, and it cannot distinguish "this whole batch is broken" from "this one row happened to be empty."

Per batch suits release decisions — a batch whose usable rate falls below threshold is held back in full. Catches systemic decay, but blocks the good records inside it.

Do both. Record-level marking first, then batch-level decision, so that when a batch is held you can name which class of record dragged it down in one step. Collapsing the two into one switch either loses data or misses decay.

Stratify the sample or the aggregate lies to you

This is the subtlest trap in the whole gate. Suppose your sample is 70% books and 30% apparel. Books fill the price field 96% of the time. Apparel, with its variants and missing variant prices, fills it 58%.

books    n=280   price fill 96%
apparel  n=120   price fill 58%
---------------------------------
total    n=400   price fill 84%   <- the number your dashboard shows

84% looks acceptable. It conceals the fact that apparel is unusable.

def stratify(records, keyfn):
    buckets = defaultdict(list)
    for r in records:
        buckets[keyfn(r)].append(r)
    return buckets

def gate_report(records, keyfn=lambda r: r.get("category", "unknown")):
    per_stratum = {k: gate(v) for k, v in stratify(records, keyfn).items()}
    worst = min(per_stratum.items(), key=lambda kv: kv[1]["usable"])
    return {"overall": gate(records), "strata": per_stratum,
            "worst": (worst[0], worst[1]["usable"])}

So the fixed regression sample must be stratified by object characteristics: category, whether the product has variants, marketplace, price band — with enough volume in each stratum to produce a standalone conclusion. Which dimensions matter depends on your business. Stratify by category if value concentrates in a few categories, by marketplace if it concentrates in a few marketplaces. What matters is that every stratum can speak for itself instead of disappearing into a total.

Set thresholds that will not get switched off

The usual way a quality gate dies: thresholds start too aggressive, the first false alarm fires, somebody disables it and never turns it back on.

def propose_thresholds(baseline, tolerance=0.05, block_gap=0.10):
    """Week 1: record, do not block. Week 2: alert below baseline, block further down."""
    return {
        "alert_fill_p0":    round(baseline["fill_p0"] - tolerance, 3),
        "block_fill_p0":    round(baseline["fill_p0"] - block_gap, 3),
        "alert_usable":     round(baseline["usable"] - tolerance, 3),
        "block_usable":     round(baseline["usable"] - block_gap, 3),
        "staleness_alert_h": round(baseline["staleness_p95"] * 1.25, 2),
    }

Observe for two weeks before setting anything. Week one records without blocking, giving you baseline distributions for all four numbers. Week two sets an alert line at a sensible tolerance below baseline and a blocking line further down. Thresholds should come from your own distribution, not from a generic best practice.

Shipping alerts two weeks later is far cheaper than shipping an alert that gets turned off — and once a gate has been disabled, rebuilding trust in it is much harder than building it the first time.

L6: five charts, and a budget guard

Trend the four numbers, alert on them, attribute cost to business objects, and run a fixed-sample regression every week. Weekly regression is the item most often skipped and the most valuable: a one-off acceptance test proves it works now, a weekly regression proves it still works.

Only five charts deserve permanent dashboard space:

Usable record rate over time — one line per object type.
P0 fill rate over time — one line per marketplace.
P95 staleness over time.
Retry amplification factor over time.
Cost per thousand usable records over time.

Five is enough; more and nobody looks. The first three answer is the data still usable, the last two answer what is it costing.

Making success rate the hero chart is a mistake. It sits above 99% for the life of the dashboard, conveys nothing, and occupies the most prominent space on the page. That is how a five-week decay from 96% to 61% happens without anybody noticing.

The effective cost formula

What you are optimising is not unit price, it is cost per thousand usable records:

effective_cost = unit_price × attempts_per_usable_record / usable_rate

That formula ties three variables together — price, stability, completeness. A cheap vendor with a poor usable rate can cost more in the end; an expensive one with full fields and few retries can be the better deal.

Record attempts per record at the collector and aggregate the mean by object type and marketplace:

def amplification(records):
    attempts = [r["meta"]["attempt"] for r in records]
    usable = [r for r in records if all(field_filled(r["payload"], f) for f in P0)]
    return {
        "avg_attempts":         round(sum(attempts) / len(attempts), 3),
        "usable_rate":          round(len(usable) / len(records), 3),
        "effective_multiplier": round(sum(attempts) / max(1, len(usable)), 3),
    }

Most teams are surprised the first time they look. Plenty assume the factor is close to 1 and measure it between 1.5 and 2.5. It often rises not because failure rates climbed but because backoff policy is triggering retries earlier than it should.

Cost attribution, minimal version

Attribution does not need a platform. At the collector, write an estimated cost into each record's metadata — call count times unit price, tagged with the object and marketplace.

UNIT_PRICE_USD = 0.001

def attributed(records):
    spend, usable = defaultdict(float), defaultdict(int)
    for r in records:
        k = (r["meta"]["object_type"], r["meta"]["marketplace"],
             r["meta"]["collected_at"][:10])
        spend[k] += r["meta"]["attempt"] * UNIT_PRICE_USD
        if all(field_filled(r["payload"], f) for f in P0):
            usable[k] += 1
    return {
        k: {"spend": round(v, 4), "usable": usable[k],
            "cost_per_1k": round(v / usable[k] * 1000, 2) if usable[k] else None}
        for k, v in spend.items()
    }

Aggregate along three dimensions — object type, marketplace, date — and you can answer all but a few cost questions: which marketplace is most expensive, which object type is burning money, which week the cost started climbing. It is common to find that the highest-volume object is not the most expensive per unit, and that the team has been optimising the wrong thing.

Budget guard: write degradation into code

DAILY_BUDGET_USD = 250.0

def sampling_policy(spent_today):
    if spent_today < 0.7 * DAILY_BUDGET_USD:
        return {"P0": 1.0,  "P1": 1.0, "P2": 1.0, "tier": "normal"}
    if spent_today < DAILY_BUDGET_USD:
        return {"P0": 1.0,  "P1": 0.5, "P2": 0.0, "tier": "p0_fidelity"}
    return {"P0": 0.25, "P1": 0.0, "P2": 0.0, "tier": "p0_only"}

When cumulative daily cost crosses the threshold, reduce sampling frequency for non-P0 objects and preserve full collection for P0. Write and test this policy in advance. Deciding under pressure leaves only two options — stop everything, or keep burning — and both are bad.

Two things that cut across every layer

Model concurrency and rate limits per operation. Planning around one global QPS is a common error. Both official channels and most serious data services use per-operation token buckets: different operations carry different quotas and different refill rates. What you need is not "how many requests per second do we send" but a table of per-operation quotas plus per-operation backoff implemented client-side. Without that table your concurrency plan is guesswork and your load test is meaningless — throughput measured on one operation tells you nothing about another.

Lineage has to trace back to the specific call. Every landed record should carry source metadata: request ID, vendor response timestamp, attempt count, contract version, collection batch ID. It is a handful of fields, but without it you cannot answer "when was this price collected." Lineage is also what makes root-cause analysis tractable — when a field starts decaying, you slice by batch, marketplace, and time instead of re-running everything and hoping.

Schema drift: three defences inside the pipeline

Upstream field changes happen without an error — renames, type changes, new enum values, merged fields. The damage never appears at the transport layer; it appears as a number in your report that is wrong without raising an error.

One: strict presence and type assertions at the normalisation layer (assert_contract above), checking type as well as existence, with failures routed to the soft-fail queue rather than skipped.

Two: weekly structural diffs. Save a response snapshot for a fixed sample every week and diff its structure:

def shape(obj, prefix=""):
    """path -> type name. Ignores values entirely, so it catches structure only."""
    out = {}
    if isinstance(obj, dict):
        for k, v in obj.items():
            out.update(shape(v, f"{prefix}.{k}" if prefix else k))
    elif isinstance(obj, list):
        for v in obj[:1]:                      # sample first element only
            out.update(shape(v, prefix + "[]"))
    else:
        out[prefix] = type(obj).__name__
    return out

def diff_shapes(prev, cur):
    return {
        "added":        {k: v for k, v in cur.items() if k not in prev},
        "removed":      {k: v for k, v in prev.items() if k not in cur},
        "type_changed": {k: (prev[k], cur[k]) for k in cur.keys() & prev.keys()
                         if prev[k] != cur[k]},
    }

Push the diff to your alert channel. The snapshot doubles as the basis for root-cause analysis — without it, when you need to know when a field stopped populating, memory is all you have.

Three: contract versioning and regression re-runs. The field contract carries a version number. When the contract changes, the idempotency key changes with it, which re-runs the affected objects. Without versioning, a contract upgrade leaves historical records looking already processed and skipping them, so old and new data sit mixed together and nobody notices.

Degradation: what happens when upstream goes down

Almost nobody writes this section into an architecture doc, and it is the difference between a chaotic incident and an orderly one. Every upstream fails at some point — vendor outage, your own quota exhausted, a network partition, tightened throttling. The question is whether your pipeline responds by delivering less data and stating the shortfall, or by pretending to deliver all of it.

With no predefined policy, teams improvise between two options: halt and wait, or let retries burn through the budget. Both are bad. Define tiers in advance:

Tier	Behaviour	Trigger
1 — P0 fidelity	P0 frequency unchanged, P1/P2 reduced	Cost over threshold or rising upstream error rate
2 — P0 only	P0 at reduced frequency, P1/P2 suspended	Upstream unavailable beyond a configured duration
3 — archive read-only	Collection stopped, downstream pointed at the last gated snapshot, labelled beyond SLO	Sustained unavailability

Trigger conditions, scope, and recovery conditions for all three belong in configuration, not in somebody's memory.

def stamp(record, tier, note):
    record["_collection"] = {"tier": tier, "note": note, "degraded": tier != "normal"}
    return record

Degraded state must be visible downstream. The dangerous part of degradation is not less data — it is downstream not knowing the data is reduced. Records released by the gate should carry a collection status marker stating which tier they were collected under and which object range was covered. The reporting layer should surface a warning when it sees a degradation marker; the model layer should adjust confidence. Hiding degraded state inside the pipeline means downstream consumes partial data as complete, which is the quiet failure from the case study above.

Where we fit, stated with a tight boundary

We build Pangolinfo as a collection layer, so let me be precise about the boundary: we own L1 and the stability that comes with it. L5, the quality gate, is yours to build regardless of whose collection service you use. If a vendor tells you otherwise, they are selling you something that cannot work.

Handing over collection means page redesigns, anti-bot evasion, proxy pool operations, and browser fingerprinting stop being your engineering problems. Our published metrics: a 3-second median latency, 99% success rate, and more than 30 million calls a day. On sponsored ad placements — the hardest object to collect for most teams — we hold 91.4% aggregate coverage across 13 marketplaces; that figure is our own measurement, published as a product metric, not an independent benchmark. Treat all four numbers the same way you would treat anyone else's: as claims to verify with the scripts above.

Three things we do not do. First, no account-domain data: orders, inventory, and your own advertising sit inside the official SP-API's authorisation scope, and we neither provide them nor provide a way around them. Second, we do not collect buyer personally identifiable information. Third, we do not collect anything behind a login. These are design boundaries, not roadmap items.

Three situations where you should not use us. One: you need a very small volume at low frequency, where a few lines of your own script beat us. Two: all of your requirements sit inside your own accounts, where SP-API is both more compliant and cheaper. Three: you need login-gated data or buyer PII — which we do not do, and no vendor should.

One special case: if the consumer is an AI agent rather than a fixed report, building a full pipeline may be over-engineering. Agents fetch on demand, in single calls, with unpredictable field scope, which does not match the pipeline's assumptions of batch processing and fixed schemas. In that shape the better move is exposing data capability as tools the agent calls when it needs them, which is why Amazon Data MCP exists — 19 tools over remote HTTP, zero installation. Agent scenarios still need quality judgement; it just moves from a gate inside the pipeline to field-validation guidance on the agent side.

Ruling out the wrong fits is what makes the remaining scope worth the maintenance you save.

The 90-minute checklist

Do it in this order. Every step has a concrete artefact.

Write the field contract (20 min). Fields the business depends on, graded P0/P1/P2. Keep P0 under 15 — more means you have not decided what matters.
Set the freshness SLO (10 min). Give P0 fields a P95 staleness ceiling and write it down. Without that number, freshness cannot be alerted on.
Build collector plus raw archive (20 min). Fetch and store only, no judgement. Far less code than most teams expect.
Run a baseline (20 min). Take 200 to 500 real ASINs and compute the four numbers with the gate scripts. This is your baseline, not an acceptance result.
Wire the gate and alerts (20 min). Record without blocking first, observe two weeks, then set alert and block lines from the baseline distribution.

Five steps and you have a pipeline that complains when data decays. Everything else — cost attribution, schema contract tests, weekly regression — bolts onto this skeleton.

One warning from experience: the step most likely to be abandoned halfway is step four, the baseline. The temptation is to skip it and turn alerts on before the baseline exists, on the grounds that you can tighten it later. But skipping the baseline means guessing thresholds, and guessed thresholds get disabled after the first false alarm. Better to ship alerts two weeks late than to ship an alert that gets switched off.

Wrapping up

The one-sentence version: collection can be outsourced; judgement cannot. Once the collection layer is gone, what you have to build yourself is a mechanism that decides whether the data is still usable — and in this architecture that mechanism is L5.

The point of the gate is not perfect data. It is that degradation becomes an alarm instead of a fact everybody accepts and stops questioning.

Everything here is runnable as written. Take the scripts, point them at whatever you are already using, and get your own four numbers before you decide anything. For product and search objects, Amazon Scraper API is the endpoint; for reviews, Amazon Review API; to skip pipeline construction end to end and let an agent fetch in one step, Amazon Data MCP. Grab a key from the console and run a baseline, or read the Amazon Data MCP technical documentation.

And run the same scripts against us. Completeness numbers a vendor reports about itself — ours included — deserve to be verified rather than believed.

If you have built something like this, I would be curious which of the three shifts bit you hardest. In my experience it is the stale one, because it is the only failure that a well-formatted response cannot hide and a status code cannot reveal.

I Compared Amazon Data APIs by Field. The Checkboxes Were Lying.

Pangolinfo — Fri, 04 Sep 2026 02:51:13 +0000

Every "best Amazon data API" article has the same table.

Vendor | Product Data | Reviews | Search | Price Monitoring
-------|--------------|---------|--------|-----------------
A      | ✓            | ✓       | ✓      | ✓
B      | ✓            | ✓       | ✓      | ✓
C      | ✓            | ✗       | ✓      | ✓

A and B look identical. You pick based on price, or whoever replied to your email fastest.

Here is the thing I learned after actually running the comparison: one of those vendors returned 12 fields. The other returned 58. Same checkmark.

This post is the method I used to find that out. It takes two days, it needs no cooperation from any vendor, and the script is about 90 lines. I am publishing it because the alternative — reading vendor docs and hoping — does not work.

Related: if you have not already, I stopped trusting "real-time" claims after one bad quarter covers the reliability side of this. This post goes one level deeper: not is it fast, but is it complete.

One scope note, because it removes an entire category of pointless comparison: everything below is about vendors competing for the same job. The official SP-API is not one of them — its authorisation model binds to a seller account, not to the marketplace, so competitor data and market-level rank were never in scope. That is a design boundary, not a missing feature. If your requirements sit near that line, read Amazon API vs Web Scraping: Stop Choosing, Build Both Routes first; it will tell you whether you need this post at all.

Why "supported" is an information black hole

A feature grid uses a binary. Real delivery is continuous, and it has at least three layers:

Layer 1: does the schema have the field?        → field coverage
Layer 2: when it has it, does it have a value?  → non-null fill rate
Layer 3: how many records pass both?            → usable record rate

Only layer 3 has anything to do with your business. Layers 1 and 2 are diagnostics.

Let me show you why layer 2 matters more than most people assume.

The null field problem

Here is a real response shape, with values removed:

{
  "asin": "B0C7V9K2XQ",
  "title": "...",
  "coupon": null,
  "deliveryEstimate": null,
  "price": { "current": 49.99, "original": null }
}

Schema validation passes. JSON parsing succeeds. Row count is correct. HTTP status is 200. Your monitoring is green.

And your coupon dimension is permanently empty.

I know a team that signed an annual contract after seeing 60+ fields and 92% coverage. Three months in, the coupon column was empty in most reports and delivery estimates were empty in most reports. Both fields existed in the schema. Both were populated less than 15% of the time. On an annual deal, switching mid-term was expensive.

Ten minutes with the script below would have caught it.

A counterintuitive result

A vendor with 95% coverage and 60% fill rate is usually worth less to you than one with 70% coverage and 98% fill rate.

The first hands you a pile of nulls. You write defensive if (x != null) branches in every downstream consumer, and you still cannot tell "this product has no coupon" from "we failed to fetch the coupon."

The second hands you records that are solid end to end. Fewer dimensions, every one of them trustworthy.

Certain absence beats false completeness. Low coverage means "I do not have this dimension," which you can plan around. Low fill rate means "I think I have this dimension and I do not," which produces bad decisions.

The three response shapes

Feed the same ASINs to different vendors and responses cluster into three shapes. This taxonomy is more useful than any ranking.

Shape A — thin, 12–18 fields

{
  "asin": "B0C7V9K2XQ",
  "title": "...",
  "brand": "...",
  "price": 49.99,
  "currency": "USD",
  "rating": 4.3,
  "reviewCount": 1284,
  "availability": "In Stock",
  "mainImage": "https://...",
  "bsr": 1842,
  "category": "Electronics",
  "url": "https://..."
}

Good for: price monitoring, basic sourcing. Useless for anything involving variants, promotions, ad placement or seller dimensions.

Shape B — mid, 30–40 fields, variants but no ad flags

{
  "asin": "B0C7V9K2XQ",
  "parentAsin": "B0C7V90001",
  "price": {"current": 49.99, "original": 69.99, "currency": "USD"},
  "coupon": {"amount": 5.00, "type": "percentage"},
  "availability": {"status": "In Stock", "deliveryEstimate": "..."},
  "rating": {"value": 4.3, "count": 1284,
             "histogram": {"5": 62, "4": 21, "3": 9, "2": 4, "1": 4}},
  "bsr": [{"category": "Electronics", "rank": 1842},
          {"category": "Portable Audio", "rank": 97}],
  "variants": [
    {"asin": "B0C7V9K2XQ", "attributes": {"color": "Black", "size": "256GB"},
     "price": 49.99, "availability": "In Stock"},
    {"asin": "B0C7V9K2XR", "attributes": {"color": "White", "size": "512GB"},
     "price": 79.99, "availability": "In Stock"}
  ],
  "seller": {"name": "...", "rating": 4.6, "isFulfilledByAmazon": true},
  "images": [{"url": "...", "variant": "MAIN"}],
  "badges": ["Amazon's Choice"]
}

Good for: sourcing, variant analysis, rating distribution. Useless for rank attribution, because ads and organic results are indistinguishable.

Shape C — complete

{
  // everything in shape B, plus:
  "variants": [
    {"asin": "B0C7V9K2XQ", "attributes": {"color": "Black", "size": "256GB"},
     "price": {"current": 49.99, "original": 69.99},
     "availability": "In Stock", "rating": {"value": 4.3, "count": 812},
     "isBuyBoxWinner": true, "offerCount": 7}
  ],
  "isSponsored": false,
  "organicPosition": 1,
  "searchContext": {"keyword": "...", "page": 1, "positionOnPage": 3,
                    "sponsoredCountAhead": 2,
                    "adTypesAhead": ["SB", "SP"]},
  "priceHistory": {"min90d": 44.99, "max90d": 69.99},
  "subscribeAndSave": {"discount": 5},
  "buyBoxWinner": {"seller": "...", "price": 49.99, "shipping": "FREE"}
}

Good for: rank attribution, ad-competitive analysis, variant-level insight, historical price ranges.

On a feature grid these three are the same checkmark. In your reporting, you are not buying data — you are buying which questions you are allowed to ask.

The method

Step 1: write the field contract

Not vendor docs. Your own list, with priorities.

P0 — the record is dead without it. Goes to the dead-letter queue.
P1 — analysis gets shallower, still works.
P2 — nice to have.

CONTRACT = {
    "P0": ["asin", "title", "price.current", "availability.status",
           "rating.value", "rating.count", "bsr", "variants"],
    "P1": ["parentAsin", "brand", "price.original", "price.currency",
           "coupon", "seller.name", "offerCount", "images",
           "badges", "isSponsored", "categoryPath"],
}

Eight to fifteen P0 fields is the sweet spot. The test is one question: if this field were missing, would you dead-letter the record? Yes means P0.

Per object, the critical fields differ:

Object	Critical fields	Why
Product	`price.current`, `bsr`, `variants`	pricing and sourcing fundamentals
Search	`page`, `items[].position`, `items[].isSponsored`, `items[].adType`, `items[].organicRank`	without the last three, your rank tracker is sorting ads and organic together
Reviews	`reviewId`, `asin` (child), `parentAsin`, `variant`, `rating`, `date`, `verifiedPurchase`	`variant` decides whether insight can be pinned to a configuration

Step 2: the flatten function (this is the whole trick)

def flatten(obj, prefix=""):
    """Flatten to a.b.c paths; sample list items; empty values are dropped."""
    out = {}
    if isinstance(obj, dict):
        for k, v in obj.items():
            out.update(flatten(v, f"{prefix}.{k}" if prefix else k))
    elif isinstance(obj, list):
        for v in obj[:50]:
            out.update(flatten(v, f"{prefix}[]"))
    elif obj is not None and obj != "":
        out[prefix] = obj          # <-- the entire trick
    return out

That last elif is the whole method. Empty values never enter the result. So when you later ask "which contract fields came back," you are asking "which contract fields came back with a value."

One function, two metrics. Coverage comes from the count. Fill rate is handled implicitly — the coupon: null case from earlier simply never lands in out.

Step 3: evaluate and aggregate

import statistics

def evaluate(record, cov_threshold=0.8):
    flat = flatten(record)
    contract = CONTRACT["P0"] + CONTRACT["P1"]
    present = [f for f in contract if f in flat]
    coverage = len(present) / len(contract)
    p0_ok = all(f in flat for f in CONTRACT["P0"])   # P0 must be 100% populated
    return {
        "coverage": round(coverage, 3),
        "usable": bool(p0_ok and coverage >= cov_threshold),
    }

def run(records, spend_usd):
    ev = [evaluate(r) for r in records]
    usable = [e for e in ev if e["usable"]]
    return {
        "samples": len(ev),
        "usable_rate": round(len(usable) / len(ev), 4) if ev else 0,
        "avg_coverage": round(statistics.mean(e["coverage"] for e in ev), 3),
        "cost_per_1k_usable": round(spend_usd / len(usable) * 1000, 2)
                              if usable else None,
    }

p0_ok uses all(), not a ratio. P0 is a hard constraint — half present is not usable.

Step 4: classify failures

def classify(record):
    if record.get("_http_status") != 200:
        return "hard"        # visible, retry it
    if not record.get("asin"):
        return "soft"        # 200 but empty body
    flat = flatten(record)
    if not all(f in flat for f in CONTRACT["P0"]):
        return "partial"     # SILENT — the expensive one
    return "ok"

Type	Symptom	Visibility	Cost
Hard	non-200, timeout	Fully visible	Low
Soft	200, empty body	Half visible	Medium
Partial	200, valid structure, P0 null	Silent	Highest
Stale	values present but old	Silent	High

Row three is why this post exists. It raises no error, trips no alert, and generates no billing dispute. It just quietly degrades your reporting.

Step 5: what the output looks like

=== Vendor A ===
  samples            : 300
  avg coverage       : 0.94
  usable rate        : 61.33%
  cost per 1k usable : $12.40
  failures           : {'ok': 184, 'partial': 97, 'soft': 14, 'hard': 5}

=== Vendor B ===
  samples            : 300
  avg coverage       : 0.71
  usable rate        : 92.67%
  cost per 1k usable : $4.85
  failures           : {'ok': 278, 'partial': 12, 'soft': 8, 'hard': 2}

A has much better coverage and two-thirds the usable rate, at 2.5x the cost per usable record.

If you only look at coverage — or at a feature grid — you pick A. Look at usable rate and the conclusion inverts.

Note vendor A's 97 partial records. Measured the traditional way, A's success rate is (184 + 97 + 14) / 300 = 98.3%. Looks excellent. With a field contract applied, real usable rate is 61%.

That gap is exactly where conventional monitoring fails. It confuses "returned" with "usable."

Sample design: why your test showed no differences

When someone tells me "I tested four vendors and they were all the same," the sample set is almost always the problem.

The most common mistake: testing fifteen bestseller ASINs. Those products have the richest data on the platform, so every vendor returns everything and all four score 100%.

You measured Amazon's data completeness for bestsellers, not the vendors' capability.

A discriminating sample has five kinds of record:

Across categories — four or more top-level categories. Page templates differ, field availability follows.
Across marketplaces — US plus at least two non-US. This is the direct test for the "US-only field" surprise.
Heavy variants — 20+ variants, to probe variant depth.
Weak-data products — new listings with no reviews, long-term out of stock, no Buy Box. This is the litmus test for fill rate.
Deep pages — for search, go through page five.

Two hundred to five hundred records per vendor is enough. Below that, noise dominates.

Four execution disciplines — break any one and the numbers stop being comparable:

same ASINs
same time window    <- compress into hours; prices and stock move
same concurrency
same retry policy   <- easiest one to get wrong

That last one: give vendor A three retries and vendor B one, and A's usable rate is inflated while its real cost is triple. You are not measuring the vendor, you are measuring your retry policy.

Five places where vendors actually diverge

After running this repeatedly, differentiation concentrates in five spots. If you are short on time, look here first.

1. Ad placement flags and organic rank

The most discriminating item. Vendors carrying isSponsored and organicRank are a minority; most return only page position.

Without them, the rank metric is systematically distorted. A team tracked a core keyword at a steady position three for four months, concluded organic had plateaued, and prepared to raise ad spend. Recomputed with isSponsored applied, the truth was organic position one with two competitors' Sponsored Brands units above it. Organic had been improving the whole time.

One boolean field put a quarter of budget decisions on an inverted premise.

2. Variant-level depth

Many vendors ship a variants array containing nothing but ASINs — no per-variant price, stock or rating. That tells you five variants exist without telling you which is selling, which is out of stock, or which is losing rating.

A review analysis once produced a high-confidence finding that users complain about battery life. It could not be actioned. Adding variant localised it to one capacity tier — concentrated in the low-spec model, near zero in the high-spec one. The response changed from "redesign the battery across the range" to "fix spec copy on the low tier." Cost fell by an order of magnitude.

Variant attribution is the field that takes an insight from plausible to executable.

3. Deep search pages

Little difference across the first two pages. Sharp divergence from page three: some vendors return duplicates, some return empty.

Consequence: competitive density is decided in the deep pages. Seeing only two pages makes you systematically underestimate competition. And when you can only see two pages, "this market is not competitive" describes your observation range, not the market.

4. Promotions and coupons

coupon, promotions and Subscribe & Save discounts are usually thin. But they matter enormously for pricing — list price alone systematically overstates what competitors transact at.

5. Seller and Buy Box detail

offerCount, buyBoxWinner and seller ratings are core metrics in heavily resold categories, and absent from many responses entirely.

Billing: why entry price tells you nothing

Four billing models, not comparable to each other:

Model	Headline	Reality
Per request	Cheapest, most seductive	Failed requests bill too; deep pages and retries multiply
Per successful result	Middle	200 with missing fields still bills — you pay full price for a broken record
Per record	Impossible to rank by eye	Only comparable as cost per thousand usable records
Monthly quota	Priciest, least friction	Overage rates are steep

A vendor billing per request can quote a third of one billing per record and cost more in practice, because deep pages fail, retries are frequent, and failures bill anyway.

The only comparable number is cost per thousand usable records. Do not do cost accounting on the vendor's billing basis — do it on your own usable record count. The multiple between the two is precisely the cost of silent failures and retries.

Make regression routine

A one-off acceptance test proves it works now. Vendor coverage drifts: page redesigns, policy changes, capacity shifts.

Run a fixed sample weekly — fifty to a hundred records is plenty and cheap — and trend four numbers:

median latency / p95 latency / usable record rate / average coverage

Chart them, and chart the failure classification as a stacked area. A rising partial share is the most valuable early warning signal you can have.

The checklist

Field contract written down, P0/P1/P2 agreed with the business side — not invented by an engineer.
Sample covers all five record types.
Four or more vendors, identical terms.
All four numbers in hand — coverage, fill rate, usable rate, cost per 1k usable.
Failures classified, partial share confirmed. This is the only class needing dedicated code.
Quotas modelled per operation type, with backoff tested against real limits.
Regression scheduled weekly, four numbers trended.

Item seven is the easiest to skip and the most valuable.

Two questions that end the sales conversation

Once the contract is written, there are two questions worth asking every vendor before signing anything. They are not gotcha questions. They are diagnostic, and the shape of the answer tells you more than the content does.

"Can you send me the field-level schema for the product object, not the object list?" Most vendors will send a page saying "product data, search data, review data." Push once. If they can send an actual field dictionary, you just saved two days of testing. If they cannot — or if what comes back is clearly a marketing page with field names sprinkled on it — you have learned something about how they think about delivery.

"If a P0 field comes back null for a week, what happens?" Listen for whether they have a category for that. Vendors who monitor fill rate answer immediately, because they already have the dashboard. Vendors who only monitor HTTP status pause. That pause is the answer.

Neither question is hostile. Both are what a serious buyer asks, and a vendor who handles them well is showing you they have been through this before.

The retry multiplier nobody puts in the pricing calculator

Here is an arithmetic that reorders rankings more often than any feature comparison does.

Say vendor A charges \$0.001 per request and vendor B charges \$0.0025 per record. On paper A looks 2.5x cheaper. Now add behaviour. A has an 82% usable rate on your sample, so you retry failures twice, and failed attempts still bill because the billing unit is the request:

effective_cost = unit_price × attempts_per_usable_record / usable_rate

A: 0.0010 × (1 + 2 × 0.18) / 0.82 = 0.00166
B: 0.0025 × 1.00            / 0.97 = 0.00258

A still wins, but by 1.55x rather than 2.5x. Now change one assumption. Suppose A's deep-page coverage is weak, so the pages you actually care about need three retries and the usable rate on page 3+ drops to 60%:

A (pages 3+): 0.0010 × (1 + 3 × 0.40) / 0.60 = 0.00367

Now A costs more than B, on precisely the task you bought it for. Nothing on the pricing page changed. Only the denominator did.

This is why I stopped normalising list prices and started computing cost per thousand usable records on my own sample. It takes an afternoon, and it is the only number that survives contact with production.

Catching staleness before it reaches your dashboard

Silent failure is the expensive class and staleness is its quietest variant. A field returning a plausible but frozen value passes every schema check you can write. Here is the smallest useful detector — it watches for values that stop moving:

from collections import defaultdict
from datetime import datetime, timedelta

history = defaultdict(list)          # field_path -> [(ts, value)]
STALE_AFTER = timedelta(days=14)

def observe(record, now=None):
    now = now or datetime.utcnow()
    for path, value in flatten(record).items():
        history[path].append((now, value))

def stale_fields(now=None, window=STALE_AFTER):
    now = now or datetime.utcnow()
    out = []
    for path, obs in history.items():
        recent = [(t, v) for t, v in obs if now - t <= window]
        if len(recent) < 5:
            continue                                  # not enough signal yet
        if len({str(v) for _, v in recent}) == 1:
            out.append((path, recent[-1][1]))
    return out

Run it weekly over a fixed sample. A price field that has not moved in two weeks across fifty ASINs is not a stable market — it is a frozen parser. The distinction matters, and no status code will tell you which one you are looking at.

Two ways to accidentally measure the wrong thing

Two production realities will distort every number in this article if you ignore them.

Concurrency changes your success rate. If you run the evaluation at 50 parallel requests and the vendor throttles at 20, the failures you observe are yours, not theirs. You will record a low usable rate that has nothing to do with field coverage, and you will carry that error into the vendor comparison. Run the test at the concurrency you actually plan to use in production, and record that number alongside the results. A measurement you cannot repeat is an anecdote.

Caches hide staleness. If responses are cached, a field can look fresh because you are being served last week's copy. A weekly regression run against a cached endpoint will report perfect stability right up until the cache expires and the parser turns out to have been broken for a month. Either bypass the cache for the measurement run, or record the cache TTL as part of the result and compare like with like — one vendor's five-minute TTL and another's twenty-four-hour TTL are not the same product, whatever the fill rates say.

Both are easy to miss for the same reason: both make the numbers look better than reality. Be suspicious of measurements that come out clean.

Wrapping up

The core idea is one sentence: write down what you need first, then see who can deliver it.

Most people do the reverse — they read the vendor docs and reverse-engineer requirements from them. The docs lead, you buy a pile of fields nobody uses, and the one field you needed is missing.

Ninety lines of Python, two days, four vendors. That is a much better deal than two weeks of reading marketing pages and emailing sales teams for a field dictionary.

If you want to run this against real responses: Amazon Scraper API covers product and search objects, Amazon Review API covers reviews, and Amazon Data MCP is the agent-facing path. Grab a key from the console and run the test; docs are at the Amazon Data MCP overview.

One last thing, and I mean it: run this on us too. Completeness numbers a vendor reports about itself — ours included — deserve to be verified.

If you have run something like this, I would be curious which failure class dominated for you. In my experience partial is the one nobody budgets for.

Amazon API vs Web Scraping: Stop Choosing. Build Both Routes.

Pangolinfo — Wed, 02 Sep 2026 06:28:55 +0000

Every team integrating Amazon data eventually hits the same debate: use the official SP-API, or scrape?

The debate never resolves cleanly, and it never will, because the question is wrong. It assumes the two are alternative implementations of the same thing, so the comparison collapses into which is more compliant, cheaper or more stable. That framing guarantees a bad decision.

Here is the actual situation: they retrieve two different classes of data. The official SP-API gives you your account. Public pages give you the market. Once you internalise that, the route debate disappears and what remains is a concrete architectural division of labour.

This post covers the authorization boundary, the public-versus-private line, what SP-API throttling actually costs in engineering time, and a hybrid architecture with runnable code.

The two domains

Category	Nature	Route
Product facts (title, price, rating, variants, BSR)	Publicly visible	Public collection
Search and ads (keyword, rank, ad type, creative)	Publicly visible	Public collection
Review content (rating, body, date, variant)	Publicly visible	Public collection
Ranks and categories	Publicly visible	Public collection
Orders and fulfillment	Account-private	SP-API (authorized)
Buyer information	Account-private + personal	SP-API (restricted PII role)
Inventory and settlement	Account-private	SP-API (authorized)
Own ad performance	Account-private	SP-API (authorized)

The route follows from the nature of the data, not from preference. This is also why "can we just use the official API" is answered no in most teams — you need market data, and it sits outside the official interface's authorization scope.

You would not use the SP-API to check a competitor's price, because it is not designed to expose that. You would not scrape your own orders, because they sit behind login and are account-private.

So the useful question is not "which one". It is: which of my requirements are account-domain and which are market-domain? For most teams the answer is both.

Where each route gets its legitimacy

Compliance discussions slide into an unanswerable "which is more legal". The two routes draw legitimacy from entirely different places, and what determines compliance is what you collect, not which method you use.

The SP-API is legitimate through an agreement chain: developer registration, seller authorization via Login with Amazon, role-based access granting minimum necessary scope. Operations touching personally identifiable information require a separate restricted role and review. What you can violate is the developer agreement and data protection policy, and the consequence is revoked authorization. The boundary is clear but narrow — only what a seller has actively authorized your application to access.

Public page collection has no contract. It sits under the platform's conditions of use, robots directives and rate limits. That is not the same as prohibition; publicly visible, non-personal page information can generally be collected, which is long-standing practice.

Risk rises with three specific things:

Login-gated content — anything visible only after signing in has clearly crossed the public line.
Personal data — buyer names, addresses and contact details are governed by data protection law.
Aggressive request rates — traffic heavy enough to affect site operation shifts the activity from reading public information to interfering with a service.

The one-line test: is the data publicly visible and non-personal? The method is irrelevant. Misusing PII through the official interface is still a violation; collecting a fully public page can still be compliant.

Here is what structured public data looks like. Every field is public and non-personal:

{
  "asin": "B0CXYZ1234",
  "marketplace": "amazon.com",
  "title": "Stainless Steel Insulated Water Bottle, 32 oz",
  "price": { "current": 34.99, "currency": "USD", "listPrice": 44.99 },
  "rating": { "average": 4.6, "count": 12847 },
  "bsr": [ { "category": "Sports & Outdoors", "rank": 128 } ],
  "availability": "In Stock",
  "sponsored": false,
  "fetchedAt": "2026-08-30T10:22:41Z"
}

No buyer information, nothing requiring seller authorization. That is the compliance basis. By contrast, orders, buyer addresses and settlement detail never appear on public pages — they are only reachable through the SP-API after explicit authorization.

What SP-API throttling actually costs

This is the part teams consistently underestimate: even though the official interface is free, throttling converts into engineering complexity.

The SP-API uses a token bucket model. Each operation has its own request rate and quota, and quota refills continuously at that rate. For some operations, initial quota and refill rate also scale with seller business volume. Three consequences:

Quotas are independent per operation. Estimating on "total call volume" is always wrong. getOrders and getInventorySummaries are separate buckets and need separate models. This is the single most common design mistake.

Being throttled is normal, not exceptional. At high polling frequency it is close to inevitable. The correct response is queueing with backoff, not letting it throw and kill the job. Treating 429 as an error also destroys your alerting signal — the team learns to ignore it, and then the real failure goes unnoticed.

Values change. Amazon adjusts rates and quotas. Verify current figures in the official documentation rather than reusing numbers from older docs. I have watched a team design quotas from a year-old document and get throttled into the ground on day one.

The code

A per-operation token bucket

import threading
import time


class TokenBucket:
    """Per-operation token bucket.

    rate:      tokens refilled per second (SP-API restore rate)
    capacity:  bucket size (SP-API maximum quota / burst)
    """

    def __init__(self, rate: float, capacity: float):
        self.rate = rate
        self.capacity = capacity
        self._tokens = capacity
        self._updated = time.monotonic()
        self._lock = threading.Lock()

    def acquire(self, tokens: float = 1.0) -> float:
        """Block until tokens are available. Returns seconds waited."""
        waited = 0.0
        while True:
            with self._lock:
                now = time.monotonic()
                self._tokens = min(
                    self.capacity,
                    self._tokens + (now - self._updated) * self.rate,
                )
                self._updated = now
                if self._tokens >= tokens:
                    self._tokens -= tokens
                    return waited
                deficit = tokens - self._tokens
                sleep_for = deficit / self.rate
            # sleep outside the lock so other threads are not blocked
            time.sleep(sleep_for)
            waited += sleep_for


class OperationLimiter:
    """Maintains an independent bucket per SP-API operation."""

    def __init__(self, config: dict):
        self._buckets = {
            op: TokenBucket(cfg["rate"], cfg["capacity"])
            for op, cfg in config.items()
        }

    def acquire(self, operation: str) -> float:
        bucket = self._buckets.get(operation)
        return bucket.acquire() if bucket else 0.0

Calls with backoff

import random
import requests

RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def call_sp_api(limiter, operation, url, headers, params, max_retries=3):
    """SP-API call with token bucket queueing and exponential backoff."""
    last_error = None

    for attempt in range(max_retries + 1):
        limiter.acquire(operation)  # queue for a token first
        try:
            resp = requests.get(url, headers=headers, params=params, timeout=30)
        except requests.RequestException as exc:
            last_error = f"network: {exc}"
            time.sleep(2 ** attempt + random.uniform(0, 0.5))
            continue

        if resp.status_code == 200:
            return resp.json(), None

        if resp.status_code in RETRYABLE_STATUS:
            retry_after = resp.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay + random.uniform(0, 0.3))
            last_error = f"http {resp.status_code}"
            continue

        # 4xx other than 429: parameter or auth problem, retrying is pointless
        return None, f"http {resp.status_code} (not retryable): {resp.text[:200]}"

    return None, f"exhausted retries: {last_error}"

Two details that matter. Prefer the server's Retry-After header over your own backoff — it is more accurate. And only retry 429 and 5xx; a 400, 401 or 403 is a parameter or authorization problem that will fail identically a hundred times while burning quota and hiding the real error.

The unified ingestion layer

from dataclasses import dataclass, field
from typing import Callable


@dataclass
class IngestResult:
    ok: bool = False
    data: dict = field(default_factory=dict)
    failure_type: str = ""   # throttled | blocked | incomplete | request_error
    missing: list = field(default_factory=list)


class UnifiedIngestion:
    """Converges SP-API and public collection into one internal model."""

    def __init__(self, field_map: dict, required_fields: list):
        self.field_map = field_map      # external field -> internal field
        self.required = required_fields

    def normalize(self, payload: dict) -> dict:
        """Allowlist normalization: keep only declared internal fields."""
        out = {}
        for src_key, dst_key in self.field_map.items():
            if src_key in payload and payload[src_key] not in (None, "", [], {}):
                out[dst_key] = payload[src_key]
        return out

    def validate(self, record: dict) -> list:
        return [f for f in self.required if f not in record]

    def ingest(self, fetcher: Callable, **kwargs) -> IngestResult:
        """fetcher returns (payload, error); non-empty error means request failure."""
        result = IngestResult()
        payload, error = fetcher(**kwargs)

        if error:
            result.failure_type = "throttled" if "429" in str(error) else "request_error"
            return result

        record = self.normalize(payload)
        missing = self.validate(record)

        if missing:
            # parsed fine but fields absent: a coverage problem, not a request problem
            result.failure_type = "incomplete"
            result.missing = missing
            return result

        result.ok = True
        result.data = record
        return result

The critical design choice is three separate failure classes:

throttled / request_error — request layer; look at quota and backoff
blocked — a 200 that is not a real page (public collection side)
incomplete — parsed fine, required field missing (a coverage problem)

These have completely different fixes. Merge them into one error log and you cannot tell whether to adjust quotas, lower concurrency or chase coverage.

Detecting blocked responses at the content layer

This is the most expensive gap on the public collection side. A blocked page returns HTTP 200. Captcha interstitials, bot-detection pages and pages that never finished rendering all come back green. If your check is the status code, that bad data enters your warehouse wearing a success badge.

BLOCK_MARKERS = [
    "captcha", "robot check", "enter the characters you see",
    "automated access", "to discuss automated access",
]


def looks_blocked(html: str) -> bool:
    """Content-layer block detection. Status code alone is not enough."""
    lowered = html[:20000].lower()
    hits = sum(1 for m in BLOCK_MARKERS if m in lowered)
    # markers present, or page implausibly short for a real product page
    return hits > 0 or len(html) < 2000

Wire this into UnifiedIngestion and blocked responses get classified as blocked instead of silently becoming data.

Compliance as code, not as a wiki page

Most teams keep compliance requirements in a document and enforce nothing in code. The result is "the doc says don't collect that, the code has been collecting it for months" — which is the worst position to be in during an audit.

Four constraints worth hardcoding:

A field allowlist. Keep only fields your internal model declares; drop everything else. Note that normalize() above is already allowlist-based rather than denylist-based, so even if an upstream response starts carrying personal data, it never lands in your store.

Path constraints. Collection jobs may only request public paths. Account pages and login redirects should be rejected and alerted on.

A hard rate ceiling at the job layer, so one runaway job cannot push overall frequency into "interfering with a service" territory.

Audit logging of source, time and field list per collection, plus a retained fetchedAt on every record. Without a capture timestamp a record cannot enter a time series and cannot be audited later.

Freshness is a budget decision

Having argued the official API is not free, the opposite error deserves equal attention: running every collection job at maximum frequency.

Field change rates vary enormously. Brand, category and variant dimensions barely move. Price, stock, Buy Box and sponsored placements move constantly. Treating them identically is the most common form of budget waste in this category.

Three tiers work in practice:

High frequency (minutes to hours) — price, stock, Buy Box holder, sponsored placements, BSR. For these, a stale value does not weaken the insight, it invalidates it. A competitor's price from yesterday cannot support a pricing decision today.

Daily — new reviews, rating changes, seller lists, best seller ranks. Trend signals that matter over weeks, not minutes.

Weekly or on demand — brand, category, variant dimensions, images, A+ content.

The top 20% of ASINs drive roughly 80% of decisions. Run those hot and the long tail cold and cost typically halves with almost no business impact. No vendor negotiation required — it is a scheduling decision in your own ingestion layer.

Implement it there rather than hoping a provider does it, and revisit it periodically. A tiering decision made eighteen months ago is usually wrong today.

When comparing prices on the collection side, use cost per thousand usable records, not per thousand requests. Failed, blocked and field-incomplete responses all bill. A plan that is 25% cheaper on the rate card is often only about 10% cheaper in reality.

And for the build decision, convert engineering hours to money first. A maintenance load of 0.3 engineer-days a month sounds trivial; priced at real internal cost over a year it frequently exceeds the API bill for a mid-sized volume — and unlike the bill, it recurs every quarter and grows with every page redesign.

What good looks like

Median latency under five seconds for on-demand fetches. p95 no more than roughly twice the median. Success rate above 98% measured at the content layer, not the status code. Field completeness above 95% against your own list.

Those are not laws of nature. They are the range where teams stop maintaining the data layer and start building on it. Below that range, the data layer becomes the most expensive dependency you have — not because of the invoice, but because of everything your engineers stop building while they babysit it.

Two fields worth treating as non-negotiable:

fetchedAt. Without a capture timestamp, a record cannot enter a time series and cannot be audited. Three months later, when a price looks wrong, you cannot tell whether it was wrong then or got mangled in processing.

isSponsored. Without it, organic rank and ad placement are indistinguishable. You think you rank third; the top two slots are ads; your true organic position is first. Every decision built on that number optimises for the wrong target.

Sponsored is also the hardest object to collect reliably, which is precisely why it needs its own evaluation line rather than an assumption that every provider has it.

The decision table

Skip the route debate and read your task off this:

Your task	Domain	Route
Sync your own orders and fulfillment	Account-private	SP-API
Manage inventory and settlements	Account-private	SP-API
Monitor competitor price and stock	Public market	Public collection / specialized API
Track keyword organic rank	Public market	Public collection / specialized API
Analyse competitor ad placements	Public market	Public collection / specialized API
Run review insights	Public market	Public collection / specialized API
Need both orders and competitor monitoring	Both	Hybrid architecture
Only your own operating data	Account-private	SP-API only

That last row deserves emphasis: if your requirements stop at your own operations, do not add public collection. Architectural complexity should match problem complexity. Converging two channels costs real money, and it is only worth paying when requirements genuinely span both domains.

One operational trap

A specific misdiagnosis shows up constantly in hybrid pipelines.

Symptom: volume for a field drops, or a window of data has holes.

First instinct: data quality degraded. So the team goes to the vendor, checks parsers, digs through logs. Frequently the real cause is quota — the token bucket saturated, jobs queued, and some timed out and were dropped.

Distinguishing them is easy if you classified failures: if throttled is climbing while blocked and incomplete are flat, it is a quota problem, not a data problem. The fixes are entirely different — tiering and scheduling for the former, coverage and parsing for the latter.

And do not alert on 429 directly. Record it as a throttled counter and watch the trend; alert only when the share crosses a sustained threshold. Otherwise the team learns to ignore the alert, and then the real failure goes unseen.

When to build nothing

Worth stating the inverse, since architecture posts have a bias toward building.

Requirements stop at your own operations — orders, inventory, your listings, your ads. Use the SP-API and stop there. Public collection buys coverage you will never use.

You need a one-off answer — a category survey, a competitor list for a deck. Do not build a pipeline. Pull the data and move on. Infrastructure built for a single question tends to outlive the question.

Volume is tiny and fields are narrow — thirty ASINs, four fields, weekly. Write a script. The break-even arithmetic only crosses once maintenance becomes recurring work, and below that line simpler is genuinely better.

Hybrid is correct when requirements span both domains. Verify that condition before committing engineering time. A decision framework is only half useful if it cannot also tell you not to build.

Wrapping up

The ordering matters more than any individual decision:

Split requirements into account-domain and market-domain.
Build the unified ingestion layer before either pipeline.
Model SP-API throttling per operation; retry only 429 and 5xx.
Detect blocked at the content layer, never on status code alone.
Classify failures three ways and put the ratios on a dashboard.

For the market side, Amazon Scraper API and Amazon Review API cover the public objects; Amazon Data MCP is the path when an agent calls data directly. Keys come from the console, and the MCP docs cover the agent path.

The full comparison, including the authorization boundary and decision table: Amazon API vs Web Scraping.

One more note on why the classification approach is worth adopting: it ends arguments that otherwise recur. Framed as compliance versus pragmatism, this debate never resolves, because nobody changes values in a meeting. Framed as "is this data public and non-personal", it resolves in half an hour.

If you are running a hybrid pipeline already, I would be curious which of the three failure classes dominates for you. In my experience incomplete is the one nobody budgets for.

I Stopped Trusting "Real-Time" Claims After One Bad Quarter. Here's the 100-Call Test I Run Instead.

Pangolinfo — Tue, 01 Sep 2026 03:32:20 +0000

A few years ago I integrated an Amazon data provider into a pricing service. The sales page said "real-time data". The docs said "high success rate". Both were technically true and both were useless.

What I eventually learned: the pricing service was reading from a cache refreshed once a day, and "high success rate" counted HTTP 200 responses, which included a meaningful number of captcha pages. We shipped two weeks of pricing recommendations built partly on blocked page responses, because our client only checked the status code.

This post is the test I wish I had run first. It takes an afternoon, it is about a hundred lines of Python, and it turns every vague vendor claim into four numbers you can put on a dashboard.

The four numbers that actually matter

Before the code, the definitions. These four are the difference between a provider you can build on and a provider you have to babysit.

Metric	Definition	Why it matters
Median latency	50th percentile response time	Your everyday experience
p95 latency	95th percentile response time	Sets your timeout and retry policy
Success rate	Parsed successfully ÷ total requests	Not HTTP 200. Determines your real unit price
Field completeness	Records with all required fields ÷ successful records	Whether the data is usable at all

The success rate definition deserves emphasis, because it is where most home-grown checks go wrong. An HTTP 200 that returns a bot-detection page is a failure, not a success. If your client only looks at the status code, that bad record flows straight into your database and you find out two weeks later when someone asks why the dashboard looks strange.

On p95: it determines your timeout. If p95 is five seconds and you set a three second timeout, you are throwing away a slice of requests that would have succeeded. Set it to ten seconds and you slow the whole pipeline. A reasonable starting point is 1.5× p95, with a retry cap of two. More retries than that usually means you are being throttled, and retrying into a throttle makes it worse.

Why per-request pricing is the wrong unit

Here is the part that changes how you read a pricing page.

A failed request still bills. A blocked page still bills. A response missing the field you need still bills.

So the unit worth comparing is cost per thousand usable records:

cost per 1k usable records = (monthly spend ÷ records parsed successfully with all required fields) × 1000

Worked example, because the gap is counterintuitive:

Plan A: $1.20 per 1k requests, 92% success, 88% field completeness. Usable share is roughly 81%, so the real cost is about $1.48 per 1k usable records.
Plan B: $1.60 per 1k requests, 99% success, 98% field completeness. Usable share is about 97%, so the real cost is about $1.65 per 1k usable records.

Plan A looks 25% cheaper on the rate card. The real gap is about 10%. Add the debugging hours Plan A will generate and the ranking usually flips.

So the question to ask a vendor is never "how much per thousand requests". It is: how do you define success rate, what is your field completeness, and do failures bill?

The four-layer pipeline

Whichever route you take, the chain has the same shape. What differs is which layers you own.

Amazon public pages (product / search / review / best sellers / sponsored)
        ↓
[Collection]   anti-bot · browser rendering · proxies and geo · retries
        ↓
[Structuring]  parsing · field normalization · type contract · failure semantics
        ↓
[Delivery]     REST API   ── or ──   MCP tools
        ↓
Your application / data pipeline / AI agent

Four routes, and how they split ownership:

Route	Collection	Structuring	Delivery	You maintain
Self-built scraper	You	You	You	Everything
General scraping API	Vendor	You	Vendor	Parsing, field drift
Amazon-native data API	Vendor	Vendor	Vendor	Business logic only
Official SP-API	Vendor	Vendor	Vendor	Quota and auth

Choosing a route is really deciding which layers you want to own. More layers means more control and more responsibility.

One thing worth stating plainly: if you only need data tied to your own seller account, use the official SP-API and skip third parties entirely. Orders, inventory, your own listings. It is the most compliant and cheapest path, and recommending anything paid for that job would be dishonest.

The real fork is whether you need public marketplace facts — competitors, categories, search results, sponsored placements. That is where the other three routes become relevant.

Build the test

Enough theory. Here is the script. It samples across marketplaces and object types, then reports all four numbers plus a breakdown of which fields went missing.

#!/usr/bin/env python3
"""
Amazon Data API acceptance test.
Samples N calls across marketplaces and object types, then reports
median latency, p95 latency, success rate and field completeness.
"""

import argparse
import os
import statistics
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field

import requests

API_BASE = "https://api.pangolinfo.com"  # confirm against current docs

# Your business-critical fields. Change these to match what you actually store.
REQUIRED_PRODUCT_FIELDS = ["asin", "title", "price", "rating", "bsr", "fetchedAt"]
REQUIRED_SEARCH_FIELDS = ["keyword", "page", "results", "sponsoredCount", "fetchedAt"]

MARKETPLACES = ["amazon.com", "amazon.co.uk", "amazon.de"]
SAMPLE_ASINS = [
    "B0CXYZ1234", "B0ABCDEFGH", "B012345678",
    "B0TEST0001", "B0TEST0002", "B0TEST0003",
]
SAMPLE_KEYWORDS = ["insulated water bottle", "standing desk", "air purifier"]


@dataclass
class CallResult:
    ok: bool = False
    latency_ms: float = 0.0
    fields_ok: bool = False
    missing: list = field(default_factory=list)
    error: str = ""


def check_fields(payload: dict, required: list) -> tuple:
    missing = [f for f in required if f not in payload or payload[f] in (None, "", [], {})]
    return (not missing), missing


def call_product(api_key: str, asin: str, marketplace: str) -> CallResult:
    result = CallResult()
    started = time.perf_counter()
    try:
        resp = requests.get(
            f"{API_BASE}/product",
            params={"asin": asin, "marketplace": marketplace},
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=30,
        )
        result.latency_ms = (time.perf_counter() - started) * 1000

        # HTTP 200 is not success. A blocked page also returns 200.
        if resp.status_code != 200:
            result.error = f"HTTP {resp.status_code}"
            return result

        payload = resp.json()
        data = payload.get("data", payload)
        result.ok = True
        result.fields_ok, result.missing = check_fields(data, REQUIRED_PRODUCT_FIELDS)
    except Exception as exc:  # noqa: BLE001
        result.latency_ms = (time.perf_counter() - started) * 1000
        result.error = f"{type(exc).__name__}: {exc}"
    return result


def call_search(api_key: str, keyword: str, marketplace: str, page: int) -> CallResult:
    result = CallResult()
    started = time.perf_counter()
    try:
        resp = requests.get(
            f"{API_BASE}/search",
            params={"keyword": keyword, "marketplace": marketplace, "page": page},
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=30,
        )
        result.latency_ms = (time.perf_counter() - started) * 1000
        if resp.status_code != 200:
            result.error = f"HTTP {resp.status_code}"
            return result
        payload = resp.json()
        data = payload.get("data", payload)
        result.ok = True
        result.fields_ok, result.missing = check_fields(data, REQUIRED_SEARCH_FIELDS)
    except Exception as exc:  # noqa: BLE001
        result.latency_ms = (time.perf_counter() - started) * 1000
        result.error = f"{type(exc).__name__}: {exc}"
    return result


def build_tasks(api_key: str, samples: int):
    """Alternate product and search calls; rotate marketplace and page depth."""
    tasks = []
    for i in range(samples):
        marketplace = MARKETPLACES[i % len(MARKETPLACES)]
        if i % 2 == 0:
            asin = SAMPLE_ASINS[i % len(SAMPLE_ASINS)]
            tasks.append(("product", call_product, (api_key, asin, marketplace)))
        else:
            keyword = SAMPLE_KEYWORDS[i % len(SAMPLE_KEYWORDS)]
            page = (i % 3) + 1  # 1..3, also probes deep-page availability
            tasks.append(("search", call_search, (api_key, keyword, marketplace, page)))
    return tasks


def percentile(values: list, pct: float) -> float:
    if not values:
        return 0.0
    ordered = sorted(values)
    idx = min(int(len(ordered) * pct / 100), len(ordered) - 1)
    return ordered[idx]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--samples", type=int, default=100)
    parser.add_argument("--concurrency", type=int, default=8)
    args = parser.parse_args()

    api_key = os.environ.get("PANGOLINFO_API_KEY")
    if not api_key:
        raise SystemExit("Set PANGOLINFO_API_KEY first")

    tasks = build_tasks(api_key, args.samples)
    results: list = []

    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
        futures = [pool.submit(fn, *params) for _, fn, params in tasks]
        for future in as_completed(futures):
            results.append(future.result())

    latencies = [r.latency_ms for r in results]
    succeeded = [r for r in results if r.ok]
    complete = [r for r in succeeded if r.fields_ok]

    total = len(results)
    success_rate = len(succeeded) / total * 100 if total else 0
    completeness = len(complete) / len(succeeded) * 100 if succeeded else 0
    usable_rate = len(complete) / total * 100 if total else 0

    print("\n=== Amazon Data API acceptance report ===")
    print(f"samples            : {total}")
    print(f"median latency     : {statistics.median(latencies):.0f} ms")
    print(f"p95 latency        : {percentile(latencies, 95):.0f} ms")
    print(f"max latency        : {max(latencies):.0f} ms")
    print(f"success rate       : {success_rate:.1f}%")
    print(f"field completeness : {completeness:.1f}%")
    print(f"usable share       : {usable_rate:.1f}%")

    missing_counter: dict = {}
    for r in succeeded:
        for field_name in r.missing:
            missing_counter[field_name] = missing_counter.get(field_name, 0) + 1
    if missing_counter:
        print("\nmissing fields (by occurrence):")
        for field_name, count in sorted(missing_counter.items(), key=lambda x: -x[1]):
            print(f"  {field_name}: {count}")

    errors: dict = {}
    for r in results:
        if r.error:
            key = r.error.split(":")[0]
            errors[key] = errors.get(key, 0) + 1
    if errors:
        print("\nerror distribution:")
        for key, count in sorted(errors.items(), key=lambda x: -x[1]):
            print(f"  {key}: {count}")

    print("\nFeed the usable share into the formula to get your real unit price:")
    print("  cost per 1k usable records = monthly spend / usable records * 1000")
    print(f"  at {usable_rate:.1f}% usable, the list price is effectively {100 / usable_rate:.2f}x higher\n")


if __name__ == "__main__":
    main()

Reading the output

A healthy run looks something like this:

=== Amazon Data API acceptance report ===
samples            : 100
median latency     : 2980 ms
p95 latency        : 5240 ms
max latency        : 8120 ms
success rate       : 99.0%
field completeness : 98.0%
usable share       : 97.0%

Two things to look at beyond the headline numbers.

The missing-fields breakdown. If gaps cluster on one field, that field has weak coverage in your target categories. That is a specific conversation with the vendor, not a reason to reject the whole provider.

The error distribution. If you see a lot of timeouts, you have a latency problem. If you see parse failures on otherwise healthy responses, you have a structuring problem. These two have completely different fixes, and lumping them into one error log is how teams spend a week debugging the wrong thing.

Then re-run it weekly. A single run proves it works today. Weekly runs prove it still works. Quality degradation is almost always gradual — drifting from 99% to 95% takes weeks, and humans notice about two weeks after it matters.

The field matrix behind the checklist

The REQUIRED_PRODUCT_FIELDS list should not be invented. It should come from a matrix of what public Amazon pages can actually be structured into.

Object	Key fields	Common gaps
Product	ASIN, title, brand, price, rating, BSR, availability, variants	Incomplete variant dimensions; lost parent-child ASIN links; sale price mixed with list price
Search	Keyword, page, organic rank, ad rank, isSponsored	Sponsored mixed with organic; deep pages (7+) truncated
Review	Rating, title, body, date, verified purchase, variant, helpful votes	Truncated bodies from summary pages; no variant attribution
Offers	Seller, price, shipping, Buy Box holder, stock	Inconsistent Buy Box logic; only first offer returned
Best Sellers	Category, rank, ASIN, rank movement	Unlabeled category-tree changes; non-continuous history
Seller	Seller ID, name, rating, listing count	Weak seller-to-brand mapping; IDs differ by marketplace
Category	Category tree, node ID, filters, item count	Node IDs vary by site; incomplete filter enums
Sponsored	Ad type (SP/SB/SD), placement, rank, creative	Ads not separated from organic; missing creative fields

Two fields I would put on every checklist.

fetchedAt. A capture timestamp. Without it the record cannot enter a time series and cannot be audited. Three months later, when a price looks wrong, you cannot tell whether it was wrong then or got mangled later.

isSponsored on search results. Without it, organic rank and ad placement are indistinguishable. You think you rank third; the top two slots are ads; your true organic position is first. Every optimization decision built on the contaminated number points the wrong way.

Not everything needs to be real time

Having argued that "real time" is widely abused, it is worth being equally clear about the opposite mistake: refreshing everything at maximum frequency.

Field change rates vary by orders of magnitude. Brand names, category assignments and variant dimensions barely move. Price, stock, Buy Box ownership and sponsored placements move constantly. Treating them identically is the most common form of budget waste in this category.

A three-tier split is usually right:

High frequency, minutes to hours — price, stock, Buy Box holder, sponsored placements, BSR. For these, a stale value does not weaken the insight, it invalidates it. A competitor's price from yesterday cannot support a pricing decision today.

Daily — new reviews, rating changes, seller lists, best seller ranks. Trend signals that matter over weeks rather than minutes.

Weekly or on demand — brand, category, variant dimensions, images, A+ content. Refreshing these hourly is pure waste with no analytical benefit.

The practical observation from many deployments: the top 20% of ASINs drive roughly 80% of the decisions. Running that 20% at high frequency and the long tail weekly typically cuts cost by half with almost no measurable business impact. It is the highest-leverage cost optimization available, and it requires no vendor negotiation — it is purely a scheduling decision in your own ingestion layer.

Two implementation notes.

First, implement the tiering in your own ingestion layer rather than hoping the provider does it. You know which ASINs matter to your business; they do not.

Second, revisit it periodically. Business priorities shift, a category that was long tail becomes strategic, and a tiering decision made eighteen months ago is usually wrong today. Put a recurring reminder on it.

REST or MCP

Worth separating, because they get conflated constantly.

REST fits deterministic, scheduled batch work. You know which ASINs to pull, how often, and which table the results land in. Data pipelines and scale collection.

MCP fits exploratory work that needs reasoning. A user or agent states a business question and the agent decides which tools to call, how many pages to check, and how to cross-validate. Research, diagnostics, one-off analysis.

One line: with REST you write a program that fetches data; with MCP you ask an agent to fetch and explain. Backend service consumer, use REST. AI agent consumer, MCP removes a lot of glue code.

The decision tree

Step 1. Is the data only about your own seller account? Yes, use the official SP-API. No, continue.

Step 2. Do you need public marketplace facts — competitors, categories, search, sponsored placements? No, revisit the requirement. Yes, continue.

Step 3. Do you have engineers who will own anti-bot, rendering and parsing maintenance long term? No, pick an Amazon-native data API. Yes, convert engineering hours to money and compare against purchase price on a per-thousand-usable-records basis.

Step 3 is where teams go wrong by comparing cash only. Price the engineering time at real internal cost. Plenty of "building is cheaper" instincts reverse once that number is on the page.

For reference, Pangolinfo publishes a median latency around 3 seconds, a 99% success rate, and more than 30 million calls per day; on sponsored placements, the published collection rate across 13 marketplaces is 91.4%. The point of those numbers is not that they are large. It is that you can reproduce or refute them with the script above.

Wrapping up

If the numbers come back bad, here is the triage.

Low success rate with high latency is usually capacity. Lower your concurrency, or ask about quotas and whether you are being throttled.

High success rate with low field completeness is a coverage problem. Identify exactly which fields are missing and ask about those specific fields in your target categories. This is often fixable, or at least scoped to particular categories you can route around.

Good numbers on page one that collapse by page three is a deep-page limitation. This one is usually a hard boundary rather than a transient issue, and it matters enormously if your business depends on tracking keywords where you rank deep.

Knowing which of the three you have determines whether the fix is configuration, a conversation, or a different provider. Collapsing them into "the data is bad" is how teams waste a week.

If you take one thing from this post, take the ordering:

Write the field checklist first (15–30 fields).
Score candidates field by field, not "do you support products?".
Convert list price to cost per thousand usable records.
Run the 100-call test and record all four numbers.
Look at the price sheet last.

If you want to run the script against real data, grab a key from the console or read the Amazon Data MCP docs. The underlying products are Amazon Scraper API for product, search, best seller, category and sponsored objects, Amazon Review API for reviews and customer voice, Amazon Data MCP when an agent needs to call data directly, and Amazon Scraper Skill for conversational workflows.

The full buyer's guide with the field matrix, three real JSON samples and the acceptance method is here: Amazon Data API: The Complete Buyer's Guide.

If you run the test, I would be curious what numbers you get — especially p95. That is the one where published specs and reality tend to diverge most.

Build an Amazon Review Monitor: ratings, complaint signals, and review media

Pangolinfo — Tue, 01 Sep 2026 02:43:34 +0000

Amazon review data is useful only when it can answer a decision: has a new quality issue emerged, what do verified buyers say, and is there visual evidence behind it? This guide shows a small, production-oriented pattern for collecting reviews, flagging complaint risk, and storing the fields needed for follow-up analysis.

The example uses the Pangolinfo Amazon Review API. It returns structured review records rather than page HTML, including star rating, title, text, verified-purchase status, helpful votes, product variant data, review images, and review video URLs.

1. Decide what to monitor

Start with a target ASIN and a clear observation rule. A useful baseline is:

retrieve the most recent page of reviews every day;
retrieve one-star reviews separately to surface issues early;
preserve image and video URLs for review validation;
keep the review ID and date so repeated runs can be deduplicated.

This keeps the workflow focused on new customer feedback instead of reprocessing the entire review history.

2. Make a structured review request

Create a Pangolinfo API Key in the Console, then make the request below. Do not put a production key into shared workflows, browser code, or public collections.

curl --request POST \
  --url https://scrapeapi.pangolinfo.com/api/v1/scrape \
  --header 'Authorization: Bearer YOUR_PANGOLINFO_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://www.amazon.com/dp/B076CLQDR4",
    "site": "",
    "bizContext": {
      "bizKey": "review",
      "pageCount": 1,
      "asin": "B076CLQDR4",
      "filterByStar": "one_star",
      "sortBy": "recent"
    },
    "format": "json",
    "formatType": "all_formats",
    "mediaType": "all_contents",
    "parserName": "amzReviewV2"
  }'

The same endpoint supports all_stars when you need a broad feedback sample. Full request and response fields are documented in the Amazon Review API reference.

3. Persist only the fields that drive decisions

For each record in data.json[].data.results[], store at least:

Field	Why it matters
`reviewId`	Deduplicate repeated scheduled runs.
`star`, `title`, `content`	Detect rating shifts and complaint themes.
`date`, `country`	Attribute changes to a time period and marketplace.
`purchased`, `vineVoice`	Separate verified-buyer feedback from other sources.
`helpful`	Prioritize issues readers find useful.
`imgs`, `videos`	Preserve evidence for quality and listing reviews.
`attributes`, `asin`	Identify a problematic size, color, or variant.

4. Turn raw reviews into alerts

A simple alert rule is more robust than a generic sentiment score:

Alert when a new verified one-star review has an image or video.
Alert when two or more reviews in the same run mention the same failure mode.
Include ASIN, review ID, rating, title, date, and evidence URLs in the alert.
Add a cooldown key such as asin:reviewId so the same review is not sent twice.

For no-code automation, the Pangolinfo n8n node and public Postman Amazon Review API collection provide starting points.

5. Use the MCP route for agent workflows

If an agent needs to combine reviews with product, keyword, niche, and AI-search data, use Amazon Data MCP instead of maintaining separate parsers. The MCP server exposes the review capability alongside the broader Amazon and AI-search toolset, while your application keeps control of its own API key and usage limits.

Practical takeaway

An effective review monitor is not an endless scrape. It is a small, repeatable process: collect recent structured feedback, retain identifiers and evidence, detect new recurring issues, and send a deduplicated alert to the team that can act on it.

We Built an Amazon Brand Operations Cockpit for a Global Electronics Brand. Here's the Architecture.

Pangolinfo — Mon, 24 Aug 2026 09:27:10 +0000

Most "data-driven Amazon" setups stop at exporting backend numbers into a spreadsheet. Exporting is not operating. This post walks through a real, anonymized project — a global consumer-electronics brand in the action-camera and pocket-gimbal category, across several major overseas markets — where we rebuilt their growth decisioning from a weekly lag into an hour-level response system: an Amazon Brand Operations Cockpit. The brand name and specific numbers are redacted; the architecture, dimensions, and SOPs are reusable.

The problem: why monthly reports can't save your growth

A monthly report is post-hoc description. Its collection grain is weekly, its presentation grain is monthly, while Amazon's competitive rhythm is hourly: deal badges, coupons, Buy Box rotation, keyword-rank jumps all happen within a single day. By the time the monthly report ships, the window is closed.

Our goal for this brand was never "a prettier report." It was: anomalies surface themselves the same day, with an attached "what to do." That requires splitting signals into three layers and bolting the third directly onto the data.

Layer 1: real-time Amazon sales signals (8 dimensions)

We locked in eight concrete dimensions, each with a data source, refresh cadence, and trigger threshold. This table is lifted from the actual collection schema:

Dimension	What it reveals	Trigger threshold (example)
BSR daily trajectory & velocity	Demand shifting before sales move	Same ASIN moving one direction for days, or a single-day drop beyond category norm
Price vs. category median	Competitor pre-emptive cuts, own squeeze	Effective price drops below median, or multiple rivals cut together
Buy Box win rate	Losing the box (reseller, logistics, price war)	Win rate in a market or time window below own baseline
Core keyword rank volatility	Rank erosion before it hits BSR	Core term falls off the first search screen, or a new term breaks into the top
Review velocity & rating drift	Quality / reputation inflection	Rating trends down, or negatives cluster in a short window
Search placement & share-of-shelf	New entrants, ad slots taken	Organic position drops, sponsored slot captured by rival
Variant / ASIN cannibalization	New variant stealing main variant traffic	New variant rises while main variant falls in sync
Cross-market divergence	Same ASIN behaving differently by market	One market anomalous while others are flat

The first five (BSR, price, Buy Box, keywords, reviews) are "lead" signals — they usually move before sales actually drop.

Implementation note: the underlying data uses the Amazon Scraper API for hourly collection covering BSR, price, coupons, inventory, and ad slots, with timestamp alignment — that alignment is what makes "velocity" and "divergence" calculations possible. Review and rating anomalies are pulled separately via the Amazon Review API so they aren't drowned out by the main-signal noise.

Layer 2: external social-demand signals (6 dimensions)

This is the off-site calibration. We plugged cross-platform social listening and broke it into six dimensions:

Share of voice by platform (TikTok / Instagram / YouTube / Reddit / X): where the conversation actually lives. The "home platform" differs completely by category and market.
Sentiment split (positive / neutral / negative): whether the narrative is helping or planting a landmine. A negative cluster often precedes a rating drop on Amazon.
Engagement by content format: short video, image, long-form, live — which format actually drives interaction. It decides where content budget goes.
Creator-tier mix (macro / micro / nano): was voice driven by top creators or spread by mid/small creators — this determines whether the channel strategy is repeatable or lucky.
Unbranded demand intensity: are users searching a brand name, or "how to shoot / which to buy" generic demand. This is the key signal for whether the category is pulling new buyers or the brand is catching them.
Competitor co-mention: who users compare together. Shifts in co-mention are the earliest substitution-threat signal.

Social is often the "cause" while Amazon on-site signals are the "result" — social heat typically leads search by 1–2 weeks.

Layer 3: decision rules (trigger → hypothesis → action)

The first two layers are only "seeing." The third is "driving." We attached every anomaly to a loop:

when dimension crosses threshold:
    hypothesis = generate_falsifiable_guess(dimension, context)
    action = map_action(hypothesis)
    owner = route_owner(dimension)
    notify(owner, anomaly + hypothesis + action)

We wired the Amazon Data MCP so an Agent reads the data and issues alerts directly — operators can ask in plain language "which ASINs are anomalous today" without writing SQL.

Three signals the monthly report missed

Signal 1 (cross-market · price): In one major overseas market, several same-category rivals pushed effective price below the category median within days, while the brand's pricing stayed put. In the first week, that market's BSR began slipping and Buy Box win rate fell with it — but all of it would only appear in the month-end report. The cockpit set "price vs. category median" to daily monitoring and triggered the same day, letting the operator judge, before the window closed, whether the rival was running a short promo or a long-term strategic cut.

Signal 2 (cross-platform · social): Social listening showed category-level "how to choose / which to buy" discussion contributed the vast majority of reach on short-video platforms, with engagement well above norm and sentiment mostly positive. But crossing social high-frequency terms with Amazon search terms exposed a glaring gap: a high-search-volume generic category term had zero Amazon ranking in the first two pages for the brand, while competitors already ranked organically near the top and ran sponsored placements. The category was acquiring new buyers on short video; the brand failed to catch the narrative into search.

Signal 3 (head-to-head · share of voice): In the launch of a new-generation pocket gimbal camera, the rival looked far stronger on the surface. The teardown revealed two things the monthly report ignored: first, the rival leaned on first-mover advantage and aggressive pricing to spike early, while the brand's own product had a steeper post-launch month-over-month acceleration — slower to heat, stronger in the back half; second, and most critically, the two products had almost zero keyword overlap in search. Zero overlap meant the generic and scenario long-tail terms the brand had never bid on were likely low-cost incremental pockets. At the same time, the rival lifted share of voice on a single platform via mid-tier creators while the brand's owned content underperformed on engagement.

Four playbooks that turn signals into action

Price-band defense SOP: threshold = effective price vs. category median ±X%. Short promo → defend keywords with ads, don't follow price; multi-day strategic cut → evaluate adjusting the price band or launching a secondary SKU. Owner: operations, daily.
Keyword rescue SOP: core term falls off the first screen → three-step: buy impression back with PPC, reinforce the related term in content and bullets, then check whether reviews/rating are dragging. Conversely, when a term first breaks into the top, increase investment immediately (most teams miss this step).
Reputation & review recovery SOP: review anomaly → cluster by theme (logistics / quality / expectation gap) → route action. Treat reviews as free user research.
Launch cadence & social reallocation SOP: use "unbranded demand intensity × platform" as the pre-launch heat map; move content budget to high-demand platforms; on Amazon, pre-stock and pre-schedule PPC by heat.

Cross-market and cross-platform attribution methodology

Lead-lag first: did the signal appear on Amazon first, or on social first? If social leads, use it as leading intel.
Horizontal matrix: read divergence with a "market × dimension" matrix to avoid single-market bias.
Allocate by elasticity: in markets where BSR is price-sensitive, prioritize price defense; in markets where it's content-sensitive, prioritize social reallocation.

Build a minimum-viable cockpit in three phases

Phase 1 (weeks 1–2): watch only daily price, BSR and Buy Box on core ASINs, with threshold alerts pushing anomalies the same day. Data via Amazon Scraper API hourly collection.
Phase 2 (weeks 3–4): add keyword rank and review anomalies (Review API), wire Amazon Data MCP for Agent auto-alerts.
Phase 3 (from month 2): connect social listening for external calibration, close the trigger→hypothesis→action loop, and write the four playbooks into the operations SOP.

A concrete "cockpit workday"

To keep the dimensions grounded, here is an anonymized real morning. Monday 9 a.m., the cockpit pushed three reds:

One major overseas market: a flagship's effective price was pushed below the category median by rivals, BSR slipped day-over-day, and Buy Box win rate dipped from its high.
Social: "how to choose / which to buy" discussion on short-video platforms rose week-over-week, while our Amazon rank on that scenario term was still blank.
Reviews: after a new variant launched, the main variant's rating dipped slightly and negatives clustered on "expectation gap."

The operator spent 15 minutes on three moves. For Market A, they first judged it a short promo — checking the rival's promo history confirmed a weekly pattern, not a strategic cut — so they defended core keywords with ads and did not follow price. They dropped the trending social scenario term into the content calendar and PPC candidates. For the main variant's "expectation gap" negatives, they published a video for expectation management. By Friday's review, Market A's BSR had recovered, the scenario term had earned organic plus sponsored placement, and the main variant's rating had stopped falling. That is the gap between "same-day response" and "monthly lag" — and all three moves would have waited until the month-end review under the old flow.

Threshold configuration: how to set each of the 8 Amazon thresholds

A cockpit lives or dies by its thresholds. Guesswork either false-alarms daily or never fires. Here is how we set each of the eight:

BSR trajectory & velocity: compute a 14-day rolling baseline per ASIN; alert when the day-over-day delta exceeds two standard deviations, or when the slope turns negative for three consecutive days. Velocity (the slope) matters more than the level.
Price vs. category median: pull the category median daily; alert when own or any tracked rival's effective price crosses ±X% of median. The median — not a fixed price — is the right reference because categories drift.
Buy Box win rate: track win rate per market per day; alert when a market's 7-day average drops below its own historical baseline by more than a band. A dip in one market while others are stable is the classic "something is wrong here" signal.
Core keyword rank: alert when a core term falls off the first search screen, or when a tracked term enters the top for the first time (the latter is an opportunity, not a threat).
Review velocity & rating: alert when daily review count spikes above baseline (possible spike of incentivized or fake reviews) or when rolling rating drops by a threshold. Cluster negatives by theme automatically.
Search placement & share-of-shelf: alert when organic position for a core term drops, or when a rival's sponsored slot appears above your organic result.
Variant / ASIN cannibalization: alert when a new variant's BSR rises while the main variant's falls in sync — a sign the launch is stealing internal traffic.
Cross-market divergence: alert when one market diverges from the median of all markets for the same ASIN.

The pattern: every threshold is "relative to a baseline or a median," never an absolute number pulled from thin air.

Second worked example: the head-to-head launch week

Signal 3 deserves its own walkthrough because it is the one most teams misread. When the rival's new pocket-gimbal launched, surface metrics said they were winning: early sales were multiples of ours. The cockpit's teardown changed the call. First, the rival's velocity came from first-mover advantage and aggressive pricing; our product's post-launch month-over-month acceleration was actually steeper — slower to heat, stronger in the back half. Second, and decisively, the two products had almost zero keyword overlap: they owned brand terms and a few generic words; we owned a different brand-term family. They were not colliding; they were in different traffic pools.

The action that followed was counter-intuitive: instead of panic-bidding on the terms where the rival was strong, we mapped the generic and scenario long-tail terms neither of us ranked for, confirmed real search demand, and quietly built content and PPC there. Those became low-cost incremental pockets. Meanwhile we flagged that the rival was lifting share of voice on a single platform via mid-tier creators while our owned content underperformed — a narrative-share problem the monthly report would never surface, addressed by reallocating content budget toward that platform's high-demand format.

Five most common mistakes

Treating a dashboard as a cockpit: seeing isn't driving; without a trigger→action loop, a dashboard is just a pricier report.
Watching only yourself, not competitors: share is taken by others; rival price, rank, launches, and lost Buy Box are the real alert sources.
Siloing social and Amazon: social heat leads search by 1–2 weeks; siloing throws away the lead.
Guesswork thresholds: without "vs. category median" or "vs. own baseline" references, thresholds either false-alarm daily or never fire.
No owner: routing an anomaly to "the team" means routing it to no one. Every anomaly must map to a specific owner.

How to measure whether the cockpit actually works

Five health metrics, themselves part of the cockpit: anomaly response time (target within a day), price-defense hit rate (correct "defend vs. follow" calls), keyword-rescue success rate (terms that return to the first screen), social→search conversion (trending social terms that earn Amazon rank), and share-recovery cycle (average time for an affected ASIN's BSR/share to return to baseline). The cockpit must both drive and self-check.

Tooling and data pipeline (no detours)

Real-time collection via the Amazon Scraper API (hourly, timestamp-aligned); review and rating anomalies via the Amazon Review API; Agent natural-language alerts via the Amazon Data MCP; the social layer via cross-platform listening for external calibration. We deliberately did not build a black-box SaaS — the data pipeline and integration are yours to wire into your own BI, Agent, and alert logic.

Why each of the 8 Amazon dimensions matters (with mini-examples)

Most teams ask for "more data" without clarifying what each dimension actually defends. Here is the per-dimension logic, because skipping any one of them leaves a blind spot.

BSR daily trajectory & velocity defends against "we only find out when sales drop." BSR reacts to demand before sales do, and velocity (consecutive same-direction moves) beats the single-day level. In the anonymized case, a flagship's BSR fell three days straight before sales moved on day four — watching sales alone costs you four days of response time.
Price vs. category median defends against "being quietly undercut." Using "relative to median" rather than an absolute price matters because categories drift; when rivals push effective price below median, your original price band effectively becomes expensive even if you didn't move.
Buy Box win rate defends against "losing the box without knowing." Resellers, logistics delays, and price wars all drop win rate; reading it per market localizes exactly where you lost.
Core keyword rank defends against "rank dropped before it hit BSR." Rank is upstream of BSR; rescue it while it's still off the first screen, not after BSR already slid.
Review velocity & rating defends against "reputation inflection drowned out." A rating decline or negative cluster often precedes a sales problem by weeks.
Search placement & share-of-shelf defends against "new entrants and ad slots taken." An organic position drop or a rival's sponsored slot above you are both direct share-loss signals.
Variant / ASIN cannibalization defends against "fighting yourself." A new variant stealing the main variant's traffic is frequently mistaken for competitor action.
Cross-market divergence defends against "global averages hiding local problems." One market anomalous while others are flat signals a local issue, not a category-wide one.

How the 6 social dimensions actually get used

Social is not "post content and watch views"; it is Amazon's leading indicator. How each dimension is operationalized:

Share of voice by platform tells you where the conversation lives. This category lives on short video, that one on long-form; being on the wrong platform wastes the entire budget.
Sentiment split gives a 1–2 week early warning of an Amazon rating risk the moment negatives cluster — long before the star rating moves.
Engagement by format decides budget allocation: if short video drives interaction, stop pouring money into image posts that quietly underperform.
Creator-tier mix tells you whether voice built from mid-tier creators is more repeatable than a single top creator's one-off spike.
Unbranded demand intensity is the switch between "category acquiring" and "brand catching." High unbranded demand plus a blank brand rank is the single biggest gap we found.
Competitor co-mention flags substitution threat the moment users start comparing you with someone new.

Writing "trigger → hypothesis → action" into your ops manual

This layer is the most important and the easiest to skip. What we built was not a "dashboard" but a "dashboard + rules": when any dimension crosses its threshold, the system first proposes a falsifiable hypothesis (short promo or long-term cut), then a standard action and an owner. The operator receives not a pile of numbers but a prioritized to-do with a recommendation attached. Without this layer, the first two are just a pricier weekly report. The hypothesis step matters because it prevents over-reaction: a price dip that is a week-long promo does not deserve the same response as a strategic repricing, and the system forces that distinction to be made explicitly rather than in someone's gut.

What this looks like as a weekly operating rhythm

Monday the cockpit surfaces three reds; the operator spends 15 minutes triaging each against its hypothesis. Tuesday through Thursday, the same loop runs daily but lighter — mostly confirming yesterday's actions are holding. Friday is the review: not "what happened last month," but "did our same-day responses hold share, and which thresholds need recalibration." That Friday recalibration is where the system learns — thresholds that false-alarmed get widened, blind spots get new dimensions added. Over a quarter, the cockpit stops being a project and becomes the team's default operating rhythm.

A note on team size

The most common objection is "we're too small for this." You are not. The minimum viable version is three dimensions (price, BSR, Buy Box) on your core ASINs, with threshold alerts pushed to one person's phone. That alone closes the "monthly lag" gap. The social layer and the full eight dimensions are phase-two upgrades, not prerequisites. The point is to start driving with a few instruments, not to wait until you can afford the whole cockpit.

A deeper look at cross-market attribution (worked)

Attribution is where most multi-market teams quietly fail, because they reverse cause and effect. The discipline we built has three moves. First, lead-lag: determine whether a signal appeared on Amazon first or on social first. In the anonymized case, the "how to choose" social spike preceded the Amazon search lift by roughly two weeks — so social was the leading indicator, and the team learned to pre-stage PPC before the search spike, rather than explaining it after. Second, the horizontal matrix: read divergence with a "market × dimension" grid. When Market A's BSR dropped, the matrix showed Markets B and C were flat — so this was a local competitive event in A (a rival promo), not a category-wide demand dip. Treating it as category-wide would have triggered a wasteful global price cut. Third, allocate by elasticity: in markets where BSR is price-sensitive, priority goes to price defense; in markets where it is content-sensitive, priority goes to social reallocation. A one-size playbook fails across markets because the lever that moves the needle differs by market.

A second anonymized mini-case: the review-velocity early warning

The first case was about price and share of voice; the second was quieter and arguably more valuable. A new variant launched, and within days the main variant's review velocity doubled while its rolling rating dipped by a hair — not enough to trip a "rating dropped" alert, but the velocity spike was anomalous versus baseline. The cockpit auto-clustered the new negatives and found them concentrated on "expectation gap": the listing promised one thing, the unboxing video implied another. The team published a short expectation-management video and tightened the bullet copy. Two weeks later, when a competitor launched a similar variant and their rating actually fell, our main variant's rating had already stabilized. The lesson: velocity — the rate of change — is a better early signal than the level, and clustering negatives by theme turns a raw rating into an actionable fix.

What to put on the cockpit home screen (a template)

If you build this, the home screen should be six to eight cards, not a wall of charts. Today's reds (the prioritized action queue). A price-vs-median heatmap across markets. Buy Box win rate by market. Keyword movers (terms that entered or left the first screen). Social share of voice by platform with sentiment. Review health (velocity + rating + top negative theme). An action queue with owner and SLA. And a response-time gauge showing the average hours from signal to action. That last card is the whole point: the cockpit is only as good as how fast the team closes the loop.

Takeaway

An Amazon Brand Operations Cockpit is not a report; it's the operating system for growth. It doesn't replace strategy, but it lets strategy be executed correctly at the right moment. The full deep-dive is here: Amazon Brand Operations Cockpit: Why Monthly Reports Can't Save Your Growth. More similar practice: Pangolinfo customer cases.

DEV Community: Pangolinfo

From API Key to Production: a 5-Gate Acceptance Checklist for an Amazon Data API

The five gates

Why field verification is its own gate

A two-week calendar

The API is one variable

Amazon Product Data JSON API: Field Contracts, Variant Drift, and Contract Tests

The case that motivates a field contract

Divide the fields into four objects first

Model three nullability states, not two

Trap 1: the split title has a boundary you cannot see

Trap 2: size holds RAM, not storage

Trap 3: the word "size" appears twice with two meanings

Diff two variants of one parent

The review payload's asin is not the ASIN you requested

Null rules: three shapes, three behaviors

inStock is free text, and unknown must not default to available

Schema diff for version drift

Contract tests: assert equivalence, not constants

Cost tiering by tolerated delay

Landing this on Pangolinfo

Verifying "Real-Time" Amazon Data APIs: 3 Clocks, 5 Cache Signatures, and a 48-Hour Protocol

The one delta that decides everything

Three clocks, and why exposure matters more than speed

Freshness tiers by object, not by vendor promise

The official ceiling: Amazon's own APIs are not magic

A real sample: B0CMZFCQ6D, captured twice

Four pitfalls that make latency measurements lie

Five signatures that a feed is a cache wearing a real-time badge

A scenario SLA table, read backwards

Seven questions every vendor must answer in writing

A 48-hour verification protocol

Poll versus push, and the webhook queue you forgot

Runtime staleness audit

A measured claim beats a printed one

Amazon Data API Node.js: Why a Job Dies at Block 17 and a Rerun Doubles Your Rows

What you will learn

Thread one: Node has no curl_cffi equivalent for fingerprints

impit claims Chrome and speaks Rust

The 2026-08 benchmark, read as a shape not a verdict

JA3 moved, so JA4 took over

Reuse the session or pay the handshake twice

Thread two: HTTP/2 changes the concurrency model

Streams replace pipelining

Concurrency is not rate

Thread three: idempotent reruns

Coverage is not fill rate

TypeScript types do not hold runtime data

Per-phase timeouts and the Node error names

Graceful shutdown keeps the rows you pulled

What one managed request already includes

Your Amazon Data API Returns 200 With Empty Fields: Eight Gaps to a Daily Job

The version that breaks in week three

Patch 1: timeouts, credentials, and retry in one class

Patch 2 and 3: field contract, P0 fields, and the quality gate

Patch 4: paging, dedupe key, and four failure classes

Patch 5: semaphore, snapshot storage, and the cost meter

Four metrics, two alerts, one pre-launch check

What stays on your side after you pick a vendor

Appendix: the four gates, and what one request includes

The 4.9x Gap Between Amazon Data API Quotes and Invoices

What Sits Between the Quote and the Record You Pay For?

Who Lands on This Page, and What Do They Need?

How Many Ways Can an Amazon Data API Bill You?

Why Does the Pricing Page Leave These Out?

What Are the Five Multipliers From One Call to One Usable Record?

Does a Failed Attempt Cost You Money?

Why Does a 200 Response Fail Your Quality Gate?

Is One Call One Record?

Are You Paying for Changes or for Polls?

Why Did You Buy Allowance You Cannot Spend?

What Is the Formula From Unit Price to Cost per Usable Record?

Which Variables Feed the Formula?

How Is the Formula Written?

Can You Run the Calculator Yourself?

What Is Each of the Three Lines For?

Can You Instrument Your Logs to Measure s, v, r, and w?

How Do You Build a Retry Policy That Does Not Inflate the Bill?

How Do You Stop Retries From Double-Billing?

What Does This Look Like at 50K, 500K, and 5M Records?

Trap 2: `size` holds RAM, not storage

The review payload's `asin` is not the ASIN you requested

`inStock` is free text, and unknown must not default to available