DEV Community: Pangolinfo

I Compared Amazon Data APIs by Field. The Checkboxes Were Lying.

Pangolinfo — Fri, 04 Sep 2026 02:51:13 +0000

Every "best Amazon data API" article has the same table.

Vendor | Product Data | Reviews | Search | Price Monitoring
-------|--------------|---------|--------|-----------------
A      | ✓            | ✓       | ✓      | ✓
B      | ✓            | ✓       | ✓      | ✓
C      | ✓            | ✗       | ✓      | ✓

A and B look identical. You pick based on price, or whoever replied to your email fastest.

Here is the thing I learned after actually running the comparison: one of those vendors returned 12 fields. The other returned 58. Same checkmark.

This post is the method I used to find that out. It takes two days, it needs no cooperation from any vendor, and the script is about 90 lines. I am publishing it because the alternative — reading vendor docs and hoping — does not work.

Related: if you have not already, I stopped trusting "real-time" claims after one bad quarter covers the reliability side of this. This post goes one level deeper: not is it fast, but is it complete.

One scope note, because it removes an entire category of pointless comparison: everything below is about vendors competing for the same job. The official SP-API is not one of them — its authorisation model binds to a seller account, not to the marketplace, so competitor data and market-level rank were never in scope. That is a design boundary, not a missing feature. If your requirements sit near that line, read Amazon API vs Web Scraping: Stop Choosing, Build Both Routes first; it will tell you whether you need this post at all.

Why "supported" is an information black hole

A feature grid uses a binary. Real delivery is continuous, and it has at least three layers:

Layer 1: does the schema have the field?        → field coverage
Layer 2: when it has it, does it have a value?  → non-null fill rate
Layer 3: how many records pass both?            → usable record rate

Only layer 3 has anything to do with your business. Layers 1 and 2 are diagnostics.

Let me show you why layer 2 matters more than most people assume.

The null field problem

Here is a real response shape, with values removed:

{
  "asin": "B0C7V9K2XQ",
  "title": "...",
  "coupon": null,
  "deliveryEstimate": null,
  "price": { "current": 49.99, "original": null }
}

Schema validation passes. JSON parsing succeeds. Row count is correct. HTTP status is 200. Your monitoring is green.

And your coupon dimension is permanently empty.

I know a team that signed an annual contract after seeing 60+ fields and 92% coverage. Three months in, the coupon column was empty in most reports and delivery estimates were empty in most reports. Both fields existed in the schema. Both were populated less than 15% of the time. On an annual deal, switching mid-term was expensive.

Ten minutes with the script below would have caught it.

A counterintuitive result

A vendor with 95% coverage and 60% fill rate is usually worth less to you than one with 70% coverage and 98% fill rate.

The first hands you a pile of nulls. You write defensive if (x != null) branches in every downstream consumer, and you still cannot tell "this product has no coupon" from "we failed to fetch the coupon."

The second hands you records that are solid end to end. Fewer dimensions, every one of them trustworthy.

Certain absence beats false completeness. Low coverage means "I do not have this dimension," which you can plan around. Low fill rate means "I think I have this dimension and I do not," which produces bad decisions.

The three response shapes

Feed the same ASINs to different vendors and responses cluster into three shapes. This taxonomy is more useful than any ranking.

Shape A — thin, 12–18 fields

{
  "asin": "B0C7V9K2XQ",
  "title": "...",
  "brand": "...",
  "price": 49.99,
  "currency": "USD",
  "rating": 4.3,
  "reviewCount": 1284,
  "availability": "In Stock",
  "mainImage": "https://...",
  "bsr": 1842,
  "category": "Electronics",
  "url": "https://..."
}

Good for: price monitoring, basic sourcing. Useless for anything involving variants, promotions, ad placement or seller dimensions.

Shape B — mid, 30–40 fields, variants but no ad flags

{
  "asin": "B0C7V9K2XQ",
  "parentAsin": "B0C7V90001",
  "price": {"current": 49.99, "original": 69.99, "currency": "USD"},
  "coupon": {"amount": 5.00, "type": "percentage"},
  "availability": {"status": "In Stock", "deliveryEstimate": "..."},
  "rating": {"value": 4.3, "count": 1284,
             "histogram": {"5": 62, "4": 21, "3": 9, "2": 4, "1": 4}},
  "bsr": [{"category": "Electronics", "rank": 1842},
          {"category": "Portable Audio", "rank": 97}],
  "variants": [
    {"asin": "B0C7V9K2XQ", "attributes": {"color": "Black", "size": "256GB"},
     "price": 49.99, "availability": "In Stock"},
    {"asin": "B0C7V9K2XR", "attributes": {"color": "White", "size": "512GB"},
     "price": 79.99, "availability": "In Stock"}
  ],
  "seller": {"name": "...", "rating": 4.6, "isFulfilledByAmazon": true},
  "images": [{"url": "...", "variant": "MAIN"}],
  "badges": ["Amazon's Choice"]
}

Good for: sourcing, variant analysis, rating distribution. Useless for rank attribution, because ads and organic results are indistinguishable.

Shape C — complete

{
  // everything in shape B, plus:
  "variants": [
    {"asin": "B0C7V9K2XQ", "attributes": {"color": "Black", "size": "256GB"},
     "price": {"current": 49.99, "original": 69.99},
     "availability": "In Stock", "rating": {"value": 4.3, "count": 812},
     "isBuyBoxWinner": true, "offerCount": 7}
  ],
  "isSponsored": false,
  "organicPosition": 1,
  "searchContext": {"keyword": "...", "page": 1, "positionOnPage": 3,
                    "sponsoredCountAhead": 2,
                    "adTypesAhead": ["SB", "SP"]},
  "priceHistory": {"min90d": 44.99, "max90d": 69.99},
  "subscribeAndSave": {"discount": 5},
  "buyBoxWinner": {"seller": "...", "price": 49.99, "shipping": "FREE"}
}

Good for: rank attribution, ad-competitive analysis, variant-level insight, historical price ranges.

On a feature grid these three are the same checkmark. In your reporting, you are not buying data — you are buying which questions you are allowed to ask.

The method

Step 1: write the field contract

Not vendor docs. Your own list, with priorities.

P0 — the record is dead without it. Goes to the dead-letter queue.
P1 — analysis gets shallower, still works.
P2 — nice to have.

CONTRACT = {
    "P0": ["asin", "title", "price.current", "availability.status",
           "rating.value", "rating.count", "bsr", "variants"],
    "P1": ["parentAsin", "brand", "price.original", "price.currency",
           "coupon", "seller.name", "offerCount", "images",
           "badges", "isSponsored", "categoryPath"],
}

Eight to fifteen P0 fields is the sweet spot. The test is one question: if this field were missing, would you dead-letter the record? Yes means P0.

Per object, the critical fields differ:

Object	Critical fields	Why
Product	`price.current`, `bsr`, `variants`	pricing and sourcing fundamentals
Search	`page`, `items[].position`, `items[].isSponsored`, `items[].adType`, `items[].organicRank`	without the last three, your rank tracker is sorting ads and organic together
Reviews	`reviewId`, `asin` (child), `parentAsin`, `variant`, `rating`, `date`, `verifiedPurchase`	`variant` decides whether insight can be pinned to a configuration

Step 2: the flatten function (this is the whole trick)

def flatten(obj, prefix=""):
    """Flatten to a.b.c paths; sample list items; empty values are dropped."""
    out = {}
    if isinstance(obj, dict):
        for k, v in obj.items():
            out.update(flatten(v, f"{prefix}.{k}" if prefix else k))
    elif isinstance(obj, list):
        for v in obj[:50]:
            out.update(flatten(v, f"{prefix}[]"))
    elif obj is not None and obj != "":
        out[prefix] = obj          # <-- the entire trick
    return out

That last elif is the whole method. Empty values never enter the result. So when you later ask "which contract fields came back," you are asking "which contract fields came back with a value."

One function, two metrics. Coverage comes from the count. Fill rate is handled implicitly — the coupon: null case from earlier simply never lands in out.

Step 3: evaluate and aggregate

import statistics

def evaluate(record, cov_threshold=0.8):
    flat = flatten(record)
    contract = CONTRACT["P0"] + CONTRACT["P1"]
    present = [f for f in contract if f in flat]
    coverage = len(present) / len(contract)
    p0_ok = all(f in flat for f in CONTRACT["P0"])   # P0 must be 100% populated
    return {
        "coverage": round(coverage, 3),
        "usable": bool(p0_ok and coverage >= cov_threshold),
    }

def run(records, spend_usd):
    ev = [evaluate(r) for r in records]
    usable = [e for e in ev if e["usable"]]
    return {
        "samples": len(ev),
        "usable_rate": round(len(usable) / len(ev), 4) if ev else 0,
        "avg_coverage": round(statistics.mean(e["coverage"] for e in ev), 3),
        "cost_per_1k_usable": round(spend_usd / len(usable) * 1000, 2)
                              if usable else None,
    }

p0_ok uses all(), not a ratio. P0 is a hard constraint — half present is not usable.

Step 4: classify failures

def classify(record):
    if record.get("_http_status") != 200:
        return "hard"        # visible, retry it
    if not record.get("asin"):
        return "soft"        # 200 but empty body
    flat = flatten(record)
    if not all(f in flat for f in CONTRACT["P0"]):
        return "partial"     # SILENT — the expensive one
    return "ok"

Type	Symptom	Visibility	Cost
Hard	non-200, timeout	Fully visible	Low
Soft	200, empty body	Half visible	Medium
Partial	200, valid structure, P0 null	Silent	Highest
Stale	values present but old	Silent	High

Row three is why this post exists. It raises no error, trips no alert, and generates no billing dispute. It just quietly degrades your reporting.

Step 5: what the output looks like

=== Vendor A ===
  samples            : 300
  avg coverage       : 0.94
  usable rate        : 61.33%
  cost per 1k usable : $12.40
  failures           : {'ok': 184, 'partial': 97, 'soft': 14, 'hard': 5}

=== Vendor B ===
  samples            : 300
  avg coverage       : 0.71
  usable rate        : 92.67%
  cost per 1k usable : $4.85
  failures           : {'ok': 278, 'partial': 12, 'soft': 8, 'hard': 2}

A has much better coverage and two-thirds the usable rate, at 2.5x the cost per usable record.

If you only look at coverage — or at a feature grid — you pick A. Look at usable rate and the conclusion inverts.

Note vendor A's 97 partial records. Measured the traditional way, A's success rate is (184 + 97 + 14) / 300 = 98.3%. Looks excellent. With a field contract applied, real usable rate is 61%.

That gap is exactly where conventional monitoring fails. It confuses "returned" with "usable."

Sample design: why your test showed no differences

When someone tells me "I tested four vendors and they were all the same," the sample set is almost always the problem.

The most common mistake: testing fifteen bestseller ASINs. Those products have the richest data on the platform, so every vendor returns everything and all four score 100%.

You measured Amazon's data completeness for bestsellers, not the vendors' capability.

A discriminating sample has five kinds of record:

Across categories — four or more top-level categories. Page templates differ, field availability follows.
Across marketplaces — US plus at least two non-US. This is the direct test for the "US-only field" surprise.
Heavy variants — 20+ variants, to probe variant depth.
Weak-data products — new listings with no reviews, long-term out of stock, no Buy Box. This is the litmus test for fill rate.
Deep pages — for search, go through page five.

Two hundred to five hundred records per vendor is enough. Below that, noise dominates.

Four execution disciplines — break any one and the numbers stop being comparable:

same ASINs
same time window    <- compress into hours; prices and stock move
same concurrency
same retry policy   <- easiest one to get wrong

That last one: give vendor A three retries and vendor B one, and A's usable rate is inflated while its real cost is triple. You are not measuring the vendor, you are measuring your retry policy.

Five places where vendors actually diverge

After running this repeatedly, differentiation concentrates in five spots. If you are short on time, look here first.

1. Ad placement flags and organic rank

The most discriminating item. Vendors carrying isSponsored and organicRank are a minority; most return only page position.

Without them, the rank metric is systematically distorted. A team tracked a core keyword at a steady position three for four months, concluded organic had plateaued, and prepared to raise ad spend. Recomputed with isSponsored applied, the truth was organic position one with two competitors' Sponsored Brands units above it. Organic had been improving the whole time.

One boolean field put a quarter of budget decisions on an inverted premise.

2. Variant-level depth

Many vendors ship a variants array containing nothing but ASINs — no per-variant price, stock or rating. That tells you five variants exist without telling you which is selling, which is out of stock, or which is losing rating.

A review analysis once produced a high-confidence finding that users complain about battery life. It could not be actioned. Adding variant localised it to one capacity tier — concentrated in the low-spec model, near zero in the high-spec one. The response changed from "redesign the battery across the range" to "fix spec copy on the low tier." Cost fell by an order of magnitude.

Variant attribution is the field that takes an insight from plausible to executable.

3. Deep search pages

Little difference across the first two pages. Sharp divergence from page three: some vendors return duplicates, some return empty.

Consequence: competitive density is decided in the deep pages. Seeing only two pages makes you systematically underestimate competition. And when you can only see two pages, "this market is not competitive" describes your observation range, not the market.

4. Promotions and coupons

coupon, promotions and Subscribe & Save discounts are usually thin. But they matter enormously for pricing — list price alone systematically overstates what competitors transact at.

5. Seller and Buy Box detail

offerCount, buyBoxWinner and seller ratings are core metrics in heavily resold categories, and absent from many responses entirely.

Billing: why entry price tells you nothing

Four billing models, not comparable to each other:

Model	Headline	Reality
Per request	Cheapest, most seductive	Failed requests bill too; deep pages and retries multiply
Per successful result	Middle	200 with missing fields still bills — you pay full price for a broken record
Per record	Impossible to rank by eye	Only comparable as cost per thousand usable records
Monthly quota	Priciest, least friction	Overage rates are steep

A vendor billing per request can quote a third of one billing per record and cost more in practice, because deep pages fail, retries are frequent, and failures bill anyway.

The only comparable number is cost per thousand usable records. Do not do cost accounting on the vendor's billing basis — do it on your own usable record count. The multiple between the two is precisely the cost of silent failures and retries.

Make regression routine

A one-off acceptance test proves it works now. Vendor coverage drifts: page redesigns, policy changes, capacity shifts.

Run a fixed sample weekly — fifty to a hundred records is plenty and cheap — and trend four numbers:

median latency / p95 latency / usable record rate / average coverage

Chart them, and chart the failure classification as a stacked area. A rising partial share is the most valuable early warning signal you can have.

The checklist

Field contract written down, P0/P1/P2 agreed with the business side — not invented by an engineer.
Sample covers all five record types.
Four or more vendors, identical terms.
All four numbers in hand — coverage, fill rate, usable rate, cost per 1k usable.
Failures classified, partial share confirmed. This is the only class needing dedicated code.
Quotas modelled per operation type, with backoff tested against real limits.
Regression scheduled weekly, four numbers trended.

Item seven is the easiest to skip and the most valuable.

Two questions that end the sales conversation

Once the contract is written, there are two questions worth asking every vendor before signing anything. They are not gotcha questions. They are diagnostic, and the shape of the answer tells you more than the content does.

"Can you send me the field-level schema for the product object, not the object list?" Most vendors will send a page saying "product data, search data, review data." Push once. If they can send an actual field dictionary, you just saved two days of testing. If they cannot — or if what comes back is clearly a marketing page with field names sprinkled on it — you have learned something about how they think about delivery.

"If a P0 field comes back null for a week, what happens?" Listen for whether they have a category for that. Vendors who monitor fill rate answer immediately, because they already have the dashboard. Vendors who only monitor HTTP status pause. That pause is the answer.

Neither question is hostile. Both are what a serious buyer asks, and a vendor who handles them well is showing you they have been through this before.

The retry multiplier nobody puts in the pricing calculator

Here is an arithmetic that reorders rankings more often than any feature comparison does.

Say vendor A charges \$0.001 per request and vendor B charges \$0.0025 per record. On paper A looks 2.5x cheaper. Now add behaviour. A has an 82% usable rate on your sample, so you retry failures twice, and failed attempts still bill because the billing unit is the request:

effective_cost = unit_price × attempts_per_usable_record / usable_rate

A: 0.0010 × (1 + 2 × 0.18) / 0.82 = 0.00166
B: 0.0025 × 1.00            / 0.97 = 0.00258

A still wins, but by 1.55x rather than 2.5x. Now change one assumption. Suppose A's deep-page coverage is weak, so the pages you actually care about need three retries and the usable rate on page 3+ drops to 60%:

A (pages 3+): 0.0010 × (1 + 3 × 0.40) / 0.60 = 0.00367

Now A costs more than B, on precisely the task you bought it for. Nothing on the pricing page changed. Only the denominator did.

This is why I stopped normalising list prices and started computing cost per thousand usable records on my own sample. It takes an afternoon, and it is the only number that survives contact with production.

Catching staleness before it reaches your dashboard

Silent failure is the expensive class and staleness is its quietest variant. A field returning a plausible but frozen value passes every schema check you can write. Here is the smallest useful detector — it watches for values that stop moving:

from collections import defaultdict
from datetime import datetime, timedelta

history = defaultdict(list)          # field_path -> [(ts, value)]
STALE_AFTER = timedelta(days=14)

def observe(record, now=None):
    now = now or datetime.utcnow()
    for path, value in flatten(record).items():
        history[path].append((now, value))

def stale_fields(now=None, window=STALE_AFTER):
    now = now or datetime.utcnow()
    out = []
    for path, obs in history.items():
        recent = [(t, v) for t, v in obs if now - t <= window]
        if len(recent) < 5:
            continue                                  # not enough signal yet
        if len({str(v) for _, v in recent}) == 1:
            out.append((path, recent[-1][1]))
    return out

Run it weekly over a fixed sample. A price field that has not moved in two weeks across fifty ASINs is not a stable market — it is a frozen parser. The distinction matters, and no status code will tell you which one you are looking at.

Two ways to accidentally measure the wrong thing

Two production realities will distort every number in this article if you ignore them.

Concurrency changes your success rate. If you run the evaluation at 50 parallel requests and the vendor throttles at 20, the failures you observe are yours, not theirs. You will record a low usable rate that has nothing to do with field coverage, and you will carry that error into the vendor comparison. Run the test at the concurrency you actually plan to use in production, and record that number alongside the results. A measurement you cannot repeat is an anecdote.

Caches hide staleness. If responses are cached, a field can look fresh because you are being served last week's copy. A weekly regression run against a cached endpoint will report perfect stability right up until the cache expires and the parser turns out to have been broken for a month. Either bypass the cache for the measurement run, or record the cache TTL as part of the result and compare like with like — one vendor's five-minute TTL and another's twenty-four-hour TTL are not the same product, whatever the fill rates say.

Both are easy to miss for the same reason: both make the numbers look better than reality. Be suspicious of measurements that come out clean.

Wrapping up

The core idea is one sentence: write down what you need first, then see who can deliver it.

Most people do the reverse — they read the vendor docs and reverse-engineer requirements from them. The docs lead, you buy a pile of fields nobody uses, and the one field you needed is missing.

Ninety lines of Python, two days, four vendors. That is a much better deal than two weeks of reading marketing pages and emailing sales teams for a field dictionary.

If you want to run this against real responses: Amazon Scraper API covers product and search objects, Amazon Review API covers reviews, and Amazon Data MCP is the agent-facing path. Grab a key from the console and run the test; docs are at the Amazon Data MCP overview.

One last thing, and I mean it: run this on us too. Completeness numbers a vendor reports about itself — ours included — deserve to be verified.

If you have run something like this, I would be curious which failure class dominated for you. In my experience partial is the one nobody budgets for.

Amazon API vs Web Scraping: Stop Choosing. Build Both Routes.

Pangolinfo — Wed, 02 Sep 2026 06:28:55 +0000

Every team integrating Amazon data eventually hits the same debate: use the official SP-API, or scrape?

The debate never resolves cleanly, and it never will, because the question is wrong. It assumes the two are alternative implementations of the same thing, so the comparison collapses into which is more compliant, cheaper or more stable. That framing guarantees a bad decision.

Here is the actual situation: they retrieve two different classes of data. The official SP-API gives you your account. Public pages give you the market. Once you internalise that, the route debate disappears and what remains is a concrete architectural division of labour.

This post covers the authorization boundary, the public-versus-private line, what SP-API throttling actually costs in engineering time, and a hybrid architecture with runnable code.

The two domains

Category	Nature	Route
Product facts (title, price, rating, variants, BSR)	Publicly visible	Public collection
Search and ads (keyword, rank, ad type, creative)	Publicly visible	Public collection
Review content (rating, body, date, variant)	Publicly visible	Public collection
Ranks and categories	Publicly visible	Public collection
Orders and fulfillment	Account-private	SP-API (authorized)
Buyer information	Account-private + personal	SP-API (restricted PII role)
Inventory and settlement	Account-private	SP-API (authorized)
Own ad performance	Account-private	SP-API (authorized)

The route follows from the nature of the data, not from preference. This is also why "can we just use the official API" is answered no in most teams — you need market data, and it sits outside the official interface's authorization scope.

You would not use the SP-API to check a competitor's price, because it is not designed to expose that. You would not scrape your own orders, because they sit behind login and are account-private.

So the useful question is not "which one". It is: which of my requirements are account-domain and which are market-domain? For most teams the answer is both.

Where each route gets its legitimacy

Compliance discussions slide into an unanswerable "which is more legal". The two routes draw legitimacy from entirely different places, and what determines compliance is what you collect, not which method you use.

The SP-API is legitimate through an agreement chain: developer registration, seller authorization via Login with Amazon, role-based access granting minimum necessary scope. Operations touching personally identifiable information require a separate restricted role and review. What you can violate is the developer agreement and data protection policy, and the consequence is revoked authorization. The boundary is clear but narrow — only what a seller has actively authorized your application to access.

Public page collection has no contract. It sits under the platform's conditions of use, robots directives and rate limits. That is not the same as prohibition; publicly visible, non-personal page information can generally be collected, which is long-standing practice.

Risk rises with three specific things:

Login-gated content — anything visible only after signing in has clearly crossed the public line.
Personal data — buyer names, addresses and contact details are governed by data protection law.
Aggressive request rates — traffic heavy enough to affect site operation shifts the activity from reading public information to interfering with a service.

The one-line test: is the data publicly visible and non-personal? The method is irrelevant. Misusing PII through the official interface is still a violation; collecting a fully public page can still be compliant.

Here is what structured public data looks like. Every field is public and non-personal:

{
  "asin": "B0CXYZ1234",
  "marketplace": "amazon.com",
  "title": "Stainless Steel Insulated Water Bottle, 32 oz",
  "price": { "current": 34.99, "currency": "USD", "listPrice": 44.99 },
  "rating": { "average": 4.6, "count": 12847 },
  "bsr": [ { "category": "Sports & Outdoors", "rank": 128 } ],
  "availability": "In Stock",
  "sponsored": false,
  "fetchedAt": "2026-08-30T10:22:41Z"
}

No buyer information, nothing requiring seller authorization. That is the compliance basis. By contrast, orders, buyer addresses and settlement detail never appear on public pages — they are only reachable through the SP-API after explicit authorization.

What SP-API throttling actually costs

This is the part teams consistently underestimate: even though the official interface is free, throttling converts into engineering complexity.

The SP-API uses a token bucket model. Each operation has its own request rate and quota, and quota refills continuously at that rate. For some operations, initial quota and refill rate also scale with seller business volume. Three consequences:

Quotas are independent per operation. Estimating on "total call volume" is always wrong. getOrders and getInventorySummaries are separate buckets and need separate models. This is the single most common design mistake.

Being throttled is normal, not exceptional. At high polling frequency it is close to inevitable. The correct response is queueing with backoff, not letting it throw and kill the job. Treating 429 as an error also destroys your alerting signal — the team learns to ignore it, and then the real failure goes unnoticed.

Values change. Amazon adjusts rates and quotas. Verify current figures in the official documentation rather than reusing numbers from older docs. I have watched a team design quotas from a year-old document and get throttled into the ground on day one.

The code

A per-operation token bucket

import threading
import time


class TokenBucket:
    """Per-operation token bucket.

    rate:      tokens refilled per second (SP-API restore rate)
    capacity:  bucket size (SP-API maximum quota / burst)
    """

    def __init__(self, rate: float, capacity: float):
        self.rate = rate
        self.capacity = capacity
        self._tokens = capacity
        self._updated = time.monotonic()
        self._lock = threading.Lock()

    def acquire(self, tokens: float = 1.0) -> float:
        """Block until tokens are available. Returns seconds waited."""
        waited = 0.0
        while True:
            with self._lock:
                now = time.monotonic()
                self._tokens = min(
                    self.capacity,
                    self._tokens + (now - self._updated) * self.rate,
                )
                self._updated = now
                if self._tokens >= tokens:
                    self._tokens -= tokens
                    return waited
                deficit = tokens - self._tokens
                sleep_for = deficit / self.rate
            # sleep outside the lock so other threads are not blocked
            time.sleep(sleep_for)
            waited += sleep_for


class OperationLimiter:
    """Maintains an independent bucket per SP-API operation."""

    def __init__(self, config: dict):
        self._buckets = {
            op: TokenBucket(cfg["rate"], cfg["capacity"])
            for op, cfg in config.items()
        }

    def acquire(self, operation: str) -> float:
        bucket = self._buckets.get(operation)
        return bucket.acquire() if bucket else 0.0

Calls with backoff

import random
import requests

RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def call_sp_api(limiter, operation, url, headers, params, max_retries=3):
    """SP-API call with token bucket queueing and exponential backoff."""
    last_error = None

    for attempt in range(max_retries + 1):
        limiter.acquire(operation)  # queue for a token first
        try:
            resp = requests.get(url, headers=headers, params=params, timeout=30)
        except requests.RequestException as exc:
            last_error = f"network: {exc}"
            time.sleep(2 ** attempt + random.uniform(0, 0.5))
            continue

        if resp.status_code == 200:
            return resp.json(), None

        if resp.status_code in RETRYABLE_STATUS:
            retry_after = resp.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay + random.uniform(0, 0.3))
            last_error = f"http {resp.status_code}"
            continue

        # 4xx other than 429: parameter or auth problem, retrying is pointless
        return None, f"http {resp.status_code} (not retryable): {resp.text[:200]}"

    return None, f"exhausted retries: {last_error}"

Two details that matter. Prefer the server's Retry-After header over your own backoff — it is more accurate. And only retry 429 and 5xx; a 400, 401 or 403 is a parameter or authorization problem that will fail identically a hundred times while burning quota and hiding the real error.

The unified ingestion layer

from dataclasses import dataclass, field
from typing import Callable


@dataclass
class IngestResult:
    ok: bool = False
    data: dict = field(default_factory=dict)
    failure_type: str = ""   # throttled | blocked | incomplete | request_error
    missing: list = field(default_factory=list)


class UnifiedIngestion:
    """Converges SP-API and public collection into one internal model."""

    def __init__(self, field_map: dict, required_fields: list):
        self.field_map = field_map      # external field -> internal field
        self.required = required_fields

    def normalize(self, payload: dict) -> dict:
        """Allowlist normalization: keep only declared internal fields."""
        out = {}
        for src_key, dst_key in self.field_map.items():
            if src_key in payload and payload[src_key] not in (None, "", [], {}):
                out[dst_key] = payload[src_key]
        return out

    def validate(self, record: dict) -> list:
        return [f for f in self.required if f not in record]

    def ingest(self, fetcher: Callable, **kwargs) -> IngestResult:
        """fetcher returns (payload, error); non-empty error means request failure."""
        result = IngestResult()
        payload, error = fetcher(**kwargs)

        if error:
            result.failure_type = "throttled" if "429" in str(error) else "request_error"
            return result

        record = self.normalize(payload)
        missing = self.validate(record)

        if missing:
            # parsed fine but fields absent: a coverage problem, not a request problem
            result.failure_type = "incomplete"
            result.missing = missing
            return result

        result.ok = True
        result.data = record
        return result

The critical design choice is three separate failure classes:

throttled / request_error — request layer; look at quota and backoff
blocked — a 200 that is not a real page (public collection side)
incomplete — parsed fine, required field missing (a coverage problem)

These have completely different fixes. Merge them into one error log and you cannot tell whether to adjust quotas, lower concurrency or chase coverage.

Detecting blocked responses at the content layer

This is the most expensive gap on the public collection side. A blocked page returns HTTP 200. Captcha interstitials, bot-detection pages and pages that never finished rendering all come back green. If your check is the status code, that bad data enters your warehouse wearing a success badge.

BLOCK_MARKERS = [
    "captcha", "robot check", "enter the characters you see",
    "automated access", "to discuss automated access",
]


def looks_blocked(html: str) -> bool:
    """Content-layer block detection. Status code alone is not enough."""
    lowered = html[:20000].lower()
    hits = sum(1 for m in BLOCK_MARKERS if m in lowered)
    # markers present, or page implausibly short for a real product page
    return hits > 0 or len(html) < 2000

Wire this into UnifiedIngestion and blocked responses get classified as blocked instead of silently becoming data.

Compliance as code, not as a wiki page

Most teams keep compliance requirements in a document and enforce nothing in code. The result is "the doc says don't collect that, the code has been collecting it for months" — which is the worst position to be in during an audit.

Four constraints worth hardcoding:

A field allowlist. Keep only fields your internal model declares; drop everything else. Note that normalize() above is already allowlist-based rather than denylist-based, so even if an upstream response starts carrying personal data, it never lands in your store.

Path constraints. Collection jobs may only request public paths. Account pages and login redirects should be rejected and alerted on.

A hard rate ceiling at the job layer, so one runaway job cannot push overall frequency into "interfering with a service" territory.

Audit logging of source, time and field list per collection, plus a retained fetchedAt on every record. Without a capture timestamp a record cannot enter a time series and cannot be audited later.

Freshness is a budget decision

Having argued the official API is not free, the opposite error deserves equal attention: running every collection job at maximum frequency.

Field change rates vary enormously. Brand, category and variant dimensions barely move. Price, stock, Buy Box and sponsored placements move constantly. Treating them identically is the most common form of budget waste in this category.

Three tiers work in practice:

High frequency (minutes to hours) — price, stock, Buy Box holder, sponsored placements, BSR. For these, a stale value does not weaken the insight, it invalidates it. A competitor's price from yesterday cannot support a pricing decision today.

Daily — new reviews, rating changes, seller lists, best seller ranks. Trend signals that matter over weeks, not minutes.

Weekly or on demand — brand, category, variant dimensions, images, A+ content.

The top 20% of ASINs drive roughly 80% of decisions. Run those hot and the long tail cold and cost typically halves with almost no business impact. No vendor negotiation required — it is a scheduling decision in your own ingestion layer.

Implement it there rather than hoping a provider does it, and revisit it periodically. A tiering decision made eighteen months ago is usually wrong today.

When comparing prices on the collection side, use cost per thousand usable records, not per thousand requests. Failed, blocked and field-incomplete responses all bill. A plan that is 25% cheaper on the rate card is often only about 10% cheaper in reality.

And for the build decision, convert engineering hours to money first. A maintenance load of 0.3 engineer-days a month sounds trivial; priced at real internal cost over a year it frequently exceeds the API bill for a mid-sized volume — and unlike the bill, it recurs every quarter and grows with every page redesign.

What good looks like

Median latency under five seconds for on-demand fetches. p95 no more than roughly twice the median. Success rate above 98% measured at the content layer, not the status code. Field completeness above 95% against your own list.

Those are not laws of nature. They are the range where teams stop maintaining the data layer and start building on it. Below that range, the data layer becomes the most expensive dependency you have — not because of the invoice, but because of everything your engineers stop building while they babysit it.

Two fields worth treating as non-negotiable:

fetchedAt. Without a capture timestamp, a record cannot enter a time series and cannot be audited. Three months later, when a price looks wrong, you cannot tell whether it was wrong then or got mangled in processing.

isSponsored. Without it, organic rank and ad placement are indistinguishable. You think you rank third; the top two slots are ads; your true organic position is first. Every decision built on that number optimises for the wrong target.

Sponsored is also the hardest object to collect reliably, which is precisely why it needs its own evaluation line rather than an assumption that every provider has it.

The decision table

Skip the route debate and read your task off this:

Your task	Domain	Route
Sync your own orders and fulfillment	Account-private	SP-API
Manage inventory and settlements	Account-private	SP-API
Monitor competitor price and stock	Public market	Public collection / specialized API
Track keyword organic rank	Public market	Public collection / specialized API
Analyse competitor ad placements	Public market	Public collection / specialized API
Run review insights	Public market	Public collection / specialized API
Need both orders and competitor monitoring	Both	Hybrid architecture
Only your own operating data	Account-private	SP-API only

That last row deserves emphasis: if your requirements stop at your own operations, do not add public collection. Architectural complexity should match problem complexity. Converging two channels costs real money, and it is only worth paying when requirements genuinely span both domains.

One operational trap

A specific misdiagnosis shows up constantly in hybrid pipelines.

Symptom: volume for a field drops, or a window of data has holes.

First instinct: data quality degraded. So the team goes to the vendor, checks parsers, digs through logs. Frequently the real cause is quota — the token bucket saturated, jobs queued, and some timed out and were dropped.

Distinguishing them is easy if you classified failures: if throttled is climbing while blocked and incomplete are flat, it is a quota problem, not a data problem. The fixes are entirely different — tiering and scheduling for the former, coverage and parsing for the latter.

And do not alert on 429 directly. Record it as a throttled counter and watch the trend; alert only when the share crosses a sustained threshold. Otherwise the team learns to ignore the alert, and then the real failure goes unseen.

When to build nothing

Worth stating the inverse, since architecture posts have a bias toward building.

Requirements stop at your own operations — orders, inventory, your listings, your ads. Use the SP-API and stop there. Public collection buys coverage you will never use.

You need a one-off answer — a category survey, a competitor list for a deck. Do not build a pipeline. Pull the data and move on. Infrastructure built for a single question tends to outlive the question.

Volume is tiny and fields are narrow — thirty ASINs, four fields, weekly. Write a script. The break-even arithmetic only crosses once maintenance becomes recurring work, and below that line simpler is genuinely better.

Hybrid is correct when requirements span both domains. Verify that condition before committing engineering time. A decision framework is only half useful if it cannot also tell you not to build.

Wrapping up

The ordering matters more than any individual decision:

Split requirements into account-domain and market-domain.
Build the unified ingestion layer before either pipeline.
Model SP-API throttling per operation; retry only 429 and 5xx.
Detect blocked at the content layer, never on status code alone.
Classify failures three ways and put the ratios on a dashboard.

For the market side, Amazon Scraper API and Amazon Review API cover the public objects; Amazon Data MCP is the path when an agent calls data directly. Keys come from the console, and the MCP docs cover the agent path.

The full comparison, including the authorization boundary and decision table: Amazon API vs Web Scraping.

One more note on why the classification approach is worth adopting: it ends arguments that otherwise recur. Framed as compliance versus pragmatism, this debate never resolves, because nobody changes values in a meeting. Framed as "is this data public and non-personal", it resolves in half an hour.

If you are running a hybrid pipeline already, I would be curious which of the three failure classes dominates for you. In my experience incomplete is the one nobody budgets for.

I Stopped Trusting "Real-Time" Claims After One Bad Quarter. Here's the 100-Call Test I Run Instead.

Pangolinfo — Tue, 01 Sep 2026 03:32:20 +0000

A few years ago I integrated an Amazon data provider into a pricing service. The sales page said "real-time data". The docs said "high success rate". Both were technically true and both were useless.

What I eventually learned: the pricing service was reading from a cache refreshed once a day, and "high success rate" counted HTTP 200 responses, which included a meaningful number of captcha pages. We shipped two weeks of pricing recommendations built partly on blocked page responses, because our client only checked the status code.

This post is the test I wish I had run first. It takes an afternoon, it is about a hundred lines of Python, and it turns every vague vendor claim into four numbers you can put on a dashboard.

The four numbers that actually matter

Before the code, the definitions. These four are the difference between a provider you can build on and a provider you have to babysit.

Metric	Definition	Why it matters
Median latency	50th percentile response time	Your everyday experience
p95 latency	95th percentile response time	Sets your timeout and retry policy
Success rate	Parsed successfully ÷ total requests	Not HTTP 200. Determines your real unit price
Field completeness	Records with all required fields ÷ successful records	Whether the data is usable at all

The success rate definition deserves emphasis, because it is where most home-grown checks go wrong. An HTTP 200 that returns a bot-detection page is a failure, not a success. If your client only looks at the status code, that bad record flows straight into your database and you find out two weeks later when someone asks why the dashboard looks strange.

On p95: it determines your timeout. If p95 is five seconds and you set a three second timeout, you are throwing away a slice of requests that would have succeeded. Set it to ten seconds and you slow the whole pipeline. A reasonable starting point is 1.5× p95, with a retry cap of two. More retries than that usually means you are being throttled, and retrying into a throttle makes it worse.

Why per-request pricing is the wrong unit

Here is the part that changes how you read a pricing page.

A failed request still bills. A blocked page still bills. A response missing the field you need still bills.

So the unit worth comparing is cost per thousand usable records:

cost per 1k usable records = (monthly spend ÷ records parsed successfully with all required fields) × 1000

Worked example, because the gap is counterintuitive:

Plan A: $1.20 per 1k requests, 92% success, 88% field completeness. Usable share is roughly 81%, so the real cost is about $1.48 per 1k usable records.
Plan B: $1.60 per 1k requests, 99% success, 98% field completeness. Usable share is about 97%, so the real cost is about $1.65 per 1k usable records.

Plan A looks 25% cheaper on the rate card. The real gap is about 10%. Add the debugging hours Plan A will generate and the ranking usually flips.

So the question to ask a vendor is never "how much per thousand requests". It is: how do you define success rate, what is your field completeness, and do failures bill?

The four-layer pipeline

Whichever route you take, the chain has the same shape. What differs is which layers you own.

Amazon public pages (product / search / review / best sellers / sponsored)
        ↓
[Collection]   anti-bot · browser rendering · proxies and geo · retries
        ↓
[Structuring]  parsing · field normalization · type contract · failure semantics
        ↓
[Delivery]     REST API   ── or ──   MCP tools
        ↓
Your application / data pipeline / AI agent

Four routes, and how they split ownership:

Route	Collection	Structuring	Delivery	You maintain
Self-built scraper	You	You	You	Everything
General scraping API	Vendor	You	Vendor	Parsing, field drift
Amazon-native data API	Vendor	Vendor	Vendor	Business logic only
Official SP-API	Vendor	Vendor	Vendor	Quota and auth

Choosing a route is really deciding which layers you want to own. More layers means more control and more responsibility.

One thing worth stating plainly: if you only need data tied to your own seller account, use the official SP-API and skip third parties entirely. Orders, inventory, your own listings. It is the most compliant and cheapest path, and recommending anything paid for that job would be dishonest.

The real fork is whether you need public marketplace facts — competitors, categories, search results, sponsored placements. That is where the other three routes become relevant.

Build the test

Enough theory. Here is the script. It samples across marketplaces and object types, then reports all four numbers plus a breakdown of which fields went missing.

#!/usr/bin/env python3
"""
Amazon Data API acceptance test.
Samples N calls across marketplaces and object types, then reports
median latency, p95 latency, success rate and field completeness.
"""

import argparse
import os
import statistics
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field

import requests

API_BASE = "https://api.pangolinfo.com"  # confirm against current docs

# Your business-critical fields. Change these to match what you actually store.
REQUIRED_PRODUCT_FIELDS = ["asin", "title", "price", "rating", "bsr", "fetchedAt"]
REQUIRED_SEARCH_FIELDS = ["keyword", "page", "results", "sponsoredCount", "fetchedAt"]

MARKETPLACES = ["amazon.com", "amazon.co.uk", "amazon.de"]
SAMPLE_ASINS = [
    "B0CXYZ1234", "B0ABCDEFGH", "B012345678",
    "B0TEST0001", "B0TEST0002", "B0TEST0003",
]
SAMPLE_KEYWORDS = ["insulated water bottle", "standing desk", "air purifier"]


@dataclass
class CallResult:
    ok: bool = False
    latency_ms: float = 0.0
    fields_ok: bool = False
    missing: list = field(default_factory=list)
    error: str = ""


def check_fields(payload: dict, required: list) -> tuple:
    missing = [f for f in required if f not in payload or payload[f] in (None, "", [], {})]
    return (not missing), missing


def call_product(api_key: str, asin: str, marketplace: str) -> CallResult:
    result = CallResult()
    started = time.perf_counter()
    try:
        resp = requests.get(
            f"{API_BASE}/product",
            params={"asin": asin, "marketplace": marketplace},
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=30,
        )
        result.latency_ms = (time.perf_counter() - started) * 1000

        # HTTP 200 is not success. A blocked page also returns 200.
        if resp.status_code != 200:
            result.error = f"HTTP {resp.status_code}"
            return result

        payload = resp.json()
        data = payload.get("data", payload)
        result.ok = True
        result.fields_ok, result.missing = check_fields(data, REQUIRED_PRODUCT_FIELDS)
    except Exception as exc:  # noqa: BLE001
        result.latency_ms = (time.perf_counter() - started) * 1000
        result.error = f"{type(exc).__name__}: {exc}"
    return result


def call_search(api_key: str, keyword: str, marketplace: str, page: int) -> CallResult:
    result = CallResult()
    started = time.perf_counter()
    try:
        resp = requests.get(
            f"{API_BASE}/search",
            params={"keyword": keyword, "marketplace": marketplace, "page": page},
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=30,
        )
        result.latency_ms = (time.perf_counter() - started) * 1000
        if resp.status_code != 200:
            result.error = f"HTTP {resp.status_code}"
            return result
        payload = resp.json()
        data = payload.get("data", payload)
        result.ok = True
        result.fields_ok, result.missing = check_fields(data, REQUIRED_SEARCH_FIELDS)
    except Exception as exc:  # noqa: BLE001
        result.latency_ms = (time.perf_counter() - started) * 1000
        result.error = f"{type(exc).__name__}: {exc}"
    return result


def build_tasks(api_key: str, samples: int):
    """Alternate product and search calls; rotate marketplace and page depth."""
    tasks = []
    for i in range(samples):
        marketplace = MARKETPLACES[i % len(MARKETPLACES)]
        if i % 2 == 0:
            asin = SAMPLE_ASINS[i % len(SAMPLE_ASINS)]
            tasks.append(("product", call_product, (api_key, asin, marketplace)))
        else:
            keyword = SAMPLE_KEYWORDS[i % len(SAMPLE_KEYWORDS)]
            page = (i % 3) + 1  # 1..3, also probes deep-page availability
            tasks.append(("search", call_search, (api_key, keyword, marketplace, page)))
    return tasks


def percentile(values: list, pct: float) -> float:
    if not values:
        return 0.0
    ordered = sorted(values)
    idx = min(int(len(ordered) * pct / 100), len(ordered) - 1)
    return ordered[idx]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--samples", type=int, default=100)
    parser.add_argument("--concurrency", type=int, default=8)
    args = parser.parse_args()

    api_key = os.environ.get("PANGOLINFO_API_KEY")
    if not api_key:
        raise SystemExit("Set PANGOLINFO_API_KEY first")

    tasks = build_tasks(api_key, args.samples)
    results: list = []

    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
        futures = [pool.submit(fn, *params) for _, fn, params in tasks]
        for future in as_completed(futures):
            results.append(future.result())

    latencies = [r.latency_ms for r in results]
    succeeded = [r for r in results if r.ok]
    complete = [r for r in succeeded if r.fields_ok]

    total = len(results)
    success_rate = len(succeeded) / total * 100 if total else 0
    completeness = len(complete) / len(succeeded) * 100 if succeeded else 0
    usable_rate = len(complete) / total * 100 if total else 0

    print("\n=== Amazon Data API acceptance report ===")
    print(f"samples            : {total}")
    print(f"median latency     : {statistics.median(latencies):.0f} ms")
    print(f"p95 latency        : {percentile(latencies, 95):.0f} ms")
    print(f"max latency        : {max(latencies):.0f} ms")
    print(f"success rate       : {success_rate:.1f}%")
    print(f"field completeness : {completeness:.1f}%")
    print(f"usable share       : {usable_rate:.1f}%")

    missing_counter: dict = {}
    for r in succeeded:
        for field_name in r.missing:
            missing_counter[field_name] = missing_counter.get(field_name, 0) + 1
    if missing_counter:
        print("\nmissing fields (by occurrence):")
        for field_name, count in sorted(missing_counter.items(), key=lambda x: -x[1]):
            print(f"  {field_name}: {count}")

    errors: dict = {}
    for r in results:
        if r.error:
            key = r.error.split(":")[0]
            errors[key] = errors.get(key, 0) + 1
    if errors:
        print("\nerror distribution:")
        for key, count in sorted(errors.items(), key=lambda x: -x[1]):
            print(f"  {key}: {count}")

    print("\nFeed the usable share into the formula to get your real unit price:")
    print("  cost per 1k usable records = monthly spend / usable records * 1000")
    print(f"  at {usable_rate:.1f}% usable, the list price is effectively {100 / usable_rate:.2f}x higher\n")


if __name__ == "__main__":
    main()

Reading the output

A healthy run looks something like this:

=== Amazon Data API acceptance report ===
samples            : 100
median latency     : 2980 ms
p95 latency        : 5240 ms
max latency        : 8120 ms
success rate       : 99.0%
field completeness : 98.0%
usable share       : 97.0%

Two things to look at beyond the headline numbers.

The missing-fields breakdown. If gaps cluster on one field, that field has weak coverage in your target categories. That is a specific conversation with the vendor, not a reason to reject the whole provider.

The error distribution. If you see a lot of timeouts, you have a latency problem. If you see parse failures on otherwise healthy responses, you have a structuring problem. These two have completely different fixes, and lumping them into one error log is how teams spend a week debugging the wrong thing.

Then re-run it weekly. A single run proves it works today. Weekly runs prove it still works. Quality degradation is almost always gradual — drifting from 99% to 95% takes weeks, and humans notice about two weeks after it matters.

The field matrix behind the checklist

The REQUIRED_PRODUCT_FIELDS list should not be invented. It should come from a matrix of what public Amazon pages can actually be structured into.

Object	Key fields	Common gaps
Product	ASIN, title, brand, price, rating, BSR, availability, variants	Incomplete variant dimensions; lost parent-child ASIN links; sale price mixed with list price
Search	Keyword, page, organic rank, ad rank, isSponsored	Sponsored mixed with organic; deep pages (7+) truncated
Review	Rating, title, body, date, verified purchase, variant, helpful votes	Truncated bodies from summary pages; no variant attribution
Offers	Seller, price, shipping, Buy Box holder, stock	Inconsistent Buy Box logic; only first offer returned
Best Sellers	Category, rank, ASIN, rank movement	Unlabeled category-tree changes; non-continuous history
Seller	Seller ID, name, rating, listing count	Weak seller-to-brand mapping; IDs differ by marketplace
Category	Category tree, node ID, filters, item count	Node IDs vary by site; incomplete filter enums
Sponsored	Ad type (SP/SB/SD), placement, rank, creative	Ads not separated from organic; missing creative fields

Two fields I would put on every checklist.

fetchedAt. A capture timestamp. Without it the record cannot enter a time series and cannot be audited. Three months later, when a price looks wrong, you cannot tell whether it was wrong then or got mangled later.

isSponsored on search results. Without it, organic rank and ad placement are indistinguishable. You think you rank third; the top two slots are ads; your true organic position is first. Every optimization decision built on the contaminated number points the wrong way.

Not everything needs to be real time

Having argued that "real time" is widely abused, it is worth being equally clear about the opposite mistake: refreshing everything at maximum frequency.

Field change rates vary by orders of magnitude. Brand names, category assignments and variant dimensions barely move. Price, stock, Buy Box ownership and sponsored placements move constantly. Treating them identically is the most common form of budget waste in this category.

A three-tier split is usually right:

High frequency, minutes to hours — price, stock, Buy Box holder, sponsored placements, BSR. For these, a stale value does not weaken the insight, it invalidates it. A competitor's price from yesterday cannot support a pricing decision today.

Daily — new reviews, rating changes, seller lists, best seller ranks. Trend signals that matter over weeks rather than minutes.

Weekly or on demand — brand, category, variant dimensions, images, A+ content. Refreshing these hourly is pure waste with no analytical benefit.

The practical observation from many deployments: the top 20% of ASINs drive roughly 80% of the decisions. Running that 20% at high frequency and the long tail weekly typically cuts cost by half with almost no measurable business impact. It is the highest-leverage cost optimization available, and it requires no vendor negotiation — it is purely a scheduling decision in your own ingestion layer.

Two implementation notes.

First, implement the tiering in your own ingestion layer rather than hoping the provider does it. You know which ASINs matter to your business; they do not.

Second, revisit it periodically. Business priorities shift, a category that was long tail becomes strategic, and a tiering decision made eighteen months ago is usually wrong today. Put a recurring reminder on it.

REST or MCP

Worth separating, because they get conflated constantly.

REST fits deterministic, scheduled batch work. You know which ASINs to pull, how often, and which table the results land in. Data pipelines and scale collection.

MCP fits exploratory work that needs reasoning. A user or agent states a business question and the agent decides which tools to call, how many pages to check, and how to cross-validate. Research, diagnostics, one-off analysis.

One line: with REST you write a program that fetches data; with MCP you ask an agent to fetch and explain. Backend service consumer, use REST. AI agent consumer, MCP removes a lot of glue code.

The decision tree

Step 1. Is the data only about your own seller account? Yes, use the official SP-API. No, continue.

Step 2. Do you need public marketplace facts — competitors, categories, search, sponsored placements? No, revisit the requirement. Yes, continue.

Step 3. Do you have engineers who will own anti-bot, rendering and parsing maintenance long term? No, pick an Amazon-native data API. Yes, convert engineering hours to money and compare against purchase price on a per-thousand-usable-records basis.

Step 3 is where teams go wrong by comparing cash only. Price the engineering time at real internal cost. Plenty of "building is cheaper" instincts reverse once that number is on the page.

For reference, Pangolinfo publishes a median latency around 3 seconds, a 99% success rate, and more than 30 million calls per day; on sponsored placements, the published collection rate across 13 marketplaces is 91.4%. The point of those numbers is not that they are large. It is that you can reproduce or refute them with the script above.

Wrapping up

If the numbers come back bad, here is the triage.

Low success rate with high latency is usually capacity. Lower your concurrency, or ask about quotas and whether you are being throttled.

High success rate with low field completeness is a coverage problem. Identify exactly which fields are missing and ask about those specific fields in your target categories. This is often fixable, or at least scoped to particular categories you can route around.

Good numbers on page one that collapse by page three is a deep-page limitation. This one is usually a hard boundary rather than a transient issue, and it matters enormously if your business depends on tracking keywords where you rank deep.

Knowing which of the three you have determines whether the fix is configuration, a conversation, or a different provider. Collapsing them into "the data is bad" is how teams waste a week.

If you take one thing from this post, take the ordering:

Write the field checklist first (15–30 fields).
Score candidates field by field, not "do you support products?".
Convert list price to cost per thousand usable records.
Run the 100-call test and record all four numbers.
Look at the price sheet last.

If you want to run the script against real data, grab a key from the console or read the Amazon Data MCP docs. The underlying products are Amazon Scraper API for product, search, best seller, category and sponsored objects, Amazon Review API for reviews and customer voice, Amazon Data MCP when an agent needs to call data directly, and Amazon Scraper Skill for conversational workflows.

The full buyer's guide with the field matrix, three real JSON samples and the acceptance method is here: Amazon Data API: The Complete Buyer's Guide.

If you run the test, I would be curious what numbers you get — especially p95. That is the one where published specs and reality tend to diverge most.

Build an Amazon Review Monitor: ratings, complaint signals, and review media

Pangolinfo — Tue, 01 Sep 2026 02:43:34 +0000

Amazon review data is useful only when it can answer a decision: has a new quality issue emerged, what do verified buyers say, and is there visual evidence behind it? This guide shows a small, production-oriented pattern for collecting reviews, flagging complaint risk, and storing the fields needed for follow-up analysis.

The example uses the Pangolinfo Amazon Review API. It returns structured review records rather than page HTML, including star rating, title, text, verified-purchase status, helpful votes, product variant data, review images, and review video URLs.

1. Decide what to monitor

Start with a target ASIN and a clear observation rule. A useful baseline is:

retrieve the most recent page of reviews every day;
retrieve one-star reviews separately to surface issues early;
preserve image and video URLs for review validation;
keep the review ID and date so repeated runs can be deduplicated.

This keeps the workflow focused on new customer feedback instead of reprocessing the entire review history.

2. Make a structured review request

Create a Pangolinfo API Key in the Console, then make the request below. Do not put a production key into shared workflows, browser code, or public collections.

curl --request POST \
  --url https://scrapeapi.pangolinfo.com/api/v1/scrape \
  --header 'Authorization: Bearer YOUR_PANGOLINFO_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://www.amazon.com/dp/B076CLQDR4",
    "site": "",
    "bizContext": {
      "bizKey": "review",
      "pageCount": 1,
      "asin": "B076CLQDR4",
      "filterByStar": "one_star",
      "sortBy": "recent"
    },
    "format": "json",
    "formatType": "all_formats",
    "mediaType": "all_contents",
    "parserName": "amzReviewV2"
  }'

The same endpoint supports all_stars when you need a broad feedback sample. Full request and response fields are documented in the Amazon Review API reference.

3. Persist only the fields that drive decisions

For each record in data.json[].data.results[], store at least:

Field	Why it matters
`reviewId`	Deduplicate repeated scheduled runs.
`star`, `title`, `content`	Detect rating shifts and complaint themes.
`date`, `country`	Attribute changes to a time period and marketplace.
`purchased`, `vineVoice`	Separate verified-buyer feedback from other sources.
`helpful`	Prioritize issues readers find useful.
`imgs`, `videos`	Preserve evidence for quality and listing reviews.
`attributes`, `asin`	Identify a problematic size, color, or variant.

4. Turn raw reviews into alerts

A simple alert rule is more robust than a generic sentiment score:

Alert when a new verified one-star review has an image or video.
Alert when two or more reviews in the same run mention the same failure mode.
Include ASIN, review ID, rating, title, date, and evidence URLs in the alert.
Add a cooldown key such as asin:reviewId so the same review is not sent twice.

For no-code automation, the Pangolinfo n8n node and public Postman Amazon Review API collection provide starting points.

5. Use the MCP route for agent workflows

If an agent needs to combine reviews with product, keyword, niche, and AI-search data, use Amazon Data MCP instead of maintaining separate parsers. The MCP server exposes the review capability alongside the broader Amazon and AI-search toolset, while your application keeps control of its own API key and usage limits.

Practical takeaway

An effective review monitor is not an endless scrape. It is a small, repeatable process: collect recent structured feedback, retain identifiers and evidence, detect new recurring issues, and send a deduplicated alert to the team that can act on it.

We Built an Amazon Brand Operations Cockpit for a Global Electronics Brand. Here's the Architecture.

Pangolinfo — Mon, 24 Aug 2026 09:27:10 +0000

Most "data-driven Amazon" setups stop at exporting backend numbers into a spreadsheet. Exporting is not operating. This post walks through a real, anonymized project — a global consumer-electronics brand in the action-camera and pocket-gimbal category, across several major overseas markets — where we rebuilt their growth decisioning from a weekly lag into an hour-level response system: an Amazon Brand Operations Cockpit. The brand name and specific numbers are redacted; the architecture, dimensions, and SOPs are reusable.

The problem: why monthly reports can't save your growth

A monthly report is post-hoc description. Its collection grain is weekly, its presentation grain is monthly, while Amazon's competitive rhythm is hourly: deal badges, coupons, Buy Box rotation, keyword-rank jumps all happen within a single day. By the time the monthly report ships, the window is closed.

Our goal for this brand was never "a prettier report." It was: anomalies surface themselves the same day, with an attached "what to do." That requires splitting signals into three layers and bolting the third directly onto the data.

Layer 1: real-time Amazon sales signals (8 dimensions)

We locked in eight concrete dimensions, each with a data source, refresh cadence, and trigger threshold. This table is lifted from the actual collection schema:

Dimension	What it reveals	Trigger threshold (example)
BSR daily trajectory & velocity	Demand shifting before sales move	Same ASIN moving one direction for days, or a single-day drop beyond category norm
Price vs. category median	Competitor pre-emptive cuts, own squeeze	Effective price drops below median, or multiple rivals cut together
Buy Box win rate	Losing the box (reseller, logistics, price war)	Win rate in a market or time window below own baseline
Core keyword rank volatility	Rank erosion before it hits BSR	Core term falls off the first search screen, or a new term breaks into the top
Review velocity & rating drift	Quality / reputation inflection	Rating trends down, or negatives cluster in a short window
Search placement & share-of-shelf	New entrants, ad slots taken	Organic position drops, sponsored slot captured by rival
Variant / ASIN cannibalization	New variant stealing main variant traffic	New variant rises while main variant falls in sync
Cross-market divergence	Same ASIN behaving differently by market	One market anomalous while others are flat

The first five (BSR, price, Buy Box, keywords, reviews) are "lead" signals — they usually move before sales actually drop.

Implementation note: the underlying data uses the Amazon Scraper API for hourly collection covering BSR, price, coupons, inventory, and ad slots, with timestamp alignment — that alignment is what makes "velocity" and "divergence" calculations possible. Review and rating anomalies are pulled separately via the Amazon Review API so they aren't drowned out by the main-signal noise.

Layer 2: external social-demand signals (6 dimensions)

This is the off-site calibration. We plugged cross-platform social listening and broke it into six dimensions:

Share of voice by platform (TikTok / Instagram / YouTube / Reddit / X): where the conversation actually lives. The "home platform" differs completely by category and market.
Sentiment split (positive / neutral / negative): whether the narrative is helping or planting a landmine. A negative cluster often precedes a rating drop on Amazon.
Engagement by content format: short video, image, long-form, live — which format actually drives interaction. It decides where content budget goes.
Creator-tier mix (macro / micro / nano): was voice driven by top creators or spread by mid/small creators — this determines whether the channel strategy is repeatable or lucky.
Unbranded demand intensity: are users searching a brand name, or "how to shoot / which to buy" generic demand. This is the key signal for whether the category is pulling new buyers or the brand is catching them.
Competitor co-mention: who users compare together. Shifts in co-mention are the earliest substitution-threat signal.

Social is often the "cause" while Amazon on-site signals are the "result" — social heat typically leads search by 1–2 weeks.

Layer 3: decision rules (trigger → hypothesis → action)

The first two layers are only "seeing." The third is "driving." We attached every anomaly to a loop:

when dimension crosses threshold:
    hypothesis = generate_falsifiable_guess(dimension, context)
    action = map_action(hypothesis)
    owner = route_owner(dimension)
    notify(owner, anomaly + hypothesis + action)

We wired the Amazon Data MCP so an Agent reads the data and issues alerts directly — operators can ask in plain language "which ASINs are anomalous today" without writing SQL.

Three signals the monthly report missed

Signal 1 (cross-market · price): In one major overseas market, several same-category rivals pushed effective price below the category median within days, while the brand's pricing stayed put. In the first week, that market's BSR began slipping and Buy Box win rate fell with it — but all of it would only appear in the month-end report. The cockpit set "price vs. category median" to daily monitoring and triggered the same day, letting the operator judge, before the window closed, whether the rival was running a short promo or a long-term strategic cut.

Signal 2 (cross-platform · social): Social listening showed category-level "how to choose / which to buy" discussion contributed the vast majority of reach on short-video platforms, with engagement well above norm and sentiment mostly positive. But crossing social high-frequency terms with Amazon search terms exposed a glaring gap: a high-search-volume generic category term had zero Amazon ranking in the first two pages for the brand, while competitors already ranked organically near the top and ran sponsored placements. The category was acquiring new buyers on short video; the brand failed to catch the narrative into search.

Signal 3 (head-to-head · share of voice): In the launch of a new-generation pocket gimbal camera, the rival looked far stronger on the surface. The teardown revealed two things the monthly report ignored: first, the rival leaned on first-mover advantage and aggressive pricing to spike early, while the brand's own product had a steeper post-launch month-over-month acceleration — slower to heat, stronger in the back half; second, and most critically, the two products had almost zero keyword overlap in search. Zero overlap meant the generic and scenario long-tail terms the brand had never bid on were likely low-cost incremental pockets. At the same time, the rival lifted share of voice on a single platform via mid-tier creators while the brand's owned content underperformed on engagement.

Four playbooks that turn signals into action

Price-band defense SOP: threshold = effective price vs. category median ±X%. Short promo → defend keywords with ads, don't follow price; multi-day strategic cut → evaluate adjusting the price band or launching a secondary SKU. Owner: operations, daily.
Keyword rescue SOP: core term falls off the first screen → three-step: buy impression back with PPC, reinforce the related term in content and bullets, then check whether reviews/rating are dragging. Conversely, when a term first breaks into the top, increase investment immediately (most teams miss this step).
Reputation & review recovery SOP: review anomaly → cluster by theme (logistics / quality / expectation gap) → route action. Treat reviews as free user research.
Launch cadence & social reallocation SOP: use "unbranded demand intensity × platform" as the pre-launch heat map; move content budget to high-demand platforms; on Amazon, pre-stock and pre-schedule PPC by heat.

Cross-market and cross-platform attribution methodology

Lead-lag first: did the signal appear on Amazon first, or on social first? If social leads, use it as leading intel.
Horizontal matrix: read divergence with a "market × dimension" matrix to avoid single-market bias.
Allocate by elasticity: in markets where BSR is price-sensitive, prioritize price defense; in markets where it's content-sensitive, prioritize social reallocation.

Build a minimum-viable cockpit in three phases

Phase 1 (weeks 1–2): watch only daily price, BSR and Buy Box on core ASINs, with threshold alerts pushing anomalies the same day. Data via Amazon Scraper API hourly collection.
Phase 2 (weeks 3–4): add keyword rank and review anomalies (Review API), wire Amazon Data MCP for Agent auto-alerts.
Phase 3 (from month 2): connect social listening for external calibration, close the trigger→hypothesis→action loop, and write the four playbooks into the operations SOP.

A concrete "cockpit workday"

To keep the dimensions grounded, here is an anonymized real morning. Monday 9 a.m., the cockpit pushed three reds:

One major overseas market: a flagship's effective price was pushed below the category median by rivals, BSR slipped day-over-day, and Buy Box win rate dipped from its high.
Social: "how to choose / which to buy" discussion on short-video platforms rose week-over-week, while our Amazon rank on that scenario term was still blank.
Reviews: after a new variant launched, the main variant's rating dipped slightly and negatives clustered on "expectation gap."

The operator spent 15 minutes on three moves. For Market A, they first judged it a short promo — checking the rival's promo history confirmed a weekly pattern, not a strategic cut — so they defended core keywords with ads and did not follow price. They dropped the trending social scenario term into the content calendar and PPC candidates. For the main variant's "expectation gap" negatives, they published a video for expectation management. By Friday's review, Market A's BSR had recovered, the scenario term had earned organic plus sponsored placement, and the main variant's rating had stopped falling. That is the gap between "same-day response" and "monthly lag" — and all three moves would have waited until the month-end review under the old flow.

Threshold configuration: how to set each of the 8 Amazon thresholds

A cockpit lives or dies by its thresholds. Guesswork either false-alarms daily or never fires. Here is how we set each of the eight:

BSR trajectory & velocity: compute a 14-day rolling baseline per ASIN; alert when the day-over-day delta exceeds two standard deviations, or when the slope turns negative for three consecutive days. Velocity (the slope) matters more than the level.
Price vs. category median: pull the category median daily; alert when own or any tracked rival's effective price crosses ±X% of median. The median — not a fixed price — is the right reference because categories drift.
Buy Box win rate: track win rate per market per day; alert when a market's 7-day average drops below its own historical baseline by more than a band. A dip in one market while others are stable is the classic "something is wrong here" signal.
Core keyword rank: alert when a core term falls off the first search screen, or when a tracked term enters the top for the first time (the latter is an opportunity, not a threat).
Review velocity & rating: alert when daily review count spikes above baseline (possible spike of incentivized or fake reviews) or when rolling rating drops by a threshold. Cluster negatives by theme automatically.
Search placement & share-of-shelf: alert when organic position for a core term drops, or when a rival's sponsored slot appears above your organic result.
Variant / ASIN cannibalization: alert when a new variant's BSR rises while the main variant's falls in sync — a sign the launch is stealing internal traffic.
Cross-market divergence: alert when one market diverges from the median of all markets for the same ASIN.

The pattern: every threshold is "relative to a baseline or a median," never an absolute number pulled from thin air.

Second worked example: the head-to-head launch week

Signal 3 deserves its own walkthrough because it is the one most teams misread. When the rival's new pocket-gimbal launched, surface metrics said they were winning: early sales were multiples of ours. The cockpit's teardown changed the call. First, the rival's velocity came from first-mover advantage and aggressive pricing; our product's post-launch month-over-month acceleration was actually steeper — slower to heat, stronger in the back half. Second, and decisively, the two products had almost zero keyword overlap: they owned brand terms and a few generic words; we owned a different brand-term family. They were not colliding; they were in different traffic pools.

The action that followed was counter-intuitive: instead of panic-bidding on the terms where the rival was strong, we mapped the generic and scenario long-tail terms neither of us ranked for, confirmed real search demand, and quietly built content and PPC there. Those became low-cost incremental pockets. Meanwhile we flagged that the rival was lifting share of voice on a single platform via mid-tier creators while our owned content underperformed — a narrative-share problem the monthly report would never surface, addressed by reallocating content budget toward that platform's high-demand format.

Five most common mistakes

Treating a dashboard as a cockpit: seeing isn't driving; without a trigger→action loop, a dashboard is just a pricier report.
Watching only yourself, not competitors: share is taken by others; rival price, rank, launches, and lost Buy Box are the real alert sources.
Siloing social and Amazon: social heat leads search by 1–2 weeks; siloing throws away the lead.
Guesswork thresholds: without "vs. category median" or "vs. own baseline" references, thresholds either false-alarm daily or never fire.
No owner: routing an anomaly to "the team" means routing it to no one. Every anomaly must map to a specific owner.

How to measure whether the cockpit actually works

Five health metrics, themselves part of the cockpit: anomaly response time (target within a day), price-defense hit rate (correct "defend vs. follow" calls), keyword-rescue success rate (terms that return to the first screen), social→search conversion (trending social terms that earn Amazon rank), and share-recovery cycle (average time for an affected ASIN's BSR/share to return to baseline). The cockpit must both drive and self-check.

Tooling and data pipeline (no detours)

Real-time collection via the Amazon Scraper API (hourly, timestamp-aligned); review and rating anomalies via the Amazon Review API; Agent natural-language alerts via the Amazon Data MCP; the social layer via cross-platform listening for external calibration. We deliberately did not build a black-box SaaS — the data pipeline and integration are yours to wire into your own BI, Agent, and alert logic.

Why each of the 8 Amazon dimensions matters (with mini-examples)

Most teams ask for "more data" without clarifying what each dimension actually defends. Here is the per-dimension logic, because skipping any one of them leaves a blind spot.

BSR daily trajectory & velocity defends against "we only find out when sales drop." BSR reacts to demand before sales do, and velocity (consecutive same-direction moves) beats the single-day level. In the anonymized case, a flagship's BSR fell three days straight before sales moved on day four — watching sales alone costs you four days of response time.
Price vs. category median defends against "being quietly undercut." Using "relative to median" rather than an absolute price matters because categories drift; when rivals push effective price below median, your original price band effectively becomes expensive even if you didn't move.
Buy Box win rate defends against "losing the box without knowing." Resellers, logistics delays, and price wars all drop win rate; reading it per market localizes exactly where you lost.
Core keyword rank defends against "rank dropped before it hit BSR." Rank is upstream of BSR; rescue it while it's still off the first screen, not after BSR already slid.
Review velocity & rating defends against "reputation inflection drowned out." A rating decline or negative cluster often precedes a sales problem by weeks.
Search placement & share-of-shelf defends against "new entrants and ad slots taken." An organic position drop or a rival's sponsored slot above you are both direct share-loss signals.
Variant / ASIN cannibalization defends against "fighting yourself." A new variant stealing the main variant's traffic is frequently mistaken for competitor action.
Cross-market divergence defends against "global averages hiding local problems." One market anomalous while others are flat signals a local issue, not a category-wide one.

How the 6 social dimensions actually get used

Social is not "post content and watch views"; it is Amazon's leading indicator. How each dimension is operationalized:

Share of voice by platform tells you where the conversation lives. This category lives on short video, that one on long-form; being on the wrong platform wastes the entire budget.
Sentiment split gives a 1–2 week early warning of an Amazon rating risk the moment negatives cluster — long before the star rating moves.
Engagement by format decides budget allocation: if short video drives interaction, stop pouring money into image posts that quietly underperform.
Creator-tier mix tells you whether voice built from mid-tier creators is more repeatable than a single top creator's one-off spike.
Unbranded demand intensity is the switch between "category acquiring" and "brand catching." High unbranded demand plus a blank brand rank is the single biggest gap we found.
Competitor co-mention flags substitution threat the moment users start comparing you with someone new.

Writing "trigger → hypothesis → action" into your ops manual

This layer is the most important and the easiest to skip. What we built was not a "dashboard" but a "dashboard + rules": when any dimension crosses its threshold, the system first proposes a falsifiable hypothesis (short promo or long-term cut), then a standard action and an owner. The operator receives not a pile of numbers but a prioritized to-do with a recommendation attached. Without this layer, the first two are just a pricier weekly report. The hypothesis step matters because it prevents over-reaction: a price dip that is a week-long promo does not deserve the same response as a strategic repricing, and the system forces that distinction to be made explicitly rather than in someone's gut.

What this looks like as a weekly operating rhythm

Monday the cockpit surfaces three reds; the operator spends 15 minutes triaging each against its hypothesis. Tuesday through Thursday, the same loop runs daily but lighter — mostly confirming yesterday's actions are holding. Friday is the review: not "what happened last month," but "did our same-day responses hold share, and which thresholds need recalibration." That Friday recalibration is where the system learns — thresholds that false-alarmed get widened, blind spots get new dimensions added. Over a quarter, the cockpit stops being a project and becomes the team's default operating rhythm.

A note on team size

The most common objection is "we're too small for this." You are not. The minimum viable version is three dimensions (price, BSR, Buy Box) on your core ASINs, with threshold alerts pushed to one person's phone. That alone closes the "monthly lag" gap. The social layer and the full eight dimensions are phase-two upgrades, not prerequisites. The point is to start driving with a few instruments, not to wait until you can afford the whole cockpit.

A deeper look at cross-market attribution (worked)

Attribution is where most multi-market teams quietly fail, because they reverse cause and effect. The discipline we built has three moves. First, lead-lag: determine whether a signal appeared on Amazon first or on social first. In the anonymized case, the "how to choose" social spike preceded the Amazon search lift by roughly two weeks — so social was the leading indicator, and the team learned to pre-stage PPC before the search spike, rather than explaining it after. Second, the horizontal matrix: read divergence with a "market × dimension" grid. When Market A's BSR dropped, the matrix showed Markets B and C were flat — so this was a local competitive event in A (a rival promo), not a category-wide demand dip. Treating it as category-wide would have triggered a wasteful global price cut. Third, allocate by elasticity: in markets where BSR is price-sensitive, priority goes to price defense; in markets where it is content-sensitive, priority goes to social reallocation. A one-size playbook fails across markets because the lever that moves the needle differs by market.

A second anonymized mini-case: the review-velocity early warning

The first case was about price and share of voice; the second was quieter and arguably more valuable. A new variant launched, and within days the main variant's review velocity doubled while its rolling rating dipped by a hair — not enough to trip a "rating dropped" alert, but the velocity spike was anomalous versus baseline. The cockpit auto-clustered the new negatives and found them concentrated on "expectation gap": the listing promised one thing, the unboxing video implied another. The team published a short expectation-management video and tightened the bullet copy. Two weeks later, when a competitor launched a similar variant and their rating actually fell, our main variant's rating had already stabilized. The lesson: velocity — the rate of change — is a better early signal than the level, and clustering negatives by theme turns a raw rating into an actionable fix.

What to put on the cockpit home screen (a template)

If you build this, the home screen should be six to eight cards, not a wall of charts. Today's reds (the prioritized action queue). A price-vs-median heatmap across markets. Buy Box win rate by market. Keyword movers (terms that entered or left the first screen). Social share of voice by platform with sentiment. Review health (velocity + rating + top negative theme). An action queue with owner and SLA. And a response-time gauge showing the average hours from signal to action. That last card is the whole point: the cockpit is only as good as how fast the team closes the loop.

Takeaway

An Amazon Brand Operations Cockpit is not a report; it's the operating system for growth. It doesn't replace strategy, but it lets strategy be executed correctly at the right moment. The full deep-dive is here: Amazon Brand Operations Cockpit: Why Monthly Reports Can't Save Your Growth. More similar practice: Pangolinfo customer cases.

Customer Service Agent End-to-End Integration: Defining the Delivery Boundary, Not Wiring More APIs

Pangolinfo — Fri, 14 Aug 2026 07:34:50 +0000

If you have read more than three "build a customer support agent" tutorials in the last year, you have seen the same script: pick a model, chunk your FAQs, drop them into a vector store, add a retriever, wrap it in a chat UI, and declare victory. I have sat in on too many post-mortems where a team shipped exactly that and then watched the agent fail the first time a customer asked "where is my refund?" instead of "what is your return policy?" The gap between a demo and a system that actually closes tickets is almost never about the model. It is about Customer Service Agent End-to-End Integration — and most teams define that phrase wrong.

The mistake is treating "end-to-end" as a plumbing problem. More connectors, more APIs, more middleware. In my experience, that framing is the single biggest reason support agents stall at the proof-of-concept stage. End-to-end is not the count of integrations you have connected. It is the delivery boundary: the exact line where the agent stops talking and starts changing something, what it is allowed to change, who confirms it, and how you undo it if it was wrong. If you cannot draw that boundary on a whiteboard in one sentence, you do not have an end-to-end agent. You have a chatbot with a longer résumé.

This article is a field guide to drawing that boundary. It is written for engineers and technical leads who are past the "hello world" stage and are now being asked to make a support agent actually do things: look up orders, draft refunds, write tickets, and survive the day a policy changes mid-incident. I am not going to teach you how to embed text. I am going to give you a framework you can implement this quarter.

The counter-consensus: Customer Service Agent End-to-End Integration is a delivery boundary, not API count

RAG tutorials quietly assume that 80% of support is "answer the question from the knowledge base." That is true only for a narrow slice of contacts: pre-sales curiosity, policy explanation, how-to. The contacts that cost you money and attrition are the ones that require state change — a refund issued, an address edited, a replacement shipped, a ticket escalated. Those are the calls that pile up in queues, and they are exactly the ones a pure RAG bot cannot touch.

So when I say Customer Service Agent End-to-End Integration, I mean the agent's ability to carry a request from "customer asked" to "real-world state changed, logged, and reversible." That journey crosses four kinds of capability that most tutorials never separate:

Reading the knowledge base (the part everyone builds).
Looking up live operational truth (order status in an ERP, inventory, shipping).
Writing an action with authority and bounds (refund, ticket, flag).
Assigning responsibility and a rollback path for every step.

Notice the last two. They are the unglamorous half. They are also, in every failure review I have run, the half that was missing. A team will happily spend six weeks tuning retrieval and then hand the agent refund authority with no approver and no undo. The integration that matters is the one that says: here is the boundary, here is who crosses it, here is the receipt.

Let me give you the mental model I use to design that boundary.

The four tables framework behind Customer Service Agent End-to-End Integration

Every production support agent I have helped ship runs on what I call the four tables. They are not literally four database tables in every case, but they are four distinct sources of truth the agent must consult on every transactional turn. If one is missing, the agent is either unsafe or useless for real work.

Table 1 — The Fact table (real-time operational truth)

This is your live order/ERP/inventory state. It is the only truth for "what is happening right now." The knowledge base is not truth; it is guidance. The Fact table answers "is the order actually shipped?" not "what does our policy say about shipping?"

Common mistakes: teams let the agent answer order-status questions from the KB ("usually ships in 3–5 days") instead of the Fact table. That is how you tell a customer their package is en route when it was returned to sender two days ago. The Fact table must be the system of record, queried live, with a timestamp the customer can see.

Table 2 — The Rule table (versioned policy)

Policy is not a paragraph you paste into the prompt. Policy is a versioned table with effective dates, priorities, and conflict-resolution logic. "Effective 2026-06-01, electronics returns within 30 days, priority over the generic 14-day rule." When two rules collide, the table decides, not the model's vibes.

I have seen a launch slip because marketing changed the return window in a blog post but never updated the agent's policy source. The agent kept quoting the old window for three weeks. A versioned Rule table with an effective-date column and a published changelog would have caught it on day one. If you cannot answer "which policy version answered this customer?" you do not have a Rule table; you have a prompt someone edited at 2 a.m.

Table 3 — The Action table (writes + authority bounds)

This table declares which writes the agent may trigger and the authority boundaries around each. "May draft a refund up to $50; may not issue it. May open a ticket in queue P2; may not ban an account." This is where most RAG-only teams freeze, because it forces a real decision about authority.

The key insight: the Action table is not about capability (can the API do it?) but about mandate (are we allowed to let the agent do it unsupervised?). Decoupling those two questions is what lets you ship incrementally instead of not at all.

Table 4 — The Responsibility table (approver, owner, rollback, audit)

This is the one nobody builds and everyone wishes they had. For every action step, it records: who approves, who owns the outcome, how to roll back, and what the audit record looks like. Missing the Responsibility table is the #1 cause of failed deployments I have reviewed. Teams build read + lookup + draft, then discover they have no defined approver when the agent wants to issue a $200 refund at 11 p.m., no owner when a write goes wrong, and no undo when a duplicate ticket is created.

Here is the skeleton I hand teams:

Step	Fact source	Rule applied	Action (R/W)	Authority	Approver	Rollback
Order lookup	ERP read replica	—	Read	Auto	—	n/a
Eligibility	Rule table v2026.06	Return window	Read	Auto	—	n/a
Refund draft	Fact + Rule	Amount cap	Write (draft)	Agent	Human (tier 2)	Void draft by ID
Ticket open	CRM	Category	Write	Agent	Auto (queue)	Close by ticket ID
Refund issue	Payment API	>$50	Write	Human only	Tier 2 agent	Reverse by txn ID

If your design fits in that table, you can build it. If it doesn't, the blank cells are your risks.

Q&A bot vs transactional agent: the line that changes everything

I want to be precise about a distinction the market blurs on purpose. A Q&A bot only reads the knowledge base and outputs text. It is stateless with respect to your business. A transactional agent reads, looks up live facts, writes a state change, and audits it. The second one delivers a real outcome; the first one delivers a sentence.

This is not a moral hierarchy. Most contacts should be handled by the Q&A bot, and that is fine. The error is pretending your Q&A bot is "end-to-end" because it can also call one read API. A customer whose package is lost does not care that your bot "integrates" with the carrier's tracking page. They care that someone changes their situation. The transactional agent is the only one who can do that, and it is the only one that needs the four tables.

A practical way to decide where to invest: tally your last 1,000 tickets by required capability, not by topic. How many needed only a text answer? How many needed a lookup? How many needed a write? You will usually find that 20–30% of volume is transactional, and that 20–30% is responsible for most of your cost and most of your escalations. That is your integration target. Do not spend your budget making the FAQ bot marginally smoother.

No ERP API? Start with tiered integration (three tiers)

The most common objection I hear: "our ERP has no API, so we can't do this." That is false, and it is the excuse that keeps support teams on manual queues for another year. You do not need a clean API to start. You need a tier you can ship today and a path to the next tier.

Tier 1 — Read replica for lookup. If the ERP stores data in a database, stand up a read-only replica or a scheduled export. The agent queries order status from there. No writes, no risk to production, near-zero blast radius. This alone deflects the "where is my order?" calls that dominate repeat contact.

Tier 2 — Agent drafts, human approves. The agent prepares the ticket, the refund suggestion, the response — and a human clicks confirm. The agent does the cognitive work; the human carries the mandate. This is where you capture most of the efficiency without handing over authority.

Tier 3 — Low-amount, low-risk actions auto-execute and log. Only after Tier 1 and 2 are stable do you let the agent auto-issue, say, refunds under a small cap or auto-close certain ticket types. Every auto-action is logged with an audit ID and is reversible.

I worked with a retailer running a ten-year-old ERP that had no API and no budget to expose one. We pointed a read replica at the order database, gave the agent order-status lookup against it, and kept all tickets and refunds human-approved. Result: it deflected 60% of repeat "status" calls within the first quarter, with zero writes to the legacy system. Nobody touched the ERP. The agent just read a copy. That is Customer Service Agent End-to-End Integration at the tier that fits your reality — not the tier in the vendor demo.

The trap is skipping Tier 1 to chase Tier 3. Teams that jump straight to auto-writes on a fragile connection ship incidents, then retreat to "the bot is off." Tiers exist so you can show value this quarter and earn authority next quarter.

Human approval is a first-class node, not a fallback

When teams finally add a human, they bolt it on as an afterthought: "if confidence is low, escalate." That is a fallback, not a node. A first-class approval node has three things a fallback lacks:

An SLA: the agent's draft waits in a queue with a target response time, and the customer sees "pending review, ~20 min." No silent hanging.
An escalation path: if the approver does not act in time, it routes up, not sideways into the void.
An evidence UI: the approver sees exactly what the agent saw — the Fact row, the Rule version, the proposed action — so they approve in seconds, not by re-investigating.

Risk tiers govern what auto-runs versus what waits:

Read-only and ticket-draft: can be automatic. Looking up an order and opening a structured ticket carries little irreversible risk.
Low-value compensation: capped and sampled. Auto-issue a small goodwill credit, but sample 5% for human review so the pattern stays honest.
Refund, address change, account ban: always human. These change money, identity, or relationship. No exception, no "just this once" override in the prompt.

Treating approval as a node changes your architecture. You build an approval service, not a branch in the prompt. The agent's job ends at "draft submitted, ID returned." The human's job starts there. That boundary is what makes the whole thing auditable.

Failure paths: three lines, not one happy path

A support agent that only handles the happy path is a liability, because support is where things go wrong on schedule. I design three lines:

Happy line — everything resolves, state changes, confirmation sent.
Degraded line — a dependency is down or uncertain. Route to a human, and the agent must state what is missing ("I can't confirm shipping because the carrier feed is delayed; routing you to an agent who can see the backup system"). Degraded is not failure; it is honest handoff.
Rollback line — a write was wrong or duplicate. Undo it by audit ID. If you cannot roll back a write, you do not ship that write. That is the rule. No rollback, no auto-execute.

The logs are what make the three lines operable. Every transactional turn records six things:

Trigger — what the customer said / what event fired.
Evidence — the Fact rows and Rule version the agent used.
System called — which downstream service, with request ID.
Read/write result — success, partial, timeout, with payload summary.
Approver — who confirmed (or "auto" with the cap that permitted it).
Rollback time — when it was undone, if applicable, with the audit ID.

If a post-mortem can't be reconstructed from those six fields, the integration was not end-to-end. It was a guess with a chat box.

Returns and exchanges capability map (eight steps)

Here is the capability map I use to pressure-test a returns/exchanges flow. Walk each step and fill the columns. If a column is blank, that is your gap.

Step	Input	Source	R/W	Failure mode	Human owner	Rollback
1. User question	"I want to return X"	Chat	Read	Unclear item	Agent (clarify)	n/a
2. Product identify	SKU / order line	Fact (order)	Read	Multiple matches	Agent	n/a
3. Order lookup	Order ID	Fact (ERP replica)	Read	Order not found	Agent	n/a
4. Policy match	Return window, category	Rule table	Read	Conflicting rules	Policy owner	n/a
5. Eligibility	Pass/fail + reason	Rule + Fact	Read	Edge case	Tier 2	n/a
6. Refund/ticket action	Amount, queue	Action table	Write (draft)	Over cap	Human approver	Void by ID
7. Notify	Email/SMS	Notify API	Write (notify)	Bounce	System retry	Resend by ID
8. Audit	Record	Log store	Write	Loss	Eng on-call	Rebuild from source

Two things to notice. Only steps 1–5 (reads) and step 7 (notify) are truly automatic. Every write that changes state — step 6 especially — is gated. And step 8, the audit, is not optional polish; it is the thing that makes steps 6 and 7 safe to exist at all. If you ship steps 1–5 and 7 but skip 8, you have a bot that sometimes does things you can't explain. That is how trust dies.

Where Pangolinfo fits: the external Amazon data layer

Let me be clear about scope, because vendors love to imply they do everything. Your internal systems — ERP, CRM, payment, warehouse — are yours to wire. That is the work above. What an external specialist can add is the Amazon-side fact layer: real-time product data, review signals, and ad-placement facts that live outside your walls.

For teams selling on Amazon, the customer's question often depends on data Amazon holds, not you. "Is this review pattern new?" "Did our ad placement change this week?" "What does the live listing say versus what we think we published?" Pangolinfo fills that external Amazon data layer — the product, review, and ad-placement real-time facts — so your agent reasons from current marketplace truth instead of a stale export. The Amazon Data MCP is the integration point we use to pipe those facts into the agent's Fact table without the team building scrapers themselves.

That is the only slice we own. Everything that writes to your customer's account is your Responsibility table, your approver, your rollback. Keep that separation and the architecture stays honest: external facts in, internal authority out.

Wiring it to the rest of your program

This article is sub-article #2 in our Enterprise AI Transformation series. It assumes the trust foundation laid in sub-article #1, Customer Service Agent Refusal, where we argue the agent must be able to refuse unsafe or unclear requests as the precondition for being allowed to act. An agent that can't say no should never be given write authority. Read those together, then read the pillar, Enterprise AI Transformation, for the full map.

The throughline: refusal is the brake, integration is the engine. You do not ship the engine without the brake.

A build order you can actually follow

If you take one thing from this piece, take a sequence:

Draw the delivery boundary in one sentence. If you can't, stop and think.
Stand up the Fact table (even a read replica counts).
Version your Rule table with effective dates.
Define the Action table: list every write, assign a mandate tier.
Build the Responsibility table: approver, owner, rollback, audit per step.
Ship Tier 1 (lookup) and Tier 2 (draft + human approve). Measure deflection.
Only then promote low-risk writes to Tier 3, capped and logged.
Wire the six-field log on every transactional turn.

Most teams invert this: they wire APIs first and define authority last, usually after an incident. The four tables put authority first, where it belongs.

The part nobody tells you

The hardest conversation in this whole project is not technical. It is the one where you tell the business: "We can make the agent answer anything, but it may only change these three things, and only with a human on two of them." That conversation feels like you are under-delivering. You are not. You are the only person in the room protecting the company from an agent that quietly issues refunds it shouldn't.

I have watched teams lose six months because they promised "full automation" and then couldn't get sign-off on a single write. The tiered model lets you ship value now and earn the rest. The four tables let you have the authority conversation with a diagram instead of a prayer. That is what Customer Service Agent End-to-End Integration actually is. Not more APIs. A boundary, drawn well.

If you are mapping this onto a live Amazon support operation and want the external marketplace facts handled for you, start with the deep dive on Customer Service Agent End-to-End Integration. It pairs with the refusal foundation and the transformation pillar above, and it is where we keep the current version of the four-tables template.

Teach Your Customer Service Agent to Say "I Don't Know" — A Refusal-First Engineering Guide

Pangolinfo — Thu, 13 Aug 2026 01:47:13 +0000

By Leo, AI & ecommerce data solutions lead at Pangolinfo

Most teams building a customer service agent measure success wrong from day one. They wire up a retrieval pipeline, point it at a vector store, and celebrate when the bot answers 92% of conversations without a human in the loop. That number is a trap. The bot you just shipped is probably confident about things it has no business being confident about — and the first time it invents a refund window or a coupon code, you lose a customer you spent months acquiring.

This article is the engineering take on a core thesis from our pillar on enterprise AI transformation: the most overlooked capability of a customer service agent is not answering more questions. It is refusing to answer when the evidence isn't there. I'll show you why this is a data problem and not a model problem, then give you a concrete, runnable design: a confidence-thresholded decision function, a policy-version resolver, an escalation-logging schema, and the integration pattern that actually keeps a bot honest.

If you just want code, skip to the Python section. But read the diagnosis first — most of your hallucination problems are not where you think they are.

The surface complaints are a smokescreen

When users dodge AI customer service, the feedback sounds like tone problems: "it has no human feel," "it doesn't understand me," "it makes things up." I've sat in too many post-mortems where a team responds to those by swapping the LLM or rewriting the prompt's personality. Stop. The root cause is almost never the model. It's the foundation under the model — an incomplete knowledge base and incomplete product data — and a missing safety behavior: controlled refusal.

Let me reframe each complaint with what's actually happening underneath.

Truth 1: "No human feel" is a recognition problem, not a tone problem

People assume human feel comes from phrasing — warmer greetings, more empathy tokens, a friendlier persona. Wrong. What a user actually wants is: you remember what I bought, where I'm stuck, and what we already discussed. When a bot greets every customer as a blank text box and opens with a policy recital, that isn't stiff tone. It's that the bot was never connected to the user's orders, logistics, and history. Polite without context is more annoying than impolite without context. The fix is a data integration, not a prompt.

Here is a concrete example from a brand I worked with. A customer who had already opened a replacement ticket for a damaged blender came back the next day and typed "any update?" The bot answered as if it were the first contact in human history: "Hello! I'd be happy to help with your blender. Could you tell me your order number and what's wrong?" The customer had to re-explain the damage, re-paste the order ID, and re-state the problem. Nothing about the bot's phrasing was unfriendly — but the bot had zero memory of the ticket it had helped create twelve hours earlier. The customer's "no human feel" complaint was really "this thing doesn't even know we were just talking." You cannot prompt your way out of that. You have to connect the agent to the ticketing system so the opening line reads "Hi Sam, I see your replacement ticket for order ORD-44190 is still in transit — want me to check the latest tracking?" That is recognition, not tone.

Truth 2: "Doesn't understand me" is a context-assembly failure

A user says "the cup arrived broken, I want a return." Behind that one sentence sit at least three system facts the agent must assemble: identify the SKU, check the return window, and decide refund versus reship. Many agents collapse the whole thing into a generic "return policy FAQ" dump. The failure isn't NLP quality — it's the inability to map one sentence onto a stack of system facts: product facts, order state, and policy version. All three have to be present and consistent, or the answer is wrong by construction.

A real failure story makes this concrete. A kitchenware seller rolled out a bot and a customer wrote: "the lid of my 8-cup French press shattered in the box, I want a replacement." The bot found the keyword "replacement," retrieved the generic returns FAQ, and replied with a link to print a return label and a note that refunds process in 5–7 days. What it failed to do was assemble the facts: the product had a known manufacturing defect on one variant (so the right move was a free reship, not a return-label-and-wait), the order was 9 days old (inside the window, so a refund was also possible but not required), and the policy version for defective-goods handling had a priority flag that beat the generic returns FAQ. The bot never pulled the variant, never checked order age, never loaded the defective-goods policy. It "understood" the word replacement and then guessed. The customer printed a label, shipped the whole set back, waited a week, and posted a one-star review about a "broken returns process." The model was fine. The context assembly was missing.

Truth 3: "Makes things up" is the most trust-destroying of all

This is the one that makes users give up for good. When evidence is thin, a model will produce a plausible, fully fabricated answer: a wrong refund window, a coupon that doesn't exist, a delivery time pulled from nowhere. One confident lie destroys more goodwill than a hundred correct answers earn. The bug isn't that the model "can hallucinate" — every LLM can. The bug is that the system never told it to stay quiet when it should.

Here is the independent take I keep repeating to teams: to judge whether a support agent is trustworthy, don't ask "how much did it get right." Ask first: "when it didn't know, did it honestly say so?" The first measures demo quality; the second measures production reliability.

A hallucination war story worth memorizing: a team seeded their bot with a discount FAQ that mentioned a "WELCOME10" code for new customers. Six months later the promotion ended and the FAQ was deleted from the source folder — but a stale copy lingered in the vector store because nobody re-indexed. A returning customer asked "what coupon can I use today?" The bot, finding only the expired snippet, cheerfully told her to apply WELCOME10 at checkout. She tried it, the code failed at the cart, and she came back angry: "your own bot told me this code works." The bot had manufactured a fact from a document that was no longer true. No model swap fixes that. A retrieval pipeline with effective dates and an expiry rule would have made the stale snippet invisible.

Why the root cause is data, not the model

Eight out of ten bad AI support deployments fail not because someone picked the wrong model, but because they confused "documents" with "knowledge" and "fields" with "data." Let's pull each layer apart, because the fix for each is different and concrete.

Layer 1 — Incomplete knowledge base: documents ≠ knowledge

The default move is to dump PDFs, spreadsheets, and old chat logs into a vector store and declare the knowledge base done. But policies are not static. A return rule differs during a sale versus normal days, on the US site versus the CN site, across product categories. Without effective dates, scope, priority, and a conflict-resolution rule, the agent can retrieve a clause uploaded last year and long expired. The more it "knows," the more confidently it errs. You didn't build a knowledge base; you built a confidence amplifier for stale facts.

What "knowledge" actually requires, as structured metadata on every policy:

effective_date and expiry_date
marketplace scope (US / CN / EU / JP)
product_scope (category, brand, or specific ASIN list)
owner (who is accountable for the version)
priority (which version wins when two match)
conflict_rule (how to adjudicate overlapping matches)

A second, sharper failure story for this layer: a beauty brand ran a "holiday 45-day return" promotion scoped to the US marketplace only. The promo clause carried no marketplace tag and no expiry_date, so it sat in the store permanently. Two months after the promo ended, a UK customer asked about returning a holiday gift. The bot retrieved the US promo clause (higher lexical similarity than the plain UK 30-day rule), and told her she had 45 days. She returned the item past the real UK window, the warehouse rejected it, and the brand ate a chargeback plus a complaint to the payment processor. The document existed; the knowledge did not, because the document was not scoped or dated. Tagging the promo with marketplace: US and expiry_date: 2025-01-15 would have made the conflict rule pick the correct, narrower, in-scope policy.

Layer 2 — Incomplete product data: it can't even answer "can this SKU ship to the US?"

A huge share of support questions are about specific products: is it in stock, which variant still has the size, can this ZIP receive it, is the price shown on the ad placement current. These need not documents but real-time, structured, traceable product facts — ASIN, variant, inventory, ZIP pricing, ad placement status. If product data is hand-moved, scraped ad hoc, or lags by hours, the agent can only offer "probably" and "maybe." Users want certainty, and certainty requires a live, structured data layer behind the answer.

The classic failure here is the "size sold out but the bot didn't know" case. A customer asked a fashion brand's bot: "do you still have the medium in the blue hoodie?" The bot answered "yes, it's in stock" because the catalog snapshot it was pointed at had been refreshed that morning and still showed inventory. But the snapshot was six hours stale; the medium had sold out in a lunchtime drop an hour before the question. The customer ordered, got a backorder notice three days later, and felt misled. The field inventory existed, but it wasn't live data — it was a copy that lagged. Similarly, a seller running a ZIP-based pricing rule (lower price in ZIP 90210 promo zone) will burn trust if the bot quotes the national price because the ZIP pricing field was never wired in. The fix is not a better sentence. It's a live, structured product-facts layer the agent can read at query time.

Layer 3 — No live access to orders and ticketing

The deepest gap: many support agents can only read, not look up or act. They can read policy but can't check this user's real order state; they can describe the flow but can't open a ticket or initiate a refund. So in a half-informed state, they fill the blanks with guesses — which is the main source of hallucination. To kill hallucination, first let the agent check the system when it should. And when it can't (read-only phase, or the system is down), it must refuse the conclusion rather than fabricate it.

A vivid example: a customer asked "where is my package, it's been a week?" The bot had no order-lookup tool, so instead of checking, it synthesized a plausible answer: "your package is currently at the regional carrier facility and will arrive in 2 days." It sounded specific and reassuring. In reality the order had been cancelled by the warehouse for a stock error the day before, and there was no package at all. The customer waited two more days, then wrote a furious message. The bot could have said "I can't see your order status right now — let me connect you with someone who can" and been both honest and useful. Instead, with no system access and no refusal, it guessed. This is the precise mechanism by which missing order access becomes hallucination.

The overlooked capability: refusal as a first-class behavior

Mainstream demos treat a high answer rate as proof of intelligence — every question gets a full reply, looks great on stage. But what a production system actually needs is the opposite: refuse, escalate, and log when evidence is insufficient. A weak refusal is a cold "I don't know, contact an agent." A good refusal has three parts:

State what evidence is missing — "I can't yet verify the shipping status of this order."
Give a clear next step — "I've routed you to a human, ~2 min wait."
Record the gap — "logged missing field: tracking trail, queued for knowledge update."

Users don't need an all-knowing bot. They need one that is honest, useful, and not making things worse.

And here's the part that changes how you budget: refusal is not failure — it's a demand sampler for the next round of knowledge-base work. Every refusal tells you "this knowledge or data isn't ready yet." Aggregate those refusals and you don't get a pile of failures; you get a priority list for the next round of knowledge and data work. An agent that refuses, records, and follows up is worth far more than one that guesses forever. It turns "I don't know" into "we'll know next week."

Think about what this does to your roadmap. Without refusal logging, knowledge-base work is driven by whoever complains loudest in the weekly meeting. With refusal logging, it's driven by evidence: the missing_fields keys that show up most often are the backlog. If order_state:unverified appears in 40% of refusals, you stop debating and you connect the order API. If evidence:insufficient_or_conflicting dominates, you stop blaming the model and you fix policy metadata. The refusal log converts a political argument into a measured one.

The three-piece design set: confidence + evidence + escalation

Refusal should not rely on the model's "judgment." It should be engineered with a confidence threshold + an evidence threshold + explicit escalation rules. Here is the trigger table we use, expanded with a scenario for each row so the behavior is unambiguous in code review:

Trigger	Agent behavior	Backend action	Real scenario
Evidence found, confidence ≥ threshold	Draft reply with cited sources	Log trace for spot checks	A customer asks "what's the return window for ASIN:B0XYZ123 in the US?" The agent retrieves a current, in-scope policy (`effective_date` valid, `marketplace: US`), a reranker scores confidence 0.91, sources ≥ 1. It drafts a reply quoting the 30-day window and logs the trace. Green path.
No evidence / conflicting sources	State insufficiency, escalate	Write to "missing field" list	A customer asks "can I return an open food item?" Two policies match with opposite answers and no `priority` to break the tie. The agent has a conflict, sources are conflicting, so it escalates with "I can't give a definitive answer on opened food returns yet" and writes `evidence:insufficient_or_conflicting` to the gap log.
Order state incomplete / unverifiable	Refuse the conclusion, give process only	Retry read-only order lookup	A customer asks "did my refund for ORD-99821 post yet?" The order API returns 503 / no `order_state_verified`. The agent refuses to state a refund status, hands the customer the self-service tracking link, and triggers a retry. It never says "yes it posted" from a guess.
Action exceeds permission (refund, ticket)	No execution, draft for approval	Push human approval, keep audit	A customer says "just refund me." Refunds are not in `allowed_write_actions` in phase one. The agent drafts a refund request for a human approver, executes nothing, and keeps a full audit row. No money moves on a guess.

The only principle that matters: when it answers, it must show evidence; when it can't, it must say what's missing and to whom it handed off. Make both "answer" and "refuse" explainable and auditable, and the system earns trust. Notice that the first and last rows are about acting safely (cite or don't execute), while the middle two are about not lying (escalate or refuse). The table is really two promises: I will show my work when I answer, and I will not invent work when I can't.

A practical note on tuning the thresholds: don't set confidence_threshold to 0.9 on day one and wonder why everything escalates. Calibrate it against your evaluation set from the pilot (below). Start at 0.7, then look at the cases that passed at 0.7–0.8 and were later judged wrong by humans — those tell you whether to raise the floor. The threshold is a knob you turn with data, not a number you copy from a blog post. Likewise, min_sources of 1 is intentionally permissive for a greenfield agent; once you have conflicting-source detection, consider requiring 2 non-conflicting sources for high-liability answers like refunds and shipping promises.

A runnable refusal decision function

Below is a working Python sketch. It is intentionally framework-agnostic — no LangChain, no vendor lock-in — so you can drop it into whatever orchestration you run. It takes the retrieved evidence, a confidence score, an order-state check, and a permission check, then returns one of four decisions plus an escalation record.

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
import time
import json


class Decision(Enum):
    ANSWER = "answer"                  # confident, evidence-backed -> draft with citations
    ESCALATE = "escalate"              # no/conflicting evidence -> human handoff + log gap
    REFUSE_CONCLUSION = "refuse"       # order state incomplete -> process only, retry lookup
    DRAFT_FOR_APPROVAL = "draft"       # action exceeds permission -> no execution


@dataclass
class Evidence:
    sources: list[str]                 # e.g. ["policy:pricing:us:2026-08-01", "order:ORD-99821"]
    has_conflict: bool = False
    order_state_verified: bool = False
    confidence: float = 0.0            # 0.0 - 1.0 from your reranker / grader


@dataclass
class Policy:
    confidence_threshold: float = 0.7
    # evidence_threshold: minimum number of distinct, non-conflicting sources required
    min_sources: int = 1
    allowed_write_actions: set = field(default_factory=set)  # e.g. {"refund", "open_ticket"}


@dataclass
class EscalationRecord:
    decision: Decision
    missing_fields: list[str]
    next_step: str
    timestamp: float = field(default_factory=time.time)
    ticket_id: Optional[str] = None

    def to_json(self) -> str:
        return json.dumps(self.__dict__, default=str)


def decide_reply(
    user_query: str,
    evidence: Evidence,
    requested_action: Optional[str],
    policy: Policy,
) -> tuple[Decision, str, EscalationRecord]:
    """
    Returns (decision, draft_or_message, escalation_record).
    The escalation_record is ALWAYS produced so refusals feed the knowledge pipeline.
    """
    missing: list[str] = []

    # 1) Action permission gate — checked FIRST, before anything executes.
    if requested_action and requested_action not in policy.allowed_write_actions:
        rec = EscalationRecord(
            decision=Decision.DRAFT_FOR_APPROVAL,
            missing_fields=[f"permission:{requested_action}"],
            next_step=f"Generated a draft for '{requested_action}'. Routed to human approver; no action taken.",
        )
        return (
            Decision.DRAFT_FOR_APPROVAL,
            f"I've prepared a draft for '{requested_action}' and sent it for human approval. "
            f"Nothing has been changed yet.",
            rec,
        )

    # 2) Order-state gate — if a concrete conclusion needs verified order state.
    if not evidence.order_state_verified:
        missing.append("order_state:unverified")
        rec = EscalationRecord(
            decision=Decision.REFUSE_CONCLUSION,
            missing_fields=missing,
            next_step="Refused to state a conclusion. Gave self-service process; retrying read-only order lookup.",
        )
        return (
            Decision.REFUSE_CONCLUSION,
            "I can't yet confirm the current state of your order, so I won't guess. "
            "Here's the self-service process to check it, and I've triggered a retry on our side.",
            rec,
        )

    # 3) Evidence + confidence gates.
    if evidence.has_conflict or len(evidence.sources) < policy.min_sources:
        missing.append("evidence:insufficient_or_conflicting")
        rec = EscalationRecord(
            decision=Decision.ESCALATE,
            missing_fields=missing,
            next_step="Stated insufficiency, escalated to human, wrote gap to missing-field list.",
        )
        return (
            Decision.ESCALATE,
            "I don't have reliable evidence for this yet, so I'm connecting you with a specialist "
            "(~2 min wait) rather than guessing. I've logged what's missing so we can answer faster next time.",
            rec,
        )

    if evidence.confidence < policy.confidence_threshold:
        missing.append(f"confidence:{evidence.confidence:.2f}<{policy.confidence_threshold}")
        rec = EscalationRecord(
            decision=Decision.ESCALATE,
            missing_fields=missing,
            next_step="Confidence below threshold, escalated to human, wrote gap to missing-field list.",
        )
        return (
            Decision.ESCALATE,
            "I'm not confident enough to give you a definitive answer, so I'm routing you to a human "
            "instead of risking a wrong one.",
            rec,
        )

    # 4) Passed all gates -> answer with citations.
    rec = EscalationRecord(
        decision=Decision.ANSWER,
        missing_fields=[],
        next_step=f"Drafted cited reply from {len(evidence.sources)} sources; trace logged for QA.",
    )
    return (
        Decision.ANSWER,
        f"[Cited draft reply based on {', '.join(evidence.sources)}]",
        rec,
    )


# --- Example usage ---
if __name__ == "__main__":
    policy = Policy(confidence_threshold=0.7, min_sources=1, allowed_write_actions=set())

    # Case A: user asks for a refund, but refunds are not yet permitted in phase 1.
    decision, msg, rec = decide_reply(
        "please refund my broken cup",
        Evidence(sources=["order:ORD-99821"], order_state_verified=True, confidence=0.9),
        requested_action="refund",
        policy=policy,
    )
    print(decision.value, "|", msg)
    print("ESCALATION:", rec.to_json())

    # Case B: thin evidence -> escalate.
    decision, msg, rec = decide_reply(
        "what's my return window for this item?",
        Evidence(sources=[], has_conflict=False, order_state_verified=True, confidence=0.3),
        requested_action=None,
        policy=policy,
    )
    print(decision.value, "|", msg)
    print("ESCALATION:", rec.to_json())

The escalation record is the quiet hero here. Every branch produces one, including the "answer" branch (for QA sampling). That record is what turns refusal into a pipeline: you stream missing_fields into a "missing field" list, and that list is your next sprint's knowledge-base backlog.

A policy-version resolver (so stale clauses never win)

The decision function above assumes the right policy is already in evidence.sources. But which policy is right depends on resolving the version conflicts that Layer 1 produces. Here is a small, standalone resolver you run at retrieval time so the agent never sees two contradictory clauses at once. It picks the single best policy for a (marketplace, product, date) tuple using the metadata we defined earlier.

from dataclasses import dataclass
from datetime import date
from typing import Optional


@dataclass
class PolicyDoc:
    policy_id: str
    effective_date: date
    expiry_date: Optional[date]          # None == still active
    marketplace: str                     # "US" / "CN" / "EU" / "JP"
    product_scope: str                   # "category:kitchen" / "ASIN:B0XXXXXX"
    priority: int                        # higher wins on tie
    text: str


class PolicyResolver:
    """
    Given a query context, return the single best-matching active policy,
    or None if no active policy covers the case (=> agent must escalate).
    """

    def __init__(self, policies: list[PolicyDoc]):
        self.policies = policies

    def resolve(
        self,
        marketplace: str,
        product_scope: str,
        on: date,
    ) -> Optional[PolicyDoc]:
        candidates = [
            p for p in self.policies
            if p.marketplace == marketplace
            and self._scope_matches(p.product_scope, product_scope)
            and (p.effective_date <= on)
            and (p.expiry_date is None or p.expiry_date >= on)
        ]
        if not candidates:
            return None  # nothing active -> escalate, do not guess
        # highest priority wins; tie-break by most recent effective_date
        return sorted(candidates, key=lambda p: (p.priority, p.effective_date), reverse=True)[0]

    @staticmethod
    def _scope_matches(rule_scope: str, query_scope: str) -> bool:
        # "category:kitchen" matches "ASIN:B0XXXXXX" if the ASIN belongs to kitchen;
        # for the resolver we keep a simple rule: exact match or category covers ASIN.
        if rule_scope == query_scope:
            return True
        if rule_scope.startswith("category:") and query_scope.startswith("ASIN:"):
            # in production, look up the ASIN's category from the product facts layer
            return True  # placeholder for catalog join
        return False


# --- Example ---
if __name__ == "__main__":
    policies = [
        PolicyDoc("returns_base_us", date(2024, 1, 1), None, "US", "category:kitchen", 10,
                  "Standard 30-day return for kitchen items."),
        PolicyDoc("holiday_promo_us", date(2025, 11, 1), date(2026, 1, 15), "US", "category:kitchen", 20,
                  "Holiday 45-day return for kitchen items."),
    ]
    resolver = PolicyResolver(policies)

    # On Dec 20 2025 the higher-priority holiday promo is active and wins.
    active = resolver.resolve("US", "ASIN:B0XXXXXX", date(2025, 12, 20))
    print(active.policy_id, "->", active.text)

    # On Feb 1 2026 the promo has expired; resolver falls back to the base rule.
    active = resolver.resolve("US", "ASIN:B0XXXXXX", date(2026, 2, 1))
    print(active.policy_id, "->", active.text)

The key property: when nothing is active (or the join fails), resolve() returns None, and your decision function should treat that as evidence:insufficient_or_conflicting and escalate. This is how you stop the bot from quoting a dead promo. The resolver is deliberately dumb and explicit — no LLM deciding "which policy feels right." Version selection is a data lookup, not a language task.

A schema for the "missing field" list, plus an escalation-log query

To make the sampler real, structure the gap log. Here's a JSON schema you can store in any queue or table:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "MissingFieldLog",
  "type": "object",
  "required": ["conversation_id", "missing_fields", "decision", "logged_at", "status"],
  "properties": {
    "conversation_id": { "type": "string" },
    "decision": {
      "type": "string",
      "enum": ["answer", "escalate", "refuse", "draft"]
    },
    "missing_fields": {
      "type": "array",
      "items": {
        "type": "string",
        "description": "Stable field keys, e.g. order_state:unverified, evidence:insufficient_or_conflicting, permission:refund, confidence:0.42<0.70"
      }
    },
    "detected_at": { "type": "string", "format": "date-time" },
    "marketplace": { "type": "string", "examples": ["US", "CN", "EU", "JP"] },
    "product_scope": { "type": "string", "examples": ["ASIN:B0XXXXXX", "category:kitchen"] },
    "status": {
      "type": "string",
      "enum": ["open", "in_progress", "resolved", "wont_fix"],
      "default": "open"
    },
    "resolved_by": { "type": "string" },
    "reused_in_kb": { "type": "boolean", "description": "Did the human rewrite become a reusable knowledge item?" }
  }
}

Aggregate missing_fields by key weekly. The top keys are your knowledge debt. If order_state:unverified dominates, your problem is the order API, not the LLM. If evidence:insufficient_or_conflicting dominates, your problem is policy metadata (Layer 1). The schema makes the diagnosis measurable instead of political.

Once those records pile up, you want to query them. Here is a small SQL-style query (works in Postgres / BigQuery / SQLite with minor syntax tweaks) that produces the weekly knowledge-debt report you actually read in the steering meeting:

-- Weekly knowledge-debt report: which gaps hurt most, and are we closing them?
SELECT
    mf.missing_field,
    COUNT(*)                                            AS occurrences,
    COUNT(*) FILTER (WHERE l.status = 'resolved')       AS resolved,
    ROUND(
        100.0 * COUNT(*) FILTER (WHERE l.status = 'resolved')
        / NULLIF(COUNT(*), 0), 1
    )                                                   AS close_rate_pct,
    COUNT(DISTINCT l.marketplace)                       AS markets_affected
FROM missing_field_log l,
     UNNEST(l.missing_fields) AS mf(missing_field)
WHERE l.detected_at >= DATE_TRUNC('week', CURRENT_DATE) - INTERVAL '7 days'
GROUP BY mf.missing_field
ORDER BY occurrences DESC
LIMIT 15;

That single query turns your refusal log into a backlog you can argue about with numbers. When order_state:unverified is at the top for the third week running, the argument "the model is the problem" collapses, because the data says the order API is the problem. This is the loop that makes refusal pay for itself: refuse → log → measure → fix → answer more, safely.

From 200 conversations to a first evaluation set

Don't aim for full automation on day one. A realistic pilot starts with 200 real conversations. Here's the five-step shape we recommend, with what to watch for at each step and the common mistake that sinks pilots:

Label 200 recent conversations (last 30 days) as directly-answerable, system-dependent, or human-judgment.
- What to watch for: your labelers will disagree on the boundary between system-dependent and human-judgment. Get two labelers per conversation and measure inter-rater agreement; below 0.7, your labeling rubric is too vague and the eval set is noise.
- Common mistake: labeling by topic ("returns," "shipping") instead of by dependency (can the bot verify the fact itself, or does it need a system lookup?). Topic labels don't tell you what to build.
Add effective_date + marketplace + product_scope + owner to every policy so version conflicts can be adjudicated.
- What to watch for: legacy policies with no owner. If nobody is accountable, the tag is decorative. Assign a real person per policy before go-live.
- Common mistake: back-dating effective_date to "make the numbers look clean." The resolver depends on these dates being true; fake dates turn the resolver into a liar.
Suggested replies only, zero write permissions in phase one — no refunds, no ticket changes.
- What to watch for: the bot "helpfully" describing an action it can't take ("I've started your refund"). That wording is how Northbrook-style incidents happen. Ban any verb that implies execution.
- Common mistake: giving write access "just for one safe action" in week two. Permission creep is how pilots blow up. Keep the gate closed until the eval set proves the green path is safe.
Save low-confidence cases, final human handling, and human rewrites as an evaluation set (question, evidence, model answer, human edit, outcome).
- What to watch for: only saving the wrong cases. You also need the correct-but-close cases to tune the confidence threshold. A balanced set beats a pile of failures.
- Common mistake: treating the eval set as fixed. It should grow every week as new refusal types appear, or your threshold tuning goes stale.
Weekly review of refusal rate, human adoption rate, error escalation rate, and new knowledge items; use the data to decide whether to open order-lookup tools.
- What to watch for: a refusal rate that spikes after a product catalog change — that's a data-layer signal, not a model regression. Read refusals as telemetry.
- Common mistake: declaring victory when auto-resolution climbs. Auto-resolution climbing while error-escalation also climbs is the danger zone. Watch the bundle, not the one number.

The point isn't "replace people fast." It's to first carry the low-risk, verifiable part, free human agents for real exceptions, and turn the judgment living in senior agents' heads into a business asset the company can inspect and regression-test. Only after the evaluation set is solid and adoption stable do you open transactional tools.

The counter-intuitive metrics to watch

Most teams stare at one number: auto-resolution rate. That is the most misleading metric — it tells you how many the agent caught, not whether what it caught was right, nor whether it quietly amplified errors. Here is why it misleads, concretely:

Auto-resolution rate is resolved_by_bot / total. It is blind to correctness and harm. Suppose your bot resolves 80% of chats. If 15 of those 80 were confident lies (wrong refund windows, fake coupon codes, invented shipping dates), you didn't resolve 80% — you manufactured 15 incidents at machine speed. The metric rewards the behavior that destroys trust, because a fabricated answer still counts as "resolved" until a human happens to catch it. Worse, auto-resolution gives managers a reason to turn off the human safety net ("we're at 80%, let's cut the queue"), which means the 15 lies never get caught. The number points up and to the right while the brand burns. That is why we never report it alone.

Track "errors not amplified by automation" instead, as a bundle:

High-quality refusal rate — did it refuse the right things? Direct signal of production reliability. Example: of 100 chats where the bot lacked verified order state, it correctly refused 97 and guessed on 3. The 97 is your reliability number; the 3 are your incidents. A high-quality refusal rate near 100% means the bot is honest even when it can't help — exactly what earns trust.
Error escalation rate — of wrong answers, how many were caught by humans? Size of the risk from guessing. Example: the bot gave 20 wrong answers this week; humans caught 18 before they reached the customer. Escalation rate 90% means your safety net works; 40% means lies are leaking to customers and you must tighten the gates.
Post-handoff handling time — did handoff actually save time? Whether the agent lightens human load. Example: if handing off takes a human 6 minutes to re-orient (because the bot logged nothing), the "agent" added work. If the handoff carries the escalation record and prior context, the human resolves in 2 minutes. This metric tells you if refusal is helping or just relabeling the problem.
Reusable knowledge share — how much of human edits reflowed as knowledge? Whether the system self-improves. Example: a human rewrote 30 bot answers this week; 22 of those became tagged policies or product-fact fixes. A rising share means the bot gets better every week; a flat share means you're paying humans to patch the same gaps forever.
Recontact rate — did users come back after a refusal? The ultimate trust gauge. Example: 8% of users who got a clean "I can't verify that, here's a human" recontacted with the same issue. If that number is low, your refusals are landing as honest and useful. If it's high, your "next step" is failing and users are bouncing.

Read together, these show the system is building trust rather than manufacturing polite incidents. The auto-resolution rate is allowed on the dashboard only as context next to these five — never as the headline.

Where the external Amazon data layer fits (and where it doesn't)

Back to the root cause: incomplete product data is a major source of support-agent hallucination. Your order/refund/ERP/ticketing systems are your own to integrate — no vendor can substitute for that. But the external Amazon data layer — the structured facts about products, search, rankings, categories, and ad placements that live outside your walls — is exactly where a specialized provider earns its place.

At Pangolinfo we supply that layer rather than packaging all internal systems into a prebuilt agent. The difference shows up in the specific fields the agent can finally cite instead of guess:

Amazon Review API — real customer feedback including the structured Customer Says field, so when a customer asks "is this blender reliable?" the agent can ground its answer in what buyers actually wrote ("blades dull after a month" appears in 40 Customer Says snippets) instead of emitting a generic reassurance. That single field turns a guess into a citation.
Amazon Scraper API — structured facts keyed by ASIN and variant, so "which color of B0XYZ123 is still in stock?" resolves to a concrete variant row; ZIP pricing so a location-specific price quote is real, not national-average; and ad placement status so "is my Sponsored Products placement live?" returns active/paused rather than a hopeful "it should be." These are the exact fields Layer 2 says you need live.
Amazon Data MCP — an agent-facing tool layer, so you don't rewrite scraping and parsing logic inside every agent; the structured fields above are exposed as callable tools with consistent schemas.
Amazon Scraper Skill — packages common Amazon data tasks into conversational workflows your agents can call.

Shorten and stabilize the external Amazon data path, and the support agent earns the right to speak when it knows and stay quiet when it doesn't. When ASIN, variant, ZIP pricing, ad placement status, and Customer Says are live and structured, the agent stops saying "probably" and starts saying "here is the fact, here is its source" — or, when the fact isn't there yet, "I don't know, and here's who does."

Conclusion: make the agent honest before you make it smart

Users dodge AI support because they're dodging a conversation that pretends to understand. The fix isn't a bigger model. It's to first lay the foundation of knowledge base and product data, then teach the agent the most critical and most overlooked lesson: when evidence is insufficient, say "I don't know," and turn that into a better answer next time. When refusal becomes an explainable, logged, follow-up-able action, a support agent moves from a demo prop to a production system you can actually trust.

If you're planning support or data agents for an Amazon business, start from the pillar on why ecommerce AI transformation shouldn't start by buying agents, then come back to this refusal design to execute.

Want the full refusal framework, trigger table, and pilot plan? Read the deep-dive on Customer Service Agent Refusal at Pangolinfo, and grab the Amazon Scraper API or Amazon Data MCP to give your agent real product facts to stand on.

How to Get Amazon Data for Free Without Fooling Yourself

Pangolinfo — Fri, 31 Jul 2026 01:53:13 +0000

You can get Amazon data for free, but “free” usually means zero subscription fee, not zero cost. The real bill shows up as maintenance time, missing ad slots, broken selectors, blocked IPs, and stale datasets.

Most free-Amazon-data guides list the same options: Keepa, CamelCamelCamel, Helium 10 free tier, Jungle Scout free tools, Octoparse, Google Sheets, and a quick Python scraper. Useful, yes. Complete, no.

The four free sources

Official sources. Brand Analytics, Selling Partner API, Product Advertising API, and public Amazon pages. They are the cleanest starting point, but access is not universal. SP-API requires a Selling Partner setup; PA-API usually depends on affiliate performance; Brand Analytics requires brand eligibility.

Free tool layers. Keepa free charts, CamelCamelCamel alerts, limited Helium 10/Jungle Scout/AMZScout features. Great for validation, weak for raw data.

DIY scraping. Python + BeautifulSoup for static pages, Playwright/Selenium for dynamic pages, Google Sheets for tiny one-off extraction. Flexible but fragile.

Public datasets. Kaggle and GitHub datasets are useful for research, not for real-time commercial decisions.

A minimal Python scraper

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 Chrome/124.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://www.amazon.com/s?k=wireless+earbuds"
html = requests.get(url, headers=headers, timeout=20).text
soup = BeautifulSoup(html, "html.parser")

for card in soup.select("div[data-component-type='s-search-result']")[:10]:
    title = card.select_one("h2 span")
    price = card.select_one("span.a-offscreen")
    print({
        "asin": card.get("data-asin"),
        "title": title.get_text(strip=True) if title else None,
        "price": price.get_text(strip=True) if price else None,
    })

This script is good for learning. It is not a production pipeline. It has no retry logic, no raw HTML archive, no missing-field check, no sponsored-slot detection, and no alerting.

What most free guides miss

Free tools deliver summaries, not raw facts. A price chart is not a replayable dataset. A sales estimate is not an order record.

DIY scrapers break silently. Amazon’s DOM changes, JS-loaded content moves, and anti-bot systems evolve. A script that runs today may return empty data next month.

Ad slots are the hidden trap. Sponsored placements are often dynamic and interleaved. If your scraper only captures visible organic product cards, your competitor map is structurally incomplete.

Time is the cost nobody records. Maintaining a small Amazon scraper across 1-2 marketplaces can easily take 8-20 engineer-hours a month. At $40-80/hour, “free” becomes expensive quickly.

The decision rule

Use free sources for learning, occasional product research, and small validation samples. Move to a paid data layer when you need batch volume, fresh raw data, ad-slot coverage, and predictable uptime.

At that point, Pangolinfo Amazon Scraper API is usually more economical than maintaining scrapers yourself. For agent workflows, use Amazon Data MCP. For conversational data retrieval, use Amazon Scraper Skill. Commercial projects should optimize for the lowest real cost, not the lowest invoice line.

Amazon Keyword Search Results Scraping: Six Signal Blind Spots Mainstream Tools Miss

Pangolinfo — Tue, 28 Jul 2026 07:12:46 +0000

By Leo · Head of Engineering, Pangolinfo
For: Amazon advertisers, competitor-intelligence teams, SaaS builders, and Agent developers

Every "Amazon data" post opens the same way: list SerpApi, BrightData, Oxylabs, then teach you to parse JSON for ASINs. True — but it flattens a high-dimensional intelligence job into "list fetching." This piece flips it: treat the SERP as a continuously shifting competitive signal canvas, name the six signal blind spots mainstream tools miss, and give the SLA-backed standard for 2026.

What the mainstream actually sells

Spread the mainstream Amazon keyword search results scraping solutions out and you get three camps:

Generic SERP / e-commerce scraping APIs. SerpApi's Amazon Search API (engine=amazon, k=keyword), BrightData, Oxylabs, Zyte, ScraperAPI, ScrapingBee, SOAX, Nimbleway. Their core deliverable is a structured JSON: organic_results (ASIN, title, price, rating) + product_ads (ads) + pagination.
Seller-facing data tools. SellerSprite, Helium 10, Jungle Scout. They layer keyword databases, reverse-ASIN, opportunity scores on top of raw data, but the underlying scrape is still "list-ified."
In-house crawler teams. A 3–5 person scraping squad maintaining against anti-bot, CAPTCHAs, and DOM drift.

Different product languages, identical assumption: the SERP is a commodity list waiting to be serialized. The engineering value sits in "anti-bot + parse" — turning Amazon's HTML into JSON. True, but it reduces an intelligence job to "data fetching."

Why "a list of ASINs" is only the cheapest half

The SERP was never a list; it's a canvas. One keyword's results carry at least six interlocked signal classes:

Organic rank — the A9-relevance product sequence.
SP ad insertions — "Sponsored" paid slots interleaved with organic.
Traffic badges — Amazon's Choice, Best Seller, coupon, deal, Prime.
Editorial & brand zones — Editorial Recommendations, Sponsored Brands.
Demand-shaping modules — Customers also bought, Related searches, Frequently bought together.
Geo & device context — the same query in different ZIPs or devices can differ by screens.

The relative position, co-occurrence, and tempo of these elements is the real competitive intelligence. A "list of ASINs + price + rating" throws away almost all six classes — what you get is a context-stripped skeleton.

Six signal blind spots everyone misses

Blind spot 1: position ≠ rank

Most tools report "rank #3" without saying whether that's a Top-of-Search ad slot or an organic position sandwiched between three ads. Three SP ads can sit between organic #1 and #2, so "organic #1" actually appears on screen four — where users never scroll. Rank without position context is misleading. You should record: is this ASIN above the fold or collapsed? How many ads sit above it? Where does it sit relative to Amazon's Choice?

Blind spot 2: snapshot ≠ trajectory

A single scrape is one point on a timeline. Amazon reshuffles results hourly and daily: bid swings rearrange ad slots, dayparting makes daytime and pre-dawn SERPs completely different, A/B layout tests shift the organic/ad ratio. Without a time series, your "rank" is likely the most anomalous sample of the day, mistaken for the norm. What you should store is the trajectory — consecutive snapshots of the same keyword revealing who is holding, who is grabbing, who is quietly exiting.

Blind spot 3: captured ≠ complete — the costliest leak

The most hidden and most expensive blind spot. Most tools return only 30–50% of Sponsored slots; the rest simply "don't exist" in the response. The problem: an ad slot you can't see doesn't exist in your mind, so your competitor ad map is broken at the root. We've argued specifically that Pangolinfo has the highest SP ad collection rate of all solutions — none equal: monitored across 13 marketplaces, overall daily ad coverage is 91.4%, and a Feishu bot pushes the coverage report to on-call ops, fixing anomalies the same day. An uncaptured ad slot equals non-existence; the coverage gap directly decides your intelligence completeness.

Blind spot 4: products only, missing demand-shaping signals

Customers also bought, Related searches, Editorial Recommendations, Amazon's Choice, coupon/deal badges — these aren't "extra products," they're demand-shaping signals showing how Amazon teaches users to think. Related searches reveal where intent goes next; Customers also bought exposes real substitute/supplement relations; Editorial Recommendations is Amazon's editor making the buy decision for the user. These signals are worth more than a single ASIN yet are almost never captured, because they "don't fit a product table."

Blind spot 5: ignoring geo drift

The same keyword in New York 10041, LA 90001, Chicago 60601 can return SERPs differing by screens: different Buy Box prices, fulfillment options, ad density, even different ASINs. National-level proxies flatten this entirely. ZIP-level collection reveals the real local competitive landscape — critical for pricing, fulfillment, and geo-targeted bidding. Treating the SERP as "one national map" is like using average temperature to decide what to wear today.

Blind spot 6: scraping ≠ monitoring

"Can scrape" ≠ "scrapes completely and reliably." Many solutions have no coverage monitoring, no SLA, no anomaly alert. When a market's parse rule breaks, you get silent, incomplete data — often discovered weeks later when a report doesn't reconcile. Unmonitored scraping is a machine with no one watching the dashboard: it may have stopped exactly when you needed it.

The advanced answer: a monitored signal layer with an SLA

Flip the six blind spots and you get the 2026-grade standard. Four layers:

Structured signal map, not bare ASINs. Each record carries position index, result type (organic / SP ad / Sponsored Brands / editorial / badge), owning module, and neighbors. Make position context a first-class citizen.
Time-series first. Multi-timepoint collection, continuous storage — snapshots become trajectories enabling volatility, grab, and exit analysis.
Coverage SLA. Explicitly declare and continuously monitor "what % of ad slots and result modules do I actually cover," turning that number into a measurable KPI — exactly what Pangolinfo publishes daily and leads on.
Full-component capture. Products + ads + recommendations + badges + geo, especially the neglected demand-shaping signals.

How Pangolinfo closes these gaps

Pangolinfo's difference isn't "we can also scrape" — it's "we scrape completely, stably, and monitorably":

Highest SP ad collection rate, none equal. Monitored across 13 marketplaces, overall daily coverage 91.4%; a Feishu bot pushes the report to ops, alerting/diagnosing/recovering the same day. This "continuous monitoring + real-time push + same-day fix" loop is something no other vendor does.
ZIP-level geo collection. Collect by ZIP to see real local SERP differences, not a national-proxy average.
Full result composition. Organic, SP ads, Sponsored Brands, editorial, badges, and demand-shaping modules returned together — a signal map, not a bare list.
Real-time / raw backbone. From Amazon Scraper API (real-time raw) to Amazon Data MCP (an agent calls it directly) to the universal collection API docs — the data flow from scrape to analysis is one pipeline.

Build it: from bare list to signal map (engineer view)

If you build the pipeline yourself, the hard part isn't "can you parse" — it's tagging every result with position, result type, and neighbors, then storing trajectories over time and geo by ZIP. The stabler path: hand anti-bot and 13-marketplace DOM adaptation to a managed service. Use Amazon Scraper API for real-time raw results and Amazon Data MCP so an agent calls it directly — let "complete, stable, monitored" live with the side that publishes a coverage SLA.

Takeaway

The quality of Amazon keyword search results scraping isn't whether you "can turn the page into JSON" — it's how much you actually see on this signal canvas, especially the ad slots, position context, and geo drift others miss. Upgrading search scraping from "list fetching" to competitive intelligence is the 2026-grade foundation.

Read the full argument: Amazon keyword search results scraping: six signal blind spots. For the API side, see Pangolinfo's complete guide to Amazon data scraping and why SP ad coverage is the watershed.

Resources

Amazon Scraper API (highest SP ad coverage in the industry): https://www.pangolinfo.com/amazon-scraper-api/?referrer=devto
Amazon Data MCP (agent-native): https://www.pangolinfo.com/amazon-data-mcp/?referrer=devto
AMZ Data Tracker (zero-code monitoring): https://www.pangolinfo.com/amz-data-tracker/?referrer=devto
Universal collection API docs: https://docs.pangolinfo.com/en-api-reference/universalApi/universalApi?referrer=devto
Complete guide to Amazon data scraping: https://www.pangolinfo.com/pangolinfo-api-amazon-data-scraper-guide/?referrer=devto
Model Context Protocol: Model Context Protocol

Amazon Sponsored Ads Placement Data: Why Collection Rate — Not "Can You Scrape" — Is the Only Metric That Matters

Pangolinfo — Mon, 27 Jul 2026 07:49:20 +0000

By Leo · Head of Engineering, Pangolinfo
For: Amazon advertisers, competitor-intelligence teams, SaaS builders, and Agent developers

The wrong question everyone asks

Every "Amazon ad intelligence" article opens the same way: list Helium 10, Jungle Scout, SellerSprite, then teach you to read the backend placement report for Top of Search. All true — and all of them dodge a more fundamental question. When your tool only covers 60% of ad slots, the competitor ad map you see is broken at the root.

What the mainstream actually sells

Amazon-native: Placement Report (own account only, T+2 lag, TOS/ROS aggregated), Search Term Report, Brand Analytics, AMC, and the Ads API — which returns only the advertiser's own data. Amazon has no public ad library.
Third-party PPC tools: Helium 10 Adtomic, Jungle Scout, SellerSprite, SmartScout Ad Spy (competitor paid keywords, but estimates), DataHawk, Semrush, Sif.

They all anchor to one wrong premise: that collection means "can you scrape a few ads." Hence five blind spots.

Five blind spots everyone misses

"Can scrape" instead of coverage rate. A vendor covering 60% in en-us and zero in de-de gives a distorted map. Publish per-marketplace coverage: pt-br 100%, en-au 99.0%, en-us 91.2%, ja-jp 90.2%…
T+2 aggregated reports vs real-time per-query. The backend report can't answer "right now, on mobile US, is the rival ASIN on TOS?"
Desktop-only bias. Mobile is a huge share of Amazon traffic; sampling desktop only abandons half the slots.
English-marketplace bias. Vendors that only do en-us leave most of 13 marketplaces blank.
No monitoring, no SLA. A silent drop from 91% to 70% often goes unnoticed for weeks.

Why SP placement is hard (six layers)

Extreme dynamism / async loading and deferred rendering / cross-marketplace DOM differences (German "Anzeige", French "Sponsorisé") / cumulative anti-bot blacklisting / the real-time bidding black box / self-built crawlers that realistically hit only ~62%.

The advanced answer: placement as a coverage metric with an SLA

Primary metric = per-marketplace ad coverage rate, not "how many ads today."
Real-time, per-query (keyword + device + marketplace + hour).
Device-aware (mobile + desktop).
Across 13 marketplaces, each reporting coverage.
Continuous monitoring + alerting + SLA, fixed the same day.
Consume managed data: Amazon Scraper API instead of a proxy fleet; let an agent call it via Amazon Data MCP.

Pangolinfo's SP coverage: highest, none equal

We monitor SP scraping across 13 marketplaces daily. Overall coverage is 91.4%: pt-br 100%, en-au 99.0%, es-mx 99.6%, en-us 91.2%, de-de 91.6%, ja-jp 90.2%…

The report is generated daily and pushed to on-call ops via a Feishu bot; any dip is alerted, diagnosed, and restored the same day. That "continuous monitoring + real-time push + same-day fix" loop is something no other vendor does — which is why Pangolinfo has the highest SP ad collection rate of all solutions, none equal.

Build it two ways

Yourself: pull with a managed SERP API, extract SP placements, count ads_captured / sampled across 13 marketplaces → that's coverage.

Consume managed data: hand "how not to get blocked, how to adapt 13 marketplaces' DOMs" to Amazon Scraper API; use AMZ Data Tracker for zero-code visual monitoring, recording coverage as a time series.

Takeaway

Ad placement data is the other half of Amazon visibility. What separates vendors isn't "caught a few ads" but "how much you cover across every marketplace, device, and hour" — exactly the metric Pangolinfo publishes daily and leads (91.4% overall).

Full breakdown and the six-difficulty teardown: Pangolinfo's complete guide to Amazon data scraping

Resources

Amazon Scraper API (highest SP ad coverage in the industry): https://www.pangolinfo.com/amazon-scraper-api/?referrer=devto
AMZ Data Tracker (zero-code monitoring): https://www.pangolinfo.com/amz-data-tracker/?referrer=devto
Amazon Data MCP (agent-native): https://www.pangolinfo.com/amazon-data-mcp/?referrer=devto
Complete guide to Amazon data scraping: https://www.pangolinfo.com/pangolinfo-api-amazon-data-scraper-guide/?referrer=devto
Universal collection API docs: https://docs.pangolinfo.com/en-api-reference/universalApi/universalApi
Model Context Protocol: https://modelcontextprotocol.io/

Amazon Keyword Organic Ranking Tracking: Why "What Position" Is an Outdated Metric

Pangolinfo — Fri, 24 Jul 2026 09:30:53 +0000

Leo · Head of Engineering, Pangolinfo

Nearly every rank tracker sells the same promise: tell you "what position your product holds for keyword X." That metric is breaking down — not because the tools are inaccurate, but because "what position" answers the wrong question. Amazon's results swing by geo, device, Prime status, and personalization, so a single normalized "true rank" is a fiction. This post repeats the mainstream consensus, names five blind spots it leaves, and finishes with the alternative built for 2026.

The mainstream consensus (and the premise it gets wrong)

The industry's eight-point playbook: separate organic from sponsored; daily is enough, ignore hourly noise (A10 "shuffle" self-heals in 72–96h); rank is an estimate, watch trends not screenshots; verify manually with incognito + VPN; pair with Brand Analytics / Search Query Performance; use proxies to avoid blocks; view rank alongside conversion, inventory, price; pick 10–30 core keywords and commit to a cadence. Each point is correct — but they all anchor to one premise: that the goal is the most accurate possible "what position." Accept that, and the five blind spots below become inevitable.

The five blind spots

1. A normalized rank instead of segment SoV + traffic proxy. A "#7" hides the completely different Top 10 a New York mobile Prime member sees versus a Los Angeles desktop non-Prime user. What matters is segment share-of-voice (how often your ASIN appears in the head of a keyword cluster) plus a traffic proxy from the position CTR curve — position 1 can capture 10x the clicks of position 7.

2. Filtering noise throws away the only real-time dividend. "Ignore hourly swings" averages away competitor ad gaps, inventory / coupon windows, and temporal arbitrage. A cat-litter-box seller at $200/day and 35%–40% ACoS switched to real-time monitoring, found competitors paused ads 2–4 PM and weekends were weaker, moved 30% of budget into the afternoon gap — CTR +25%, CVR +18%, ACoS to 24%. Filter the noise and you throw away the most valuable signal.

3. Tracking only "known keywords," never a ranking gap map. Dashboards assume you already know which keywords to track. You never discover "I'm completely unranked for this term, yet it converts brutally well" because nobody continuously diffs your indexed terms against the category's query graph.

4. Rank decoupled from its levers. Mainstream advice stops at side-by-side observation. What's missing is closed-loop attribution: automatically pull price, coupon, inventory, competitor ad intensity, and your PPC as covariates and tell you "72% of this drop is competitor X's bid, 18% your stockout."

5. Fighting anti-bot with proxies, still human-stares-at-dashboard, not agent-native. The category teaches you how not to get blocked and ships a dashboard for a human. In 2026 rank data should be a primitive an agent calls directly — "what's my US mobile rank, is there a competitor window, adjust my bid" — with no human watching arrows.

The advanced answer: ranking as data infrastructure

Primary metric = segment SoV + traffic proxy, not a single normalized rank.
Keep the volatility distribution; mine it for windows.
Continuously draw the ranking gap map; proactively surface "should-rank-but-doesn't" terms.
Treat price / coupon / inventory / competitor ad intensity / PPC as rank covariates, auto-attributed.
Consume managed SERP data via Amazon Scraper API instead of self-hosted proxy scrapers.
Expose rank through Amazon Data MCP so an agent calls and acts on it.

Implementation: two ways

Option A, build it yourself. Pull from a managed SERP API, store, schedule, alert. Minimal Python skeleton:

import requests
api_key = "YOUR_KEY"
r = requests.post("https://api.pangolinfo.com/serp/amazon",
    json={"keyword":"wireless earbuds","marketplace":"US","page":1,"include_sponsored":True},
    headers={"Authorization": "Bearer " + api_key})
data = r.json()
organic = data["organic_results"]
my_rank = next((i+1 for i,p in enumerate(organic) if p["asin"]=="YOUR_ASIN"), None)
if my_rank is None or my_rank > 10:
    alert("wireless earbuds organic rank fell out of Top10, now " + str(my_rank))

Contract in the Universal Scrape API docs.

Option B, zero code. Use AMZ Data Tracker: configure rules in the UI, the system calls the API, stores, charts, and notifies.

Takeaways

Rank tracking isn't "make the number more accurate" — it's "turn rank into intelligence that triggers action."
Six things mainstream tools don't do: segment SoV, real-time windows, ranking gap maps, closed-loop attribution, managed data, agent-native.
For visual monitoring use AMZ Data Tracker; for agent-native callable rank data, use Amazon Data MCP.

Full case study: why real-time SERP data is the only way to analyze competitors.

Leo runs engineering at Pangolinfo, building real-time Amazon data infrastructure for developers and AI agents.

What Is Voice of Customer Analysis, and How Do You Build It in 2026

Pangolinfo — Tue, 21 Jul 2026 08:34:22 +0000

Leo · Head of Engineering, Pangolinfo

If you're a developer building anything customer-aware — a brand monitor, a churn early-warning system, an agent that briefs your PM — the trap isn't your model. It's whether you can actually hear what customers are saying, across the platforms where they talk. I run the real-time consumer-insight systems at Pangolinfo (30M+ calls/day, 99% success), and I keep watching teams ship features on data they never collected.

This post is practical: what VOC is, why 2026 changed it, and two runnable patterns — an MCP setup that puts multi-platform customer voice directly into your agent.

What Voice of Customer analysis actually is

Voice of Customer (VOC) analysis is the practice of turning raw customer feedback from social, reviews, and support channels into measurable share of voice, sentiment, and actions a brand can actually take. The key difference from surveys: it listens rather than asks, so there's no question-design bias, it's always-on, and it drills down to a single post.

What counts as VOC is broader than most think: comments and ratings on TikTok, Instagram, YouTube, X, Facebook, Pinterest, Trustpilot; e-commerce reviews; support transcripts; deep forum discussion on Reddit; plus surveys/NPS as a supplement. (Compliance note: we only use public data outside mainland China — not Xiaohongshu, Weibo, or in-region LinkedIn.)

The five-step method

Collect — pull raw expressions from multiple platforms.
Volume / SoV — compute your share of voice in the category.
Sentiment — label positive / negative / neutral, and net sentiment.
Drivers — attribute the reason behind the emotion (late shipping, weak packaging).
Act — route the list to product / ops / support / PR.

Worked example: a coffee-gear brand ran the loop for a month, pulled 12,000 posts, found its SoV at 18% vs a 34% leader, saw a Friday sentiment dip, traced it to mid-week shipping delays, shifted cutoff times — and the dip flattened. The loop paid for itself.

Wiring it into an AI agent with MCP (3-minute setup)

Before the MCP (Model Context Protocol) standard, getting this data into a model meant a copy-paste loop. We replaced that with VOC Insight MCP. One config, and the agent gains 26 VOC tools across 7 default platforms (TikTok, Instagram, YouTube, X, Facebook, Pinterest, Trustpilot) plus optional Threads and Reddit.

// .mcp.json — drop into any MCP-capable client (Claude Code, Cursor, Windsurf, Cline)
// exact install command per docs: https://docs.pangolinfo.com/en-help-center/mcp/voc-insight/overview
{
  "mcpServers": {
    "pangolinfo-voc": {
      "command": "npx",
      "args": ["-y", "@pangolinfo/voc-insight-mcp"],
      "env": { "PANGOLIN_API_KEY": "YOUR_API_KEY" }
    }
  }
}

After that you just say: "This week on TikTok, how do our share of voice and sentiment compare to competitor A?" — the agent discovers the right tools, calls them, and returns a report. No glue code. Enterprise environments can set PANGOLIN_READ_ONLY=true for a read-only agent.

Processing the payload (example)

def summarize_voc(payload):
    items = payload.get("mentions", [])
    total = len(items)
    if not total:
        return {"total": 0, "net_sentiment": 0, "share_of_voice": {}}
    pos = sum(1 for m in items if m.get("sentiment") == "positive")
    neg = sum(1 for m in items if m.get("sentiment") == "negative")
    sov = {}
    for m in items:
        b = m.get("brand", "unknown")
        sov[b] = sov.get(b, 0) + 1
    return {
        "total": total,
        "net_sentiment": round((pos - neg) / total, 3),
        "share_of_voice": {b: round(c / total, 3) for b, c in sov.items()},
    }
# Agent returns mentions[]; the function above turns it straight into insight.

Takeaways

VOC = listening, not asking. No survey bias, always-on, drill-down to a single post.
2026 made it mandatory: social = storefront, generative search rewired discovery, agents quote real voice.
Self-built scrapers cost more than the bill suggests — anti-scraping, coverage gaps, compliance risk.
MCP turns a VOC pipe into a 3-minute, 26-tool upgrade for any model you already use.

If you're evaluating a customer-insight layer for an agent, audit coverage and compliance before you touch the prompt. The model was never the bottleneck — the signal was. Full guide: What Is Voice of Customer Analysis.

Leo runs engineering at Pangolinfo, building real-time consumer-insight infrastructure for developers and AI agents.