Every team that runs Amazon collection in-house knows the loop by heart.
Write the parser. Ship it. Amazon changes the page. The parser throws. You get paged. You fix the parser. Two weeks later, repeat.
Run that loop enough times and somebody says out loud what everyone is thinking: can we stop maintaining scrapers?
Yes. You can. But here is the part that does not show up in any vendor pitch deck, and it took us a while to internalise: outsourcing collection removes the parsers and the proxy pool. It does not remove data quality governance. Plenty of teams migrate, look up six months later, and realise their engineers did not get free time — they got a different thing to maintain, one that is harder to notice.
The difference is loud versus quiet.
A broken parser is loud. It throws, it returns nothing, the logs fill up, you know within minutes. Data quality degradation is quiet. HTTP 200, valid JSON, schema validation passes, twelve fields are null, and your dashboard looks healthy while drifting into a lie.
This post is the architecture we converged on for that quiet half of the problem: six layers, what each one owns, the retry bugs that triple your bill with no alert, the quality gate you have to build yourself, and a 90-minute checklist to get a first version running. All the code runs as written.
Scope note: this is not the "should I use an API or scrape at all" post. If you have not settled that question, read Amazon API vs Web Scraping: Stop Choosing, Build Both Routes first — it answers the prior question, and it will save you reading the wrong half of this one.
Step zero: compute your baseline before you compare anything
This is the step everybody skips, and it is the reason most cost conversations about data collection degenerate into vibes.
Add up the engineering hours your team spent on collection maintenance over the last eight weeks and divide by eight. That weekly number is the denominator for every decision that follows, because it puts a monthly API invoice on the same scale as the headcount cost of keeping the scrapers alive.
# baseline.py — how much of your week does collection already eat?
import csv
from collections import defaultdict
from datetime import date, timedelta
WINDOW_WEEKS = 8
def weekly_maintenance_hours(csv_path, since):
"""Expects rows: date, hours, category (bugfix/false_alarm/ops_q/deps)."""
by_week = defaultdict(float)
for row in csv.DictReader(open(csv_path)):
d = date.fromisoformat(row["date"])
if d < since:
continue
by_week[d.isocalendar()[1]] += float(row["hours"])
weeks = list(by_week.values()) or [0.0]
return sum(weeks) / WINDOW_WEEKS, weeks
def break_even(monthly_fee_usd, loaded_hourly_usd, weeks_per_month=4.33):
"""How many hours/week does the fee have to buy back to be worth it?"""
return round(monthly_fee_usd / (weeks_per_month * loaded_hourly_usd), 2)
Most teams are surprised the first time they run it, and the reason is instructive. The total is not just bug-fixing time. It includes chasing false alarms, answering questions from operations, updating dependencies, re-learning the scraper's quirks each time you touch it, and the context-switch cost of every one of those interruptions. Those add up to far more than the actual fixes.
Once you have the number, break_even() gives you the honest question. If a managed service costs $3,000 a month and your loaded engineering rate is $90/hour, it needs to buy back about 7.7 hours a week. If your measured baseline is 14 hours a week, the maths is not close. If it is 3, it is not close either — in the other direction.
Skip this step and you will argue about price without ever establishing what you are paying for.
Four routes out of parsing, and what each one costs
No vendor names and no second-hand prices here. Prices move; cost structure does not.
| Route | Dominant cost | Who owns maintenance | Typical failure mode | Best fit |
|---|---|---|---|---|
| Self-built scraper stack | Engineer debugging time, far above servers | All of it, in-house | Layout change breaks parsing | Very long-tail pages, dedicated team |
| Managed platform / self-hosted framework | Compute time plus parser upkeep | Runtime outsourced, logic in-house | Harder anti-bot means a bigger bill | Existing expertise, custom logic |
| Official SP-API | Authorisation ops, per-operation rate planning | In-house, but low failure rate | Throttling and expired grants | Your own operational data |
| Vertical Amazon data API | Per-call billing plus self-built quality governance | Collection outsourced, judgement in-house | Fields going null, data going stale | Market and competitor data |
Two columns deserve your attention: maintenance ownership and typical failure mode. Wherever maintenance lives is where your engineers will spend their staring time. Whatever the failure mode is, that is what your alerts need to be built around. Most teams compare only the cost column and then discover after go-live that they have zero monitoring for the failure mode they just signed up for.
Self-built: the cost lands in the wrong bucket
Headless browser cluster, proxy pool, parsers. Maximum control, no per-record billing, marginal cost flattens as you scale. But you think you are spending money on servers and you are spending it on engineering time — in a live scraper stack, servers are almost always the smaller line item. And anti-bot evasion is a permanent arms race: the trick that works today is not promised to work in six months.
Managed platforms and self-hosted frameworks
Scrapy, Playwright clusters, hosted actor marketplaces. Fast to first result, mature ecosystem, much of the parsing logic already written. But maintenance did not transfer. The platform took over browsers and runtime; you still own parsing logic, selectors, retry policy, and anti-bot strategy. Worse, most of these platforms bill by compute duration, which means the harder the anti-bot, the higher your bill. Cost correlating with difficulty is a miserable property for a budget model.
Official SP-API
Stable, compliant, explicit schema, no anti-bot fight. But coverage is bounded by its authorisation model. It gives you orders, inventory, fulfilment, and your own advertising data for accounts a seller has authorised by name in the grant. It does not give you competitor prices, market-wide search rankings, or other sellers' reviews — not "not yet supported," but not offered by design. SP-API is one component of a pipeline, never the whole thing.
Vertical Amazon data API
You hand parsing and anti-bot to a specialist and receive structured JSON. Here maintenance transfers: page redesign becomes the vendor's problem, the proxy pool becomes the vendor's problem, your engineers stop watching selectors. What does not transfer is quality governance — and field coverage varies from vendor to vendor in ways that never appear on a feature grid.
The realistic end state is a mix. Your own operational data through SP-API, market and competitor data through a data API, maybe a thin self-built layer for the longest-tail pages. There is a simple test for drawing the boundary: if the question is "did I get authorisation," use the official channel; if it is "what is happening in the marketplace right now," use a data service. One is an authorisation problem, the other a collection problem, and their failure modes, cost structures, and compliance obligations are different enough that forcing both onto one technical path raises complexity on both sides.
What moved: three shifts, all of them quieter
This is the premise for the architecture below. If you do not accept it, everything after this will look over-engineered.
Parse failure → fields going null with no error
When a scraper fails it is loud. When an API fails without raising an error, you get HTTP 200, valid JSON, schema validation passes, and twelve of the fields are null. The failure signal moved from the transport layer to the business-semantics layer — and nobody writes business-semantics validation for you.
{
"asin": "B0C7V9K2XQ",
"title": "…",
"price": null,
"availability": { "status": null },
"rating": { "value": 4.3, "count": null }
}
Every check you have in place passes. Row count correct. Status 200. Dashboard green. Your price column is empty.
Getting blocked → data going stale with no alert
Anti-bot blocking is an unambiguous failure. The API-side equivalent is freshness decay: the vendor adjusts a cache policy, collection frequency drops in one marketplace, an upstream job backs up — and the value you receive describes the world as it was three days ago. Format valid. Content no longer true. The only defence is a freshness SLO plus monitoring the staleness distribution.
Proxy cost → retry-amplified cost
In the self-built era your costs were proxies and machines, and most of that spend was fixed. In the API era your cost is unit price × attempts, and attempts get amplified by failure rate, backoff policy, and idempotency design together. A pipeline with a 70% usable record rate and 1.8 average attempts is paying 2.57× list price. Unmonitored, that multiplier drifts — nobody changed any code — and one day the invoice is 40% higher.
The story that made this concrete for us
A six-person data team replaced their in-house scraper with a third-party API and reclaimed around two engineering days a week. Everyone was happy. In month three, operations started saying the competitor prices in the report looked odd.
Investigation showed the fill rate for the price field in two marketplaces had fallen from 96% to 61%. Over five weeks. With no alert. Because every request returned 200 and the success-rate metric never dropped below 99%.
Fixing it took two weeks — a week and a half of which was archaeology. With no raw response archive, nobody could establish which day it started, so they had to bisect the problem by re-running history and eyeballing output.
The lesson is not that they should not have migrated. It is that quality monitoring has to be built in the same project as the migration. Ship raw archiving and field-level fill-rate monitoring alongside the switch, and that decay surfaces in week one as a parameter change instead of two weeks of forensics.
The six-layer architecture
This is not theory. It is the shape teams running stable pipelines converge on, and the rule is one responsibility per layer so a failure can be localised.
Business goal
│ field contract · freshness SLO · cost budget
▼
┌──────────────────────────────────────────┐
│ L1 Collect fetch raw, no judgement │
├──────────────────────────────────────────┤
│ L2 Raw archive verbatim, only replay │
├──────────────────────────────────────────┤
│ L3 Queue+retry idempotency · backoff·DLQ │
├──────────────────────────────────────────┤
│ L4 Normalise contract check · upsert │
├──────────────────────────────────────────┤
│ L5 Quality gate four numbers decide │
├──────────────────────────────────────────┤
│ L6 Observe panels · alerts · weekly │
└──────────────────────────────────────────┘
│
▼
Downstream: reports / models / agents / alerts
Layer zero: the inputs are contracts, not requirements
Before any code, write three things down, each decidable by a machine:
- A field contract — the fields the business depends on, graded P0/P1/P2.
- A freshness SLO — the P95 staleness ceiling for P0 fields, say six hours.
- A cost budget — the ceiling per thousand usable records.
CONTRACT = {
"P0": ["asin", "title", "price.amount", "availability.status"],
"P1": ["brand", "rating.value", "review_count", "bsr.rank_main"],
}
FRESHNESS_FIELD, SLO_P95_HOURS = "collected_at", 6.0
These three are the basis for every automated judgement downstream. Without them the gate has to guess its thresholds, and a guessed threshold gets switched off after the first false alarm.
The most common mistake in writing a field contract is wanting everything. A contract listing 80 fields all marked P0 is equivalent to no contract: the usable record rate collapses toward zero, the gate screams on every run, somebody turns it off. P0 means a record missing this field cannot go into the downstream report — no more than 15 fields in most contracts. P1 means nice to have. P2 means useful in a few reports, may be null. This grading is not bureaucracy; it determines retry policy, degradation tiers, and gate thresholds.
L1: the collector fetches, it does not judge
Draw this boundary hard. The collector sends the request, takes the response, hands it onward. No field validation, no business judgement, no cleaning. The reason is attributability: if business logic lives inside the collector, you cannot tell whether bad data was fetched wrong or judged wrong.
import json, time, uuid
from datetime import datetime, timezone
def collect(client, item, timeout):
"""L1 contract: return raw body + call metadata. Never inspect the payload."""
request_id = str(uuid.uuid4())
started = time.perf_counter()
status, body = client.get(item.endpoint, params=item.params, timeout=timeout)
meta = {
"request_id": request_id,
"object_type": item.object_type,
"marketplace": item.marketplace,
"status": status,
"duration_ms": int((time.perf_counter() - started) * 1000),
"attempt": item.attempt,
"collected_at": datetime.now(timezone.utc).isoformat(),
}
return {"raw": body, "meta": meta}
The collector should emit two things: the raw response body, and metadata about the call — duration, status code, timestamp, attempt number. That last one is what makes cost attribution possible later.
L2: raw archive is the only basis for reproducibility
Write responses verbatim, partitioned by object, marketplace, and date, retained 30 to 90 days.
import gzip, os
ARCHIVE_ROOT = "/var/data/raw"
def archive(res):
meta = res["meta"]
day = meta["collected_at"][:10]
path = os.path.join(ARCHIVE_ROOT, meta["object_type"], meta["marketplace"], day)
os.makedirs(path, exist_ok=True)
name = f"{meta['request_id']}-{meta['status']}.json.gz"
with gzip.open(os.path.join(path, name), "wt", encoding="utf-8") as fh:
json.dump({"meta": meta, "body": res["raw"]}, fh)
return os.path.join(path, name)
The value shows up only when something breaks: if you suspect a record is wrong, can you go back to that exact response body from three months ago? If not, every root-cause analysis you run is guesswork. Compressed JSON is cheap.
There is a practical rule for retention: it has to be longer than your detection cycle. If your team takes three weeks on average to notice an anomaly, 30 days lets you look back once and 90 days lets you compare across two incidents. Compressed raw JSON almost always costs less than recomputing aggregate metrics over the same window, so err on the generous side.
L3: queue and retry — route by error class
This layer owns scheduling, concurrency control, throttling, and retries. The key design is routing by error class. Mixing the three classes below into one retry loop is the single biggest cause of runaway retry cost.
class Retryable(Exception): pass # 429 / 5xx / timeout
class NonRetryable(Exception): pass # 401 / 422 / bad params
class SoftFailure(Exception): pass # 200 but every P0 field is empty
def classify(status, body, p0_fields):
if status in (408, 425, 429) or 500 <= status < 600:
return Retryable(f"http_{status}")
if status in (400, 401, 403, 404, 422):
return NonRetryable(f"http_{status}")
if status == 200:
if not body:
return SoftFailure("empty_body")
if not any(field_filled(body, f) for f in p0_fields):
return SoftFailure("p0_empty")
return None
return NonRetryable(f"unexpected_{status}")
import random, time
BASE_SECONDS, CAP_SECONDS, MAX_ATTEMPTS = 0.5, 30.0, 5
def backoff(attempt):
# CAP stops runaway tail latency; random() breaks up retry storms
return min(CAP_SECONDS, BASE_SECONDS * 2 ** (attempt - 1)) * (0.5 + random.random())
def run_item(client, item, p0_fields, dlq, softq):
for attempt in range(1, MAX_ATTEMPTS + 1):
item.attempt = attempt
try:
res = collect(client, item, timeout=item.timeout)
err = classify(res["meta"]["status"], res["raw"], p0_fields)
if err:
raise err
archive(res) # archive before parsing, always
return res
except Retryable:
if attempt == MAX_ATTEMPTS:
dlq.push(item, reason="attempts_exhausted")
return None
time.sleep(backoff(attempt))
except NonRetryable as e:
dlq.push(item, reason=str(e)) # retrying 401 ten times buys nothing
return None
except SoftFailure as e:
softq.push(item, reason=str(e), raw=res)
return None
Two details people get wrong here.
Backoff needs jitter. Exponential backoff without it produces retry storms: a batch fails together, waits together, retries together, and saturates the rate limit on the same millisecond. Jitter spreads the retries out. You also need the ceiling, or tail latency runs away.
Timeouts matter more than retry counts. Teams tune retry counts down to the last attempt and set timeouts to whatever number came to hand, but timeouts move cost and throughput more. Too short, and requests that would have succeeded get marked failed and retried — inflating cost and duplicating work. Too long, and failing requests hold concurrency slots, the queue backs up, and total throughput drops.
Derive the value from your own latency distribution, per operation. Do not inherit a default.
def percentiles(values, ps=(50, 90, 95, 99)):
s = sorted(values)
out = {}
for p in ps:
idx = max(0, min(len(s) - 1, int(round(p / 100 * len(s))) - 1))
out[f"p{p}"] = s[idx]
return out
# from your own call logs, per operation
lat = percentiles([m["duration_ms"] for m in call_log if m["op"] == "product"])
CONNECT_TIMEOUT_S = 3.0 # TCP + TLS: a few seconds, always
READ_TIMEOUT_S = max(10.0, lat["p99"] / 1000 * 1.5) # follows the response distribution
Take P99 as the baseline and add headroom. And separate connect timeout from read timeout: the former is a small constant, the latter follows the response distribution. In data services some complex objects are slow, so a blanket short timeout raises your failure rate on every one of them.
Idempotency keys must include the contract version
Many teams use object ID + marketplace + date, which works right up until the contract changes. Add a new P0 field and historical records no longer satisfy the new contract — but without a version component they look already processed and get skipped.
import hashlib
CONTRACT_VERSION = "2026-08-01"
def idempotency_key(asin, marketplace, data_date, contract_version=CONTRACT_VERSION):
return hashlib.sha1(
f"{asin}|{marketplace}|{data_date}|{contract_version}".encode()
).hexdigest()
Include the version and a contract upgrade triggers a full re-run without a human in the loop — which is the correct behaviour, and the one you want at 3am without having to think about it.
A dead-letter queue is not a bin
Dead-letter contents have to be consumed, or the queue is just slow deletion.
from collections import defaultdict
def dlq_report(items):
counts = defaultdict(int)
for it in items:
counts[it["reason"]] += 1
total = sum(counts.values()) or 1
return {k: (v, round(v / total, 3)) for k, v in
sorted(counts.items(), key=lambda kv: -kv[1])}
Minimum bar: aggregate by failure reason, produce a daily list, alert on shifts in the distribution. The distribution reveals problems earlier than the failure rate does on most teams — a sudden rise in attempts_exhausted points to degrading upstream success while your headline success metric is still above threshold.
L4: normalise and store — strict types, idempotent upsert
FIELD_TYPES = {
"asin": str,
"title": str,
"price.amount": (int, float),
"availability.status": str,
"rating.value": (int, float),
"review_count": int,
"bsr.rank_main": int,
}
def dig(rec, path):
cur = rec
for part in path.split("."):
if not isinstance(cur, dict) or part not in cur:
return None
cur = cur[part]
return cur
def assert_contract(record):
"""Strict: presence AND type. A str→dict rename must not slip through."""
missing, wrong_type = [], []
for field, expected in FIELD_TYPES.items():
value = dig(record, field)
if value is None:
missing.append(field)
elif isinstance(value, bool) or not isinstance(value, expected):
# bool check matters: isinstance(True, int) is True in Python
wrong_type.append((field, type(value).__name__))
return missing, wrong_type
That isinstance(value, bool) guard is not paranoia. isinstance(True, int) returns True in Python, so a field that flips from review_count: 1284 to review_count: true passes a naive type check and lands in your warehouse as 1.
A field changing from string to object is the most common breaking change, and implicit coercion in a dynamically typed language lets it pass unnoticed. Route validation failures to the soft-fail queue; never skip them without a log entry.
Then write with idempotency:
INSERT INTO product_daily
(asin, marketplace, data_date, contract_version, payload, p0_missing)
VALUES (%(asin)s, %(marketplace)s, %(data_date)s, %(contract_version)s,
%(payload)s, %(p0_missing)s)
ON CONFLICT (asin, marketplace, data_date, contract_version)
DO UPDATE SET payload = EXCLUDED.payload,
p0_missing = EXCLUDED.p0_missing,
updated_at = now();
Processing the same object for the same day twice must produce the same result. This layer also records which P0 fields were missing, for the next layer to consume.
The quality gate: four numbers plus a freshness SLO
This is the layer you must build yourself and cannot buy. Its job is to convert "is this data usable" from a judgement call into quantities a machine can compute.
Coverage and fill rate are different numbers
Presence is not value. Coverage asks whether the path exists at the schema level; fill rate asks whether a value arrived. Both low means the field is not supported. High coverage with low fill rate is the dangerous combination — the field is on the contract, the vendor populates it on a small share of calls, and schema validation waves it through.
Related: the field-level comparison of two Amazon data APIs runs the same ASIN against both vendors and counts fields. One returned twelve, the other fifty-eight, and both pricing pages carried the same checkmark.
def field_present(rec, path):
cur = rec
for part in path.split("."):
if not isinstance(cur, dict) or part not in cur:
return False
cur = cur[part]
return True
def field_filled(rec, path):
if not field_present(rec, path):
return False
cur = rec
for part in path.split("."):
cur = cur[part]
if cur is None:
return False
if isinstance(cur, str) and cur.strip() == "":
return False
if isinstance(cur, list) and len(cur) == 0:
return False
return True
def coverage(records, fields):
total = len(records) * len(fields)
return sum(1 for r in records for f in fields if field_present(r, f)) / total if total else 0.0
def fill_rate(records, fields):
total = len(records) * len(fields)
return sum(1 for r in records for f in fields if field_filled(r, f)) / total if total else 0.0
Usable record rate judges the whole record
The two metrics above are field-level, but downstream consumes records. Define usable record rate as the share of records where every P0 field is filled. It is the only metric that maps onto "can this row go into the report," and it should be the gate's primary threshold.
from datetime import datetime, timezone
def usable_rate(records, p0_fields):
if not records:
return 0.0
ok = sum(1 for r in records if all(field_filled(r, f) for f in p0_fields))
return ok / len(records)
def staleness_p95(records, ts_field, now=None):
"""Return P95 age in hours. None means no usable timestamps at all."""
now = now or datetime.now(timezone.utc)
deltas = []
for r in records:
ts = r.get(ts_field)
if not ts:
continue
t = datetime.fromisoformat(str(ts).replace("Z", "+00:00"))
deltas.append((now - t).total_seconds() / 3600)
if not deltas:
return None
deltas.sort()
return deltas[max(0, int(round(0.95 * len(deltas))) - 1)]
Related: I stopped trusting "real-time" claims after one bad quarter covers the freshness side in much more depth, including the 100-call test I use to establish whether a "real-time" claim means anything.
Write the freshness SLO as a distribution, not a boolean
"Is the data real time" cannot be alerted on, because real time has no quantified definition. The executable version is: P95 staleness of P0 fields is at most X hours.
Use P95 rather than the mean, because the mean hides the tail. A distribution with a 20-minute P50 and a 31-hour P95 produces an excellent average while a meaningful slice of your report is a day old. That is the failure the 96%→61% team walked into — they were watching the mean of a success metric, not the tail of a freshness one.
Five quantities, one function
P0 = ["asin", "title", "price.amount", "availability.status"]
P1 = ["brand", "rating.value", "review_count", "bsr.rank_main"]
def gate(records):
m = {
"n": len(records),
"coverage_p0": coverage(records, P0),
"fill_p0": fill_rate(records, P0),
"fill_p1": fill_rate(records, P1),
"usable": usable_rate(records, P0),
"staleness_p95": staleness_p95(records, FRESHNESS_FIELD),
}
m["pass"] = (
m["fill_p0"] >= 0.95
and m["usable"] >= 0.90
and m["staleness_p95"] is not None
and m["staleness_p95"] <= SLO_P95_HOURS
)
return m
Note the explicit is not None. When the timestamp field is absent, staleness_p95 returns None. Skip that check and None <= 6.0 raises a TypeError in Python 3 — but in an implementation with light validation it evaluates to False and fails the batch with no error. That second outcome is worse: it makes data with no timestamp at all look like it failed a freshness check it never took, which sends you debugging the wrong layer.
Record level marks, batch level decides
One design choice changes whether the gate survives contact with production: judge per record or per batch.
Per record suits pre-write cleaning — a record failing P0 is not written. Clean, but it loses data, and it cannot distinguish "this whole batch is broken" from "this one row happened to be empty."
Per batch suits release decisions — a batch whose usable rate falls below threshold is held back in full. Catches systemic decay, but blocks the good records inside it.
Do both. Record-level marking first, then batch-level decision, so that when a batch is held you can name which class of record dragged it down in one step. Collapsing the two into one switch either loses data or misses decay.
Stratify the sample or the aggregate lies to you
This is the subtlest trap in the whole gate. Suppose your sample is 70% books and 30% apparel. Books fill the price field 96% of the time. Apparel, with its variants and missing variant prices, fills it 58%.
books n=280 price fill 96%
apparel n=120 price fill 58%
---------------------------------
total n=400 price fill 84% <- the number your dashboard shows
84% looks acceptable. It conceals the fact that apparel is unusable.
def stratify(records, keyfn):
buckets = defaultdict(list)
for r in records:
buckets[keyfn(r)].append(r)
return buckets
def gate_report(records, keyfn=lambda r: r.get("category", "unknown")):
per_stratum = {k: gate(v) for k, v in stratify(records, keyfn).items()}
worst = min(per_stratum.items(), key=lambda kv: kv[1]["usable"])
return {"overall": gate(records), "strata": per_stratum,
"worst": (worst[0], worst[1]["usable"])}
So the fixed regression sample must be stratified by object characteristics: category, whether the product has variants, marketplace, price band — with enough volume in each stratum to produce a standalone conclusion. Which dimensions matter depends on your business. Stratify by category if value concentrates in a few categories, by marketplace if it concentrates in a few marketplaces. What matters is that every stratum can speak for itself instead of disappearing into a total.
Set thresholds that will not get switched off
The usual way a quality gate dies: thresholds start too aggressive, the first false alarm fires, somebody disables it and never turns it back on.
def propose_thresholds(baseline, tolerance=0.05, block_gap=0.10):
"""Week 1: record, do not block. Week 2: alert below baseline, block further down."""
return {
"alert_fill_p0": round(baseline["fill_p0"] - tolerance, 3),
"block_fill_p0": round(baseline["fill_p0"] - block_gap, 3),
"alert_usable": round(baseline["usable"] - tolerance, 3),
"block_usable": round(baseline["usable"] - block_gap, 3),
"staleness_alert_h": round(baseline["staleness_p95"] * 1.25, 2),
}
Observe for two weeks before setting anything. Week one records without blocking, giving you baseline distributions for all four numbers. Week two sets an alert line at a sensible tolerance below baseline and a blocking line further down. Thresholds should come from your own distribution, not from a generic best practice.
Shipping alerts two weeks later is far cheaper than shipping an alert that gets turned off — and once a gate has been disabled, rebuilding trust in it is much harder than building it the first time.
L6: five charts, and a budget guard
Trend the four numbers, alert on them, attribute cost to business objects, and run a fixed-sample regression every week. Weekly regression is the item most often skipped and the most valuable: a one-off acceptance test proves it works now, a weekly regression proves it still works.
Only five charts deserve permanent dashboard space:
- Usable record rate over time — one line per object type.
- P0 fill rate over time — one line per marketplace.
- P95 staleness over time.
- Retry amplification factor over time.
- Cost per thousand usable records over time.
Five is enough; more and nobody looks. The first three answer is the data still usable, the last two answer what is it costing.
Making success rate the hero chart is a mistake. It sits above 99% for the life of the dashboard, conveys nothing, and occupies the most prominent space on the page. That is how a five-week decay from 96% to 61% happens without anybody noticing.
The effective cost formula
What you are optimising is not unit price, it is cost per thousand usable records:
effective_cost = unit_price × attempts_per_usable_record / usable_rate
That formula ties three variables together — price, stability, completeness. A cheap vendor with a poor usable rate can cost more in the end; an expensive one with full fields and few retries can be the better deal.
Record attempts per record at the collector and aggregate the mean by object type and marketplace:
def amplification(records):
attempts = [r["meta"]["attempt"] for r in records]
usable = [r for r in records if all(field_filled(r["payload"], f) for f in P0)]
return {
"avg_attempts": round(sum(attempts) / len(attempts), 3),
"usable_rate": round(len(usable) / len(records), 3),
"effective_multiplier": round(sum(attempts) / max(1, len(usable)), 3),
}
Most teams are surprised the first time they look. Plenty assume the factor is close to 1 and measure it between 1.5 and 2.5. It often rises not because failure rates climbed but because backoff policy is triggering retries earlier than it should.
Cost attribution, minimal version
Attribution does not need a platform. At the collector, write an estimated cost into each record's metadata — call count times unit price, tagged with the object and marketplace.
UNIT_PRICE_USD = 0.001
def attributed(records):
spend, usable = defaultdict(float), defaultdict(int)
for r in records:
k = (r["meta"]["object_type"], r["meta"]["marketplace"],
r["meta"]["collected_at"][:10])
spend[k] += r["meta"]["attempt"] * UNIT_PRICE_USD
if all(field_filled(r["payload"], f) for f in P0):
usable[k] += 1
return {
k: {"spend": round(v, 4), "usable": usable[k],
"cost_per_1k": round(v / usable[k] * 1000, 2) if usable[k] else None}
for k, v in spend.items()
}
Aggregate along three dimensions — object type, marketplace, date — and you can answer all but a few cost questions: which marketplace is most expensive, which object type is burning money, which week the cost started climbing. It is common to find that the highest-volume object is not the most expensive per unit, and that the team has been optimising the wrong thing.
Budget guard: write degradation into code
DAILY_BUDGET_USD = 250.0
def sampling_policy(spent_today):
if spent_today < 0.7 * DAILY_BUDGET_USD:
return {"P0": 1.0, "P1": 1.0, "P2": 1.0, "tier": "normal"}
if spent_today < DAILY_BUDGET_USD:
return {"P0": 1.0, "P1": 0.5, "P2": 0.0, "tier": "p0_fidelity"}
return {"P0": 0.25, "P1": 0.0, "P2": 0.0, "tier": "p0_only"}
When cumulative daily cost crosses the threshold, reduce sampling frequency for non-P0 objects and preserve full collection for P0. Write and test this policy in advance. Deciding under pressure leaves only two options — stop everything, or keep burning — and both are bad.
Two things that cut across every layer
Model concurrency and rate limits per operation. Planning around one global QPS is a common error. Both official channels and most serious data services use per-operation token buckets: different operations carry different quotas and different refill rates. What you need is not "how many requests per second do we send" but a table of per-operation quotas plus per-operation backoff implemented client-side. Without that table your concurrency plan is guesswork and your load test is meaningless — throughput measured on one operation tells you nothing about another.
Lineage has to trace back to the specific call. Every landed record should carry source metadata: request ID, vendor response timestamp, attempt count, contract version, collection batch ID. It is a handful of fields, but without it you cannot answer "when was this price collected." Lineage is also what makes root-cause analysis tractable — when a field starts decaying, you slice by batch, marketplace, and time instead of re-running everything and hoping.
Schema drift: three defences inside the pipeline
Upstream field changes happen without an error — renames, type changes, new enum values, merged fields. The damage never appears at the transport layer; it appears as a number in your report that is wrong without raising an error.
One: strict presence and type assertions at the normalisation layer (assert_contract above), checking type as well as existence, with failures routed to the soft-fail queue rather than skipped.
Two: weekly structural diffs. Save a response snapshot for a fixed sample every week and diff its structure:
def shape(obj, prefix=""):
"""path -> type name. Ignores values entirely, so it catches structure only."""
out = {}
if isinstance(obj, dict):
for k, v in obj.items():
out.update(shape(v, f"{prefix}.{k}" if prefix else k))
elif isinstance(obj, list):
for v in obj[:1]: # sample first element only
out.update(shape(v, prefix + "[]"))
else:
out[prefix] = type(obj).__name__
return out
def diff_shapes(prev, cur):
return {
"added": {k: v for k, v in cur.items() if k not in prev},
"removed": {k: v for k, v in prev.items() if k not in cur},
"type_changed": {k: (prev[k], cur[k]) for k in cur.keys() & prev.keys()
if prev[k] != cur[k]},
}
Push the diff to your alert channel. The snapshot doubles as the basis for root-cause analysis — without it, when you need to know when a field stopped populating, memory is all you have.
Three: contract versioning and regression re-runs. The field contract carries a version number. When the contract changes, the idempotency key changes with it, which re-runs the affected objects. Without versioning, a contract upgrade leaves historical records looking already processed and skipping them, so old and new data sit mixed together and nobody notices.
Degradation: what happens when upstream goes down
Almost nobody writes this section into an architecture doc, and it is the difference between a chaotic incident and an orderly one. Every upstream fails at some point — vendor outage, your own quota exhausted, a network partition, tightened throttling. The question is whether your pipeline responds by delivering less data and stating the shortfall, or by pretending to deliver all of it.
With no predefined policy, teams improvise between two options: halt and wait, or let retries burn through the budget. Both are bad. Define tiers in advance:
| Tier | Behaviour | Trigger |
|---|---|---|
| 1 — P0 fidelity | P0 frequency unchanged, P1/P2 reduced | Cost over threshold or rising upstream error rate |
| 2 — P0 only | P0 at reduced frequency, P1/P2 suspended | Upstream unavailable beyond a configured duration |
| 3 — archive read-only | Collection stopped, downstream pointed at the last gated snapshot, labelled beyond SLO | Sustained unavailability |
Trigger conditions, scope, and recovery conditions for all three belong in configuration, not in somebody's memory.
def stamp(record, tier, note):
record["_collection"] = {"tier": tier, "note": note, "degraded": tier != "normal"}
return record
Degraded state must be visible downstream. The dangerous part of degradation is not less data — it is downstream not knowing the data is reduced. Records released by the gate should carry a collection status marker stating which tier they were collected under and which object range was covered. The reporting layer should surface a warning when it sees a degradation marker; the model layer should adjust confidence. Hiding degraded state inside the pipeline means downstream consumes partial data as complete, which is the quiet failure from the case study above.
Where we fit, stated with a tight boundary
We build Pangolinfo as a collection layer, so let me be precise about the boundary: we own L1 and the stability that comes with it. L5, the quality gate, is yours to build regardless of whose collection service you use. If a vendor tells you otherwise, they are selling you something that cannot work.
Handing over collection means page redesigns, anti-bot evasion, proxy pool operations, and browser fingerprinting stop being your engineering problems. Our published metrics: a 3-second median latency, 99% success rate, and more than 30 million calls a day. On sponsored ad placements — the hardest object to collect for most teams — we hold 91.4% aggregate coverage across 13 marketplaces; that figure is our own measurement, published as a product metric, not an independent benchmark. Treat all four numbers the same way you would treat anyone else's: as claims to verify with the scripts above.
Three things we do not do. First, no account-domain data: orders, inventory, and your own advertising sit inside the official SP-API's authorisation scope, and we neither provide them nor provide a way around them. Second, we do not collect buyer personally identifiable information. Third, we do not collect anything behind a login. These are design boundaries, not roadmap items.
Three situations where you should not use us. One: you need a very small volume at low frequency, where a few lines of your own script beat us. Two: all of your requirements sit inside your own accounts, where SP-API is both more compliant and cheaper. Three: you need login-gated data or buyer PII — which we do not do, and no vendor should.
One special case: if the consumer is an AI agent rather than a fixed report, building a full pipeline may be over-engineering. Agents fetch on demand, in single calls, with unpredictable field scope, which does not match the pipeline's assumptions of batch processing and fixed schemas. In that shape the better move is exposing data capability as tools the agent calls when it needs them, which is why Amazon Data MCP exists — 19 tools over remote HTTP, zero installation. Agent scenarios still need quality judgement; it just moves from a gate inside the pipeline to field-validation guidance on the agent side.
Ruling out the wrong fits is what makes the remaining scope worth the maintenance you save.
The 90-minute checklist
Do it in this order. Every step has a concrete artefact.
- Write the field contract (20 min). Fields the business depends on, graded P0/P1/P2. Keep P0 under 15 — more means you have not decided what matters.
- Set the freshness SLO (10 min). Give P0 fields a P95 staleness ceiling and write it down. Without that number, freshness cannot be alerted on.
- Build collector plus raw archive (20 min). Fetch and store only, no judgement. Far less code than most teams expect.
- Run a baseline (20 min). Take 200 to 500 real ASINs and compute the four numbers with the gate scripts. This is your baseline, not an acceptance result.
- Wire the gate and alerts (20 min). Record without blocking first, observe two weeks, then set alert and block lines from the baseline distribution.
Five steps and you have a pipeline that complains when data decays. Everything else — cost attribution, schema contract tests, weekly regression — bolts onto this skeleton.
One warning from experience: the step most likely to be abandoned halfway is step four, the baseline. The temptation is to skip it and turn alerts on before the baseline exists, on the grounds that you can tighten it later. But skipping the baseline means guessing thresholds, and guessed thresholds get disabled after the first false alarm. Better to ship alerts two weeks late than to ship an alert that gets switched off.
Wrapping up
The one-sentence version: collection can be outsourced; judgement cannot. Once the collection layer is gone, what you have to build yourself is a mechanism that decides whether the data is still usable — and in this architecture that mechanism is L5.
The point of the gate is not perfect data. It is that degradation becomes an alarm instead of a fact everybody accepts and stops questioning.
Everything here is runnable as written. Take the scripts, point them at whatever you are already using, and get your own four numbers before you decide anything. For product and search objects, Amazon Scraper API is the endpoint; for reviews, Amazon Review API; to skip pipeline construction end to end and let an agent fetch in one step, Amazon Data MCP. Grab a key from the console and run a baseline, or read the Amazon Data MCP technical documentation.
And run the same scripts against us. Completeness numbers a vendor reports about itself — ours included — deserve to be verified rather than believed.
If you have built something like this, I would be curious which of the three shifts bit you hardest. In my experience it is the stale one, because it is the only failure that a well-formatted response cannot hide and a status code cannot reveal.
Top comments (0)