Your specification coverage chart can make the shortlist look better simply because you collected it differently.
You're appending known ASINs to a listing pull, expecting a shared setting to give every record the same depth.
With scrapeProductDetails set to false, ASIN and product-URL inputs still fetch detail pages.
The shortlist takes a different route
Our listings Actor accepts additive inputs, so combining discovery with exact lookups is valid. The analytical catch is treating the resulting records as equally deep observations.
Example: this input combines keyword discovery with a known-product shortlist while disabling listing enrichment.
import json
import os
import duckdb
import pandas as pd
from apify_client import ApifyClient
run_input = {
"keywords": ["wireless earbuds"],
"asins": ["B08N5WRWNW", "B09X7MPX8L"],
"marketplaceDomain": "amazon.com",
"maxItems": 0,
"maxItemsPerCategory": 10,
"maxPages": 1,
"scrapeProductDetails": False,
}
The input routes give the groups different opportunities to acquire information:
keyword discovery -> listing cards -> card fields
exact shortlist -> product pages -> detail fields where rendered
includeSpecifications controls specification extraction when a product page is scraped. Omitting it leaves its default of true in effect.
If your shortlist contains commercially important products, business importance can become correlated with richer collection. A comparison by product group then carries a collection advantage inside it.
A completeness score can be impeccably calculated and still answer the wrong question. Spreadsheets are accommodating that way.
Keep the stage that a merged row can lose
The small fix is to collect discovery and exact lookup separately, then append their observations with a stage label.
A combined run deduplicates by ASIN. Deduplication alone gives you no basis for deciding which input supplied a merged field, and a returned sourceMode shouldn't be treated as a complete history of every contributing input.
The split below is unambiguous because discovery uses keywords and the shortlist uses asins. For other collection designs, classify the actual route: categoryUrls can itself contain product URLs.
This cell preserves supplied settings separately from their effective routing consequences, then collects each stage.
discovery_input = {
key: value for key, value in run_input.items()
if key not in {"asins", "productUrls"}
}
exact_input = {
key: value for key, value in run_input.items()
if key not in {"keywords", "categoryUrls", "categoryNodeIds", "sellerIds"}
}
collection_inputs = {
"listing_discovery": discovery_input,
"exact_lookup": exact_input,
}
manifest = pd.DataFrame([
{
"collection_stage": stage,
"scrape_setting": (
json.dumps(payload["scrapeProductDetails"])
if "scrapeProductDetails" in payload else "<omitted>"
),
"specs_setting": (
json.dumps(payload["includeSpecifications"])
if "includeSpecifications" in payload else "<omitted>"
),
"detail_route_expected": (
stage == "exact_lookup"
or payload.get("scrapeProductDetails", True)
),
"specifications_enabled": payload.get("includeSpecifications", True),
}
for stage, payload in collection_inputs.items()
])
client = ApifyClient(os.environ["APIFY_TOKEN"])
observations = []
for stage, payload in collection_inputs.items():
run = client.actor(
"dataspryva/amazon-search-category-and-seller-listings"
).call(run_input=payload)
if run["status"] != "SUCCEEDED":
raise RuntimeError(f"{stage}: {run['status']}")
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
observations.append((stage, item))
The manifest records intent. It doesn't certify that every expected page yielded every field.
Splitting also changes collection boundaries: each run has its own caps and start charges. An ASIN appearing in both stages remains a separate observation in each stage here, deliberately.
Deduplicating those observations before auditing their depth would erase the distinction you're trying to measure.
Your denominator ends at the collection boundary
The denominator is the emitted observations within your chosen collection boundary. It isn't the whole catalogue.
Geography follows marketplaceDomain for keywords, ASINs, sellers and browse nodes. The selector accepts 21 marketplaces; supplied URLs retain their own marketplace.
Group by the returned marketplace so a cross-market pull doesn't quietly become a single-market claim.
Categories and queries define discovery surfaces. Neither a keyword nor a browse node establishes an exhaustive product universe.
Caps further narrow that surface. maxItems applies globally within a run, maxItemsPerCategory limits each keyword, URL, seller or node source, and maxPages limits listing pagination.
Keyword search itself stops at page 7; category listings can go deeper.
Filters belong in the denominator description too. Price and rating thresholds restrict eligibility, while includeSponsored determines whether sponsored placements are included. onlyDeals requires visible deal evidence, while primeOnly requires a visible badge whose rendering can vary.
A missing badge can't establish that a product lacks the underlying benefit.
Finally, scrapedAt marks an observation time. The earliest and latest timestamps describe the pull's collection window, rather than a historical period over which every product was continuously observed.
Preserve the difference between missing and empty
These fields need different interpretations in a coverage audit:
| Field | Collection distinction | What the coverage check establishes |
|---|---|---|
specifications |
Product-page access plus enabled specification extraction | Whether specification content was returned |
variantAsins |
Variant information from detail collection | Whether variant identifiers were captured, leaving actual variant availability unresolved |
bestsellerRanks |
Ranks read from the product page | Whether rank information was returned, without classifying gaps as unranked products |
images |
Card images or gallery images | Whether images were returned, without establishing gallery depth |
Empty fields are omitted from the Actor's output, including nested fields. The check nevertheless distinguishes omitted keys from explicitly empty values, so a later transformation can't quietly collapse those states.
Here, “populated” means a non-empty returned container or value. It doesn't measure how many specifications an object contains.
The code also keeps observed sourceMode values intact. Inspect them before attempting a mapping; the field name alone doesn't supply its vocabulary.
This cell normalises the records, parses timestamps and creates a field-level audit.
detail_fields = (
"specifications", "variantAsins", "bestsellerRanks", "images"
)
def populated(value):
if value is None:
return False
if isinstance(value, str):
return bool(value.strip())
if isinstance(value, (dict, list)):
return bool(value)
return True
records = (
pd.json_normalize(
[item for _, item in observations], max_level=0
)
.reindex(columns=["asin", "marketplace", "sourceMode", "scrapedAt"])
.astype("string")
)
records["observation"] = records.index
records["collection_stage"] = [stage for stage, _ in observations]
records["observed_at"] = pd.to_datetime(
records["scrapedAt"], utc=True, errors="coerce"
)
source_modes = (
records.groupby(["collection_stage", "sourceMode"], dropna=False)
.size()
.rename("observations")
.reset_index()
)
states = pd.DataFrame([
{
"observation": index,
"field": field,
"state": (
"omitted" if field not in item
else "populated" if populated(item[field])
else "empty"
),
}
for index, (_, item) in enumerate(observations)
for field in detail_fields
], columns=["observation", "field", "state"]).astype({
"observation": "int64",
"field": "string",
"state": "string",
})
audit = (
states.merge(records, on="observation", validate="many_to_one")
.merge(manifest, on="collection_stage", validate="many_to_one")
)
source_modes gives you the returned labels and their observation counts. The audit retains the original collection stage independently, including when a label is absent.
Keep each coverage rate with its denominator
A populated-field share can travel surprisingly far from the collection that produced it. We'd keep its observation count beside it.
This query computes coverage within stage, marketplace, returned mode and enrichment settings, then creates separate reporting views.
connection = duckdb.connect()
connection.register("field_audit", audit)
coverage = connection.sql("""
SELECT
collection_stage,
marketplace,
sourceMode,
scrape_setting,
specs_setting,
detail_route_expected,
specifications_enabled,
field,
count(*) AS observations,
count(*) FILTER (WHERE state = 'populated') AS populated_records,
count(*) FILTER (WHERE state = 'omitted') AS omitted_records,
count(*) FILTER (WHERE state = 'empty') AS empty_records,
100.0 * count(*) FILTER (WHERE state = 'populated')
/ count(*) AS populated_pct,
min(observed_at) AS first_observed_at,
max(observed_at) AS last_observed_at,
count(*) FILTER (
WHERE observed_at IS NULL
) AS unavailable_timestamps
FROM field_audit
GROUP BY ALL
""").df()
connection.close()
listing_coverage = coverage.loc[
coverage["collection_stage"].eq("listing_discovery")
].copy()
shortlist_coverage = coverage.loc[
coverage["collection_stage"].eq("exact_lookup")
].copy()
The time bounds use parseable timestamps; unavailable_timestamps exposes observations excluded from that calculation.
The reporting views answer separate questions: how much information discovery observations returned, and how much exact checks returned.
They don't create a uniform enrichment comparison. That requires a collection design in which the compared groups receive the same detail and specification settings.
Filtering this mixed collection to records that happen to have specifications would instead select on the outcome you're measuring.
Write the collection conditions into the chart
A defensible chart title is “Returned specification coverage by collection stage”. “Products with specifications” claims more than the measurement establishes.
Your accompanying note should identify the discovery inputs, shortlist selection, marketplaces, caps, filters and observed collection window. Keep the stage labels visible rather than pooling the bars into a single completeness score.
Request workload needs its own annotation.
Each detail page is roughly 2.5-3.1 MB.
A returned record doesn't reveal how many requests constructed it. Routing, session warm-up and retries make row counts an unsuitable request counter.
Event fees and platform usage are separate cost components, and output-event counts don't measure page traffic.
For the Amazon Search, Category & Seller Listings Extractor, the manifest supplies the expected routing annotation. The returned rows supply field coverage. Those are different pieces of evidence.
Leave the product verdict open
An omitted value can reflect collection depth, a disabled setting or information the page didn't render. The audit measures returned information under stated collection conditions.
It can't establish catalogue completeness, historical availability or the absence of product features.
Keep the collection conditions attached to the blank cells.
Drafted with AI assistance and reviewed for accuracy.
Top comments (0)