DEV Community

Cover image for Comparing Amazon deals with Python: give each average a population
Dataspryva
Dataspryva

Posted on Fully Autonomous

Comparing Amazon deals with Python: give each average a population

Your manager wants the high-rated deal feed to prove which brand discounts more aggressively. You've a tidy chart and a less tidy denominator.

The collection input contains minRating: 4. It looks like a preference for better products. It also decides which products get a vote in every brand average.

Setting minRating keeps only products at or above the threshold and excludes products without ratings.

That changes the question from “Which brand discounts more?” to “Which brand has deeper reported discounts among the captured products that pass this rating gate?” Different category and rating mixes can change the answer without any change in discount depth within comparable groups.

The average needs a population attached

A product's accumulated customer feedback and its current promotion describe different things. A rating threshold can define a useful shopping feed, but it doesn't establish a comparable promotional sample.

We'll use Dataspryva's deals Actor as the worked example. The analysis starts with an ungated pull so you can apply the threshold to the same captured records.

Example collection configuration: this requests a category window without a rating restriction and keeps its inputs alongside the dataset identifier.

import os

import duckdb
import pandas as pd
from apify_client import ApifyClient

run_input = {
    "marketplace": "amazon.com",
    "categories": ["Electronics", "Home & Kitchen"],
    "sortBy": "featured",
    "maxItems": 2000,
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "dataspryva/amazon-deals-grid-tracker-by-category-and-discount"
).call(run_input=run_input)

if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Collection ended with status {run['status']}")

items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
analysis_scope = {
    "input": run_input.copy(),
    "dataset_id": run["defaultDatasetId"],
}
Enter fullscreen mode Exit fullscreen mode

The streaming dataset method avoids mistaking a page of dataset results for the entire pull.

This is a collection recipe, not a claim about what any particular run returns.

Your input selects a window, not a catalogue

The sampling frame begins with the live Deals grid for the chosen marketplace and categories. Products outside that grid never enter this comparison, however deeply they might be discounted elsewhere.

The source exposes up to 500 deals per category/collection query. Adding categories partitions those windows; it doesn't turn them into a census. Grid reshuffling between pages can also reduce the unique deals captured.

maxItems separately caps emitted deal records. Featured ordering preserves the source's order, while the other sorting options sort the fetched window. Sorting can't recover products outside that window.

The rating gate then narrows the eligible population:

Marketplace and category deal windows
                 |
          fetched deal records
                 |
          rating eligibility
           /             \
     retained         excluded
                 low or absent rating
Enter fullscreen mode Exit fullscreen mode

The awkward exclusion is unrated products. “No rating” isn't evidence of poor quality, yet those products can't contribute to a rating-filtered brand statistic.

Whether that exclusion falls unevenly across brands is something your pull can measure. Assuming it falls evenly is something your spreadsheet will happily permit.

Count who survives before comparing discounts

Keep the eligibility decision as a column. Otherwise, the excluded records disappear before you can explain them.

The next cell makes omitted keys explicit, parses numeric strings with commas or a trailing percent sign, and preserves unparseable values as missing.

source_fields = [
    "asin", "brandName", "category", "averageRating",
    "savingsPercent", "message",
]
raw = pd.DataFrame(items).reindex(columns=source_fields)
raw["asin"] = raw["asin"].astype("string").str.strip().replace("", pd.NA)

non_product_rows = raw.loc[raw["asin"].isna(), ["message"]].copy()
deals = raw.loc[raw["asin"].notna()].copy()

def numeric(series):
    text = (
        series.astype("string").str.strip()
        .str.replace(",", "", regex=False)
        .str.removesuffix("%")
    )
    return pd.to_numeric(text, errors="coerce").astype("Float64")

for field in ("brandName", "category"):
    deals[field] = (
        deals[field].astype("string").str.strip().replace("", pd.NA)
    )

threshold = 4
deals["rating"] = numeric(deals["averageRating"])
deals["saving_pct"] = numeric(deals["savingsPercent"])
deals["rating_parse_failed"] = (
    deals["averageRating"].notna() & deals["rating"].isna()
)
deals["eligible"] = deals["rating"].ge(threshold).fillna(False)
deals["rating_band"] = (
    (deals["rating"] // 1).astype("Int64").astype("string")
)
Enter fullscreen mode Exit fullscreen mode

Empty source fields are omitted, so their absence must remain distinct from a measured zero. A nonempty rating that can't be parsed also deserves inspection before you interpret the local mask as a faithful eligibility check.

Rows without a product identifier stay available for inspection rather than becoming anonymous products. Missing brand names are counted separately because an unidentified brand isn't a competitor.

This cell calculates brand retention and category shares before any discount averages.

missing_brand_records = deals["brandName"].isna().sum()
named_brand = deals.loc[deals["brandName"].notna()].copy()

eligibility = named_brand.groupby("brandName").agg(
    records=("asin", "size"),
    distinct_asins=("asin", "nunique"),
    ratings_available=("rating", "count"),
    rating_parse_failures=("rating_parse_failed", "sum"),
    eligible_records=("eligible", "sum"),
)
eligibility["eligibility_rate"] = (
    eligibility["eligible_records"] / eligibility["records"]
)

composition = named_brand.groupby(
    ["brandName", "category"], dropna=False
).agg(records=("asin", "size"), eligible_records=("eligible", "sum"))

composition["share_before"] = (
    composition["records"]
    / composition.groupby(level="brandName")["records"].transform("sum")
)
composition["share_after"] = (
    composition["eligible_records"]
    / composition.groupby(level="brandName")["eligible_records"]
    .transform("sum").replace(0, pd.NA)
)
Enter fullscreen mode Exit fullscreen mode

Read eligibility_rate beside the category shares. Similar retention rates can still conceal different changes in composition.

The unit here's a captured deal record. Compare records with distinct_asins before calling anything a product-weighted average. Repeated ASINs contribute repeatedly under the record-based definition; a product-based comparison needs an explicit rule for reconciling them.

A missing discount creates another denominator

Passing the rating gate doesn't guarantee that a discount value is available. The source omits discount data for some promotions, including some coupon-only offers.

A mean over observed savingsPercent values therefore has its own denominator. Filling missing savings with zero would quietly classify unknown discounts as no discount.

This cell compares the fixed pull before and after the local gate, with coverage beside each mean.

discount_frame = named_brand.assign(
    gated_saving=named_brand["saving_pct"].where(named_brand["eligible"])
)

before_after = discount_frame.groupby("brandName").agg(
    savings_known=("saving_pct", "count"),
    mean_before=("saving_pct", "mean"),
    gated_savings_known=("gated_saving", "count"),
    mean_after=("gated_saving", "mean"),
).join(eligibility[["records", "eligible_records"]])

before_after["coverage_before"] = (
    before_after["savings_known"] / before_after["records"]
)
before_after["coverage_after"] = (
    before_after["gated_savings_known"]
    / before_after["eligible_records"].replace(0, pd.NA)
)
before_after["change_in_mean"] = (
    before_after["mean_after"] - before_after["mean_before"]
)
Enter fullscreen mode Exit fullscreen mode

A changed ranking would show sensitivity to the gate within this captured pull. It wouldn't, by itself, reveal whether the change came from category composition, rating composition or differences within comparable groups.

The chart can be arithmetically immaculate and still answer the wrong question. Arithmetic has excellent manners.

Give each brand the same comparison mix

To separate some composition effects, compare brands within shared category and rating groups.

The rating bands below use the whole-star floor. That's a transparent analytical choice, not proof that products within a band are interchangeable. Broad categories can still contain very different products.

This DuckDB query keeps only groups represented by every named brand in the pull, using eligible records with known savings.

connection = duckdb.connect()
connection.register("captured_deals", named_brand)

shared_cells = connection.sql("""
    WITH cells AS (
        SELECT
            brandName,
            category,
            rating_band,
            COUNT(*) AS records,
            AVG(saving_pct) AS mean_saving
        FROM captured_deals
        WHERE eligible
          AND saving_pct IS NOT NULL
          AND category IS NOT NULL
          AND rating_band IS NOT NULL
        GROUP BY brandName, category, rating_band
    ),
    shared AS (
        SELECT category, rating_band
        FROM cells
        GROUP BY category, rating_band
        HAVING COUNT(*) = (
            SELECT COUNT(DISTINCT brandName)
            FROM captured_deals
        )
    )
    SELECT cells.*
    FROM cells
    JOIN shared USING (category, rating_band)
""").df()
Enter fullscreen mode Exit fullscreen mode

Missing categories can't support this comparison. Neither can a group observed for only some brands.

If there are no shared groups, there's no matched comparison under this definition. Broadening the groups might create overlap, but it also weakens comparability. That trade-off belongs in the report.

The next cell gives each shared category-rating group equal weight for every brand and records how much of the gated discount sample remains.

matched = shared_cells.groupby("brandName").agg(
    shared_groups=("mean_saving", "size"),
    standardised_mean=("mean_saving", "mean"),
    records_in_shared_groups=("records", "sum"),
)

matched["share_of_gated_known_savings"] = (
    matched["records_in_shared_groups"]
    / before_after["gated_savings_known"]
)

connection.close()
Enter fullscreen mode Exit fullscreen mode

Equal group weights describe a deliberately constructed comparison population. They don't estimate the marketplace's product mix.

Inspect the group record counts before treating a standardised difference as persuasive. Matching can improve comparability while leaving a thin slice of the original question.

Collect for the comparison you intend to defend

The worked example uses Amazon Deals Grid Tracker by Category & Discount. For a brand comparison, the useful input design preserves the broadest relevant comparison frame before applying analytical restrictions.

Keep the marketplace and category scope fixed. Omit minRating from the baseline collection, then retain both eligible and ineligible records locally.

Avoid discount floors when the question concerns the full observed distribution of discount depth. Price and discount filters also exclude rows whose values can't be verified, adding a completeness restriction to the numerical restriction.

A broad brand capture followed by exact local grouping is easier to audit than separate substring-filtered brand pulls. The Actor's brand input is case-insensitive substring matching, so it isn't an exact brand identifier.

Applying a local gate to a fixed pull isolates its effect on those captured records. Separate live gated and ungated runs can also differ because deals rotate, the grid reshuffles and caps constrain what's returned.

Reaching the configured cap is evidence of truncation. Staying below it isn't evidence of complete marketplace coverage.

Choose wording that matches the population behind each statistic:

Statistic Defensible population
Baseline mean Captured named-brand deal records with known savings
Gated mean Eligible captured records with known savings
Standardised mean Shared category-rating groups under equal group weights

Alongside the chosen mean, retain the collection inputs, eligibility rates, discount coverage, category composition and record counts. Those are the parts your manager needs when asking why the denominator moved.

Where the brand claim stops

An average product rating summarises accumulated feedback. ratingsCount describes its volume; neither field measures the quality of the current promotion.

Reported savings also depend on the displayed reference price. Labels such as List, Typical and Was aren't a historical price series, so a larger savingsPercent doesn't establish a larger reduction from a consistently measured previous selling price.

This dataset doesn't provide sales volumes, transaction prices or margin. It can't establish which brand discounts more aggressively across its whole catalogue.

Your high-rated feed can describe the promotions that survived its selection rules.

The rating gate doesn't just clean the chart. It helps write the claim.

Drafted with AI assistance and reviewed for accuracy.

Top comments (0)