Your first-seen product count is not a launch count

You're about to label a chart “new product launches”. Your feed uses includeNewOnly and onlyNewProducts, so the wording feels defensible.

The catch: the former keeps products Sephora flags as new; the latter skips product IDs already returned under the same stateKey. Together, they select previously unseen product IDs carrying a current merchandising flag.

First observed is a collection event, not a launch date.

What does your new product count actually measure?

The tempting question is how many products launched during your reporting period. The answer available from this feed is narrower: how many eligible product IDs your collection encountered for the first time.

Those events can coincide. The settings don't establish that they did.

A product might enter your feed because it appeared in the category you monitor, because you expanded your coverage, or because its ID was absent from your saved history. None of those explanations requires a launch during the reporting period.

For an observation series, we'd start by removing both novelty gates and recording the retailer flag separately. Example input for a skincare collection:

run_input = {
    "mode": "category",
    "categories": ["skincare"],
    "market": "us",
    "sortBy": "newArrivals",
    "includeNewOnly": False,
    "onlyNewProducts": False,
    "stateKey": "us-skincare-observation",
    "resetState": False,
    "includeDetails": True,
    "includeImages": False,
    "maxItems": 10000,
    "maxItemsPerQuery": 0,
}

This changes the analytical question. You can observe products whether or not they currently carry the new flag, then calculate your own first-observation history.

It doesn't turn a category collection into a complete catalogue census.

A product ID can hide a new shade

Saved discovery state works at productId grain. A shade or size can be added beneath an existing parent product without creating another parent ID.

With incremental suppression enabled, that continuing product can be skipped before your dataset gives you another opportunity to inspect its variants.

The eligibility rule is easier to see as an intersection; this diagram doesn't imply an execution order:

Products reachable through your chosen inputs
    ∩ products carrying the retailer's new flag
    ∩ product IDs absent from the named saved state
    → eligible discovery records

Existing product ID + newly added shade
    → still an existing product ID

That creates a particular bias in an assortment story. A count of new parent IDs can miss expansion within continuing products. A brand adding options beneath existing products and a brand introducing new parent products need not move your metric in the same way.

You should check which pattern matters in your coverage. The dataset can't settle it through parent-ID discovery alone.

The same issue affects the denominator: “share of launches expanding shade choice” can't be calculated from a feed that systematically suppresses the continuing products where such expansion might occur.

Ordering becomes selection when the cap bites

The example orders listings by newArrivals and permits up to 10000 emitted records. Its per-source cap is 0, which removes that particular restriction; the overall cap still applies.

If a collection reaches its overall cap, ordering helps determine which records you observe. “Newest first” remains a listing order, not a supplied launch timestamp.

Record count is therefore a useful warning signal. Reaching the cap means you should investigate coverage before comparing periods. Finishing below it doesn't prove completeness.

For broader coverage, divide a declared scope into stable categories or brands and deduplicate by product ID within the selected market. Keep the partitions stable across reporting periods, and inspect capped partitions separately.

Otherwise, a change in collection coverage can make a discovery chart rise while the underlying assortment has done nothing especially newsworthy.

Keep the collection rule beside the rows

Your dataset needs a companion manifest. It should preserve the inputs, the collection time, and the baseline against which “first” has meaning.

The following cell fetches records, retains the complete chosen input, and handles omitted identity, flag and timestamp fields explicitly:

import os
from uuid import uuid4

import pandas as pd
import duckdb
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "dataspryva/sephora-catalog-price-and-rating-finder"
).call(run_input=run_input)

if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Collection ended with status {run['status']}")

items = []
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    items.append(item)

batch_id = uuid4().hex
baseline_epoch = "us-skincare-start"
collected_at = pd.Timestamp.now(tz="UTC")

manifest = pd.DataFrame([{
    **run_input,
    "batch_id": batch_id,
    "baseline_epoch": baseline_epoch,
    "collected_at": collected_at,
    "emitted_records": len(items),
    "cap_reached": len(items) >= run_input["maxItems"],
}])

records = pd.DataFrame(items).reindex(
    columns=["productId", "skuId", "isNew", "scrapedAt"]
)
records["productId"] = records["productId"].astype("string").str.strip()
records["skuId"] = records["skuId"].astype("string").str.strip()
records["isNew"] = records["isNew"].map(
    lambda value: value if isinstance(value, bool) else pd.NA
).astype("boolean")
records["scrapedAt"] = pd.to_datetime(
    records["scrapedAt"], utc=True, errors="coerce"
)

unidentified_records = records.loc[
    records["productId"].isna() | records["productId"].eq("")
].copy()

observations = records.loc[
    records["productId"].notna() & records["productId"].ne("")
].copy()
observations = observations.assign(
    batch_id=batch_id,
    baseline_epoch=baseline_epoch,
    observed_at=collected_at,
)

Here, observed_at means when your collector received the batch. scrapedAt remains the source record's collection timestamp, parsed independently.

That distinction matters when a run spans a reporting boundary. Choose a clock, name it, and keep using it.

An omitted isNew stays unknown. Turning absence into False would quietly manufacture a claim about retailer labelling.

Your opening batch is inventory, not discovery

The next cell creates an in-memory archive and appends the batch in bulk:

db = duckdb.connect()
db.register("incoming_manifest", manifest)
db.register("incoming_observations", observations)

db.execute("""
    CREATE TABLE run_history AS
    SELECT * FROM incoming_manifest WHERE false
""")
db.execute("""
    CREATE TABLE product_history AS
    SELECT * FROM incoming_observations WHERE false
""")
db.execute("INSERT INTO run_history SELECT * FROM incoming_manifest")
db.execute("INSERT INTO product_history SELECT * FROM incoming_observations")

db.execute("""
    CREATE VIEW first_products AS
    SELECT baseline_epoch, productId,
           min(observed_at) AS first_observed_at
    FROM product_history
    GROUP BY baseline_epoch, productId
""")

This notebook starts with empty history, so its initial batch establishes a baseline. It can't reconstruct earlier encounters. A scheduled collector needs a durable archive that subsequent batches append to.

Now calculate independent measures, including unknown retailer flags:

product_measures = db.sql("""
    SELECT
        r.baseline_epoch,
        r.batch_id,
        r.collected_at,
        r.batch_id = first_value(r.batch_id) OVER (
            PARTITION BY r.baseline_epoch
            ORDER BY r.collected_at, r.batch_id
        ) AS is_baseline,
        count(DISTINCT p.productId) AS observed_products,
        count(DISTINCT p.productId) FILTER (
            WHERE p.isNew IS TRUE
        ) AS retailer_new_products,
        count(DISTINCT p.productId) FILTER (
            WHERE p.isNew IS NULL
        ) AS products_with_unknown_new_flag,
        count(DISTINCT p.productId) FILTER (
            WHERE p.observed_at = f.first_observed_at
        ) AS first_observed_products
    FROM run_history r
    LEFT JOIN product_history p
        USING (baseline_epoch, batch_id)
    LEFT JOIN first_products f
        USING (baseline_epoch, productId)
    GROUP BY r.baseline_epoch, r.batch_id, r.collected_at
""").df()

Keep baseline batches out of a chart of subsequent discoveries, or display them explicitly as baseline inventory. Otherwise, starting collection becomes your biggest apparent launch event. Convenient for the chart, awkward for the correction.

Resetting the Actor's saved state forgets previously seen IDs for that key. It doesn't erase encounters from your independent archive. If your series relies on that saved state, mark the reset as a baseline break; don't splice the resulting rediscoveries into an uninterrupted trend.

Separate product discovery from variant discovery

The Sephora Catalog Price & Rating Finder includes a size and shade matrix in detailed product records. Revisiting continuing products is what makes that detail useful for change analysis.

A listing record's top-level skuId isn't an enumeration of that matrix. A newly observed value can identify something to inspect, but a change in the represented variant isn't sufficient evidence of a newly launched shade.

This cell finds newly observed top-level product/SKU pairs within continuing products:

sku_observations_to_review = db.sql("""
    WITH first_skus AS (
        SELECT baseline_epoch, productId, skuId,
               min(observed_at) AS first_sku_observed_at
        FROM product_history
        WHERE skuId IS NOT NULL AND skuId <> ''
        GROUP BY baseline_epoch, productId, skuId
    )
    SELECT s.*, p.first_observed_at AS product_first_observed_at
    FROM first_skus s
    JOIN first_products p
        USING (baseline_epoch, productId)
    WHERE s.first_sku_observed_at > p.first_observed_at
""").df()

These are review candidates, not a count of variant additions. Establishing additions requires comparable variant inventories from repeated detail observations, with stable variant identities and a clear policy for missing detail.

An absent matrix should remain an unknown observation, rather than becoming evidence that every previous shade disappeared.

Give each measure its own caption

Before publication, match the chart label to the actual unit of observation:

Measure	What qualifies	What the caption needs to preserve
First-observed products	A productId absent from your earlier observations within the same baseline	Declared market and scope, collection schedule, baseline and coverage changes
Retailer-new products	An observed product with isNew equal to True	The flag applies at observation time; omitted flags remain unknown
SKU review candidates	A previously unobserved top-level skuId within a continuing product	A candidate for inspection, not a verified variant addition

Keep the measures in separate tables or chart series. Their overlap can be useful, but combining them into a single “launches” column discards the distinction your reporting needs.

Capped batches also belong in the published methodology. A lower discovery count after a change in coverage isn't automatically evidence of slower product introduction.

The launch date is still missing

Neither the retailer flag nor your observation history establishes when a product originally launched, when a shade first became available, or whether it debuted elsewhere before entering your coverage.

Even comparable variant inventories only locate an observed change between collection points. They don't recover the precise moment it happened, and missing detail can leave the change unresolved.

Your chart can date the encounter. It can't backdate the launch.