Your creator count may be a pagination ceiling in disguise

Your creator chart looks suspiciously tidy when every category stops at the same count. Before you publish a market comparison, check whether you've measured competition or pagination.

Dailymotion's searchUsers and byPlaylist modes stop at 1,000 records per query because their endpoints don't accept date filters.

Setting maxItems to 10,000 enables date-window sweeps for video search, topics and channel uploads. It doesn't expand those creator-search or playlist endpoints. A valid input can still ask for more than the selected endpoint can return.

Keep each query attached to its rows

The distinction matters before you count anything: searching for creators returns channels; fetching a channel's uploads returns videos. More video records don't establish that you've discovered more competitors.

For a coverage audit, keep the input beside the returned records. Don't rely on reconstructing selection rules from titles or profile descriptions later.

Example input: this cell requests a creator search and retains its status, selection and records.

import os

from apify_client import ApifyClient
import pandas as pd
import duckdb

client = ApifyClient(os.environ["APIFY_TOKEN"])

def fetch(run_input):
    run = client.actor(
        "dataspryva/dailymotion-video-discovery-by-topic-and-channel"
    ).call(run_input=run_input)
    items = list(
        client.dataset(run["defaultDatasetId"]).iterate_items()
    )
    return {
        "input": run_input,
        "status": run["status"],
        "items": items,
    }

run_input = {
    "mode": "searchUsers",
    "searchQueries": ["space news"],
    "sortBy": "popular",
    "maxItems": 10000,
}
directory_pull = fetch(run_input)

Streaming the dataset avoids treating a dataset retrieval page as the entire collection. Otherwise, you can accidentally audit the retrieval method's coverage rather than the Actor's.

The query also matters. Keyword discovery defines a matching population, not every account operating in a market. Accounts that don't match your search remain outside the selection, however far pagination goes.

Channel uploads answer a different question

The word “channel” does awkward double duty here. Channel discovery searches account records; a channel sweep walks an already selected account's video history.

The coverage boundary follows the endpoint, not the noun.

Selection	Returned unit	Date-window expansion
search, byTopic	Video	Supported
byUser	Video from a supplied channel	Supported
searchUsers	Channel matching a keyword	Unsupported
byPlaylist	Video from a supplied playlist	Unsupported

Playlist reading also starts from a known URL or ID. There's no playlist keyword-search mode, so a collection of supplied playlists isn't a discovered playlist directory.

Example input: this cell fetches uploads from a supplied channel, keeping that pull separate from creator discovery.

uploads_input = {
    "mode": "byUser",
    "usernames": ["euronews-en"],
    "sortBy": "recent",
    "maxItems": 10000,
}
uploads_pull = fetch(uploads_input)
pulls = [directory_pull, uploads_pull]

Each run here has a single query or channel seed. That makes the returned count attributable to its selection. Combining several queries inside a run would make a run-level count harder to interpret against a per-query boundary.

For larger date-window sweeps, ordering becomes chronological, newest first or oldest first. You can't treat an expanded pull as a deeper relevance ranking or a longer most-viewed leaderboard.

Geography and dates don't travel between modes

A country input isn't a universal market boundary. It filters videos in search and trending; it doesn't geographically restrict creator discovery or a channel's uploads.

For a creator-market chart, that leaves an analytical choice: classify returned channels using their reported country, or classify videos using their owners' country. Those describe different populations, and neither establishes where the audience lives.

The code below keeps those country meanings explicit. Missing country values stay missing rather than becoming an “other country” bucket with an unjustifiably confident label.

Dates need the same care. A video's publishedAt describes publication, while a channel's createdAt describes account creation. Neither can stand in for the other when you define “active creators”.

Publication filters use roughly five-minute buckets, so neighbouring content can appear around a day boundary. If your report uses strict calendar periods, apply that boundary to parsed timestamps and retain a count of excluded records.

Only publicly accessible videos are returned. Private and unlisted material sits outside that frame, rather than inside a missing-value category.

Preserve missing fields before counting

Video records carry recordType="video" and use videoId; creator-search records carry recordType="channel" and use channelId. Counting whichever identifier happens to be present would hide a record-type mismatch.

Optional metadata is another coverage dimension. Fields such as tags, description and firstFrameUrl can be omitted because the source doesn't carry them. An analysis requiring those fields selects a narrower population than the original pull.

This cell builds audit tables with explicit missing values and parses publication timestamps; the snake-case columns are analysis columns created here.

manifest_rows = []
record_rows = []

for pull_id, pull in enumerate(pulls):
    config = pull["input"]
    is_directory = config["mode"] == "searchUsers"
    selector_key = "searchQueries" if is_directory else "usernames"
    manifest_rows.append({
        "pull_id": pull_id,
        "input_mode": config["mode"],
        "selector": config[selector_key][0],
        "expected_type": "channel" if is_directory else "video",
        "status": pull["status"],
        "requested": config["maxItems"],
        "ceiling": min(config["maxItems"], 1000)
                   if is_directory else config["maxItems"],
    })
    for item in pull["items"]:
        record_rows.append({
            "pull_id": pull_id,
            "record_type": item.get("recordType"),
            "entity_id": item.get(
                "channelId" if is_directory else "videoId"
            ),
            "reported_country": item.get(
                "country" if is_directory else "ownerCountry"
            ),
            "published_at": None if is_directory
                            else item.get("publishedAt"),
        })

manifest = pd.DataFrame(manifest_rows)
records = pd.DataFrame(record_rows, columns=[
    "pull_id", "record_type", "entity_id",
    "reported_country", "published_at",
])
for column in ["record_type", "entity_id", "reported_country"]:
    records[column] = (
        records[column].astype("string").str.strip().replace("", pd.NA)
    )
records["published_at"] = pd.to_datetime(
    records["published_at"], utc=True, errors="coerce"
)

Keeping the manifest separate preserves empty pulls. Otherwise, a query returning no records disappears from the audit, which is an impressively efficient way to erase inconvenient coverage gaps.

Measure the boundary, not an imaginary missing tail

A count at the endpoint ceiling is a warning, not proof that additional matching records exist. A count below it doesn't establish a complete market either: selection, accessibility and run completion still matter.

There's no defensible missing-tail estimate available from the returned count alone.

This DuckDB check separates row volume, distinct usable identities, missing geography and observed publication range, while preserving pulls with no records.

connection = duckdb.connect()
connection.register("manifest", manifest)
connection.register("records", records)

audit = connection.execute("""
    WITH counts AS (
        SELECT
            m.*,
            count(r.pull_id) AS rows_returned,
            count(DISTINCT r.entity_id) FILTER (
                WHERE r.record_type = m.expected_type
            ) AS distinct_entities,
            count(r.pull_id) FILTER (
                WHERE r.record_type IS DISTINCT FROM m.expected_type
                   OR r.entity_id IS NULL
            ) AS unusable_rows,
            count(r.reported_country) FILTER (
                WHERE r.record_type = m.expected_type
                  AND r.entity_id IS NOT NULL
            ) AS rows_with_country,
            min(r.published_at) FILTER (
                WHERE r.record_type = 'video'
            ) AS earliest_publication,
            max(r.published_at) FILTER (
                WHERE r.record_type = 'video'
            ) AS latest_publication
        FROM manifest AS m
        LEFT JOIN records AS r USING (pull_id)
        GROUP BY ALL
    )
    SELECT *,
        rows_returned >= ceiling AS boundary_reached,
        CASE
            WHEN status <> 'SUCCEEDED' THEN 'run incomplete'
            WHEN unusable_rows > 0 THEN 'check record identity'
            WHEN rows_returned >= ceiling THEN 'boundary reached'
            ELSE 'below boundary; completeness unknown'
        END AS assessment
    FROM counts
    ORDER BY pull_id
""").df()

audit gives you measured coverage diagnostics for your pulls. Publication endpoints describe the observed span, not continuous coverage between those timestamps.

The boundary check uses returned rows because that's the collection limit being examined. Distinct identities answer a separate question: how many usable entities your dataset represents.

Put the selection rule beside the chart

The worked example uses Dailymotion Video Discovery by Topic & Channel. The reusable lesson is to carry selection metadata into the report instead of discarding it once ingestion succeeds.

For creator discovery, a defensible chart label is “Distinct channels returned by keyword search”. Alongside it, state the query, ordering, retrieval time and whether the audit reached the endpoint boundary.

With popular ordering, the search is ordered by follower count. If collection stops at the boundary, the returned set can't establish the distribution of accounts beyond that point.

For uploads, label the measure as videos from the supplied channels, with the publication window and any language, tag or duration restrictions. The input channel list is part of the population definition.

The distinction is small enough to fit beside a chart:

keyword search → returned channel IDs → observed creator set
known channel  → returned video IDs   → observed upload catalog

If you add narrower keyword queries, deduplicate channel IDs across pulls. The combined set may expand discovery, but lexical slices can overlap and omit accounts in related ways. They don't become a representative sample merely because you used more queries.

The market outside the returned rows

These records can describe public channels and videos reached through your selections. They can support comparisons within that observed set, provided missing metadata and collection boundaries stay visible.

They don't establish total creator volume, the number of commercially active competitors, or the size of an unseen tail. More uploads from known channels can deepen a catalogue without broadening creator discovery.

An unknown tail is still unknown. A round number doesn't make it a market size.