Deduplication does not refund your search result fees

Your clean URL count can make the acquisition fee look wrong just before you publish the report. You've applied drop_duplicates("url"), and now the cost per retained page looks uncomfortable.

Each query is scraped independently up to maxResults, while result fees follow emitted dataset items. Overlapping queries can therefore retrieve the same destination without adding another URL to your analytical corpus.

Deduplicating destination URLs reduces the corpus, but it does not reverse result events already emitted.

The awkward part is explaining that distinction after you've thrown away the retrieval history. A tidy table is surprisingly good at hiding how much work produced it.

The acquisition needs its own ledger

We need separate records of what you requested, what came back and what you retained. The final URL table can't stand in for all of them.

For this example, we'll use the Actor's web mode with related queries. The input requests overlapping subject matter; it doesn't presume that repeated URLs will occur.

Example acquisition cell: this keeps the input manifest and fetches the dataset through the streaming interface.

import os
from decimal import Decimal

import pandas as pd
import duckdb
from apify_client import ApifyClient

run_label = "web-month"
run_input = {
    "mode": "webSearch",
    "queries": ["人工智能", "人工智能 最新进展"],
    "maxResults": 50,
    "timeRange": "month",
    "language": "simplified",
    "includeSerpFeatures": True,
}
manifests = {run_label: run_input.copy()}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "dataspryva/baidu-web-image-and-news-results-extractor"
).call(run_input=run_input)
items = list(
    client.dataset(run["defaultDatasetId"]).iterate_items()
)

The stream matters because reconciliation needs the whole exported dataset. A partial export would introduce another gap before deduplication even starts.

Keep the manifest with the acquisition. A query string alone doesn't identify its retrieval frame: the same words under different filters are different requests.

The retrieval frame ends before the web does

This pull measures a slice of public search results. It doesn't enumerate every page about the topic.

These are the boundaries that affect the denominator:

Boundary	Included or excluded	Consequence for your corpus
Geography and script	There is no geographic input filter. language selects Chinese script and does not translate the query.	Script selection cannot establish where a publisher or subject is located.
Result surface	Web, news, images and autocomplete are separate modes.	An organic web corpus excludes articles, images or suggestions found only through another surface.
Publication window	timeRange filters web results; dateFrom and dateTo override it for web and news.	A publication window is not an archive of everything searchable during that period.
Retrieval ceiling	maxResults is a per-query maximum. Source exhaustion or a security check can stop retrieval earlier.	A short return does not establish that the topic has little coverage.
Conditional fields and rows	Dates and other unavailable fields are omitted. Enabled search features arrive as separate rows without reducing the organic result count.	Dataset rows, dated pages and organic URLs need separate counts.

Different date windows, site restrictions, file formats, script selections and the mobile view can expose different pages. The Actor falls back to the mobile view when needed; there's no mobile-selection input in the schema.

That makes partitioning useful for discovery. It also makes overlap worth measuring.

A union can grow while its acquisition efficiency falls. Neither observation, by itself, tells you whether another query was worthwhile.

Keep the retrieval trail beside the corpus

We'll retain exact URL strings as the identity rule. No tracking-parameter removal, redirect resolution or inferred canonicalisation is hidden inside this check.

This cell creates a corpus and a bridge back to the queries that retrieved each URL, while keeping every exported row available for accounting.

rows = (
    pd.DataFrame.from_records(items)
    .reindex(columns=[
        "query", "resultType", "url", "scrapedAt", "publishedAt"
    ])
    .assign(
        run_label=run_label,
        item_number=lambda frame: range(len(frame)),
        query=lambda frame: frame["query"].astype("string"),
        resultType=lambda frame: frame["resultType"].astype("string"),
        url=lambda frame: frame["url"].astype("string"),
        scrapedAt=lambda frame: pd.to_datetime(
            frame["scrapedAt"], errors="coerce", utc=True
        ),
        publishedAt=lambda frame: pd.to_datetime(
            frame["publishedAt"], errors="coerce", utc=True
        ),
    )
)

organic_mask = (
    rows["resultType"].eq("organic").fillna(False)
    & rows["url"].str.strip().ne("").fillna(False)
)
organic = rows.loc[organic_mask].copy()
corpus = organic[["url"]].drop_duplicates().reset_index(drop=True)
bridge = organic[["run_label", "query", "url"]].drop_duplicates()

row_types = (
    rows.groupby(["run_label", "resultType"], dropna=False)
    .size()
    .rename("exported_rows")
    .reset_index()
)

Missing fields become missing values in your analytical table. Missing or unfamiliar row types remain visible in row_types; they don't quietly become organic pages.

The bridge removes duplicate query-to-URL relationships. The untouched rows table preserves repeated emissions within those relationships.

Those tables answer different questions. Keeping both prevents a provenance join from quietly becoming a billing ledger.

Measure overlap without erasing empty queries

Start the coverage check from the requested queries, rather than the returned rows. Otherwise a query returning nothing disappears from the audit.

This cell measures organic retrievals, exact URL diversity and available publication timestamps for each requested query.

planned = pd.DataFrame([
    {
        "run_label": label,
        "query": query,
        "requested_cap": manifest["maxResults"],
    }
    for label, manifest in manifests.items()
    for query in manifest["queries"]
])

coverage = duckdb.sql("""
    SELECT
        p.run_label,
        p.query,
        p.requested_cap,
        COUNT(o.url) AS organic_rows,
        COUNT(DISTINCT o.url) AS unique_urls,
        COUNT(o.url) - COUNT(DISTINCT o.url)
            AS repeated_url_rows,
        COUNT(o.url) = p.requested_cap AS reached_requested_cap,
        COUNT(o.publishedAt) * 1.0 / NULLIF(COUNT(o.url), 0)
            AS published_at_share,
        MIN(o.scrapedAt) AS first_scraped_at,
        MAX(o.scrapedAt) AS last_scraped_at
    FROM planned AS p
    LEFT JOIN organic AS o
        ON p.run_label = o.run_label AND p.query = o.query
    GROUP BY p.run_label, p.query, p.requested_cap
""").df()

url_reach = (
    bridge.groupby("url")
    .size()
    .rename("retrieval_frames")
    .reset_index()
)

coverage measures repetition within each query. url_reach measures how many query frames reached each retained URL.

Don't sum the per-query unique counts to obtain corpus size. Their overlap is precisely what you're trying to preserve.

Reaching the requested cap flags possible truncation, not completeness. Falling below it also leaves an unresolved question: the source may have exhausted results, or retrieval may have been interrupted. The status message identifies affected queries when that happens.

The timestamp share measures field availability. Treating rows without publishedAt as outside the publication window would impose a new exclusion after retrieval.

Reconcile charged events before pricing the corpus

Use charged-event counts from the acquisition's billing record. Don't reconstruct them from unique URLs, and don't silently substitute exported row counts.

Start-event counts also need their own entry: charging depends on memory, with one event per GB and a minimum of one event. Batching queries may change overhead, but “one run” isn't a sufficient billing calculation.

These labelled arithmetic examples isolate event fees:

• FREE tier, one start event and 100 result events: $0.10 + 100 x $0.00099 = $0.199, plus platform usage.
• GOLD tier, the same event counts: $0.03 + 100 x $0.00049 = $0.079, plus platform usage.

For the collected pull, the following cell leaves billing values unknown until you supply them. Use numeric counts and, if available, the actual platform usage charge.

billing = {
    "tier": "FREE",
    "result_events": None,
    "start_events": None,
    "platform_usage_usd": None,
}
event_prices = {
    "FREE": (Decimal("0.00099"), Decimal("0.10")),
    "GOLD": (Decimal("0.00049"), Decimal("0.03")),
}
result_rate, start_rate = event_prices[billing["tier"]]

event_fees_usd = (
    None
    if billing["result_events"] is None
    or billing["start_events"] is None
    else (
        Decimal(str(billing["result_events"])) * result_rate
        + Decimal(str(billing["start_events"])) * start_rate
    )
)

A difference between charged result events and exported rows is a reconciliation question. Deduplication can't explain that difference because it happens downstream of both.

Feature rows need their own category too. They belong in the export accounting even when your research corpus retains only organic destination URLs.

Put both denominators in the report

The worked example uses Baidu Web, Image & News Results Extractor. The accounting pattern applies whenever retrieval frames overlap and acquisition charges precede analytical deduplication.

This cell puts the acquisition and corpus denominators alongside the supplied billing counts, leaving unavailable values and undefined ratios empty.

def ratio_or_missing(numerator, denominator):
    if numerator is None or denominator is None or denominator == 0:
        return None
    return Decimal(str(numerator)) / Decimal(str(denominator))

report = pd.Series({
    "run_status": run["status"],
    "exported_dataset_rows": len(rows),
    "organic_retrieval_rows": len(organic),
    "retained_exact_urls": len(corpus),
    "charged_result_events": billing["result_events"],
    "charged_start_events": billing["start_events"],
    "charged_results_minus_exported_rows": (
        None
        if billing["result_events"] is None
        else billing["result_events"] - len(rows)
    ),
    "exported_rows_per_retained_url": ratio_or_missing(
        len(rows), len(corpus)
    ),
    "event_fees_usd": event_fees_usd,
    "event_fees_per_exported_row_usd": ratio_or_missing(
        event_fees_usd, len(rows)
    ),
    "event_fees_per_retained_url_usd": ratio_or_missing(
        event_fees_usd, len(corpus)
    ),
    "platform_usage_usd": billing["platform_usage_usd"],
    "total_acquisition_usd": (
        None
        if event_fees_usd is None
        or billing["platform_usage_usd"] is None
        else event_fees_usd
        + Decimal(str(billing["platform_usage_usd"]))
    ),
})

For a chart caption, state the surface, queries, filters, acquisition window and identity rule. Name the denominator explicitly: “event fees per retained exact organic URL” carries a different meaning from “event fees per exported dataset row”.

Keep platform usage beside those figures. Without it, neither ratio represents total acquisition cost.

For a corpus assembled across acquisitions, retain distinct run labels and manifests, reconcile each acquisition at its applicable tier, then divide the combined fees by the combined exact-URL corpus. Averaging per-run ratios would give small and large acquisitions equal weight.

A URL is still a narrow unit of evidence

Exact deduplication can't establish document identity across tracking parameters, redirects or syndicated copies. It also can't tell you which relevant pages the search engine never exposed.

Nor does retrieval frequency establish authority or prevalence. A page reached through several overlapping queries has several retrieval relationships, not several independent pieces of evidence.

Your manager can reasonably ask what each retained URL cost. The clean corpus can supply the denominator.

It can't remember the acquisition bill for you.