Your sentiment chart can move when extraction changes

Your weekly sentiment chart moved. Marketing wants an explanation, and you're checking the model before sending the report.

Start with the articles that reached it. A full-text cap leaves later articles in the dataset with fullTextStatus=skipped_cap; publisher refusals also remain as records. Filtering for usable bodies changes which coverage your chart represents.

A full-text sentiment trend reflects both article sentiment and the selection of articles whose bodies were available.

Example input: this starts a keyword collection with separate record and extraction budgets.

import { ApifyClient } from "apify-client";
import { z } from "zod";

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run_input = {
  mode: "keyword",
  queries: ["artificial intelligence"],
  dateRange: "7d",
  maxItems: 500,
  extractFullText: true,
  maxFullText: 100,
};

const run = await client
  .actor("nimbryxo-studio/google-news-keyword-and-full-text-article-scraper")
  .call(run_input);

if (run.status !== "SUCCEEDED") {
  throw new Error(`Inspect unfinished collection: ${run.status}`);
}

const { items } = await client.dataset(run.defaultDatasetId).listItems();

That success check belongs in the ingestion job. It doesn't certify every publisher response.

Which symptom explains the missing bodies?

The Actor caps publisher requests so a large metadata collection can't quietly become a large page-fetching job. It also preserves articles when extraction fails, keeping the metadata useful.

Your downstream filter decides whether those preserved records remain visible.

Observable symptom	Cause you can establish	Easy misdiagnosis	Fix and its cost
skipped_cap	The article was beyond the shared extraction budget	Publisher blocking	Increase the budget or select a separate extraction cohort; more requests
short	A small body was extracted	A complete, unusually brief article	Exclude it or maintain a separate scoring series; less coverage or more reporting work
paywalled	A subscription wall was detected	A temporary request failure	Retain metadata and exclude unavailable bodies; a persistent coverage gap
blocked	The publisher refused the request	A confirmed subscription wall	Inspect access conditions; review work without guaranteed recovery
rate_limited, timeout, error	The request encountered the named condition	Permanent publisher exclusion	Consider bounded retries; additional requests without guaranteed success
empty, not_found	No body was found, or the page wasn't found	The extraction cap	Inspect the URL and page; manual work or reduced coverage
unresolved	The publisher URL couldn't be resolved	Publisher refusal	Supply a verified publisher URL; URL repair work
Missing status or unfamiliar status	The row doesn't establish a recognized outcome	An ordinary extraction failure	Check saved inputs and preserve the raw value; investigation

A small body can be a preview. It doesn't establish why the body is small.

That distinction matters when someone proposes retrying every excluded article. A subscription wall is unlikely to improve through enthusiasm.

Preserve statuses before classifying them

Empty fields are omitted. Validate optional fields accordingly, and retain unfamiliar status strings rather than rejecting the entire dataset.

This schema validates the fields used by the audit.

const ArticleSchema = z.object({
  url: z.string().url(),
  domain: z.string().optional(),
  publishedAt: z.string().optional(),
  scrapedAt: z.string().optional(),
  sourceUrl: z.string().optional(),
  fullText: z.string().optional(),
  fullTextStatus: z.string().optional(),
});

type Article = z.infer<typeof ArticleSchema>;
const rows = z.array(ArticleSchema).parse(items);

const knownStatuses = new Set([
  "ok", "short", "empty", "blocked", "paywalled", "rate_limited",
  "not_found", "timeout", "error", "unresolved", "skipped_cap",
]);

function outcome(row: Article): string {
  if (row.fullTextStatus === undefined) return "missing_status";
  return knownStatuses.has(row.fullTextStatus)
    ? row.fullTextStatus
    : "unknown_status";
}

const byOutcome = new Map<string, Article[]>();
for (const row of rows) {
  const key = outcome(row);
  const bucket = byOutcome.get(key) ?? [];
  bucket.push(row);
  byOutcome.set(key, bucket);
}

Each table symptom now has a corresponding bucket. Missing and unfamiliar values remain separate, with the original string retained on each row.

Keep the submitted input beside the run identifier in your ingestion records. Items alone can't prove that extraction was enabled. A metadata-only run intentionally omits snippet, because the feed description repeats the headline.

For this input, extraction is enabled. Missing statuses therefore deserve investigation rather than a convenient metadata-only label.

Separate budget exhaustion from retry candidates

maxItems limits emitted unique articles. maxFullText limits page fetching across the run, within its allowed range of 1–500.

Increasing the record budget doesn't increase the extraction budget. The resulting dataset can offer broader metadata coverage while your body-based analysis remains constrained.

These selectors distinguish the remediation queues without claiming that every request failure is temporary.

const capSkipped = byOutcome.get("skipped_cap") ?? [];
const shortBodies = byOutcome.get("short") ?? [];
const paywalled = byOutcome.get("paywalled") ?? [];
const blocked = byOutcome.get("blocked") ?? [];
const unresolved = byOutcome.get("unresolved") ?? [];

const retryCandidates = rows.filter((row) =>
  ["rate_limited", "timeout", "error"].includes(row.fullTextStatus ?? "")
);

const pageReview = rows.filter((row) =>
  ["empty", "not_found"].includes(row.fullTextStatus ?? "")
);

const statusReview = [
  ...(byOutcome.get("missing_status") ?? []),
  ...(byOutcome.get("unknown_status") ?? []),
];

const okWithoutBody = rows.filter(
  (row) => row.fullTextStatus === "ok" && !row.fullText?.trim()
);

A nonempty capSkipped queue establishes a budget effect. The other queues establish different extraction outcomes, which can coexist in the same run.

Treat okWithoutBody as an ingestion or contract investigation. Don't silently score the headline to make that queue disappear.

For unresolved records, inspect url alongside sourceUrl. Resolution failure can preserve the original news link, so the presence of a URL doesn't prove that a publisher page was reached.

Keep excluded articles in the denominator

Your sentiment service may only need eligible text. Your coverage audit needs every emitted article.

Otherwise, dropping failed extractions before aggregation makes the remaining dataset look perfectly healthy. The denominator has left the building.

This aggregation retains extraction outcomes by publisher and publication day.

function publicationDay(value: string | undefined): string {
  const timestamp = value === undefined ? NaN : Date.parse(value);
  return Number.isFinite(timestamp)
    ? new Date(timestamp).toISOString().slice(0, 10)
    : "(missing-or-invalid)";
}

function summarize(input: Article[], daily: boolean) {
  const groups = new Map<string, {
    domain: string;
    period: string;
    emitted: number;
    eligible: number;
    outcomes: Record<string, number>;
  }>();

  for (const row of input) {
    const domain = row.domain ?? "(missing)";
    const period = daily ? publicationDay(row.publishedAt) : "(all)";
    const key = JSON.stringify([domain, period]);
    const group = groups.get(key) ?? {
      domain, period, emitted: 0, eligible: 0, outcomes: {},
    };

    group.emitted += 1;
    group.eligible += Number(
      row.fullTextStatus === "ok" && Boolean(row.fullText?.trim())
    );
    const status = outcome(row);
    group.outcomes[status] = (group.outcomes[status] ?? 0) + 1;
    groups.set(key, group);
  }

  return [...groups.values()];
}

const coverageByPublicationDay = summarize(rows, true);

Here, “eligible” is your scoring policy: a substantial body with nonblank text. It isn't an additional Actor status.

Use publication time for the coverage period. Keep scrapedAt for collection provenance. A rerun's collection time shouldn't move an older article into a different editorial period.

This denominator measures emitted articles, not every article published about your topic. Date filtering, deduplication, feed availability, and collection limits operate before your scoring decision.

Make the scoring basis travel with the result

ok indicates a substantial extracted body. It doesn't promise complete publisher coverage, nor does it establish that every available word was extracted.

Choose an explicit policy for short. For a body-based trend, exclusion is easier to interpret than silently mixing previews with substantial articles.

This creates a decision record for every article, including exclusions.

const scoringDecisions = rows.map((row) => {
  const eligible =
    row.fullTextStatus === "ok" && Boolean(row.fullText?.trim());

  return {
    url: row.url,
    domain: row.domain ?? null,
    publishedAt: row.publishedAt ?? null,
    scrapedAt: row.scrapedAt ?? null,
    extractionStatus: row.fullTextStatus ?? null,
    textBasis: eligible ? "body" : null,
    eligible,
    text: eligible ? row.fullText! : null,
  };
});

These are application-owned records. Store them beside your eventual scores in Postgres, carrying the run identifier and model version in your own ingestion envelope.

If you also need headline or preview sentiment, give each its own basis and series. A fallback can keep a workflow moving while quietly changing the question your chart answers.

The branch should remain visible.

emitted article
  ├─ substantial body → body sentiment + provenance
  └─ other outcome    → exclusion reason + coverage audit

Compare publisher weights before interpreting sentiment

A publisher's share of scored articles can change even when its share of collected articles doesn't.

That can move aggregate sentiment without any change in the sentiment model. Check this possibility before explaining the movement as a change in coverage tone.

This optional comparison reads a historical dataset and computes both publisher composition and cap exposure.

function profile(input: Article[]) {
  const groups = summarize(input, false);
  const eligibleTotal = groups.reduce((sum, group) => sum + group.eligible, 0);
  const skipped = input.filter(
    (row) => row.fullTextStatus === "skipped_cap"
  ).length;

  return {
    emitted: input.length,
    capShare: input.length ? skipped / input.length : null,
    bodyCoverage: input.length ? eligibleTotal / input.length : null,
    publishers: groups.map((group) => ({
      ...group,
      scoredShare: eligibleTotal ? group.eligible / eligibleTotal : null,
    })),
  };
}

const previousDatasetId = process.env.PREVIOUS_DATASET_ID;
const previousRows = previousDatasetId
  ? z.array(ArticleSchema).parse(
      (await client.dataset(previousDatasetId).listItems()).items
    )
  : undefined;

const comparison = previousRows
  ? { previous: profile(previousRows), current: profile(rows) }
  : null;

An absent baseline produces no comparison, rather than a reassuring comparison against the current run itself.

Use complete dataset reads for these rates. Keep query scope, edition, publication windows, extraction policy, and model settings available when assessing comparability.

The worked example here's Google News Keyword & Full-Text Article Scraper. The reusable pattern is preserving selection outcomes beside analytical results.

When publisher weights move, calculate a companion trend using publishers represented in both periods and a consistent weighting policy. Label it separately.

That comparison reduces a composition effect. It doesn't prove that the underlying articles are equivalent.

Spend the next request on the right uncertainty

Raise the extraction budget when skipped records explain the coverage loss and broader fetching serves the report. The cost is additional publisher requests and platform usage.

For a deliberate cohort, select URLs and submit them through articleUrls mode. It always attempts extraction, but the run-wide budget still applies. A separate run also brings another start charge and charges for emitted results.

Keep selection rules explicit. Choosing articles after reading their apparent sentiment would introduce another source of bias.

For paywalls, accept the unavailable body. The Actor reads publicly served content and doesn't bypass access walls. A preview or blocked response can still leave a publisher underrepresented.

For request failures, retry selectively with a budget. A timeout is a retry candidate, not evidence that retrying will recover the page.

The uncomfortable catch: the capped subset isn't established as a random sample. Output order isn't a sampling guarantee. Even stable publisher weights can hide changes in topics or article types within each publisher.

Before you send the report

• Confirm the finished run and retain its submitted input.
• Count every status, including missing and unfamiliar values.
• Compare body coverage and publisher composition across periods.
• Keep exclusions and text basis beside scoring results.
• Retry request failures separately from access refusals.
• Label any companion trend built from a comparable cohort.

A higher sentiment score is a finding. A different set of readable articles is a different explanation.