Building a production-grade data pipeline around Google News requires looking past the happy path documented in a tool's README. When you integrate a scraper into your system, you are not just querying an API: you are orchestrating browser instances, resolving complex redirect chains, and attempting to parse arbitrary HTML from thousands of unique publisher domains.
By analyzing the input and output schemas of the google-news-scraper alongside the architectural constraints of the Apify platform, we can identify exactly where this integration will break. Checked against the Actor's input schema and Apify docs on 2026-09-09, this guide outlines the silent failure modes of this scraper and provides the defensive Python code needed to handle them.
How do you prevent synchronous timeout errors during deep scrapes?
To prevent synchronous timeout errors, you must bypass Apify's synchronous run endpoint when requesting full-text extraction or large result sets. This endpoint carries a hard cap of 300 seconds, and if your scrape takes longer than 5 minutes, the platform cuts the connection and returns an HTTP 408 error, leaving your calling application without data.
When you configure the input schema with extractFullText set to true or includeImages set to true, the scraper initiates a headless Playwright browser instance to render each individual article page. If you set maxResultsPerQuery to a high number, the scraper must resolve redirects from Google News, spin up browser pages, load the target publisher sites, and extract the text. Even with a high maxConcurrency setting, this process can exceed 5 minutes.
To bypass this limit, you must trigger the run asynchronously. Instead of waiting for a synchronous HTTP response, your code POSTs the input configuration to the asynchronous runs endpoint, receives a run ID, and polls the run status.
Here is a defensive Python implementation using the official apify-client library to start an asynchronous run, monitor its status, and handle timeouts gracefully:
import time
from apify_client import ApifyClient
def run_news_scraper_safely(
queries: list,
max_results: int = 20,
extract_text: bool = True,
timeout_seconds: int = 600
) -> list:
# Always pass an explicit input dict via API. Apify's Console UI prefill
# is ignored by API calls; only schema defaults are applied automatically.
run_input = {
"queries": queries,
"maxResultsPerQuery": max_results,
"language": "en",
"country": "US",
"extractFullText": extract_text,
"includeImages": extract_text,
"maxConcurrency": 5
}
client = ApifyClient("YOUR_APIFY_API_TOKEN")
# Start the Actor asynchronously. This returns immediately with a run object.
run = client.actor("crawlerbros/google-news-scraper").call(
run_input=run_input,
wait_secs=0 # 0 makes the call asynchronous
)
run_id = run["id"]
start_time = time.time()
while True:
elapsed = time.time() - start_time
if elapsed > timeout_seconds:
raise TimeoutError(f"Scraper run {run_id} exceeded client-side limit of {timeout_seconds}s")
current_run = client.run(run_id).get()
status = current_run.get("status")
if status == "SUCCEEDED":
dataset_id = current_run["defaultDatasetId"]
dataset_items = client.dataset(dataset_id).list_items().items
return dataset_items
elif status in ["FAILED", "ABORTED", "TIMED-OUT"]:
raise RuntimeError(f"Scraper run {run_id} terminated with status: {status}")
time.sleep(10)
Why does my API call ignore my input prefill configurations?
Your API calls ignore input prefill configurations because Apify only applies prefill values within its Console UI. If you configure settings in the Apify Console and expect those same parameters to apply when you trigger the scraper via curl or an SDK without passing an explicit payload, the platform falls back to the schema's raw defaults.
To ensure your production pipeline behaves exactly like your UI tests, you must treat the input schema as a strict blueprint and construct your JSON payload programmatically. This ensures that parameter overrides like custom date ranges are explicitly sent in every request.
Below is an example of a fully defined input configuration that overrides the default behaviors, targeting specific dates and excluding noise words:
{
"queries": ["renewable energy", "solar power"],
"maxResultsPerQuery": 30,
"language": "en",
"country": "US",
"dateFrom": "2026-08-01",
"dateTo": "2026-08-31",
"excludeWords": ["opinion", "editorial", "sponsored"],
"extractFullText": true,
"includeImages": false,
"maxConcurrency": 8
}
How do you handle sentinel values and nulls in the output schema?
You handle sentinel values and nulls in the output schema by implementing strict validation logic on the returned dataset. When you scrape news, you cannot assume every field will be populated. The schema allows several optional properties to return as null or empty strings depending on the publisher's site architecture, paywalls, or your input settings.
Specifically, if extractFullText or includeImages are set to false, the fields fullText, imageUrl, and author will explicitly return as null. Even if these options are enabled, paywalled articles (such as those from premium publishers) will block the Playwright scraper, resulting in a null value for fullText. Additionally, some publisher sites do not use open-graph tags, causing imageUrl to remain empty, or they omit author names entirely.
If your downstream consumption pipeline expects structured data, your ingestion code must validate the presence of these fields before processing.
Here is a Python function designed to parse the raw dataset items and flag incomplete or paywalled records:
def process_and_validate_articles(dataset_items: list) -> list:
valid_articles = []
for item in dataset_items:
# Mandatory fields that should always exist
url = item.get("url")
title = item.get("title")
domain = item.get("domain")
if not url or not title:
# Skip corrupted or completely unparsed items
continue
# Check for paywalls or extraction failures
full_text = item.get("fullText")
has_full_text = isinstance(full_text, str) and len(full_text.strip()) > 0
# Check for author presence
author = item.get("author")
has_author = isinstance(author, str) and author.strip() != ""
# Construct clean payload for downstream consumer
cleaned_record = {
"query": item.get("query"),
"title": title,
"url": url,
"domain": domain,
"published_at": item.get("publishedAt"),
"snippet": item.get("snippet", ""),
"full_text": full_text if has_full_text else None,
"image_url": item.get("imageUrl"),
"author": author if has_author else "Unknown",
"is_paywalled_or_empty": not has_full_text,
"scraped_at": item.get("scrapedAt")
}
valid_articles.append(cleaned_record)
return valid_articles
How do you export scraped news data to long-term storage?
To save your news data to long-term storage, you must export the dataset immediately upon run completion. Unnamed storages on the platform are temporary, meaning they will eventually expire and be deleted automatically.
By fetching the dataset items directly using the Apify API client and sending them to your own database or web service, you guarantee that your data is preserved indefinitely. This also keeps your downstream workflows decoupled from Apify's internal storage lifecycles.
Here is a Python script that executes the Actor, waits for its completion, and pushes the resulting items to an external API endpoint:
import requests
from apify_client import ApifyClient
def archive_scraped_news(queries: list, archive_endpoint: str):
client = ApifyClient("YOUR_APIFY_API_TOKEN")
try:
run = client.actor("crawlerbros/google-news-scraper").call(
run_input={
"queries": queries,
"maxResultsPerQuery": 10,
"extractFullText": False
},
wait_secs=250 # Keep it under the 300s platform sync cap
)
if run.get("status") == "SUCCEEDED":
dataset_id = run["defaultDatasetId"]
items = client.dataset(dataset_id).list_items().items
response = requests.post(
archive_endpoint,
json={"query_group": queries, "articles": items},
headers={"Content-Type": "application/json"}
)
response.raise_for_status()
print(f"Successfully archived {len(items)} articles.")
except Exception as e:
print(f"Data archiving failed: {e}")
How do you safely automate runs on a recurring schedule?
To automate runs on a recurring schedule, you must create a Platform Schedule and ensure that the target Actor has been executed manually at least once. Platform rules dictate that schedules cannot be established for an Actor that has never run.
Additionally, new schedules are created in a disabled state by default, so you must explicitly enable them in the console or via the API. Because a request queue can only be processed by a single Actor run at one time, scheduled tasks must be configured to run sequentially or have isolated dataset environments to avoid conflicts.
To configure your automated tasks programmatically, you should pass a fully structured JSON payload. Below is the structured representation of a single scraped news article as it appears when written to the default dataset:
{
"query": "artificial intelligence",
"title": "AI Achieves New Milestone in Scientific Research",
"url": "https://www.nature.com/articles/d41586-026-00123-4",
"source": "Nature",
"domain": "nature.com",
"publishedAt": "2026-03-03T09:15:00+00:00",
"snippet": "Researchers at MIT report a new breakthrough in machine learning...",
"fullText": "Full article body text here...",
"imageUrl": "https://nature.com/images/article-hero.jpg",
"author": "Dr. Jane Smith",
"language": "en",
"scrapedAt": "2026-03-03T12:00:00.000000+00:00"
}
Google News Scraper limitations and failure modes
The google-news-scraper has built-in limitations that you must design around. The most significant of these is the paywall barrier. If a news site requires a subscription or blocks scraper traffic behind an aggressive login wall, the underlying extraction library (trafilatura) cannot read the raw article HTML. It will return null for the full text, even though the primary metadata (headline, publication date, and source) was successfully collected from Google News.
Second, the system is constrained by concurrency. The input field maxConcurrency controls the number of parallel Playwright pages running on your container. Setting this value too high on a low-memory container will cause the container to run out of memory and crash, terminating the run before writing to the dataset.
Finally, the scraper automatically deduplicates results across queries within a single run. If you run a query for "AI breakthroughs" and "artificial intelligence" in the same execution, articles that appear in both result sets are returned only once. If your downstream database relies on a strict mapping of queries to articles for keyword tracking, you will miss relationships because the second query's record was discarded during the deduplication phase.
What is the true cost of running this scraper at scale?
To calculate the cost of running this scraper, we must look at the platform's documented compute unit (CU) formula and pricing tiers. A compute unit is defined as CU = (memory_mb / 1024) * duration_hours. Because this scraper launches Playwright browser instances to retrieve full texts and images, it requires a larger memory footprint than a simple HTTP parser.
On the Free plan ($0) and Starter plan ($19/mo), Apify charges $0.20 per CU. On the Scale plan ($199/mo), the rate drops to $0.16 per CU, and on the Business plan ($999/mo), it is $0.13 per CU.
If you run the scraper without full-text extraction (extractFullText: false), the memory consumption is minimal because the Actor only processes Google News search result pages. It runs quickly and consumes little memory. However, enabling extractFullText or includeImages forces the container to launch Playwright, loading heavy third-party sites with advertisements and tracking scripts. Under this configuration, the memory must be scaled up, and the run duration increases.
Additionally, if you use proxies, you must factor in data transfer costs. On Free and Starter plans, residential proxies cost $8/GB. On the Scale plan, the price is $7.50/GB, and on the Business plan, it is $7/GB. Because the scraper README notes that "no proxy is needed" to pull the initial list from Google News, you can avoid this cost entirely by using the platform's default network settings. However, if you choose to route your Playwright pages through residential proxies to bypass publisher blocks, loading heavy news pages with images will consume gigabytes of bandwidth.
To control this financial risk, you should always pass the maxTotalChargeUsd query parameter when triggering runs via the API. This is exposed to the Actor code as ACTOR_MAX_TOTAL_CHARGE_USD. When the specified cost threshold is reached, the platform stops processing further requests, protecting you from unexpected bills.
Here is how you can defensively trigger a run with a hard spending limit parameter using curl:
curl --request POST \
--url "https://api.apify.com/v2/acts/crawlerbros~google-news-scraper/runs?token=YOUR_APIFY_API_TOKEN&maxTotalChargeUsd=2.50" \
--header 'content-type: application/json' \
--data '{
"queries": ["quantum computing"],
"maxResultsPerQuery": 50,
"extractFullText": true,
"maxConcurrency": 5
}'
How do you handle storage expiration on the free tier?
If you are running your scrapers on Apify's free tier, you face strict retention policies that can break your historical reporting. On the free plan, Apify retains only the 10 most recent runs, and they are kept for a maximum of 4 months. Any older runs, along with their unnamed datasets, are permanently deleted.
To prevent data loss, your automated pipeline must migrate data off the platform immediately upon execution or use named storages. Named storages are explicitly exempt from automatic deletion, meaning they are never purged under the standard retention limits.
By ensuring your scraper writes to named datasets, or by hosting your own extraction script to pull raw data on a regular cycle, you can protect your data pipeline against accidental data expiration.
How to execute local testing and programmatic verification?
To verify your scraper integration without consuming platform resources unnecessarily during initial staging, you can write integration tests in Python that query the dataset structures. Since the Google News Scraper returns a structured array of articles, you should write a suite that validates schema shapes, tests dates against your input boundaries, and inspects full-text extraction outputs.
Testing programmatically allows you to detect silent extraction failures early. For example, if a major publisher updates its DOM structure and blocks Playwright-based extraction, your test suite should raise alerts if fullText fields on that publisher's domain are consistently returning null.
Below is a complete verification script that asserts schema compliance on a retrieved dataset:
import datetime
def verify_dataset_integrity(dataset_items: list, date_from_str: str = None):
"""
Asserts schema compliance on the returned Google News dataset items.
"""
assert len(dataset_items) > 0, "Verification failed: Dataset is empty"
for idx, item in enumerate(dataset_items):
# Verify base metadata structure
assert "url" in item and item["url"], f"Missing url in item index {idx}"
assert "title" in item and item["title"], f"Missing title in item index {idx}"
assert "domain" in item and item["domain"], f"Missing domain in item index {idx}"
assert "query" in item and item["query"], f"Missing search query in item index {idx}"
# Verify date ranges if a constraint was configured
if date_from_str:
target_date = datetime.datetime.strptime(date_from_str, "%Y-%m-%d").date()
published_at_str = item.get("publishedAt")
if published_at_str:
# ISO date parsing
published_date = datetime.datetime.fromisoformat(
published_at_str.replace("Z", "+00:00")
).date()
assert published_date >= target_date, (
f"Item {idx} published at {published_date} is older than filter {target_date}"
)
print(f"Passed integrity verification on {len(dataset_items)} items.")
What are the limits of concurrent browser instances in headless scraping?
Running headless browser instances is highly resource-intensive and will cause failure if concurrency settings are not carefully configured. The input field maxConcurrency specifies how many Playwright browser pages are opened simultaneously to resolve article redirect links, retrieve raw HTML, and extract body text.
If you scale maxConcurrency to its upper limit of 20 without scaling the container memory size accordingly, the container will run out of memory. This leads to a hard termination of the container process by the platform's kernel before your script can save the output data.
Because memory scales linearly with concurrency when running multiple browser profiles, your pipeline should monitor the memory footprint of the Actor runs and dynamically scale concurrency settings based on the available platform resources. It is typically safer to run at a lower concurrency level of 5 to 10 to ensure stable execution within standard compute environments.
The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com
Top comments (0)