DEV Community: Crawler Bros

Filtering by marketCap and peFilter is Cheaper Than Filtering After the Run

Crawler Bros — Fri, 09 Oct 2026 10:03:19 +0000

Ingesting raw equity screening data into a downstream analytical warehouse frequently creates an expensive pipeline bottleneck. When engineers write scrapers to pull broad lists of public companies, they often collect every listed security across major exchanges and defer filtering—such as valuation multiples or capitalization thresholds—to subsequent transformation layers in dbt, DuckDB, or Snowflake. Pulling thousands of records daily when your ingestion model only targets profitable mid-cap companies inflates network payloads, fills object stores with discarded records, and increases extraction costs.

The Finviz Stock Screener Scraper resolves this by exposing Finviz's native screening engine directly through an extraction API. Rather than writing custom DOM scrapers that break whenever Finviz updates its table layouts, you can apply technical, fundamental, and descriptive filters upfront.

Why Scraping Unfiltered Financial Tables Breaks Down

Scraping Finviz directly through headless browser sessions or fragile HTML parsing scripts presents several operational headaches:

Pagination and Table Volatility: Finviz organizes screener results into paginated HTML tables containing 20 rows per view. Navigating through hundreds of pages requires continuous DOM parsing, session state handling, and dynamic waiting to avoid throttling.
Missing Field Inconsistencies: Financial tables have variable schema density. Growth tech stocks lack price-to-earnings metrics, debt-free firms omit debt-to-equity ratios, and newly listed equities lack 5-year averages. Custom scripts frequently throw parsing errors or output null values that misalign columnar arrays.
Redundant Data Payloads: Extracting all 8,000+ US-listed equities just to extract the 150 that meet a specific valuation profile wastes pipeline bandwidth.

By structuring the extraction request directly around Finviz's server-side filtering parameters, you restrict the payload upstream. If you only care about profitable companies with low debt and positive earnings growth, you can pass those conditions directly to the scraper configuration.

Query Construction with Native Schema Parameters

The actor accepts an extensive list of configuration fields that match Finviz's native screener filters. Instead of extracting everything and querying downstream, you can construct a targeted JSON payload.

Key Input Fields

mode: Sets the operational mode. The default is "screener", but you can also set it to "stockOverview", "newsForTicker", "insiderTrading", or "groupPerformance".
marketCap: Filters by capitalization tier. Options include "mega", "large", "mid", "small", "micro", and "nano".
peFilter: Constrains trailing price-to-earnings ratios using values such as "profitable", "low", "u15", "u20", or "o25".
sector: Limits the universe to one of 11 sectors (for example, "Technology" or "Healthcare").
debtEquityFilter: Sets balance-sheet solvency bounds (such as "u0.5" or "low").
includeExtendedMetrics: A boolean flag. When set to true, the standard screener row is enriched with ~45 additional fundamental and technical data points, including forwardPE, peg, ps, pb, roe, roa, debtEq, sma20, sma50, and rsi.

Finviz omits empty fields from its output, meaning a metric is only present in a returned JSON record if Finviz published a value for it. This preserves sparse financial data without forcing you to handle synthetic null rows.

Step-by-Step Implementation

Setting up an automated run requires defining the input object and executing the actor through an HTTP client or SDK.

1. Define the Run Configuration

Construct a JSON object selecting mid-cap companies with a price-to-earnings ratio under 20 and a debt-to-equity ratio under 0.5 within the Technology sector, returning extended valuation metrics:

{
  "mode": "screener",
  "sector": "Technology",
  "marketCap": "mid",
  "peFilter": "u20",
  "debtEquityFilter": "u0.5",
  "includeExtendedMetrics": true,
  "sortBy": "marketcap",
  "sortDescending": true
}

2. Execute via the Python SDK

You can trigger this job and fetch results directly into your pipeline using the official client library:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run_input = {
    "mode": "screener",
    "sector": "Technology",
    "marketCap": "mid",
    "peFilter": "u20",
    "debtEquityFilter": "u0.5",
    "includeExtendedMetrics": True,
    "sortBy": "marketcap",
    "sortDescending": True
}

run = client.actor("crawlerbros/finviz-scraper").call(run_input=run_input)
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    print(
        item.get("ticker"),
        item.get("company"),
        item.get("marketCap"),
        item.get("peRatio"),
        item.get("roe")
    )

Each record contains the base fields (ticker, company, sector, industry, country, marketCap, peRatio, price, change, volume) along with the extended metrics requested via includeExtendedMetrics.

Output Structure and Schema Details

When running in screener mode with extended metrics enabled, records contain both basic trading data and deep fundamentals:

{
  "ticker": "EXAMPLE",
  "company": "Example Semiconductor Inc.",
  "sector": "Technology",
  "industry": "semiconductors",
  "country": "USA",
  "marketCap": "8.45B",
  "peRatio": "16.42",
  "forwardPE": "14.10",
  "peg": "1.25",
  "ps": "3.12",
  "pb": "2.85",
  "price": "45.20",
  "change": "1.25%",
  "volume": "1,245,800",
  "avgVolume": "1.10M",
  "debtEq": "0.32",
  "roe": "18.40%",
  "roa": "11.20%",
  "rsi": "54.12",
  "sma50": "43.10",
  "finvizUrl": "https://finviz.com/quote.ashx?t=EXAMPLE",
  "recordType": "stock",
  "scrapedAt": "2025-05-15T14:30:00.000Z"
}

If you switch mode to "stockOverview" and supply a specific "ticker", the output pivots to full company fundamentals, adding fields like enterpriseValue, bookPerShare, shortRatio, and historical performance metrics.

Pricing and Cost Dynamics

The actor operates under a PAY_PER_EVENT model:

Each result saved to the default dataset costs $0.005 on the FREE tier ($0.00433 on BRONZE, $0.00367 on SILVER, and $0.003 on GOLD, PLATINUM, and DIAMOND).
An Actor Start event costs $0.005 per GB of memory allocated to the run.
Platform usage for the run is billed separately at your Apify plan's rates.

Applying targeted parameters like marketCap, peFilter, and sector upfront keeps the output focused strictly on qualifying securities. Because each output item incurs a per-event result fee, narrowing the search space on Finviz's servers directly lowers the number of billed result events compared to scraping all listed equities and cleaning them downstream.

What This Tool Does Not Solve

This tool does not provide real-time, low-latency tick data or intraday order book depth; it reflects Finviz's web snapshot and screener updates, making it unsuitable for high-frequency trading execution pipelines.

For batch workflows, daily quantitative screens, and automated market monitoring, configuring filters natively before extraction provides a structured data stream without the overhead of maintaining local parsing scripts. An open question remains whether to run separate focused screens per sector or to merge them into a single cross-market pipeline.

Finviz Stock Screener Scraper is the Actor behind these examples. If a selector in your own version breaks, compare your output against the fields listed in its README first.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-09. Check the Actor page for the current rates.

Why TikTok Ads Library Scraper Pro Returns Empty Media URLs

Crawler Bros — Fri, 09 Oct 2026 07:49:04 +0000

As data engineers, we often integrate with third-party tools and APIs, navigating their stated capabilities and their undocumented constraints. The real skill isn't just knowing what a tool does, but understanding how it fails, and how to build defensively against those failure modes. This isn't always obvious from product overviews or simple "getting started" guides.

Let's look at a specific example: the TikTok Ads Library Scraper Pro on Apify. This tool, found at https://apify.com/crawlerbros/tiktok-ads-library-scraper-pro, provides access to TikTok's public ad transparency library. While its functionality for scraping ad data by query, advertiser, region, and date range is well-documented, the subtle implications of its input schema and operational environment tell a deeper story about where things can go wrong.

What Input Schema Constraints Mean for Data Completeness?

The tiktok-ads-library-scraper-pro input schema contains implicit contracts about what combinations of inputs will yield complete data, and what will result in partial or empty fields. Consider the quickSearch boolean field. When true, it explicitly skips "ad-detail fetch and only return data from the listing card." The README further clarifies that this omission includes "full targeting breakdowns."

This means if your downstream application expects fields like targetingByLocation, ageBuckets, or genderSelection, setting quickSearch to true will lead to missing data for those specific fields. This isn't a failure in the traditional sense, but a deliberate design choice with consequences for data completeness.

To avoid this, always explicitly set quickSearch: false if your data model requires the full targeting breakdown. If you're building a generalized data pipeline, you might need a conditional branch to handle the two different output shapes, or ensure all runs that feed into a particular downstream consumer consistently request full details.

{
  "query": "sustainability",
  "region": "all",
  "quickSearch": false,
  "maxItems": 100
}

In the above input, quickSearch: false signals the intent to retrieve full targeting breakdowns. Neglecting this could silently introduce None or empty values for detail-rich fields, leading to breaks in downstream analytics or reporting that assumes their presence.

Why Does TikTok Ads Library Scraper Pro Sometimes Return Empty Media URLs?

A common scenario where mediaUrl might be None or an empty string, despite the ad appearing in the results, is when an ad's compliance status changes. The FAQ states, "Ads with auditStatus: rejected or removed are still listed in the transparency database but have their creative assets stripped." This means you can get valid adId, adText, and targeting data, but no actual creative media to download.

This behavior implies a need for defensive programming when consuming the mediaUrl field. Instead of assuming its presence, always check for null or empty values before attempting to process or download the media. Your code should explicitly handle auditStatus values that indicate the media might be unavailable, perhaps by logging the adId for manual review or by skipping media-dependent processing steps.

Here's a Python snippet demonstrating how to detect and handle this in your data processing:

import apify_client

# Initialize the Apify client
client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

# Assume we have a run ID or directly fetch dataset items
# This example fetches the latest run's dataset
actor_id = "crawlerbros/tiktok-ads-library-scraper-pro"
run = client.actor(actor_id).call(run_input={
    "query": "shoes",
    "quickSearch": false,
    "maxItems": 20
})

# Access the dataset and iterate items
dataset = client.dataset(run["defaultDatasetId"])
for item in dataset.iterate_items():
    ad_id = item.get("adId")
    media_url = item.get("mediaUrl")
    audit_status = item.get("auditStatus")

    if audit_status in ["rejected", "removed"] and not media_url:
        print(f"Ad {ad_id}: Media URL is empty due to audit status '{audit_status}'. Skipping media download.")
    elif media_url:
        print(f"Ad {ad_id}: Media URL available: {media_url}")
        # Proceed with media download or processing
    else:
        print(f"Ad {ad_id}: Media URL is unexpectedly empty. Audit status: {audit_status}")

This approach safeguards your pipeline against unexpected None values for media and allows you to build logic around the known reasons for their absence.

How Do Proxy Limits Affect Data Collection Stability?

The useApifyProxy input parameter, defaulting to true, implies that network stability and rate limiting are real concerns when interacting with TikTok's transparency library. The description for this field explicitly states, "Recommended on cloud, avoids IP rate-limiting. The actor automatically rotates the proxy session on errors." While robust, proxy sessions have finite lifespans: datacenter proxies persist for around 26 hours, while residential proxies last approximately 30 minutes.

For long-running or frequently scheduled data collection jobs, a prolonged run could theoretically exhaust the stability of a single proxy session, even with automatic rotation. If a run attempts to use a stale session for too long without successful rotation, it could encounter prolonged rate limiting or connection errors, leading to incomplete datasets or failed runs.

While the actor handles rotation, understanding this underlying mechanism is crucial for diagnosing inexplicable connection issues on very long runs or when observing performance degradation over extended periods. For most typical runs, the default proxy handling is sufficient. However, if you are collecting millions of records over many hours, monitor run logs for frequent proxy rotation messages or networking errors, which could indicate stress on the proxy pool.

# Example of explicitly enabling proxy usage, though it's the default
# If you were to override with a custom proxy (not recommended unless
# you have a specific, known-good proxy setup), you'd define proxyConfiguration
# For general use, rely on the default `useApifyProxy: true`.
run_input = {
    "query": "fashion",
    "region": "all",
    "quickSearch": false,
    "useApifyProxy": True, # Explicitly stating the default for clarity
    "maxItems": 5000
}

This configuration is generally robust. However, if your data pipeline involves very high-volume, continuous scraping against TikTok's ad library, be aware that even managed proxy services have underlying session durations. If a run lasts for days, you might consider splitting it into smaller, sequential runs or monitoring for proxy-related warnings.

When Does a Run Exceed the Synchronous 300-Second Cap?

When calling an Actor via the Apify platform's synchronous POST /v2/acts/<actor_id>/run endpoint, there's a hard timeout of 300 seconds (5 minutes). If the Actor run doesn't complete and return its results within this window, the API call will return an HTTP 408 error. This doesn't mean the Actor itself stops; it continues running in the background, but your client application loses its synchronous connection to the result.

For tiktok-ads-library-scraper-pro, several input parameters can increase run duration, making it prone to hitting this cap:

maxItems: A large number of items can significantly extend the run time, especially when quickSearch is false (which requires an additional request per ad).
quickSearch: false: Each ad requires a separate detail page fetch, multiplying the request count and thus the duration.
requestDelaySecs: Increasing this value (the pause between successive API calls) directly extends run time, though it reduces rate-limit risk.

If your application needs to retrieve more than a few hundred items with quickSearch: false, or if you're using a higher requestDelaySecs, you should use the asynchronous POST /v2/acts/<actor>/runs endpoint. This allows you to start the run, get an immediate runId, and then poll for its completion or use webhooks.

import apify_client

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")
actor_id = "crawlerbros/tiktok-ads-library-scraper-pro"

# This run is likely to exceed 300 seconds if maxItems is large and quickSearch is false
run_input = {
    "query": "sustainable fashion",
    "quickSearch": False, # Each ad needs a detail page fetch
    "maxItems": 5000,     # Many items
    "requestDelaySecs": 2 # Slightly slower requests
}

# Start the run asynchronously
# Note: client.actor(actor_id).call() defaults to asynchronous if needed,
# but it's good practice to understand the underlying mechanism for long runs.
run = client.actor(actor_id).call(run_input=run_input)
print(f"Run started with ID: {run['id']}")

# Now, poll for the run's status
run_id = run["id"]
while True:
    run_info = client.run(run_id).get()
    if run_info["status"] == "SUCCEEDED":
        print(f"Run {run_id} finished successfully.")
        break
    elif run_info["status"] in ["FAILED", "ABORTED"]:
        print(f"Run {run_id} failed or was aborted.")
        break
    print(f"Run {run_id} is still {run_info['status']}... Waiting.")
    # In a real application, implement a sensible sleep/retry strategy
    import time
    time.sleep(10)

# Once finished, you can access the dataset
dataset = client.dataset(run_info["defaultDatasetId"])
# ... process data ...

This asynchronous approach prevents your client from timing out while the Actor diligently scrapes in the background. It's a fundamental pattern for any data extraction job that might take longer than a few minutes.

What Are the Cost Implications of Input Parameters?

Understanding the cost structure of tiktok-ads-library-scraper-pro requires looking at its "PAY_PER_EVENT" model. The Actor's primary charge event is "result" (apify-default-dataset-item), costing $0.002 per event. There's also an "Actor Start" event, billed at $0.005 per GB of memory allocated to the run, which is a flat per-event charge for run initiation. On top of these, you also pay for Apify platform usage, billed separately.

The maxItems input field directly multiplies the number of "result" events. Each ad emitted into the default dataset incurs this per-item cost. Therefore, setting maxItems to a high number will proportionally increase the event-based cost.

Similarly, quickSearch: false effectively doubles the amount of work the Actor does per ad, by making an additional detail page request. While the pricing model doesn't explicitly state a separate charge for "detail page fetches," it is reasonable to infer that the increased complexity and resources consumed by quickSearch: false contribute to the overall platform usage, even if the "result" event count remains tied to maxItems.

Consider the tiers for the "result" event: FREE $0.002, BRONZE $0.00167, SILVER $0.00133, GOLD $0.001, PLATINUM $0.001, DIAMOND $0.001. These indicate that unit costs decrease for users in higher Apify tiers. However, the event count itself is driven by your input parameters.

To manage costs effectively:

Use maxItems conservatively, especially during development and testing.
Only set quickSearch: false when you genuinely need the full targeting breakdown, as it increases resource consumption and thus platform usage.
Monitor your run's item count in the Apify Console.

# A simple representation of how input choices affect event counts
def estimate_event_cost(max_items, quick_search_enabled):
    # This is a simplified model for illustration.
    # Actual platform usage adds to this, and specific API behavior might vary.
    result_event_price = 0.002 # Using the FREE tier price for example
    actor_start_cost_per_gb = 0.005 # per GB

    results_cost = max_items * result_event_price
    # Assume 1GB memory for simplicity, so one Actor Start event.
    start_cost = 1 * actor_start_cost_per_gb

    # If quickSearch is false, it uses more resources, increasing platform usage,
    # though not necessarily directly more "result" events.
    # The README implies resource consumption, which is tied to platform usage.
    if not quick_search_enabled:
        print("Note: quickSearch: false increases resource consumption (platform usage).")

    total_event_cost = results_cost + start_cost
    return total_event_cost

# Example cost estimation:
# Scenario 1: Quick search, few items
print(f"Estimated event cost for 50 items (quickSearch=true): ${estimate_event_cost(50, True):.4f}")

# Scenario 2: Detailed search, more items
print(f"Estimated event cost for 1000 items (quickSearch=false): ${estimate_event_cost(1000, False):.4f}")

This demonstrates how maxItems directly drives result event costs, while quickSearch impacts the resource-based platform usage component of the total bill.

Can a Single Request Queue Fan-Out to Multiple Runs?

A common pattern for distributing workloads is to have multiple workers pull from a single queue. However, Apify's RequestQueue has a specific constraint: "A request queue can only be PROCESSED by one Actor or task run at a time, though multiple runs may add to it." This means you cannot fan out processing from a single shared RequestQueue across multiple concurrent tiktok-ads-library-scraper-pro runs.

If your use case requires processing segments of a large ad list concurrently (e.g., scraping specific ad IDs across different regions in parallel), you would need to:

Divide your adIds list into separate chunks.
Start distinct tiktok-ads-library-scraper-pro runs, each with its own segment of adIds in the input. Each run would then manage its own implicit RequestQueue internally.
Collect results from each run's separate default dataset.

Attempting to configure multiple Actor runs to pull from the same explicit RequestQueue for processing will lead to runs waiting for exclusive access, negating the benefits of parallelization.

# Example of splitting ad IDs for parallel runs (conceptual Python)
def trigger_parallel_ad_id_scrapes(ad_ids_list, client):
    chunk_size = 100 # Example chunk size
    chunks = [ad_ids_list[i:i + chunk_size] for i in range(0, len(ad_ids_list), chunk_size)]

    run_ids = []
    for i, chunk in enumerate(chunks):
        run_input = {
            "adIds": chunk,
            "quickSearch": False,
            "region": "all" # Or make this dynamic per chunk
        }
        print(f"Starting run {i+1} with {len(chunk)} ad IDs.")
        run = client.actor("crawlerbros/tiktok-ads-library-scraper-pro").call(run_input=run_input)
        run_ids.append(run["id"])
    return run_ids

# Example usage:
# all_ad_ids = ["id1", "id2", ..., "idN"] # Imagine a large list
# client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")
# started_runs = trigger_parallel_ad_id_scrapes(all_ad_ids, client)
# print(f"Started {len(started_runs)} parallel runs: {started_runs}")
# You would then monitor and collect results from each of these runs independently.

This pattern ensures each run operates with its own distinct input, effectively circumventing the single-processor constraint of a shared RequestQueue.

What Happens to Unnamed Storages and How Does It Affect Data Retention?

When you initiate a run of tiktok-ads-library-scraper-pro without explicitly creating and naming a Dataset or KeyValueStore, the platform uses unnamed storages. A critical limitation for data retention is that "unnamed storages expire." Specifically, on the Free plan, "only the 10 most recent runs are retained, for 4 months." After this, your data can be deleted.

For any production-grade data pipeline built around tiktok-ads-library-scraper-pro, you absolutely must ensure your datasets are explicitly named or immediately fetched and stored elsewhere. If you rely on the default, unnamed datasets, you risk losing valuable historical ad data, especially if you run the Actor frequently (pushing older runs out of the "10 most recent" window) or if you need data older than four months.

This is particularly important for competitive intelligence, compliance monitoring, or market sizing use cases, where historical trend analysis is key.

import apify_client
import json

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")
actor_id = "crawlerbros/tiktok-ads-library-scraper-pro"

# Define a named dataset for long-term storage
dataset_name = "tiktok-ads-library-scraper-pro-my-project-ads-2023-10-09"
dataset = client.datasets().get_or_create(name=dataset_name)

run_input = {
    "query": "electric vehicles",
    "region": "all",
    "quickSearch": false,
    "maxItems": 500
}

# When calling the Actor, specify the output dataset to be the named one
# The ApifyClient library handles passing this internally when you call the actor
run = client.actor(actor_id).call(run_input=run_input, dataset_id=dataset["id"])

print(f"Run started. Results will be stored in named dataset: {dataset['name']} (ID: {dataset['id']})")
print(f"Run ID: {run['id']}")

# After the run completes, you can fetch from the named dataset
# For simplicity, we'll just show fetching from the named dataset
# In a real scenario, you'd wait for the run to complete as shown in the async example
# items = client.dataset(dataset["id"]).iterate_items()
# for item in items:
#     print(json.dumps(item, indent=2))

By explicitly creating and linking to a named dataset, you guarantee data persistence, regardless of how many subsequent runs occur or how old the data becomes. This is a fundamental safeguard against accidental data loss.

Limitations and Caveats

While tiktok-ads-library-scraper-pro is a powerful tool, it operates within specific boundaries that developers must acknowledge:

Region Coverage is Limited: The transparency library, by design, covers EU/EEA countries, UK, Switzerland, and Turkey due to DSA mandates. If your use case requires data from other global regions (e.g., US, Canada, APAC), this Actor will not fulfill that requirement. The region input field explicitly lists these supported regions, and attempting to specify an unsupported region will likely yield no results or an error.
Impression Data is Bucketed: TikTok provides impression data in ranges like 1K-10K or 1M-10M, not exact counts. The minImpressions Pro filter operates on these parsed numeric values. This means granular impression analysis requiring precise numbers is not possible; your metrics will need to adapt to these bucketed ranges.
No Native AWS S3 or Slack Integration: For integrating results into broader data ecosystems, remember that the Apify platform does not offer native AWS S3 or Slack integrations. You will need to route data through webhooks, or use external tools like n8n, Make, or Zapier to connect the Actor's output to these services. This adds a layer of orchestration if your existing stack heavily relies on these services.
Schedules are Disabled by Default: If you plan to automate tiktok-ads-library-scraper-pro runs with schedules, be aware that new schedules are created in a DISABLED state. They also require the Actor to have run at least once before they can be scheduled. This means an extra step of enabling the schedule and an initial manual (or API-triggered) run are necessary during setup.

These points highlight areas where the Actor's capabilities are constrained by its source data or platform design, requiring forethought in your integration strategy.

Checked against the Actor's input schema and Apify docs on 2026-10-09.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Automating PR Newswire Ingestion Without Silent Date-Filtering Drops

Crawler Bros — Thu, 08 Oct 2026 19:01:57 +0000

Ingesting corporate press releases into an analytical data pipeline often sounds simple until you hit the nuances of PR Newswire's web interface. If you attempt to automate query searches using native URL parameters like startdate and enddate, the PR Newswire server silently ignores them. A search request intended to fetch historical releases from six months ago still returns today's headlines at the top of the feed.

When building downstream pipelines for investor relations monitoring or competitive intelligence, silent query drops like this break ingestion logic. You end up pulling redundant records, missing older announcements, or having to scrape untrusted HTML elements. Solving this requires programmatic pagination alongside deterministic client-side filtering.

The PR Newswire Scraper actor addresses these specific edge cases by standardizing access across feeds, organization newsrooms, and article pages.

Understanding the Ingestion Modes

PR Newswire structures data across several formats: search listing pages, general RSS feeds, dedicated company newsrooms, and full-article detail pages. The actor exposes five operational behaviors via the mode parameter:

search: Executes free-text search queries across headlines and body text via paginated HTML results.
byCategory: Filters PR Newswire's live general RSS feed against 16 specific taxonomy slugs (such as business-technology, health, or financial-services).
byOrganization: Directly paginates an issuing entity's dedicated newsroom page (for example, prnewswire.com/news/<slug>/), allowing deeper historical collection for a single company than general search indices typically keep accessible.
latest: Pulls the newest releases across all categories from the live RSS feed, continuing into paginated HTML listing pages once the standard ~20 RSS items are exhausted.
byUrls: Accepts a list of exact press release URLs and extracts full-page content, including full body text, dateline locations, media contact details, and mentioned stock tickers.

For financial signals and entity extraction, combining a discovery mode (search or byOrganization) with byUrls provides a clean two-step extraction path.

How Client-Side Date and Language Filtering Works

Because PR Newswire does not respect server-side date range limits in search queries, the actor applies client-side filtering using the dateFrom and dateTo inputs (formatted as YYYY-MM-DD).

When running in search mode, the actor extracts each card's displayed Eastern Time, normalizes it to an ISO 8601 UTC timestamp (publishedAt), and compares it directly against the requested window. Records falling outside the bounds are discarded before emission. The same logic applies to the language parameter (such as en, de, or fr), verifying the language code tag prior to persisting output.

To minimize redundant HTTP requests during pagination, configure the pageSize parameter. Setting pageSize to "100" instead of the default "25" reduces network overhead when scanning backward toward your dateFrom boundary.

Here is an example payload configured for competitor monitoring:

{
  "mode": "search",
  "searchQuery": "data pipeline",
  "pageSize": "100",
  "language": "en",
  "dateFrom": "2024-01-01",
  "dateTo": "2024-03-31",
  "maxItems": 100
}

Running an Ingestion Run via Python

The Apify API allows you to trigger runs and pull results directly into your data warehouse or processing scripts. The following script demonstrates fetching company-specific releases using byOrganization and passing their links to extract full body text and stock tickers.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Step 1: Query a specific organization's newsroom
newsroom_run = client.actor("crawlerbros/pr-newswire-scraper").call(
    run_input={
        "mode": "byOrganization",
        "organizationSlug": "microsoft-corporation",
        "maxItems": 10,
        "pageSize": "50"
    }
)

dataset_items = client.dataset(newsroom_run["defaultDatasetId"]).list_items().items
target_urls = [item["url"] for item in dataset_items if "url" in item]

# Step 2: Fetch full article text and structured metadata
detail_run = client.actor("crawlerbros/pr-newswire-scraper").call(
    run_input={
        "mode": "byUrls",
        "urls": target_urls
    }
)

for release in client.dataset(detail_run["defaultDatasetId"]).iterate_items():
    title = release.get("title")
    tickers = release.get("tickerSymbols", [])
    body = release.get("bodyText", "")
    print(f"Title: {title}")
    print(f"Tickers found: {tickers}")
    print(f"Body length: {len(body)} chars\n")

The output for mode=byUrls populates fields that listing pages do not provide. These include sourceOrganization, datelineLocation, mediaContactEmail, and tickerSymbols (formatted as EXCHANGE:SYMBOL, such as NASDAQ:MSFT).

Actor Execution Costs

This actor operates on a pay-per-event pricing model alongside standard platform usage. Charges break down into explicit platform events:

Actor Start: A flat charge of $0.005 per GB of memory allocated to the run.
Dataset Result: Charged per emitted record saved to the dataset. On the FREE tier, this is $0.005 per result. Users on Apify discount tiers pay adjusted per-event prices: BRONZE is $0.00433, SILVER is $0.00367, and GOLD, PLATINUM, or DIAMOND tiers pay $0.003 per result.

In addition to event charges, runs incur Apify platform usage billed at your account plan's standard rates. Total run expense is not calculated from event charges alone.

Limitations and Operational Caveats

One functional limitation involves mode=byCategory. PR Newswire discontinued upstream category-specific RSS endpoints, meaning all category requests now read from a shared ~20-item firehose feed filtered downstream by industry metadata. If you query a narrow industry slug like multicultural or heavy-industry-manufacturing during a quiet publishing window, the run may legitimately return fewer than 5 items, or even 0.

For reliable historical analysis across an entire vertical, do not rely on byCategory; use mode=search with explicit industry keywords and date parameters instead.

If you want to reproduce this, the Actor is PR Newswire Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-08. Check the Actor page for the current rates.

AliExpress Scraper Returns Zero Records on CSR Pages from `ja` or `ko`

Crawler Bros — Thu, 08 Oct 2026 13:43:54 +0000

Large language models (LLMs) are increasingly capable of acting as autonomous agents, performing complex tasks by breaking them down into sub-problems and calling external tools. Exposing Apify Actors as callable tools for these agents presents specific integration challenges and opportunities, particularly around how input schemas translate into tool signatures and the predictable limits an agent will encounter.

This article details how an AI agent can invoke the AliExpress Scraper via Apify's Managed Cloud Platform (MCP), focusing on the critical ?tools= configuration, the derived input schema for an agent, and the operational limitations an agent must be designed to handle for robust operation.

What happens when a requested region serves a CSR-only page?

If an AI agent requests a regional storefront that occasionally serves a client-side rendered (CSR) page, such as ja or ko, the AliExpress Scraper Actor will return zero records. This occurs because these pages omit the _init_data_ JSON state that the Actor relies on for parsing. For reliable data extraction, agents should prioritize regions known to consistently serve server-side rendered (SSR) JSON, such as com, us, de, fr, es, it, nl, pt, pl, tr, ar, vi, th, or id.

The aliexpress-scraper Actor is designed to extract structured data from AliExpress, offering various modes for querying. When integrating this Actor as an AI agent tool, the agent needs to understand not just the functionality, but also the nuances of its failure modes. A particularly sharp edge arises with certain regional storefronts. The Actor's region input field dictates which AliExpress subdomain to query. While many regions reliably provide server-side rendered (SSR) HTML containing embedded JSON for data extraction, some, like ja (Japan) or ko (Korea), can occasionally serve a client-side rendered (CSR) page. This means the critical _init_data_ JSON state, which the Actor uses for parsing, is entirely absent from the initial HTML.

When this happens, the Actor run will complete successfully, but it will yield zero records in its dataset. An AI agent, unaware of this specific limitation, might interpret this as a successful run with no matching data, rather than a parsing failure due to a rendering discrepancy. To mitigate this, an agent should be programmed to:

Prioritize known reliable regions: When given a choice, default to regions like com, us, or de which are explicitly documented to ship SSR JSON reliably.
Monitor record counts: If a run completes for a known problematic region and returns zero records, the agent should infer a parsing failure rather than an empty dataset and potentially retry with a different region or notify a human.
Use autoEscalateOnBlock carefully: While autoEscalateOnBlock helps with rate limits by switching to residential proxies, it doesn't solve the CSR-only page problem, as the underlying page structure remains unparseable.

Here's how an agent might select a region, preferring reliable ones:

def select_aliexpress_region(desired_region: str) -> str:
    """
    Selects an AliExpress region, prioritizing known reliable SSR storefronts.
    """
    reliable_regions = ["com", "us", "de", "fr", "es", "it", "nl", "pt", "pl", "tr", "ar", "vi", "th", "id"]
    if desired_region in reliable_regions:
        return desired_region
    elif desired_region in ["ja", "ko", "he", "ru"]: # Known problematic or special cases
        print(f"Warning: Region '{desired_region}' may serve CSR-only pages, potentially returning zero records.")
        # For 'ru', special handling for language
        if desired_region == "ru":
            print("Note: For Russian language results, use region='com' with language='ru_RU'.")
            return "com" # Redirect internally to .com with language override
        return desired_region # Still allow, but with warning
    else:
        print(f"Unknown region '{desired_region}', defaulting to 'com'.")
        return "com"

# Example agent usage
agent_desired_region = "ja"
actual_region_to_use = select_aliexpress_region(agent_desired_region)
print(f"Agent will attempt to scrape region: {actual_region_to_use}")

How do you expose an Apify Actor as a tool for an AI agent?

You expose an Apify Actor as an AI agent tool by calling the Apify MCP server at https://mcp.apify.com and specifying the Actor using the ?tools=owner/actor-name query parameter. This endpoint translates the Actor's input schema into a tool signature that an agent can understand and execute, provided the agent has the necessary Apify API token for authenticated calls.

The core of enabling AI agents to use Apify Actors lies in Apify's Managed Cloud Platform (MCP). The MCP server acts as an intermediary, presenting Actors as structured tools. For the aliexpress-scraper Actor, the endpoint https://mcp.apify.com?tools=crawlerbros/aliexpress-scraper is the entry point. When an agent queries this endpoint, MCP responds with a structured description of the aliexpress-scraper tool, derived directly from its input schema. This description includes the tool's name, a natural language description, and most critically, its parameter schema, which is a JSON Schema representation of the Actor's input.

An AI agent's orchestration logic would then parse this schema to understand what arguments (mode, searchQuery, region, etc.) the aliexpress-scraper tool expects, their types, and any constraints or default values. Executing the tool involves making a POST request to the MCP server with the appropriate X-Apify-Api-Token header and a JSON body corresponding to the Actor's input.

Here's a simplified representation of how the aliexpress-scraper input schema translates into a tool signature for an agent:

{
  "name": "aliexpress-scraper",
  "description": "Scrape AliExpress search results, product details, store profiles, and customer reviews. Multi-region (com / us / ru / es / fr / de / it / nl / pt / pl / ar / tr / ko / ja / vi / th / id / he), multi-currency, with sort, price, rating, and ship-from/ship-to filters.",
  "input_schema": {
    "type": "object",
    "properties": {
      "mode": {
        "type": "string",
        "description": "What to scrape. search: text-query results. byProduct: product detail by ID. byStore: store profile by ID. byReviews: customer reviews for product IDs. byUrl: parse any AliExpress",
        "enum": ["search", "byProduct", "byStore", "byReviews", "byUrl"],
        "default": "search"
      },
      "searchQuery": {
        "type": "string",
        "description": "Free-text search query, e.g. \"phone case\". Required when mode=search."
      },
      "productIds": {
        "type": "array",
        "items": { "type": "string" },
        "description": "AliExpress numeric product IDs (e.g. 1005010155028387)."
      },
      "region": {
        "type": "string",
        "description": "AliExpress regional sub-domain to query.",
        "default": "com"
      },
      "currency": {
        "type": "string",
        "description": "ISO-4217 currency code. AliExpress may override it with the proxy IP's local currency.",
        "default": "USD"
      },
      "language": {
        "type": "string",
        "description": "Locale that AliExpress should render text in (b_locale cookie).",
        "default": "en_US"
      },
      "priceMin": {
        "type": "integer",
        "description": "Minimum price (in the selected currency). 0 disables.",
        "default": 0
      },
      "maxItems": {
        "type": "integer",
        "description": "Hard cap on emitted records.",
        "default": 25
      },
      "maxPages": {
        "type": "integer",
        "description": "Maximum pagination pages per search query (60 items per page).",
        "default": 3
      }
    },
    "required": ["mode"]
  }
}

This structured tool description allows the agent to dynamically construct valid requests. However, it's crucial for the agent to also understand the implications of default values versus explicitly passed values. The prefill values specified in the Actor's console UI are not applied to API calls; only the default values in the schema are. An agent should always construct a complete input dictionary, explicitly setting all necessary parameters, to avoid relying on implicit server-side prefill behavior that won't be triggered by an API call.

When should an AI agent switch from synchronous to asynchronous Actor execution?

An AI agent should switch from synchronous to asynchronous Actor execution when a single run is expected to exceed 300 seconds. The synchronous run endpoint for Apify Actors has a hard-coded timeout of 300 seconds (5 minutes); exceeding this limit will result in an HTTP 408 response, forcing the agent to restart or abandon the task. For longer-running tasks, the agent must initiate the Actor run via a POST request to /v2/acts/<actor>/runs and then poll the run's status or use a webhook.

AI agents, particularly when interacting with external services, need to be mindful of execution limits. For Apify Actors, a critical threshold is the 300-second (5-minute) timeout for synchronous runs. If an agent attempts to call an Actor using the synchronous endpoint and the Actor runs for longer than 300 seconds, the connection will be terminated with an HTTP 408 error. This isn't a graceful shutdown; the Actor will continue running on Apify's platform, but the agent's connection will be lost, making it impossible to retrieve the results synchronously.

For tasks that inherently take longer – such as scraping thousands of AliExpress products across many search pages, or fetching reviews for a large list of product IDs – an AI agent must opt for asynchronous execution. This involves:

Initiating the run: Make a POST request to https://api.apify.com/v2/acts/crawlerbros/aliexpress-scraper/runs (with an API token). This returns a runId.
Polling for status: Periodically (e.g., every 10-30 seconds) make GET requests to https://api.apify.com/v2/actor-runs/<runId> to check the status field.
Retrieving results: Once the status indicates SUCCEEDED or FAILED, retrieve the run's dataset items via the dataset URL provided in the run object.

This asynchronous pattern adds complexity but is essential for robust operation. An agent's decision-making logic should incorporate an estimate of run duration based on input parameters (e.g., maxItems, maxPages, number of productIds). If these parameters suggest a run might approach or exceed the 300-second threshold, the agent should proactively choose the asynchronous approach. For instance, maxPages=3 for mode=search implies up to 180 items (60 items/page), which is unlikely to hit the limit. However, if maxItems is set to a much higher number, or productIds contains hundreds of entries, an asynchronous execution strategy becomes imperative.

import requests
import time

APIFY_API_TOKEN = "YOUR_APIFY_API_TOKEN"
ACTOR_ID = "crawlerbros/aliexpress-scraper"

def run_aliexpress_scraper_async(run_input: dict):
    """
    Initiates an asynchronous run of the AliExpress Scraper Actor and polls for its completion.
    """
    print("Initiating asynchronous Actor run...")
    start_response = requests.post(
        f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs",
        headers={"Authorization": f"Bearer {APIFY_API_TOKEN}"},
        json=run_input
    ).json()

    run_id = start_response["data"]["id"]
    print(f"Actor run started with ID: {run_id}")

    dataset_id = start_response["data"]["defaultDatasetId"]
    print(f"Default Dataset ID: {dataset_id}")

    # Poll for run status
    while True:
        run_status_response = requests.get(
            f"https://api.apify.com/v2/actor-runs/{run_id}",
            headers={"Authorization": f"Bearer {APIFY_API_TOKEN}"}
        ).json()
        status = run_status_response["data"]["status"]
        print(f"Current run status: {status}")

        if status in ["SUCCEEDED", "FAILED", "ABORTED"]:
            print(f"Run finished with status: {status}")
            break
        time.sleep(10) # Wait 10 seconds before polling again

    if status == "SUCCEEDED":
        # Retrieve results from the dataset
        results_url = f"https://api.apify.com/v2/datasets/{dataset_id}/items"
        print(f"Retrieving results from: {results_url}")
        results = requests.get(results_url, headers={"Authorization": f"Bearer {APIFY_API_TOKEN}"}).json()
        print(f"Retrieved {len(results)} items.")
        return results
    else:
        print("Actor run did not succeed.")
        return []

# Example: A run that might exceed 300 seconds
# The actual duration depends on network conditions, AliExpress response times, etc.
long_run_input = {
  "mode": "search",
  "searchQuery": "laptop bag",
  "region": "com",
  "maxPages": 10, # Max items could be 600, potentially a longer run
  "maxItems": 500
}

How does the `maxTotalChargeUsd` parameter protect against unexpected costs?

The maxTotalChargeUsd parameter protects against unexpected costs by setting an upper limit on the monetary charge for an Actor run. When the total cost incurred by the run reaches this cap, the Actor run is terminated. This provides a crucial safeguard for AI agents operating with budget constraints, although termination is not instantaneous, and some resources may still be consumed briefly after the cap is tripped.

Cost control is a critical aspect of autonomous agent operation, especially when calling external APIs or Actors that incur charges. The aliexpress-scraper Actor, like many on Apify, charges based on specific events. To prevent an agent from inadvertently spending beyond a defined budget, the maxTotalChargeUsd query parameter (or ACTOR_MAX_TOTAL_CHARGE_USD environment variable within the Actor) is invaluable.

An AI agent, before initiating a potentially large scrape, can estimate the likely cost based on the number of expected output records and the per-result price. It can then set maxTotalChargeUsd to a value that reflects its allocated budget for that specific task. If, for any reason (e.g., more results found than expected, higher proxy usage), the actual cost begins to approach this limit, the Apify platform will automatically terminate the run.

It's crucial for the agent to understand that this termination is not immediate. The Actor will continue to consume resources for a brief period while it's winding down. Therefore, setting maxTotalChargeUsd slightly above the absolute hard limit can provide a small buffer. This mechanism allows for fine-grained financial control and prevents runaway costs in scenarios where an agent's logic might accidentally request an overly expansive scrape.

Consider a scenario where an agent is instructed to find "cheap phone cases" but due to a misinterpretation, it searches for a very generic term and sets maxItems extremely high. Without maxTotalChargeUsd, this could lead to a significant bill. With it, the agent ensures that even if its logic falters, the financial impact is contained.

# Example of setting maxTotalChargeUsd for a run
import requests

APIFY_API_TOKEN = "YOUR_APIFY_API_TOKEN"
ACTOR_ID = "crawlerbros/aliexpress-scraper"

def run_aliexpress_scraper_with_budget(run_input: dict, max_charge_usd: float):
    """
    Runs the AliExpress Scraper Actor with a maximum total charge limit.
    """
    print(f"Initiating Actor run with max charge: ${max_charge_usd}")
    # The maxTotalChargeUsd parameter can be added to the query string
    start_response = requests.post(
        f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs?maxTotalChargeUsd={max_charge_usd}",
        headers={"Authorization": f"Bearer {APIFY_API_TOKEN}"},
        json=run_input
    ).json()

    # The agent would then monitor this run as in the asynchronous example
    # and handle potential 'ABORTED' status if the budget is hit.
    print(f"Run started: {start_response['data']['id']}")
    print("Agent should poll run status and check for 'ABORTED' due to budget.")

# Agent wants to scrape some data but doesn't want to spend more than $5 on this particular task.
budget_constrained_input = {
  "mode": "search",
  "searchQuery": "smartwatch",
  "region": "us",
  "maxItems": 1000 # Potentially many results, but capped by budget
}

What are the cost considerations when using the AliExpress Scraper?

The AliExpress Scraper charges for two events: a "result" event at $0.005 per single item in the default dataset, and an "Actor Start" event at $0.005 per GB of memory allocated to the run. These event charges are in addition to platform usage fees for run time and memory, which are billed separately at the user's Apify plan rates.

The number of "result" events scales directly with maxItems, maxPages, and maxReviewsPerProduct, with discounts available for FREE, BRONZE, SILVER, GOLD, PLATINUM, and DIAMOND tier users.

Understanding the cost structure is crucial for any AI agent that aims to operate efficiently and predictably. The aliexpress-scraper Actor uses a PAY_PER_EVENT pricing model.

Here's a breakdown:

Result Event: Each record that the Actor pushes to its default dataset (e.g., a product, a store profile, or a customer review) costs $0.005. This is the most significant variable cost.
- mode=search: The maxItems and maxPages inputs directly multiply the number of potential result events. Since each page can yield up to 60 items, maxPages=3 could result in up to 180 product records, each triggering a result event.
- mode=byProduct: Each ID in productIds typically fetches one product record.
- mode=byReviews: Each ID in productIds can generate maxReviewsPerProduct review records. If enrichReviews is true for mode=search, then each product record also generates maxReviewsPerProduct review records.
- The price per result event varies by Apify discount tier: FREE ($0.005), BRONZE ($0.00433), SILVER ($0.00367), GOLD ($0.003), PLATINUM ($0.003), DIAMOND ($0.003). An agent should be aware of the user's tier to accurately estimate costs.
Actor Start Event: This event is charged once per run at $0.005 per GB of memory allocated. For most runs, this is a small, fixed cost.
Platform Usage: Beyond these event charges, the user also pays for the underlying Apify platform resources consumed by the run (memory and run time). These are billed separately at the user's Apify plan's rates. An AI agent should consider these too, although they are harder to predict precisely without direct measurement.

An intelligent agent can use these details to make informed decisions. For example, if the goal is to get a brief overview of products, setting maxItems to a low number significantly reduces the "result" event cost. If only product details are needed, mode=byProduct with a specific list of productIds might be more cost-effective than a broad search query followed by post-processing.

def estimate_aliexpress_scraper_event_cost(run_input: dict, user_tier: str = "FREE") -> float:
    """
    Estimates the event-based cost for an AliExpress Scraper run.
    Does not account for platform usage.
    """
    result_price_per_item = {
        "FREE": 0.005,
        "BRONZE": 0.00433,
        "SILVER": 0.00367,
        "GOLD": 0.003,
        "PLATINUM": 0.003,
        "DIAMOND": 0.003
    }.get(user_tier.upper(), 0.005) # Default to FREE tier if unknown

    estimated_results = 0
    if run_input.get("mode") == "search":
        max_items = run_input.get("maxItems", 25)
        max_pages = run_input.get("maxPages", 3)
        # Assuming 60 items per page max, but capped by maxItems
        estimated_products = min(max_items, max_pages * 60)
        estimated_results += estimated_products

        if run_input.get("enrichReviews", False):
            max_reviews_per_product = run_input.get("maxReviewsPerProduct", 0)
            if max_reviews_per_product > 0:
                estimated_results += estimated_products * max_reviews_per_product

    elif run_input.get("mode") == "byProduct":
        estimated_results += len(run_input.get("productIds", []))

    elif run_input.get("mode") == "byReviews":
        max_reviews_per_product = run_input.get("maxReviewsPerProduct", 0)
        if max_reviews_per_product > 0:
            estimated_results += len(run_input.get("productIds", [])) * max_reviews_per_product

    elif run_input.get("mode") == "byStore":
        estimated_results += len(run_input.get("storeIds", []))

    elif run_input.get("mode") == "byUrl":
        estimated_results += len(run_input.get("urls", [])) # Simplified assumption, depends on URL type

    # Actor Start cost (assuming 1GB memory for simplicity, minimum 1 event)
    actor_start_cost = 0.005

    total_event_cost = (estimated_results * result_price_per_item) + actor_start_cost
    return total_event_cost

# Example agent cost estimation
example_input_search = {
  "mode": "search",
  "searchQuery": "bluetooth speaker",
  "maxItems": 100,
  "maxPages": 2,
  "enrichReviews": True,
  "maxReviewsPerProduct": 5
}

What are the key limitations an AI agent must account for when using AliExpress Scraper?

An AI agent using the AliExpress Scraper must account for several limitations: client-side rendering issues for product details and some regional storefronts (potentially leading to empty records), best-effort currency localization (AliExpress may override requested currency based on proxy IP), and the absence of variation/SKU details or shipping costs due to API call signing requirements. Aggressive crawls can also be rate-limited, requiring explicit proxy usage.

Beyond the cost structure and execution model, an AI agent needs a deep understanding of the aliexpress-scraper's functional limitations to avoid generating incorrect assumptions or making repeated, futile requests. These limitations are intrinsic to how AliExpress renders its content and how the Actor can access it:

Client-Side Rendering (CSR) for Product Details: The richest product data (like detailed descriptions, variation/SKU specifics) is often loaded via client-side JavaScript after the initial page render, often using signed API calls. The aliexpress-scraper primarily relies on server-side rendered (SSR) JSON embedded in the HTML. This means:
- Product detail pages scraped via mode=byProduct will be sparser than records from mode=search. An agent needing full product data should prioritize mode=search for initial discovery or explicitly understand that byProduct yields less detail.
- Variation/SKU details (colors, sizes) and shipping calculations are generally not captured. An agent cannot ask for specific SKU information or shipping costs and expect a reliable return.
Regional Storefront CSR Issue: As discussed, certain regional storefronts (ja, ko, he) sometimes serve CSR-only pages, causing runs to return zero records. An agent should be programmed to detect this failure mode and potentially re-attempt with a more reliable region or language setting (e.g., language=ru_RU with region=com for Russian content).
Currency Localization is Best-Effort: While the currency input is sent, AliExpress might override it based on the proxy IP's geolocation. For deterministic currency results, an agent must explicitly enable useProxy=true and configure a residential proxy in the desired country, incurring additional proxy costs and potentially shorter proxy session lifespans (~30 minutes for residential).
Rate Limiting: AliExpress can rate-limit aggressive scraping. The autoEscalateOnBlock feature (default true) helps by switching to residential proxies, but for sustained, high-volume scraping, an agent should proactively enable useProxy=true with a residential proxy group from the outset.
Request Queue Consumption: If an agent architecture involves multiple Actor runs processing items from a shared request queue, it's critical to remember that only one Actor or task run can process a request queue at a time. While multiple runs can add to a queue, fan-out processing from a single queue does not work, which might impact complex parallel scraping strategies.

How do unnamed storage expirations impact data retention for AI agents?

Unnamed storage expirations significantly impact data retention for AI agents, as data from default datasets, request queues, and key-value stores from older runs will be automatically deleted. On the free Apify plan, only the 10 most recent runs are retained for four months. To ensure long-term data persistence, AI agents must be configured to use named storages, which are exempt from automatic deletion.

For AI agents managing ongoing data collection or requiring access to historical data, the Apify platform's storage policies are a critical design consideration. By default, Actor runs create "unnamed" storages. This includes the default dataset where the aliexpress-scraper outputs its records, any request queues it uses, and temporary key-value stores. These unnamed storages are subject to expiration. Specifically, on the free plan, only the data from the 10 most recent runs is retained, and even then, only for four months. Older data, or data from runs beyond the 10-run limit, is automatically purged.

This has direct implications for an AI agent's operational strategy:

Ephemeral Data: If an agent's task is purely analytical and the results are consumed immediately, then unnamed storages might suffice. However, if the agent needs to track changes over time, perform comparative analysis across multiple runs, or simply maintain a historical record, relying on unnamed storages is a recipe for data loss.
Long-term Persistence: For any long-term data retention, the agent must be designed to specify output/datasetId (or requestQueueId, keyValueStoreId) with a named identifier when initiating an Actor run. Named storages are explicitly created by the user (or the agent) and are exempt from automatic deletion.
Cost of Retention: While named storages offer persistence, they still incur storage costs on the Apify platform. An intelligent agent needs to balance the need for data retention with the associated storage expenses.

An agent that periodically scrapes AliExpress data for market analysis, for example, would ideally create a named dataset (e.g., my-product-trends-2026) and instruct the aliexpress-scraper to output its data there for every run. This ensures that even if hundreds of runs occur over months, the collected data remains accessible.

import requests

APIFY_API_TOKEN = "YOUR_APIFY_API_TOKEN"
ACTOR_ID = "crawlerbros/aliexpress-scraper"

def run_aliexpress_scraper_with_named_dataset(run_input: dict, dataset_name: str):
    """
    Initiates an Actor run, pushing results to a named dataset for persistence.
    The dataset will be created if it doesn't exist.
    """
    print(f"Ensuring named dataset '{dataset_name}' exists or is created...")
    # This call creates or retrieves the named dataset
    dataset_info = requests.post(
        f"https://api.apify.com/v2/datasets?token={APIFY_API_TOKEN}",
        json={"name": dataset_name}
    ).json()
    named_dataset_id = dataset_info["data"]["id"]
    print(f"Using dataset ID: {named_dataset_id}")

    # Now, initiate the Actor run, explicitly specifying the named dataset
    # The 'output/datasetId' input field is for this purpose
    run_input_with_dataset = run_input.copy()
    run_input_with_dataset["output/datasetId"] = named_dataset_id

    print(f"Initiating Actor run with output to named dataset '{dataset_name}'...")
    start_response = requests.post(
        f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs",
        headers={"Authorization": f"Bearer {APIFY_API_TOKEN}"},
        json=run_input_with_dataset
    ).json()

    run_id = start_response["data"]["id"]
    print(f"Actor run started with ID: {run_id}")
    # The agent would then poll this run and retrieve results from the named dataset
    # using its ID, instead of relying on the defaultDatasetId from the run object.

# Example: An agent wanting to track "phone case" prices over time
tracking_input = {
  "mode": "search",
  "searchQuery": "phone case",
  "region": "com",
  "maxItems": 10
}

How do proxy session limitations affect long-running AliExpress Scraper tasks?

Proxy session limitations can affect long-running AliExpress Scraper tasks because datacenter proxies persist for 26 hours, while residential proxies typically last around 30 minutes. If an AI agent uses residential proxies for sustained scraping due to rate limiting or currency localization needs, it must be prepared to handle frequent proxy rotation and potential session drops during extended runs.

When an AI agent uses the aliexpress-scraper for tasks that involve a significant number of requests or require specific IP geo-locations, the choice and management of proxies become crucial. AliExpress can rate-limit aggressive crawls, and autoEscalateOnBlock provides a fallback to residential proxies. However, there are important distinctions in proxy session longevity that an agent must consider:

Datacenter Proxies: These proxies have longer session lifespans, typically around 26 hours. They are generally suitable for less sensitive or lower-volume scraping where IP changes are not frequently required.
Residential Proxies: These proxies are more resilient to detection and rate limiting, but their sessions are significantly shorter, usually around 30 minutes. This short lifespan means that for long-running tasks, the proxy IP will change frequently.

If an AI agent enables useProxy=true with a residential proxy group from the outset, or if autoEscalateOnBlock triggers the use of residential proxies, the agent's logic needs to be robust enough to handle these proxy rotations. While the Actor itself is designed to manage proxy changes internally, the agent should be aware that the egress IP address used for requests will change often, which can impact consistent currency localization if AliExpress relies heavily on the IP's geolocation. For truly deterministic currency, the agent must ensure a residential proxy from the exact target country is selected, and acknowledge that even then, the underlying proxy IP will rotate.

This limitation means that an agent cannot assume a stable IP identity for the duration of a multi-hour scrape if residential proxies are involved. It might need to implement logic to detect if proxy-dependent features (like specific currency localization) become inconsistent and potentially adjust its strategy, such as restarting segments of the scrape or issuing warnings.

Checked against the Actor's input schema and Apify docs on 2026-10-08.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Regional Job Market Analysis Across Nigeria's Six Geopolitical Zones

Crawler Bros — Thu, 08 Oct 2026 10:07:11 +0000

Analyzing hiring patterns in emerging labor markets often breaks down when job boards lack standard API endpoints or enforce inconsistent metadata schemas. For data engineers tracking workforce shifts in Nigeria, MyJobMag serves as a primary aggregator. However, manually querying every regional hub or relying on basic string searches leaves structural gaps—such as missing employment types or inconsistent state designations across local employers.

Programmatic extraction requires navigating multiple browse axes while maintaining accurate data schemas. By leveraging structural fallbacks across categorical endpoints, you can build reliable datasets for regional employment trends without managing complex scraping infrastructure.

Handling Structural Gaps in Multi-Axis Aggregation

Scraping job listings across different geographical levels introduces specific data hygiene challenges. On MyJobMag, listings vary depending on how an employer posts a position. A job listing might explicitly specify a state like Lagos in its metadata, while another posting in the same state might only supply a city name or rely on the category page context.

The MyJobMag Scraper addresses this by supporting thirteen distinct operation modes. Instead of running a generic, prone-to-miss keyword search, you can isolate collection across ten specific browse axes:

Geographical filters: byLocation (38 state options), byRegion (6 geopolitical zones), and byCity (100 specific city options).
Categorical filters: byField (51 job functions), byIndustry (38 industry options), byEducation, byType, byDate, byCompany, and byTitle.
Specialized endpoints: search (full-text search), byFeatured (curated listings), and byUrls (direct target list).

A key technical detail lies in how the scraper handles incomplete structured data on individual job pages. When browsing a dedicated axis—such as using mode="byType" with employmentType="remote"—the scraper applies mode-implied fallbacks. If an individual posting omits its explicit remote flag in its on-page schema, the scraper injects the target filter value into the output record. This ensures downstream analysis scripts receive complete categorical context without losing track of how the record was selected.

Conversely, the actor does not overwrite explicit on-page listings with filter defaults. For example, MyJobMag's upstream platform occasionally cross-lists a job based in Ebonyi state under a Lagos location page. Rather than silently overwriting the location string to match the requested filter, the scraper preserves the actual on-page details (addressLocality and addressRegion), exposing upstream data quirks directly in the scraped dataset.

Constructing Regional Ingestion Pipelines

To build a clean pipeline that extracts full details across regional zones, configure the actor's input payload to match your target taxonomy. The schema accepts granular controls that let you target specific subsets before initiating payload parsing.

For instance, if you need to extract full listing text for technology roles across the South-West geopolitical zone, you combine mode, region, and secondary string matching.

Here is a Python configuration using the Apify API client to execute a targeted extraction:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_KEY")

run_input = {
    "mode": "byRegion",
    "region": "south-west",
    "containsKeyword": "developer",
    "includeFullDetails": True,
    "maxItems": 100
}

run = client.actor("crawlerbros/myjobmag-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Title: {item.get('title')}")
    print(f"Location: {item.get('location')}")
    print(f"Posted: {item.get('datePosted')}")
    print("---")

Setting includeFullDetails to true instructs the scraper to follow each candidate URL from the initial listing page. This extra step parses the underlying page DOM to extract comprehensive fields, including descriptionHtml, descriptionText, validThrough, industry, educationLevel, and companyLogoUrl. Omitting empty values ensures output JSON payloads contain only valid, populated attributes rather than null-padded objects.

Step-by-Step Dataset Extraction

Setting up an automated run requires defining the precise entry parameters for the input schema.

Select the Extraction Mode: Choose the core mode that matches your primary aggregation axis (search, byLocation, byRegion, byCity, byField, etc.).
Define Mode-Specific Parameters: Supply the parameter string matching your chosen mode. If mode is set to byLocation, populate location with a state key like "abuja" or "lagos". If using mode="byCompany", pass a string like "pz-cussons" to the companySlug field.
Apply Substring Filters: Optionally pass a value to containsKeyword. This performs an additional case-insensitive filter across the title, company name, and body text across all operational modes.
Configure Payload Depth: Set includeFullDetails to true if your analysis requires complete job descriptions and ISO 8601 timestamps (datePosted, validThrough). Set it to false if you only need index-level fields like title, companyName, and sourceUrl.
Set Execution Boundaries: Pass an integer to maxItems (ranging from 1 to 1000) to place a hard cap on emitted records and control dataset size.

{
  "mode": "byField",
  "field": "research-data-analysis",
  "employmentType": "full-time",
  "containsKeyword": "analyst",
  "includeFullDetails": true,
  "maxItems": 50
}

The resulting output matches a predictable schema structure ready for database ingestion:

{
  "title": "Senior Data Analyst",
  "companyName": "Example Enterprise",
  "location": "Lagos",
  "addressLocality": "Ikeja",
  "addressRegion": "Lagos",
  "addressCountry": "NG",
  "category": "Research / Data Analysis",
  "employmentType": "full-time",
  "datePosted": "2026-03-28T08:00:00Z",
  "descriptionText": "Full role details and operational requirements...",
  "sourceUrl": "https://www.myjobmag.com/job/senior-data-analyst",
  "recordType": "job",
  "scrapedAt": "2026-03-30T10:15:00Z"
}

Platform Pricing Dynamics

Event pricing for the actor operates on a fixed per-event structure tied to dataset emissions and run initialization. Each dataset output record generated as a "result" event costs $0.005 on the FREE tier ($0.00433 on BRONZE, $0.00367 on SILVER, and $0.003 on GOLD, PLATINUM, and DIAMOND tiers).

An "Actor Start" charge incurs $0.005 per GB of memory allocated to the run upon execution. Platform usage for the run is billed separately at your Apify plan's rates.

When calculating expected costs for large extraction runs, evaluate the volume of records required against your target dataset boundaries. Fetching 500 records on a standard execution generates 500 "result" events alongside the single "Actor Start" event, with any platform usage calculated independently by the platform billing system.

Inherent Extraction Limitations

While the actor normalizes metadata across diverse browse targets, it cannot synthesize information that employers omit entirely from their listings. If an employer submits a job listing without specifying salary ranges, explicit education requirements, or precise localized addresses, those key-value pairs are omitted from the output JSON rather than populated with placeholder strings. Pipelines expecting deterministic, fully-populated schemas for every record must handle variable dictionary keys downstream.

How do your downstream data pipelines reconcile conflicting geographic designations when cross-referencing multi-source labor market feeds?

Everything above runs on MyJobMag Scraper. Start with a small input and a low result limit before you widen the run -- the output shape is easier to check that way.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-08. Check the Actor page for the current rates.

Why Preventing Duplicate SEMrush Scraper Output Needs Named Storage

Crawler Bros — Thu, 08 Oct 2026 07:44:25 +0000

How can I prevent duplicate processing of SEMrush Scraper output?

To prevent duplicate output from the SEMrush Free Website Stats Scraper, your data pipeline needs a robust mechanism for tracking what has already been processed. Maintain a list of all domains for which you have successfully retrieved and processed SEMrush stats, storing this information in persistent storage. Before initiating a new run, filter your input list of domains against this stored state.

While Apify offers named Key-Value Stores as a suitable solution for this, similar results can be achieved using other persistent storage solutions like Redis, a relational database, or cloud-based object storage. Named Key-Value Stores, unlike unnamed storages which expire, are persistent and provide a reliable place to store your processed domain identifiers and their scrapedAt timestamps. This approach, leveraging persistent storages, is crucial for long-running or scheduled pipelines, as free-tier accounts only retain the 10 most recent run storages for four months.

Here's how a Python application might manage this state before feeding a list of domains to the semrush-scraper Actor:

from apify_client import ApifyClient
import os

# Initialize ApifyClient with your API token
# For local testing, ensure APIFY_TOKEN is set in environment variables
apify_client = ApifyClient(os.environ["APIFY_TOKEN"])

# Define the name for your state storage
PROCESSED_DOMAINS_KV_STORE = "semrush-processed-domains"

def load_processed_domains(client: ApifyClient) -> set[str]:
    """Loads the set of domains previously processed from a named Key-Value Store."""
    try:
        kv_store = client.get_or_create_key_value_store(store_name=PROCESSED_DOMAINS_KV_STORE)
        processed_data = kv_store.get_record(key="processed_domains")
        if processed_data and processed_data["value"]:
            return set(processed_data["value"])
        return set()
    except Exception as e:
        print(f"Error loading processed domains: {e}")
        return set()

def save_processed_domains(client: ApifyClient, domains: set[str]):
    """Saves the current set of processed domains to a named Key-Value Store."""
    try:
        kv_store = client.get_or_create_key_value_store(store_name=PROCESSED_DOMAINS_KV_STORE)
        kv_store.set_record(key="processed_domains", value=list(domains), content_type="application/json")
    except Exception as e:
        print(f"Error saving processed domains: {e}")

# Example usage:
# all_domains_to_check = ["wikipedia.org", "github.com", "example.com", "apify.com"]
#
# processed = load_processed_domains(apify_client)
# print(f"Previously processed domains: {processed}")
#
# domains_for_new_run = [d for d in all_domains_to_check if d not in processed]
# print(f"Domains for current run: {domains_for_new_run}")
#
# # In a real scenario, you'd now pass domains_for_new_run to the Actor.
# # After the Actor runs and data is consumed, you'd update 'processed' and save it.

After a scraper run completes, fetch its dataset items, process them, and then persist the identifiers of successfully handled items into a named Apify Key-Value Store or an external database to prevent reprocessing in future runs. This post-run update of your state ensures that only newly acquired or modified data is considered for downstream actions. Once the SEMrush Free Website Stats Scraper Actor run finishes, you need to retrieve its results from the default dataset and then update your state.

Here's how you might fetch the dataset items and update your set of processed domains, including handling domains for which SEMrush reported no data (notFound):

from apify_client import ApifyClient
import os

apify_client = ApifyClient(os.environ["APIFY_TOKEN"])
PROCESSED_DOMAINS_KV_STORE = "semrush-processed-domains"

# load_processed_domains and save_processed_domains functions are defined as above
# ...

def process_and_update_state(client: ApifyClient, run_id: str, current_processed_domains: set[str]):
    """
    Fetches dataset items from a run, processes them, and updates the set of processed domains.
    """
    run = client.get_run(run_id=run_id)
    dataset = client.get_dataset(dataset_id=run["defaultDatasetId"])

    newly_processed_domains = set()
    for item in dataset.iterate_items():
        domain = item.get("domain")
        if domain:
            # Here you would typically send 'item' to your downstream system
            # For demonstration, we just mark it as processed
            print(f"Processing data for domain: {domain}. Not found: {item.get('notFound', False)}")
            newly_processed_domains.add(domain)
        else:
            print(f"Warning: Dataset item missing 'domain' field: {item}")

    updated_processed_domains = current_processed_domains.union(newly_processed_domains)
    save_processed_domains(client, updated_processed_domains)
    print(f"State updated. Total processed domains: {len(updated_processed_domains)}")

# Example of how you would integrate this:
#
# # 1. Load existing state
# existing_processed = load_processed_domains(apify_client)
# all_input_domains = ["apify.com", "example.com", "nodata.org", "wikipedia.org"] # Your full list
#
# # 2. Filter input domains
# domains_for_run = [d for d in all_input_domains if d not in existing_processed]
#
# if domains_for_run:
#     print(f"Starting run for {len(domains_for_run)} new domains.")
#     run_input = {
#         "domains": domains_for_run,
#         "mode": "full",
#         "proxyConfiguration": {"useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US"}
#     }
#     actor_run = apify_client.actor("crawlerbros/semrush-scraper").call(run_input=run_input)
#     print(f"Actor run started with ID: {actor_run['id']}")
#
#     # 3. After the run completes (or after fetching its status), process output and update state
#     # In a real pipeline, you'd wait for run completion or use a webhook.
#     # For this example, let's assume 'actor_run' is already a completed run object.
#     # process_and_update_state(apify_client, actor_run["id"], existing_processed)
# else:
#     print("No new domains to process.")

By storing the set of domain values for every item successfully pulled from the dataset, you create a robust processing history. Even if a run fails midway or a downstream system experiences an outage, subsequent runs can always pick up where the last successful processing left off, preventing redundant work and ensuring data consistency. The notFound status, when present in an item, indicates that SEMrush has no data for that domain. This is still a processed state; you've checked the domain, and the Actor has reported its status. Therefore, such domains should also be added to your processed_domains set to avoid repeatedly checking them in future runs.

What unique identifier works best for SEMrush Website Stats?

The domain field from the scraper's output, once normalized, serves as the most effective unique identifier for tracking individual records and preventing duplicates across pipeline runs. Each record represents the stats for a single domain, making domain a natural candidate for an ID.

The semrush-scraper output for each domain includes a domain field that has already been normalized (e.g., https://www.custify.com/ becomes custify.com). This ensures consistency and makes it suitable for use as a primary key in a database or a unique identifier in your state management logic.

Consider the typical structure of a single output item from the semrush-scraper:

{
  "type": "semrush_website_stats",
  "domain": "wikipedia.org",
  "authorityScore": 92,
  "visits": 1100000000,
  "visitsText": "1.1B",
  "organicSearchTraffic": 485000000,
  "organicSearchTrafficText": "485M",
  "referringDomains": 3900000,
  "referringDomainsText": "3.9M",
  "referringDomainsChange": 1.2,
  "backlinks": 2960000000,
  "backlinksText": "2.96B",
  "backlinksChange": 0.5,
  "asOf": "July 2026",
  "sourceUrl": "https://www.semrush.com/website/wikipedia.org",
  "scrapedAt": "2026-09-28T10:30:00.000Z"
}

The domain field is consistently present and represents the specific website whose statistics were scraped. For tracking historical data, domain can be combined with asOf (the data month) or scrapedAt (the UTC ISO timestamp of the scrape) to create a composite key, allowing you to store multiple data points for the same domain over time without collisions. When a domain is not found by SEMrush, the output includes notFound: true and a reason, but crucially, the domain field is still present, allowing you to record that specific domain was checked and no data was available.

How do I orchestrate runs beyond the 300-second synchronous limit?

For semrush-scraper runs that might exceed five minutes due to enableBrowserFallback for many domains, use the asynchronous run API endpoint (POST /v2/acts/<actor>/runs) and implement a webhook or polling mechanism to receive results. This avoids the hard-capped synchronous endpoint which returns HTTP 408 past 300 seconds.

The semrush-scraper README explicitly states that some domains, particularly those where SEMrush doesn't have a public overview page, trigger a "browser fallback" which involves launching a real Chrome instance, warming Google cookies, navigating to SEMrush, and rotating through residential proxy IPs. This process can easily take "60+ seconds for some domains." If your input domains list contains many such cases, a batch run is very likely to exceed the 300-second (5-minute) synchronous API timeout.

To reliably run the scraper for larger batches or domains requiring browser fallback, you must initiate the Actor asynchronously and then either poll its status or configure a webhook to be notified upon completion. This kind of orchestration could also be handled by dedicated workflow management systems like Airflow, Prefect, or custom-built solutions using cloud functions and message queues.

from apify_client import ApifyClient
import os

apify_client = ApifyClient(os.environ["APIFY_TOKEN"])
ACTOR_ID = "crawlerbros/semrush-scraper"

def run_semrush_scraper_async(domains_list: list[str], webhook_url: str = None):
    """
    Initiates an asynchronous run of the SEMrush scraper and optionally sets up a webhook.
    """
    run_input = {
        "domains": domains_list,
        "mode": "full",
        "enableBrowserFallback": True,
        "proxyConfiguration": {"useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US"}
    }

    # Define the webhook if a URL is provided
    webhooks = []
    if webhook_url:
        webhooks.append({
            "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"],
            "requestUrl": webhook_url,
            "payloadTemplate": "{\"runId\": {{run.id}}, \"datasetId\": {{run.defaultDatasetId}}, \"status\": \"{{run.status}}\", \"actorId\": \"{{run.actorId}}\"}"
        })

    # Call the Actor asynchronously
    print(f"Initiating asynchronous run for {len(domains_list)} domains...")
    actor_run = apify_client.actor(ACTOR_ID).call(
        run_input=run_input,
        webhooks=webhooks,
        timeout_secs=0, # Crucial: setting timeout_secs to 0 ensures asynchronous execution
        wait_for_finish=0 # Also ensures async, does not wait for finish
    )
    print(f"Actor run initiated. Run ID: {actor_run['id']}")
    print(f"View run details at: https://console.apify.com/actors/{ACTOR_ID}/runs/{actor_run['id']}")
    return actor_run['id']

# Example usage:
# domains_to_scrape = ["github.com", "youtube.com", "example.com", "nodata.org", "wikipedia.org", "apify.com"]
# my_webhook_endpoint = "https://your-n8n-or-service-webhook-url.com/semrush-callback"
#
# if domains_to_scrape:
#    run_id = run_semrush_scraper_async(domains_to_scrape, my_webhook_endpoint)
#    print(f"Scraper run {run_id} started asynchronously. Webhook configured to {my_webhook_endpoint}.")
# else:
#    print("No domains to scrape.")

When you call an Actor and set timeout_secs=0 (or omit it and rely on the default behavior for POST requests), the API immediately returns the runId, and the Actor execution continues in the background. You can then use this runId to poll the run status or, more efficiently, rely on the webhook to trigger your downstream processing once the run completes, fails, or is aborted. This asynchronous pattern is fundamental for building robust pipelines that aren't constrained by short API timeouts.

How does the `mode` input parameter affect the SEMrush Scraper's output?

The mode input parameter controls the scope of data returned by the semrush-scraper, allowing you to retrieve either a full set of metrics or narrow the output to authority_only, backlinks_only, or traffic_only. This enables efficient data retrieval by only fetching the specific information relevant to your use case.

The semrush-scraper offers four distinct mode options, which directly influence which fields are populated in the output for each domain. By default, the mode is set to full, providing a comprehensive set of metrics. However, if your application only requires a specific subset of data, selecting a more granular mode can potentially optimize the scraping process by reducing the amount of data that needs to be extracted and processed from the raw SEMrush page. While the Actor's pricing is per result item, fetching less data might reduce downstream processing load and storage.

Here's how each mode impacts the output fields:

full (default): This mode returns all available metrics for a domain. This includes authorityScore, visits (and visitsText), organicSearchTraffic (and organicSearchTrafficText), referringDomains (and referringDomainsText, referringDomainsChange), and backlinks (and backlinksText, backlinksChange). It also always includes type, domain, asOf, sourceUrl, and scrapedAt.

    {
      "domains": ["wikipedia.org"],
      "mode": "full"
    }
    ```
{% endraw %}


    The resulting output item would contain all fields, such as:
{% raw %}


```json
    {
      "type": "semrush_website_stats",
      "domain": "wikipedia.org",
      "authorityScore": 92,
      "visits": 1100000000,
      "visitsText": "1.1B",
      "organicSearchTraffic": 485000000,
      "organicSearchTrafficText": "485M",
      "referringDomains": 3900000,
      "referringDomainsText": "3.9M",
      "referringDomainsChange": 1.2,
      "backlinks": 2960000000,
      "backlinksText": "2.96B",
      "backlinksChange": 0.5,
      "asOf": "July 2026",
      "sourceUrl": "https://www.semrush.com/website/wikipedia.org",
      "scrapedAt": "2026-09-28T10:30:00.000Z"
    }
    ```
{% endraw %}


*   **{% raw %}`authority_only`{% endraw %}**: This mode focuses solely on retrieving the {% raw %}`authorityScore`{% endraw %}. Other metric-related fields will be omitted from the output.
{% raw %}


```json
    {
      "domains": ["google.com"],
      "mode": "authority_only"
    }
    ```
{% endraw %}


    An example output for {% raw %}`authority_only`{% endraw %}:
{% raw %}


```json
    {
      "type": "semrush_website_stats",
      "domain": "google.com",
      "authorityScore": 100,
      "asOf": "July 2026",
      "sourceUrl": "https://www.semrush.com/website/google.com",
      "scrapedAt": "2026-09-28T10:30:00.000Z"
    }
    ```
{% endraw %}


*   **{% raw %}`backlinks_only`{% endraw %}**: This mode retrieves {% raw %}`backlinks`{% endraw %}, {% raw %}`backlinksText`{% endraw %}, {% raw %}`backlinksChange`{% endraw %}, {% raw %}`referringDomains`{% endraw %}, {% raw %}`referringDomainsText`{% endraw %}, and {% raw %}`referringDomainsChange`{% endraw %}. Traffic and authority score data are excluded.
{% raw %}


```json
    {
      "domains": ["apify.com"],
      "mode": "backlinks_only"
    }
    ```
{% endraw %}


    An example output for {% raw %}`backlinks_only`{% endraw %}:
{% raw %}


```json
    {
      "type": "semrush_website_stats",
      "domain": "apify.com",
      "referringDomains": 10000,
      "referringDomainsText": "10K",
      "referringDomainsChange": 0.8,
      "backlinks": 500000,
      "backlinksText": "500K",
      "backlinksChange": 1.1,
      "asOf": "July 2026",
      "sourceUrl": "https://www.semrush.com/website/apify.com",
      "scrapedAt": "2026-09-28T10:30:00.000Z"
    }
    ```
{% endraw %}


*   **{% raw %}`traffic_only`{% endraw %}**: This mode exclusively fetches {% raw %}`visits`{% endraw %}, {% raw %}`visitsText`{% endraw %}, {% raw %}`organicSearchTraffic`{% endraw %}, and {% raw %}`organicSearchTrafficText`{% endraw %}. Backlink and authority score data are omitted.
{% raw %}


```json
    {
      "domains": ["github.com"],
      "mode": "traffic_only"
    }
    ```
{% endraw %}


    An example output for {% raw %}`traffic_only`{% endraw %}:
{% raw %}


```json
    {
      "type": "semrush_website_stats",
      "domain": "github.com",
      "visits": 450000000,
      "visitsText": "450M",
      "organicSearchTraffic": 200000000,
      "organicSearchTrafficText": "200M",
      "asOf": "July 2026",
      "sourceUrl": "https://www.semrush.com/website/github.com",
      "scrapedAt": "2026-09-28T10:30:00.000Z"
    }
    ```
{% endraw %}


Choosing the appropriate {% raw %}`mode`{% endraw %} is a key consideration for optimizing your data pipeline, especially when dealing with large volumes of domains and specific data requirements. It ensures that you only retrieve the data you truly need, potentially reducing processing overhead and clarifying your data analysis focus.

## How can I specify the proxy configuration for SEMrush Scraper runs?

You can specify the proxy configuration for {% raw %}`semrush-scraper`{% endraw %} runs using the {% raw %}`proxyConfiguration`{% endraw %} input field, which allows you to define whether to use Apify Proxy, which proxy groups to use, and even target specific countries. This is essential because SEMrush's reCAPTCHA v3 often rejects datacenter IPs, necessitating the use of residential proxies for successful scraping. Other proxy solutions could be integrated via custom Actor development, but the {% raw %}`semrush-scraper`{% endraw %} is designed around Apify Proxy.

The {% raw %}`proxyConfiguration`{% endraw %} input object is critical for ensuring the {% raw %}`semrush-scraper`{% endraw %} can bypass reCAPTCHA v3, which SEMrush uses to guard its free website checker. The Actor's README explicitly states that reCAPTCHA v3 "rejects datacenter IPs" by scoring them as 0.0 (bot). Therefore, using residential proxies is a prerequisite for reliable scraping, particularly for domains that trigger the browser fallback.

The {% raw %}`proxyConfiguration`{% endraw %} object has several key properties:

*   **{% raw %}`useApifyProxy`{% endraw %} (boolean)**: A flag to indicate whether to use the Apify Proxy. For this Actor, it should almost always be {% raw %}`true`{% endraw %}.
*   **{% raw %}`apifyProxyGroups`{% endraw %} (array of strings)**: Specifies which proxy groups from the Apify Proxy to use. The Actor's default and recommendation is {% raw %}`["RESIDENTIAL"]`{% endraw %}. Residential proxies simulate real user traffic from real residential IP addresses, making them much less likely to be blocked by services like reCAPTCHA v3.
*   **{% raw %}`apifyProxyCountry`{% endraw %} (string)**: Allows you to specify a particular country for the residential proxy IPs. For example, {% raw %}`US`{% endraw %} for United States. Apify Proxy supports country targeting, and even US state granularity (e.g., {% raw %}`US_CA`{% endraw %} for California). This can be useful if your target audience or the specific SEMrush instance you are hitting is geographically sensitive, or if you want to ensure the most stable connection from a certain region. If left unspecified, Apify Proxy will select from available residential IPs globally.

Here's an example of how you might set the {% raw %}`proxyConfiguration`{% endraw %} in your input:
{% raw %}


```json
{
  "domains": ["example.com", "apify.com"],
  "mode": "full",
  "enableBrowserFallback": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

If you omit the proxyConfiguration field entirely, the Actor defaults to using {"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US"}, which is a sensible default for most use cases. However, explicitly defining it ensures clarity and allows for customization if, for instance, you need to scrape from a non-US region.

Keep in mind that using residential proxies contributes to your Apify platform usage costs, which are separate from the Actor's event-based pricing. Residential proxy sessions typically persist for around 30 minutes, meaning the Actor might rotate through several IPs during a longer run, especially for domains requiring extensive browser fallback. This automatic rotation is managed by the Apify Proxy and the Actor's internal logic, abstracting away the complexity of proxy management from your pipeline.

How can I connect to downstream systems like n8n using webhooks?

Integrate the semrush-scraper output directly into downstream systems like n8n by configuring a webhook on Actor completion to push dataset item URLs or directly trigger a workflow for data extraction and transformation. This eliminates the need for manual polling or complex custom API integrations on your end.

Webhooks provide a real-time, event-driven mechanism to connect Apify Actor runs with other services. When an Actor run finishes (or any other specified event occurs), Apify sends an HTTP POST request to a URL you define. This payload can contain details about the run, including the ID of the default dataset where the results are stored. This event-driven approach could also be replicated with custom orchestration logic using message queues, but webhooks provide a direct integration.

For n8n, a powerful workflow automation tool, this integration is particularly seamless. n8n has a dedicated Apify Trigger node that can listen for ACTOR.RUN.SUCCEEDED or ACTOR.RUN.FAILED events. When an Actor run completes, the Apify platform sends a notification to your n8n webhook URL, providing the run ID and default dataset ID. Your n8n workflow can then use the Apify node to fetch the dataset items and proceed with transformation, storage, or further processing.

Here's an example of how you might create a webhook for an Actor run using the Apify API (though the apify-client in Python simplifies this, as shown in the previous section):

curl -X POST "https://api.apify.com/v2/webhooks?token=YOUR_API_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
           "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"],
           "requestUrl": "https://your-n8n-webhook-url.com/semrush-data-processor",
           "payloadTemplate": "{\"runId\": {{run.id}}, \"datasetId\": {{run.defaultDatasetId}}, \"status\": \"{{run.status}}\", \"actorId\": \"{{run.actorId}}\"}"
         }'

The payloadTemplate is highly customizable. You can include any details from the run object, allowing your downstream system to receive precisely the information it needs to identify the source of the data and fetch it. This event-driven architecture is far more efficient than polling, reducing idle waiting times and immediately kicking off processing as soon as data is available.

For setting up webhooks programmatically, ensure your webhooks array in the call method of apify_client.actor() is correctly structured, specifying the eventTypes, requestUrl, and payloadTemplate.

Designing input for scalability and partial failures

Structure the semrush-scraper input by pre-filtering domains based on prior processing state and by considering smaller batch sizes, which can isolate failures and simplify recovery compared to monolithic runs. This strategy enhances resilience and makes debugging easier when issues arise within a large input set.

The semrush-scraper accepts an array of domains as its primary input. While it's tempting to throw a massive list of domains into a single run, this has several drawbacks for robust pipelines:

Single Point of Failure: If a single run fails (e.g., due to a platform issue, network instability, or hitting maxTotalChargeUsd), you might lose context on which domains were successfully processed and which were not, complicating recovery.
Debugging Complexity: Diagnosing issues within a run that processed thousands of domains is much harder than examining a run with a few hundred.
Timeout Risk: As discussed, domains requiring browser fallback can be slow. A large batch amplifies the risk of exceeding platform timeouts, even with asynchronous calls.
Cost Control: While maxTotalChargeUsd can terminate a run, splitting into smaller batches gives you more granular control over spend and allows for pause-and-review points.

Instead, combine your state management (filtering already processed domains) with batching. This means taking your domains_for_new_run (after filtering) and splitting them into smaller, manageable chunks (e.g., 500-1000 domains per chunk). You can then initiate separate asynchronous Actor runs for each batch. A critical platform constraint to remember here is that a request queue can only be processed by one Actor or task run at a time. So, if your original domain list was in a request queue, you'd need a different strategy, like the one shown, where the orchestrator directly passes domains arrays as input. Batching and orchestration could also be managed by external systems like AWS Step Functions, Azure Logic Apps, or Google Cloud Workflows, rather than solely within Apify.

from apify_client import ApifyClient
import os

apify_client = ApifyClient(os.environ["APIFY_TOKEN"])
ACTOR_ID = "crawlerbros/semrush-scraper"
PROCESSED_DOMAINS_KV_STORE = "semrush-processed-domains"

# load_processed_domains function is defined as above
# ...

def chunk_list(lst: list, n: int):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

def orchestrate_batched_semrush_runs(all_domains_to_check: list[str], batch_size: int = 500, webhook_url: str = None):
    """
    Orchestrates batched, asynchronous runs of the SEMrush scraper with state management.
    """
    existing_processed = load_processed_domains(apify_client)
    domains_for_new_runs = [d for d in all_domains_to_check if d not in existing_processed]

    if not domains_for_new_runs:
        print("No new domains to process.")
        return []

    print(f"Found {len(domains_for_new_runs)} new domains for processing.")
    run_ids = []
    for i, domain_batch in enumerate(chunk_list(domains_for_new_runs, batch_size)):
        print(f"Starting batch {i+1} with {len(domain_batch)} domains...")
        run_input = {
            "domains": domain_batch,
            "mode": "full",
            "enableBrowserFallback": True,
            "proxyConfiguration": {"useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US"}
        }

        webhooks = []
        if webhook_url:
            webhooks.append({
                "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"],
                "requestUrl": webhook_url,
                "payloadTemplate": "{\"runId\": {{run.id}}, \"datasetId\": {{run.defaultDatasetId}}, \"status\": \"{{run.status}}\", \"actorId\": \"{{run.actorId}}\", \"batchNumber\": " + str(i+1) + "}"
            })

        actor_run = apify_client.actor(ACTOR_ID).call(
            run_input=run_input,
            webhooks=webhooks,
            timeout_secs=0,
            wait_for_finish=0
        )
        run_ids.append(actor_run['id'])
        print(f"Batch {i+1} started with Run ID: {actor_run['id']}")
    return run_ids

# Example of full pipeline orchestration:
#
# full_list_of_all_domains = ["domain1.com", "domain2.org", "domainN.net"] # Your complete list
# my_callback_webhook = "https://your-n8n-or-service-webhook-url.com/semrush-batch-callback"
#
# initiated_run_ids = orchestrate_batched_semrush_runs(full_list_of_all_domains, batch_size=200, webhook_url=my_callback_webhook)
# print(f"All batches initiated with run IDs: {initiated_run_ids}")
#
# # Your webhook handler would then need to process each batch's dataset and update the master state.

This batched approach allows for parallel processing of smaller, more manageable units. If one batch run fails, others can still succeed. Your state management logic (updating processed_domains) would then need to aggregate results from all completed batches.

Understanding the `semrush-scraper` event-based pricing

The semrush-scraper charges $0.002 for each "result" item emitted, with potential discounts based on platform usage, plus a $0.005 "Actor Start" charge per GB of memory allocated once per run, in addition to standard Apify platform usage. These event prices are not subject to subscription plan rates.

Understanding the cost model is crucial for any pipeline. The semrush-scraper uses a PAY_PER_EVENT pricing model, meaning you are charged for specific actions or outputs generated by the Actor, on top of the general Apify platform usage (which covers things like compute, proxy, and storage).

Here are the specific charged events and their prices:

"result" (apify-default-dataset-item): This event is charged for every single result record that the Actor pushes to its default dataset.
- Base Price: $0.002 per event (per item). Discounts may apply based on platform usage.
- Impact of input: The number of "result" events scales directly with the number of domains you provide in the input. If you request stats for 1,000 domains and all yield a result, you will incur 1,000 "result" charges.
"Actor Start" (apify-actor-start): This event is charged when the Actor begins its execution.
- Price: $0.005 per GB of memory allocated to the run. Even if the Actor uses less than 1GB, the minimum charge is for 1GB.
- Impact of input: This is a flat charge per run based on memory allocation and does not directly scale with the number of domains input, but rather with the resources required for the Actor to initiate and operate.

It's important to remember that these event charges are separate from your general Apify platform usage, which is billed based on your specific Apify plan's rates for compute, proxy, and storage. The event prices themselves do not change based on your Apify subscription plan, though your overall platform usage costs will. When considering the total cost of a run, you must factor in both the event charges and the platform usage. For example, using residential proxies (which this Actor requires for many domains) will contribute to your platform proxy usage.

What are the limitations and considerations for SEMrush data pipelines?

The semrush-scraper is limited to free-tier SEMrush data, requires residential proxies which can increase run duration, and relies on a dynamic anti-detection browser fallback that makes run times variable and potentially long for "notFound" domains. These factors necessitate careful pipeline design to manage cost and performance expectations.

While powerful, the semrush-scraper (and any scraping tool) operates within specific boundaries:

Free-Tier Data Only: The Actor only scrapes SEMrush's publicly available "Website Traffic Checker" data. It does not provide access to paid SEMrush features like full keyword research, competitor analysis, or detailed backlink profiles. If your use case requires deeper SEMrush insights, this Actor will not suffice, and you'll need to consider a paid SEMrush API subscription.
Residential Proxy Requirement: SEMrush employs reCAPTCHA v3, which heavily penalizes datacenter IP addresses. As such, the Actor explicitly requires Apify Proxy Residential IPs. This is prefilled and necessary for the Actor to function effectively, particularly for domains triggering the browser fallback. While Apify Proxy handles the IP rotation and management, it adds to your platform usage cost. Other proxy solutions exist, but integrating them would require modifying the Actor's code or building a custom scraper. Residential proxy sessions typically persist for around 30 minutes, which the Actor accounts for by rotating IPs, but it can still influence the consistency of very long-running browser fallback operations. You can target specific countries (e.g., US or even US_TX for Texas) via apifyProxyCountry if your use case demands it.
Variable and Potentially Long Run Times: As noted in the Actor's FAQ, domains that SEMrush doesn't have a public overview page for (the "false 404" cases) require the "browser fallback" mechanism. This involves launching a real Chrome browser, warming Google cookies, navigating to SEMrush, and rotating through up to 8 residential proxy IPs to find one with a sufficient reCAPTCHA score. This significantly extends run duration for those specific domains (potentially 60+ seconds per domain), impacting the overall run time for batches with many such domains. Fast-path domains, conversely, complete in under 5 seconds.
No Nulls/Sentinels: The output schema explicitly states, "Empty fields are omitted, no nulls, no sentinels." This means your downstream parsing logic should anticipate the absence of certain fields rather than expecting null values. For example, if a domain has no organic search traffic, the organicSearchTraffic field might simply not be present in the JSON record.
Data Freshness: The asOf field provides the data month (e.g., July 2026), indicating the freshness of the SEMrush data. This is useful for tracking changes over time but means you're scraping data that SEMrush itself last updated, which might not be real-time.

These limitations aren't showstoppers, but they are crucial for designing a realistic and efficient data pipeline. Account for the variable run times, the specific data available, and the proxy requirements when planning your infrastructure, budget, and data processing logic.

Checked against the Actor's input schema and Apify docs on 2026-10-08.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Building a GOG Price Index for 2,000 Titles Without Authentication

Crawler Bros — Wed, 07 Oct 2026 19:02:37 +0000

Aggregating Digital Storefront Metadata for PC Video Games

Tracking pricing shifts and catalog movements across PC game storefronts requires structured, reliable catalog data. While storefront web interfaces present catalog information for humans, automated collection requires targeting the underlying public API endpoints that serve pricing, platform compatibility, developers, and player ratings.

GOG.com offers a vast catalog of DRM-free titles, but collecting historical price points and availability across thousands of games manually or via raw custom web scrapers presents ongoing maintenance challenges. Dynamic web pages change DOM elements frequently, requiring scraper maintenance whenever the presentation layer updates.

The GOG.com Game Scraper addresses this by directly querying GOG's public catalog API. It extracts rich game metadata—including prices, user ratings, developers, screenshots, and platform support—without requiring account authentication or custom proxy configurations.

Programmatic Access to the GOG Catalog API

The scraper functions as an interface to GOG's public catalog API, exposing three main modes of operation: catalog searches by title, browsing by genre, and direct lookup using specific game IDs.

Because the underlying catalog endpoints do not require session authentication, queries run directly against public endpoints. The returned dataset normalizes GOG's raw response structures into flat, structured JSON records suitable for ingestion into analytical pipelines or database systems.

Key attributes extracted for each game entry include:

Identification: Internal GOG game ID, official title, and store URL.
Pricing Data: Current price, base/original price, discount percentage, and currency.
Metadata: Genres, developer, publisher, release date, and community ratings.
System Details: Supported operating systems (Windows, macOS, Linux).
Media Assets: Screenshot URLs, cover images, and promotional banners.

Because the Actor fetches clean data directly from API responses, execution remains fast and resilient against visual layout redlines or front-end redesigns.

Scraper Configuration and Input Parameters

Running the Actor involves supplying JSON input configuration depending on whether your objective is broad monitoring or targeted tracking.

Searching by Title or Keywords

To track specific franchises or titles, pass a search term string to target GOG's catalog search endpoint:

{
  "search": "Cyberpunk"
}

Filtering by Genre

When building a broader index of specific market segments—such as indie strategy games or classical RPGs—you can direct the scraper to aggregate games published within designated store categories:

{
  "genre": "role-playing"
}

Direct Retrieval by Game ID

If your system already tracks GOG catalog identifiers, direct ID retrieval bypasses search relevance algorithms entirely, returning exact records for specified items:

{
  "gameIds": [
    "1207658924",
    "1440708155"
  ]
}

Step-by-Step Walkthrough: Executing a Catalog Collection Run

Setting up an automated run via Apify's API or Python SDK takes a few minutes.

Initialize the Actor Run: Call the Actor through the Apify platform API or Python client, passing your chosen input JSON configuration.
Monitor Execution: The Actor connects to GOG's API endpoints, handles request pagination, and formats incoming game objects.
Consume Dataset Output: Upon completion, the run populates a default dataset containing standard JSON items for each game processed.

Here is a practical Python snippet demonstrating how to trigger a run and fetch results using the official apify-client library:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Set up input to scrape GOG RPG titles
run_input = {
    "genre": "role-playing"
}

# Run the gog-scraper Actor
run = client.actor("crawlerbros/gog-scraper").call(run_input=run_input)

# Fetch and print results from the dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
for game in dataset_items:
    print(f"Title: {game.get('title')} | Price: {game.get('price')} {game.get('currency')}")

A typical JSON result record returned by the dataset contains structured fields ready for analysis:

{
  "id": "1207658924",
  "title": "The Witcher 3: Wild Hunt",
  "price": 9.99,
  "basePrice": 39.99,
  "discount": 75,
  "currency": "USD",
  "developer": "CD PROJEKT RED",
  "publisher": "CD PROJEKT RED",
  "rating": 4.9,
  "genres": ["Role-playing", "Story Rich", "Open World"],
  "platforms": {
    "windows": true,
    "mac": false,
    "linux": false
  },
  "url": "https://www.gog.com/game/the_witcher_3_wild_hunt"
}

Actor Execution Pricing Structure

The Actor operates under a pay-per-event pricing model based on output generation and execution initialization.

Each result item emitted to the default dataset costs $0.005 on the FREE tier.
Discount-tier pricing reduces this per-item cost for higher platform tiers: $0.00433 on BRONZE, $0.00367 on SILVER, and $0.003 on GOLD, PLATINUM, and DIAMOND tiers.
The Actor Start event costs $0.005 per GB of memory allocated to the run upon invocation.

Platform usage for the run is billed separately at your Apify plan's rates.

For instance, if a run collects 1,000 game records on the FREE tier using 1 GB of memory, the event charges include $0.005 for the start event plus 1,000 individual result events at $0.005 each ($5.00 total for results), alongside standard platform usage billed separately.

System Limitations and Boundaries

While effective for catalog and price tracking, this approach does NOT handle user-specific data, such as regional price differences requiring authenticated account sessions, personalized user libraries, purchase history, or restricted regional storefront listings.

If you want to reproduce this, the Actor is GOG.com Game Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-07. Check the Actor page for the current rates.

Filtering by maxPriceEur is cheaper than post-filtering rental listings

Crawler Bros — Wed, 07 Oct 2026 10:02:46 +0000

Housing aggregators like Nestpick pull inventory from platforms such as Spotahome, Blueground, and HousingAnywhere. Working with mid- to long-term rental data across multiple international markets presents two distinct data engineering challenges: handling cross-currency comparisons and resolving the real external booking URLs hidden behind aggregator redirects.

If you attempt to fetch an entire city's housing inventory and filter the prices client-side, you process and pay for records that get discarded immediately. Furthermore, cross-currency comparisons break if you rely on local listing prices, as a listing in Seoul returns values in South Korean Won while a listing in Berlin returns Euros.

The Nestpick Scraper solves this by providing server-side filtering on normalized Euro values, alongside a dedicated enrichment mode that extracts final partner URLs.

Server-Side EUR Normalization Reduces Fetch Costs

When querying housing inventory in cities with non-EUR local currencies, Nestpick normalizes prices to Euros internally. The actor exposes this normalization directly in its query interface using minPriceEur and maxPriceEur.

Applying these parameters in your actor input strips non-matching properties at the source, ensuring you only receive data items that meet your target criteria. Applying post-processing filters on raw extraction outputs results in higher item charges for data you drop.

Consider this request configured to extract verified studios in Seoul priced under €2,000 per month:

{
  "mode": "search",
  "city": "Seoul",
  "category": "apartments",
  "propertyType": "studio",
  "maxPriceEur": 2000,
  "verifiedOnly": true,
  "sortBy": "priceAsc",
  "maxItems": 50
}

The output yields normalized records where priceEur is calculated alongside the raw local price:

{
  "listingId": "21514575",
  "title": "Furnished Studio Apartment in Gangnam",
  "address": "Gangnam-gu, Seoul",
  "propertyType": "studio",
  "providerName": "Spotahome",
  "providerCode": "spotahome",
  "price": 2200000,
  "priceCurrency": "KRW",
  "priceEur": 1520,
  "sizeSqm": 32,
  "rooms": 1,
  "availableFrom": "2026-04-01",
  "minStayMonths": 3,
  "amenities": ["wifi", "air_conditioning", "washing_machine"],
  "verified": true,
  "sourceUrl": "https://www.nestpick.com/pick/21514575/",
  "city": "Seoul",
  "searchCategory": "apartments",
  "recordType": "listing",
  "scrapedAt": "2026-03-30T10:00:00.000Z"
}

By leveraging maxPriceEur: 2000, any listing exceeding that threshold is omitted before the dataset item is generated.

Resolving Direct Booking Links with `listingDetails` Mode

A search payload returns Nestpick's aggregated view of a listing, but market research or lead generation workflows often require the original source URL (e.g., the exact direct page on Blueground or HousingAnywhere).

To obtain these target links without scraping every partner platform individually, you can execute a secondary pass using mode: "listingDetails". You supply either an array of target listing IDs via listingIds or full Nestpick listing URLs via listingUrls.

{
  "mode": "listingDetails",
  "listingIds": ["21514575", "20393877"]
}

Executing this request updates the dataset schema for those records, appending provider-specific identifiers and the resolved booking target:

{
  "listingId": "21514575",
  "providerPropertyCode": "sp-kr-8841",
  "cityCode": "seoul",
  "externalListingUrl": "https://www.spotahome.com/seoul/for-rent:studios/21514575",
  "recordType": "listingDetail"
}

This two-step workflow keeps initial discovery runs fast and targeted, allowing you to run detail enrichment only on properties that pass your preliminary pipeline criteria.

Processing Search vs. Detail Pipeline Runs

To collect and enrich listings using Nestpick Scraper, follow these steps:

Define Your Initial Search Parameters: Construct a JSON payload setting mode to "search". Pass your target city, select a category (rooms or apartments), and narrow results using schema filters such as propertyType, minRooms, minSqm, verifiedOnly, or specific options from the 24 supported values in amenity (e.g., balcony, pets, elevator).
Execute Search Run: Run the actor to populate your default dataset with normalized records capped by your maxItems value.
Extract Listing IDs for Enrichment: Parse the dataset output for listingId values matching properties you need to process further.
Execute Listing Details Run: Re-run the actor with mode set to "listingDetails" and pass the collected IDs into listingIds. Extract the externalListingUrl field from the output dataset to route users or downstream parsers directly to the underlying booking platform.

Event-Based Execution Pricing

Nestpick Scraper uses pay-per-event pricing on the Apify platform.

Each result costs $0.005 on the FREE tier, plus a one-time start charge of $0.005 per GB of memory allocated to the run. Platform usage for the run is billed separately at your Apify plan's rates.

Under Apify's discount tiers, the per-result event price drops based on your account tier:

FREE: $0.005 per result
BRONZE: $0.00433 per result
SILVER: $0.00367 per result
GOLD: $0.003 per result
PLATINUM: $0.003 per result
DIAMOND: $0.003 per result

Filtering out unwanted items before extraction using parameter inputs directly limits the number of result events generated during a execution.

System Limitations and Constraints

This scraper relies entirely on server-side parameters mapped directly to Nestpick's public search API. As a result, client-side UI filters that lack corresponding backend GET parameters cannot be targeted.

Specifically, filtering by "Corporate Housing" is not supported because Nestpick applies that filter client-side in JavaScript rather than exposing it through a dedicated URL parameter or server query flag. Pipeline architectures requiring corporate housing tags must collect property records under general categories and evaluate property attributes downstream.

Nestpick Scraper is what these steps drive. The README covers the inputs this article skipped, including the ones that change how much a run costs.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-07. Check the Actor page for the current rates.

Ingesting FlightAware Waypoint and Airport Boards Without API Keys

Crawler Bros — Tue, 06 Oct 2026 19:00:37 +0000

Flight operations teams and logistics tracking pipelines frequently need real-time data on gate changes, runway taxi times, and waypoint trajectories. Integrating commercial aviation APIs often comes with steep enterprise pricing tiers, strict licensing terms, and per-query limits that break ingestion jobs during disruption spikes.

When scraping public web interfaces directly, teams run into aggressive rate limits, dynamic rendering hurdles, and nested page layouts. The FlightAware Flight Tracking Scraper handles these ingestion challenges by parsing FlightAware endpoints directly over HTTP without requiring account logins or private API keys.

Understanding the Four Data Collection Modes

The actor structures flight tracking into four distinct scraping modes:

flightStatus: Pulls live flight data including gate departure and arrival milestones, terminal data, estimated and actual takeoff/landing timestamps (in ISO UTC), aircraft details, and live telemetry (currentLatitude, currentLongitude, altitudeFl, groundspeedKts, and headingDegrees).
flightHistory: Captures the last two weeks of historical flight instances for a specific flight ident or aircraft registration, including status flags (Cancelled, Scheduled, Landed), duration metrics, and local time zones.
airportBoard: Scrapes current arrivals, departures, en-route traffic, or scheduled departures for a specified IATA or ICAO airport code.
flightTrackLog: Returns waypoint-by-waypoint position reports and milestone events for the most recent flight instance.

This approach does not scrape arbitrary historical track logs from months prior; flightTrackLog exclusively extracts data for the most recently tracked instance of a flight ident.

Ingesting Waypoint Telemetry for Route Analysis

For post-flight performance metrics or flight path visualisations, you need sequential position reports rather than single snapshot coordinates. The flightTrackLog mode emits two types of records distinguished by the recordKind field:

waypoint: Sequential coordinates containing latitude, longitude, altitudeFeet, groundspeedKts, verticalRateFpm, and verticalDirection (Climbing, Descending, or Level).
event: Milestone records logging operational events such as Left Gate, Taxi Time, Departure, Arrival, and Gate Arrival.

Here is an example input configuration to pull the track log of an active flight:

{
  "mode": "flightTrackLog",
  "flightIdent": "AAL100",
  "maxItems": 200
}

Because long-haul flights generate several hundred telemetry points, setting maxItems close to the upper limit of 200 ensures your pipeline captures the descent profile and runway approach waypoints.

Processing Live Hub Traffic with Airport Board Scraping

Monitoring inbound and outbound congestion at specific airports requires structured access to departure and arrival boards. The airportBoard mode accepts either 3-letter IATA codes (JFK) or 4-letter ICAO codes (KJFK).

You can isolate specific operations by combining boardType with airline filters:

{
  "mode": "airportBoard",
  "airportCode": "KJFK",
  "boardType": "departures",
  "airlineIcao": "DAL",
  "maxItems": 50
}

When parsing the airport board output, note that FlightAware's web board only displays current on-screen rows (typically around 20 per section, varying by airport hub size). The scraper emits all available rows up to your maxItems cap without artificial pagination.

Automating Ingestion with Python

You can run this scraper programmatically from your orchestration platform (such as Dagster, Airflow, or Prefect) using the official client library.

Setup and Execution

Install the client:

pip install apify-client

Execute the run and iterate through the output records:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "mode": "flightStatus",
    "flightIdent": "UAL1",
    "maxItems": 10
}

# Run the actor and wait for completion
run = client.actor("crawlerbros/flightaware-flight-tracking-scraper").call(run_input=run_input)

# Fetch dataset items
dataset_items = client.dataset(run["defaultDatasetId"]).iterate_items()

for item in dataset_items:
    flight_id = item.get("flightIdent")
    status = item.get("flightStatus", "scheduled")

    actual_dep = item.get("actualDepartureTime")
    sched_dep = item.get("scheduledDepartureTime")

    print(f"Flight {flight_id} Status: {status}")
    print(f"Scheduled Gate Departure: {sched_dep} | Actual: {actual_dep}")

    # Live position metrics for airborne flights
    if status == "airborne":
        lat = item.get("currentLatitude")
        lon = item.get("currentLongitude")
        alt = item.get("altitudeFl")
        print(f"Coordinates: {lat}, {lon} at FL{alt}")

The output dataset omits empty fields automatically, meaning telemetry keys like currentLatitude or distanceRemainingMiles will only be present on the record when the flight is actively airborne.

Execution Pricing Model

The scraper operates on a pay-per-event pricing model alongside standard platform usage. Every run incurs charges based on specific operational events:

Actor Start: $0.005 per GB of memory allocated to the run.
Dataset Result: $0.005 per emitted item on the default tier.

Users on discounted Apify tiers pay lower result rates:

BRONZE: $0.00433 per item
SILVER: $0.00367 per item
GOLD, PLATINUM, and DIAMOND: $0.003 per item

Platform usage consumed by the run (compute duration and memory allocation) is billed separately at your specific Apify account plan rates.

Handling Proxy Fallbacks and Rate Limits

FlightAware monitors direct scraper traffic and periodically blocks or rate-limits requests from standard datacenter IPs. The scraper includes an automated proxy fallback mechanism. By default, it attempts a direct HTTP request to minimise latency, but if FlightAware triggers a challenge or rate-limit response, the actor automatically routes requests through the built-in Apify proxy configuration.

For historical route analysis across multiple days, you can exclude disrupted flights by setting "includeCancelled": false in flightHistory mode, saving processing time downline when calculating typical scheduled run times.

Pipelines that require continuous real-time coordinate streams at one-second intervals should use direct ADS-B receiver networks instead, as this HTTP scraping workflow is designed for polling discrete status updates and post-flight operational summaries.

Runs in this article used FlightAware Flight Tracking Scraper. Its README is the reference for input fields and output structure; this post is only one path through them.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-06. Check the Actor page for the current rates.

Handling Dynamic S3 Logos and Non-Browser 302s on CareerJunction

Crawler Bros — Tue, 06 Oct 2026 10:02:59 +0000

Tracking hiring trends in South Africa requires pulling live structured data from localized job boards. CareerJunction host tens of thousands of active postings across 26 job categories, but extracting these listings programmatically presents two technical roadblocks: strict HTTP client blocking that 302-redirects non-browser requests to /not-found, and short-lived Amazon S3 image URLs that break logo assets within 24 hours of scraping.

The CareerJunction Jobs Scraper extracts live records from this platform while handling browser-fingerprint challenges and image rehosting automatically.

Bypassing the `/not-found` Redirect on CareerJunction Listings

Standard HTTP requests sent via basic curl or lightweight Python libraries like requests fail when hitting CareerJunction listing URLs. Even if you pass a modern browser User-Agent header, the server's bot-detection infrastructure checks for specific TLS/browser fingerprints. If the client fails this check, the server issues a 302 redirect to a /not-found endpoint, hiding the underlying job data.

To observe this issue, curl a direct job listing URL from a terminal:

curl -I -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)" \
  "https://www.careerjunction.co.za/digital-coordinator-job-2642426.aspx"

Instead of a 200 OK response with HTML content, the server returns:

HTTP/1.1 302 Found
Location: https://www.careerjunction.co.za/not-found

The scraper bypasses this security layer by using browser-like fetch signatures without requiring paid proxies or manual cookie management. This allows direct ingestion of the listing's target content and underlying JSON payloads.

Resolving 24-Hour Expiry on Company Logos

CareerJunction serves employer logos via presigned AWS S3 URLs. When a scraper extracts the raw HTML src attribute, it captures a temporary URL structured like this:

https://cj-employer-logos.s3.amazonaws.com/logo_123.png?AWSAccessKeyId=AKIA...&Expires=1710000000&Signature=...

Because these presigned links expire after roughly 24 hours, storing the raw URL in a database results in broken asset links the next day.

To solve this, the scraper fetches the logo binary during execution, rehosts it in permanent platform storage, and yields two distinct fields in the output dataset:

companyLogoUrl: A permanent, rehosted image URL safe for long-term database storage.
companyLogoUrlOriginal: The raw, presigned S3 link captured directly from CareerJunction.

Structured Field Extraction and Output Schema

CareerJunction packs metadata—including employment type, seniority level, and Employment Equity status—into a single composite string on the listing page (such as "Permanent Senior EE position"). The scraper parses these raw strings into structured fields.

When a field is missing on the source page (e.g., when an employer does not disclose salary information), the key is completely omitted from the output JSON record rather than returning empty strings or null values.

A fully parsed item record contains the following keys:

{
  "jobId": "2642426",
  "title": "Senior Backend Developer",
  "companyName": "Tech Corp",
  "companyUrl": "https://www.careerjunction.co.za/companies/tech-corp",
  "companyLogoUrl": "https://api.apify.com/v2/key-value-stores/.../records/logo.png",
  "companyLogoUrlOriginal": "https://cj-employer-logos.s3.amazonaws.com/...",
  "location": "Cape Town",
  "country": "South Africa",
  "countryCode": "ZA",
  "industry": "Information Technology",
  "positionText": "Permanent Senior EE position",
  "employmentType": "Permanent",
  "seniorityLevel": "Senior",
  "isEmploymentEquity": true,
  "salaryText": "R80 000 - R100 000 per month",
  "salaryMin": 80000,
  "salaryMax": 100000,
  "salaryCurrency": "ZAR",
  "salaryPeriod": "Monthly",
  "descriptionHtml": "<p>Full job description...</p>",
  "descriptionText": "Full job description...",
  "datePosted": "2026-03-20T08:00:00.000Z",
  "validThrough": "2026-04-20T08:00:00.000Z",
  "sourceUrl": "https://www.careerjunction.co.za/senior-backend-developer-job-2642491.aspx",
  "recordType": "job",
  "scrapedAt": "2026-03-30T10:15:00.000Z"
}

Configuring Search Modes and Filters

The scraper operates in two distinct operational modes via the mode parameter:

search: Discovers listings dynamically via keyword queries, server-side region codes, and job categories.
byUrls: Directly processes an explicit array of CareerJunction listing URLs provided in the jobUrls input parameter.

Filtering Options

Server-Side Regional Filtering: The region parameter accepts specific region keys like westernCape, gauteng, or workFromHome. These correspond to CareerJunction's native query parameters for fast server-side target selection.
Client-Side Substring Matching: The location parameter filters strings post-fetch (e.g., matching "Sandton" or "Rosebank" within a broader region).
Client-Side Attribute Filtering: employmentTypes, seniorityLevels, employmentEquityOnly, minSalary, and postedWithinDays narrow down results post-extraction. If a listing's raw positionText does not follow standard parsing conventions, the scraper passes the record through rather than silently dropping data.

Example Input Configuration

This JSON input targets recent IT jobs in the Western Cape with detailed descriptions enabled:

{
  "mode": "search",
  "keywords": "developer",
  "category": "informationTechnology",
  "region": "westernCape",
  "sortBy": "newest",
  "employmentEquityOnly": false,
  "minSalary": 50000,
  "postedWithinDays": 7,
  "fetchFullDetails": true,
  "maxItems": 100
}

Running the Scraper Step-by-Step

Step 1: Define Input Schema

Create a JSON configuration file input.json containing your target parameters and search restrictions:

{
  "mode": "search",
  "keywords": "accountant",
  "category": "finance",
  "region": "gauteng",
  "maxItems": 20
}

Step 2: Execute the Actor

Run the scraper using the Apify JavaScript SDK or Python API client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("crawlerbros/careerjunction-scraper").call(
    run_input={
        "mode": "search",
        "keywords": "accountant",
        "category": "finance",
        "region": "gauteng",
        "maxItems": 20,
    }
)

dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
for item in dataset_items:
    print(f"{item.get('title')} - {item.get('companyName')}")

Step 3: Consume Dataset Output

Retrieve extracted records from the resulting default dataset. Disable fetchFullDetails ("fetchFullDetails": false) if you only require search-level card data (title, company, location, salary text) to increase execution speed.

Event Cost Structure

The scraper operates under a Pay-Per-Event billing model. Charges are calculated based on explicit runtime events:

Result event (apify-default-dataset-item): Each result costs $0.005 on the FREE tier ($0.00433 on BRONZE, $0.00367 on SILVER, and $0.003 on GOLD, PLATINUM, and DIAMOND tiers).
Actor Start event (apify-actor-start): Charged at $0.005 per GB of memory allocated to the run.

Platform usage for the run is billed separately at your Apify plan's rates.

Limitations and Operational Bounds

While the scraper handles job search pages, direct job URLs, and listing details, it cannot scrape dedicated employer/company profile pages under the /companies/... route. CareerJunction protects company profile pages behind an additional bot-challenge layer that blocks direct HTTP collection methods. Accessing company profile data requires direct navigation through standard job detail listings instead.

Runs in this article used CareerJunction Jobs Scraper. Its README is the reference for input fields and output structure; this post is only one path through them.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-10-06. Check the Actor page for the current rates.

Why Yelp Scraper Returns Incomplete Data Without Error

Crawler Bros — Tue, 06 Oct 2026 07:41:38 +0000

As data engineers, we often encounter tools that promise straightforward data extraction. The Yelp Scraper (available on the Apify Store: https://apify.com/crawlerbros/yelp-scraper) is one such tool, designed to pull business data, reviews, and ratings from Yelp. While its input and output schemas seem clear, the implicit failure modes, the ways it can return incomplete or misleading data without an explicit error, are often where the real work begins. This article will focus on those less-obvious constraints and how to defensively code against them.

When does `runTimeoutSecs` lead to silently incomplete data?

The runTimeoutSecs parameter (default 1800 seconds or 30 minutes) acts as a wall-clock cap on the run. When this limit is reached, the Actor stops new fetches, pushes everything it has collected so far, and exits cleanly. This means your dataset will contain whatever was collected up to that point, which might be a partial result set if the scraper was in the middle of processing a large searchLimit or many directUrls. Crucially, this is not an error state from the Actor's perspective.

This mechanism is particularly relevant when you're using the synchronous run API. The synchronous run endpoint hard-caps at 300 seconds and returns an HTTP 408 (Request Timeout) past that. This means if your client makes a synchronous request and the Actor run on the platform exceeds 300 seconds, the client's connection will terminate with an HTTP 408, and it will not receive any further data for that specific request. However, the Actor itself will continue to run on the Apify platform until its runTimeoutSecs is met. For runs expected to last longer than 5 minutes, you must use the asynchronous run endpoint and poll for its status or configure a webhook.

Consider this scenario:

import apify_client
from datetime import datetime, timedelta
import time

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

actor_id = "crawlerbros/yelp-scraper"
run_input = {
    "searchTerms": ["coffee shop"],
    "locations": ["London, UK"],
    "searchLimit": 100, # A high limit to challenge the timeout
    "runTimeoutSecs": 400 # Just over the 300s synchronous cap
}

try:
    print(f"Starting Actor {actor_id} with runTimeoutSecs: {run_input['runTimeoutSecs']}")

    # This call would typically be client.actor(actor_id).call(run_input=run_input)
    # The Apify Python client's .call() method might internally poll for longer than 300 seconds
    # if runTimeoutSecs allows, handling the asynchronous nature.
    # However, if using a raw HTTP client for a synchronous call, a 300s hard cap is real
    # and would return HTTP 408 to the client after 300 seconds.
    run = client.actor(actor_id).call(run_input=run_input)

    # If the run finishes within the synchronous client's timeout, you get results.
    # If it exceeds 300s, an HTTP client would typically throw a timeout error,
    # but the run would still be happening on Apify until its runTimeoutSecs is met.

    print(f"Run completed with status: {run['status']}")
    print(f"Access dataset items via: {client.dataset(run['defaultDatasetId']).url}")

except Exception as e:
    print(f"An error occurred during the run call, possibly a client-side timeout: {e}")
    # In a real synchronous HTTP client timeout, you'd get an error,
    # but the run would still be happening on Apify. You'd need to
    # find its dataset ID to retrieve any collected data.

print("Always use asynchronous execution for runs expected to exceed 300 seconds.")
print("Checked against the Actor's input schema and Apify docs on 2026-10-06")

In this case, runTimeoutSecs of 400 seconds combined with a synchronous API call might lead to a TimeoutError on the client side if not handled by the client library, because the synchronous run endpoint would return HTTP 408 after 300 seconds. The Actor on the platform would continue for its full 400 seconds, pushing data to the default dataset, but your client would never receive a direct response containing the dataset ID for that specific request. Always design your long-running data pipelines to use asynchronous execution and robust status checking.

How do I handle missing review data from `reviewLimit`?

The reviewLimit input parameter controls the maximum number of reviews to collect per business. However, reviews are extracted only from the reviews visible on the first page load of a business listing. Even if you set reviewLimit up to 50, only reviews rendered on the page will be captured. If a business has fewer reviews visible on the first page than your reviewLimit, or if reviewLimit is set to 0, the output for reviews will contain fewer items than expected or be an empty array.

This behavior implies that even if you set reviewLimit: 50, you might only get a handful of reviews, or even fewer if the business itself has only a few. The reviews array in the output will reflect what was actually collected. To detect this defensively, compare the length of the reviews array for each business against your reviewLimit. If len(business['reviews']) < review_limit (and review_limit > 0), it might indicate either a natural scarcity of reviews for that business or that the scraper couldn't load more than the first page.

Here's a Python snippet to check for this:

import apify_client

# Initialize Apify client
client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

# Example run configuration
actor_id = "crawlerbros/yelp-scraper"
run_input = {
    "searchTerms": ["pizza"],
    "locations": ["New York, NY"],
    "reviewLimit": 10, # Requesting up to 10 reviews
    "searchLimit": 1
}

# Start the Actor and wait for it to finish
run = client.actor(actor_id).call(run_input=run_input)

# Fetch results from the default dataset
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "reviews" in item and run_input["reviewLimit"] > 0:
        actual_reviews = len(item["reviews"])
        expected_reviews = run_input["reviewLimit"]
        if actual_reviews < expected_reviews:
            print(f"Business '{item['name']}' returned {actual_reviews} reviews, less than the requested {expected_reviews}.")
            # You might want to log this, trigger an alert, or re-process
        elif actual_reviews == 0 and run_input["reviewLimit"] > 0:
            print(f"Business '{item['name']}' returned no reviews despite reviewLimit > 0.")
    elif "reviews" not in item:
        print(f"Business '{item['name']}' has no 'reviews' field, potentially indicating an issue.")

print("Checked against the Actor's input schema and Apify docs on 2026-10-06")

Why does Yelp Scraper return fewer results than my `searchLimit`?

The Yelp Scraper prioritizes collecting results until the searchLimit is reached, the results on Yelp's side run out, or the runTimeoutSecs is hit. If your run terminates early, it's often due to the runTimeoutSecs being too low for the requested searchLimit and number of searchTerms and locations combinations. Yelp also uses Cloudflare, and persistent blocking can lead to fewer results if too many proxy sessions fail.

When setting searchLimit, it's crucial to understand how the scraper works. Yelp search pages use infinite scroll, and the Actor simulates this to collect results. This process takes time. The documentation states that "each scroll round adds a few seconds." If you set a high searchLimit (e.g., 50 per search term/location combination) but keep the default runTimeoutSecs of 1800 seconds (30 minutes) for a complex query, the run might hit its timeout before all results can be fetched. The Actor will "stop launching new fetches at the runTimeoutSecs mark, pushes everything it has collected so far, and exits cleanly." This results in a partial dataset without an explicit failure.

Consider this input, designed to fetch many results:

{
    "searchTerms": ["restaurant", "cafe"],
    "locations": ["New York, NY", "Los Angeles, CA", "Chicago, IL"],
    "searchLimit": 50,
    "reviewLimit": 5,
    "runTimeoutSecs": 1800
}

This configuration attempts 6 distinct searches (2 searchTerms * 3 locations). If each search aims for 50 results, that's potentially 300 business records. Given that each search requires pagination and multiple scroll events, the 30-minute default timeout can easily be exceeded, especially under network latency or minor Cloudflare challenges. To avoid this, you need to either increase runTimeoutSecs significantly or reduce your searchLimit for exploratory runs. For large jobs, using the asynchronous run endpoint and polling or webhooks is also critical, as the synchronous run endpoint has a hard cap of 300 seconds.

What happens if the `proxy` fails, and how do I detect it?

The proxy input field is explicitly marked as "REQUIRED" because "Yelp uses Cloudflare protection. Residential proxy is mandatory; datacenter IPs are reliably blocked." If the proxy configuration is incorrect or the residential proxies themselves are exhausted or blocked by Cloudflare, the run will likely return a yelp_blocked record in the dataset and then terminate cleanly. This is a partial success state from the Actor's perspective, but a data failure from a user's.

A yelp_blocked record has type, reason, message, and scrapedAt fields. It's crucial to check for the presence of these records in your output dataset, especially if the expected number of business results is significantly lower than anticipated. While the Actor exits "successfully" with this record, your downstream processes should interpret it as a trigger for re-running the job, possibly with different proxy settings or a delay.

Here's an example of how to check for yelp_blocked records in your output:

import apify_client

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

actor_id = "crawlerbros/yelp-scraper"
run_input = {
    "searchTerms": ["invalidsearchterm"], # A term likely to return nothing, or cause blocks
    "locations": ["Nowhere, US"],
    "proxy": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}

# To get a real dataset ID, you would run the Actor first:
run = client.actor(actor_id).call(run_input=run_input)
mock_dataset_id = run["defaultDatasetId"]

blocked_records = []
for item in client.dataset(mock_dataset_id).iterate_items():
    if item.get("type") == "yelp_blocked":
        blocked_records.append(item)

if blocked_records:
    print(f"Detected {len(blocked_records)} 'yelp_blocked' records in the output:")
    for record in blocked_records:
        print(f"  Reason: {record.get('reason')}, Message: {record.get('message')}")
    print("Consider re-running the Actor with fresh residential proxy sessions or different input.")
else:
    print("No 'yelp_blocked' records detected. Run likely completed without explicit blocks.")

print("Checked against the Actor's input schema and Apify docs on 2026-10-06")

It's also worth noting that residential proxy sessions typically persist for around 30 minutes. For long-running scrapes, especially those hitting a high searchLimit across many locations or direct URLs, proxy sessions might expire and need to be refreshed by the underlying Apify Proxy service. This is usually handled transparently, but severe, sustained blocking can indicate an issue where the proxy pool itself is struggling.

How can silent data loss from storage limitations be avoided?

Apify manages storage, and unnamed storages, like the default dataset created by Yelp Scraper runs, have retention policies. On the free plan, "only the 10 most recent runs are retained, for 4 months." If you are running the Yelp Scraper frequently on a free plan without explicitly naming your datasets, older data will be automatically deleted. This can lead to silent data loss if you aren't actively consuming and archiving your results.

To prevent this, ensure that any critical datasets generated by your Yelp Scraper runs are explicitly named. Named storages are always exempt from deletion. For example, if you schedule daily runs, you could dynamically name your dataset based on the run date.

Here's how you might create a named dataset before running the Actor:

import apify_client
from datetime import datetime

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

# Create a named dataset
timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
dataset_name = f"yelp-scraper-data-{timestamp}"
named_dataset = client.datasets().get_or_create(name=dataset_name)

actor_id = "crawlerbros/yelp-scraper"
run_input = {
    "searchTerms": ["bakery"],
    "locations": ["Paris, France"],
    "searchLimit": 10,
    "reviewLimit": 2
}

# Start the Actor and specify the named dataset for output
run = client.actor(actor_id).call(
    run_input=run_input,
    dataset_id=named_dataset["id"] # Direct the output to our named dataset
)

print(f"Run finished. Data saved to named dataset: {dataset_name} (ID: {named_dataset['id']})")
print(f"You can access it at: {client.dataset(named_dataset['id']).url}")
print("Checked against the Actor's input schema and Apify docs on 2026-10-06")

Using a named dataset ensures that your historical data remains accessible, regardless of your Apify plan or the number of subsequent runs.

What are the cost implications of Yelp Scraper's input parameters?

The Yelp Scraper follows a PAY_PER_EVENT pricing model. This means you are charged for specific named events that occur during its execution, in addition to general Apify platform usage. Understanding how your input parameters influence these events is key to managing costs. Platform usage for the run (memory and run time) is billed separately at your Apify plan rates.

The primary charge event for this Actor is "result" (technical name: apify-default-dataset-item). This event is emitted for "Single result in the default dataset." Therefore, the total number of business records (items) pushed to the dataset directly multiplies this cost.

The "result" event costs $0.01 per event for FREE tier users.
BRONZE tier users pay $0.00833 per event.
SILVER tier users pay $0.00667 per event.
GOLD, PLATINUM, and DIAMOND tier users pay $0.005 per event.

There is also an "Actor Start" event (technical name: apify-actor-start) which costs $0.05 per GB of memory allocated to the run, charged once per run. This means even a run that returns zero business results will still incur this initial charge.

searchLimit: This parameter directly impacts the number of "result" events. If you set searchLimit: 10 for a single search term and location, and 10 businesses are found, you will incur 10 "result" events. If you have multiple searchTerms and locations (e.g., 2 searchTerms * 3 locations * 10 searchLimit = 60 potential results), the "result" events will scale proportionally, leading to higher costs.
directUrls: Similarly, each business successfully scraped via a directUrls entry will count as one "result" event.
reviewLimit: While reviewLimit affects the content of each business result (by adding reviews to the business object), it does not generate separate "result" events. Each business record, regardless of how many reviews it contains, is still a single dataset item.

These event charges do not change based on run time or memory, but remember that platform usage for the run (memory and run time) is billed separately at your Apify plan rates.

How to manage scheduled Yelp Scraper runs?

When creating new schedules for the Yelp Scraper on the Apify platform, they are DISABLED by default. You must explicitly enable them after creation. Additionally, an Actor "must have run at least once before it can be scheduled." These are important considerations for setting up automated data extraction pipelines.

Consider a scenario where you want to schedule daily scrapes:

Run the Actor manually once: Before you can schedule the yelp-scraper Actor, you must execute it at least once. This initializes its internal state on the platform for scheduling purposes.
Create a schedule: Define your cron expression and other schedule parameters. Remember that new schedules are DISABLED by default.
Enable the schedule: After creation, navigate to the schedule in the Apify Console and explicitly enable it.

This process ensures that your automated tasks will actually execute. For example, to set up a run for every day at 3 AM UTC:

{
    "name": "Daily Yelp Scrape",
    "actorId": "crawlerbros/yelp-scraper",
    "cronExpression": "0 3 * * *",
    "isEnabled": true,
    "input": {
        "searchTerms": ["restaurants"],
        "locations": ["London, UK"],
        "searchLimit": 10
    }
}

This JSON represents the configuration for a schedule, which you would typically set up through the Apify Console or API. Note that isEnabled: true must be set after creation, as schedules default to disabled.

What are the key limitations and caveats for data engineers?

Beyond the specific failure modes, it's essential to understand the inherent limitations of web scraping, especially with a target like Yelp that employs active anti-bot measures.

Cloudflare Protection: Yelp uses Cloudflare, which means continuous vigilance against bot detection. While the Actor mandates residential proxies to counter this, occasional blocks are still possible. If a run does get blocked across all available residential proxy sessions, it will exit successfully but with a yelp_blocked record in the dataset, allowing you to detect this programmatically.
Dynamic HTML Changes: Yelp, like any dynamic website, can change its HTML structure without notice. The README mentions that "Yelp uses dynamic class names that can change," and "some fields may occasionally be empty if Yelp changes their HTML structure." This means even a perfectly configured run might return null or empty strings for fields like phone, description, or specific review fields if the scraper's parsing logic temporarily breaks. Defensively, your downstream pipelines should always account for the possibility of missing data for any non-essential field.
Review Extraction Limit: The scraper explicitly states that reviews are extracted only from "the reviews visible on the business page (first page of reviews)." Even if Yelp displays a total reviewCount of thousands, you will only receive the reviews visible on that initial page, capped by your reviewLimit. There is no current mechanism to paginate through all reviews for a single business using this Actor.
Search Result Reliability: "Search results depend on Yelp's ranking algorithm and may vary." This means repeated identical searches might yield slightly different orderings or even different sets of businesses over time. For auditing or consistency-critical tasks, this variability needs to be considered.
Actor Task Consumption: While multiple runs can add to a request queue, a single request queue can only be processed by one Actor or task run at a time. This impacts design if you considered fanning out a single queue across multiple parallel Yelp Scraper runs.
Input Prefill vs Default: The prefill values in the input schema are only visible in the Console UI. When making API calls or using existing Actor tasks, only the default values are applied if a field is omitted. Always pass an explicit input dictionary via API to ensure your desired configuration.

These limitations mean that robust data pipelines built around the Yelp Scraper require more than just calling an API. They need monitoring for yelp_blocked records, checks for partial or missing fields in the output, and an understanding of storage retention policies and proxy behavior.

Example Input

Search for businesses

{
    "searchTerms": ["pizza", "sushi"],
    "locations": ["New York, NY", "San Francisco, CA"],
    "searchLimit": 5,
    "reviewLimit": 3
}

Scrape specific businesses

{
    "directUrls": [
        "https://www.yelp.com/biz/prince-street-pizza-new-york-2",
        "https://www.yelp.com/biz/joes-pizza-new-york"
    ],
    "reviewLimit": 10
}

Checked against the Actor's input schema and Apify docs on 2026-10-06

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Paginating GoFundMe Search Results Across 20 Countries

Crawler Bros — Mon, 05 Oct 2026 19:14:48 +0000

Tracking disaster response, emergency medical relief, or localized community fundraising at scale requires structured access to GoFundMe campaigns. Querying individual pages through a headless browser introduces unnecessary rendering overhead, high resource consumption, and slow execution times.

GoFundMe exposes its public campaign search index through direct HTTP endpoints—the same index that powers browser search on gofundme.com/s. The GoFundMe Campaign Scraper targets this index directly without requiring authentication or proxies. Understanding how to use its filtering options and pricing parameters allows you to extract campaign data cleanly while controlling execution costs.

Filtering Campaigns at the Search Index Level

Extracting targeted campaign data without fetching thousands of irrelevant records relies on passing structured parameters directly to GoFundMe’s internal search queries. The Actor supports two primary operational modes via the mode parameter: search (keyword search across campaigns) and byUrls (direct lookup of specific campaign links, short URLs, or slugs).

When using mode: "search", passing generic search terms can return thousands of low-value, zero-donation records. To narrow the returned dataset, you can apply categorical, geographical, and temporal parameters simultaneously:

{
  "mode": "search",
  "searchQuery": "flood relief",
  "categories": ["Accidents & Emergencies", "Non-Profits & Charities"],
  "countries": ["US", "CA"],
  "minCurrentAmount": 1000,
  "hasDonationsOnly": true,
  "sortBy": "mostFunded",
  "maxItems": 100
}

Key Schema Parameters

categories: An array restricting results to predefined GoFundMe categories such as "Medical, Illness & Healing", "Rent, Food & Monthly Bills", or "Environment".
countries: An array of ISO 3166-1 alpha-2 country codes (supporting 20 countries, including "US", "GB", "DE", "CA", and "AU").
minCurrentAmount / maxCurrentAmount: Integer bounds for the total funds raised so far in the campaign's local currency.
createdAfter / createdBefore: Date filters (YYYY-MM-DD) that restrict results based on the campaign launch date.
registeredCharityOnly: A boolean flag that filters out personal fundraisers, returning only campaigns tied to a verified registered charity.

By pre-filtering results using hasDonationsOnly: true or minDonationCount, you eliminate unverified or abandoned campaigns before they are emitted into your dataset.

Structure of the Extracted Payload

Every campaign returned by the scraper emits a standardized JSON object to the default dataset. Fields that contain no data for a given campaign are automatically omitted from the payload rather than returning null values.

{
  "campaignId": 84930281,
  "title": "Emergency Flood Relief for Local Families",
  "description": "Providing immediate shelter and essentials...",
  "organizerName": "Jane Doe",
  "beneficiaryName": "Community Relief Fund",
  "category": "Accidents & Emergencies",
  "city": "Austin",
  "state": "TX",
  "country": "US",
  "goalAmount": 50000,
  "currentAmount": 34200,
  "amountToGoal": 15800,
  "goalProgressPercent": 68.4,
  "currency": "USD",
  "donationCount": 412,
  "heartCount": 89,
  "createdAt": "2024-02-10T14:32:00Z",
  "hasDonations": true,
  "isPopular": true,
  "isRegisteredCharity": false,
  "campaignUrl": "https://www.gofundme.com/f/emergency-flood-relief-for-local-families",
  "recordType": "campaign",
  "scrapedAt": "2024-03-01T10:00:00Z"
}

The payload includes spatial properties (latitude, longitude, zip), activity metrics (recentDonationCount, socialShareCount), and logical flags (isCrisisResponse, donationsClosed).

How to Execute a Search Run

Executing a targeted extraction run can be configured programmatically using the Apify Python client or REST API.

Step 1: Define the Input Payload

Create a JSON configuration establishing your search scope and volume limits. Set maxItems to hard-cap the number of emitted records.

input_data = {
    "mode": "search",
    "searchQuery": "wildfire",
    "countries": ["US"],
    "categories": ["Accidents & Emergencies"],
    "createdAfter": "2024-01-01",
    "sortBy": "newest",
    "maxItems": 200
}

Step 2: Initialize and Run the Actor

Pass the input payload to the Actor endpoint.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("crawlerbros/gofundme-campaign-scraper").call(run_input=input_data)

Step 3: Fetch Dataset Items

Retrieve the results from the dataset allocated to the completed run.

dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for campaign in dataset_items:
    print(f"{campaign['title']} - Raised: {campaign['currentAmount']} {campaign['currency']}")

Calculating Pay-Per-Event Execution Costs

This Actor uses a PAY_PER_EVENT billing model. Users pay per specific event type plus standard platform usage.

The event costs for this Actor are structured as follows:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run. Charged once when the Actor starts running.
Result (apify-default-dataset-item): Charged for every dataset item returned.

The base result price per dataset item across Apify discount tiers is:

FREE: $0.005
BRONZE: $0.00433
SILVER: $0.00367
GOLD: $0.003
PLATINUM: $0.003
DIAMOND: $0.003

For example, on the FREE tier, running a job with 1 GB of allocated memory that extracts 100 campaign records results in:

Actor Start event: $0.005
Result events: 100 × $0.005 = $0.500
Total event charges: $0.505 (plus platform usage).

On the GOLD tier, the same 100-result extraction incurs:

Actor Start event: $0.005
Result events: 100 × $0.003 = $0.300
Total event charges: $0.305 (plus platform usage).

Platform usage is billed separately at the rates defined by your Apify plan. Setting maxItems prevents unexpected charges by stopping execution once the target number of records is met.

Sorting Limitations on the Public Search Index

GoFundMe’s public search index does not expose a globally sorted multi-million campaign database for metrics like total funding or donation counts. When setting sortBy to options other than relevance (such as mostFunded, mostDonations, or highestGoal), GoFundMe reorders records within each fetched page of results rather than across all campaigns globally.

If your use case requires a strict, global "Top 100 Most Funded Campaigns" dataset, fetch a broader subset of records by setting maxItems higher (e.g., 500) and perform the global sort in python or SQL post-ingestion.

Additionally, this approach reads from the public search index and does not fetch individual donor names or granular transaction-level donation histories.

Runs in this article used GoFundMe Campaign Scraper. Its README is the reference for input fields and output structure; this post is only one path through them.

DEV Community: Crawler Bros

Filtering by marketCap and peFilter is Cheaper Than Filtering After the Run

Why Scraping Unfiltered Financial Tables Breaks Down

Query Construction with Native Schema Parameters

Key Input Fields

Step-by-Step Implementation

1. Define the Run Configuration

2. Execute via the Python SDK

Output Structure and Schema Details

Pricing and Cost Dynamics

What This Tool Does Not Solve

Why TikTok Ads Library Scraper Pro Returns Empty Media URLs

What Input Schema Constraints Mean for Data Completeness?

Why Does TikTok Ads Library Scraper Pro Sometimes Return Empty Media URLs?

How Do Proxy Limits Affect Data Collection Stability?

When Does a Run Exceed the Synchronous 300-Second Cap?

What Are the Cost Implications of Input Parameters?

Can a Single Request Queue Fan-Out to Multiple Runs?

What Happens to Unnamed Storages and How Does It Affect Data Retention?

Limitations and Caveats

Automating PR Newswire Ingestion Without Silent Date-Filtering Drops

Understanding the Ingestion Modes

How Client-Side Date and Language Filtering Works

Running an Ingestion Run via Python

Actor Execution Costs

Limitations and Operational Caveats

AliExpress Scraper Returns Zero Records on CSR Pages from `ja` or `ko`

What happens when a requested region serves a CSR-only page?

How do you expose an Apify Actor as a tool for an AI agent?

When should an AI agent switch from synchronous to asynchronous Actor execution?

How does the maxTotalChargeUsd parameter protect against unexpected costs?

What are the cost considerations when using the AliExpress Scraper?

What are the key limitations an AI agent must account for when using AliExpress Scraper?

How do unnamed storage expirations impact data retention for AI agents?

How do proxy session limitations affect long-running AliExpress Scraper tasks?

Regional Job Market Analysis Across Nigeria's Six Geopolitical Zones

Handling Structural Gaps in Multi-Axis Aggregation

Constructing Regional Ingestion Pipelines

Step-by-Step Dataset Extraction

Platform Pricing Dynamics

Inherent Extraction Limitations

Why Preventing Duplicate SEMrush Scraper Output Needs Named Storage

How can I prevent duplicate processing of SEMrush Scraper output?

What unique identifier works best for SEMrush Website Stats?

How do I orchestrate runs beyond the 300-second synchronous limit?

How does the mode input parameter affect the SEMrush Scraper's output?

How can I connect to downstream systems like n8n using webhooks?

Designing input for scalability and partial failures

Understanding the semrush-scraper event-based pricing

What are the limitations and considerations for SEMrush data pipelines?

Building a GOG Price Index for 2,000 Titles Without Authentication

Aggregating Digital Storefront Metadata for PC Video Games

Programmatic Access to the GOG Catalog API

Scraper Configuration and Input Parameters

Searching by Title or Keywords

Filtering by Genre

Direct Retrieval by Game ID

Step-by-Step Walkthrough: Executing a Catalog Collection Run

Actor Execution Pricing Structure

System Limitations and Boundaries

Filtering by maxPriceEur is cheaper than post-filtering rental listings

Server-Side EUR Normalization Reduces Fetch Costs

Resolving Direct Booking Links with listingDetails Mode

Processing Search vs. Detail Pipeline Runs

Event-Based Execution Pricing

System Limitations and Constraints

Ingesting FlightAware Waypoint and Airport Boards Without API Keys

Understanding the Four Data Collection Modes

Ingesting Waypoint Telemetry for Route Analysis

Processing Live Hub Traffic with Airport Board Scraping

Automating Ingestion with Python

Setup and Execution

Execution Pricing Model

Handling Proxy Fallbacks and Rate Limits

Handling Dynamic S3 Logos and Non-Browser 302s on CareerJunction

Bypassing the /not-found Redirect on CareerJunction Listings

Resolving 24-Hour Expiry on Company Logos

Structured Field Extraction and Output Schema

Configuring Search Modes and Filters

Filtering Options

How does the `maxTotalChargeUsd` parameter protect against unexpected costs?

How does the `mode` input parameter affect the SEMrush Scraper's output?

Understanding the `semrush-scraper` event-based pricing

Resolving Direct Booking Links with `listingDetails` Mode

Bypassing the `/not-found` Redirect on CareerJunction Listings

When does `runTimeoutSecs` lead to silently incomplete data?

How do I handle missing review data from `reviewLimit`?

Why does Yelp Scraper return fewer results than my `searchLimit`?

What happens if the `proxy` fails, and how do I detect it?