DEV Community

Cover image for Why Facebook Ads Library Scraper Runs Drop Connections Past 300 Seconds
Crawler Bros
Crawler Bros

Posted on

Why Facebook Ads Library Scraper Runs Drop Connections Past 300 Seconds

#python #javascript #node #programming

The Architectural Cost of Meta’s Ad Transparency Gates

To scrape the Facebook Ad Library at scale, you must bypass aggressive, IP-based blocking and structural pagination limits. Meta gates its ad library behind residential IP checks and occasionally forces redirect-to-login walls depending on the origin network. While the facebook-ads-library-scraper handles these structural hurdles out of the box using built-in residential proxies, it operates under a strict execution model.

For developers, treating this scraper as a simple REST API endpoint that you can query synchronously is a common architectural mistake. This article unpacks how the scraper's input parameters, the Apify platform's execution limits, and the pay-per-event pricing model interact. Understanding these relationships is critical to building predictable data pipelines that do not drop records or run up unexpected costs.

What happens when a synchronous scraping run exceeds 300 seconds?

When a synchronous run exceeds 300 seconds, the Apify platform hard-caps the connection and returns an HTTP 408 Request Timeout response. The scraper does not stop running immediately on the platform, but your calling client loses the connection and fails to receive the data payload. To prevent this, you must initiate runs asynchronously for large search volumes and poll the execution run endpoint or use a webhook.

If you make an API request to trigger the scraper synchronously (for example, by calling the run endpoint with the expectation of an immediate data return), you are bound by this 5-minute platform ceiling. When the cap is tripped, your HTTP client receives an error, even though the Actor container continues to run in the background.

To avoid the 300-second timeout on larger jobs, you should trigger the run asynchronously by POSTing to the runs endpoint and handling the output out-of-band. Below is a Python implementation utilizing the official apify-client that initiates an asynchronous run, avoids the 300-second barrier, and polls for the results safely.

from apify_client import ApifyClient

# Initialize the client with your API token
client = ApifyClient("APIFY_API_TOKEN_HERE")

run_input = {
    "searchTerms": ["e-commerce", "retail"],
    "country": "US",
    "adActiveStatus": "active",
    "adType": "all",
    "mediaType": "all",
    "resultsPerSearch": 150
}

# Start the actor asynchronously, bypassing the synchronous HTTP 408 timeout
run = client.actor("crawlerbros/facebook-ads-library-scraper").call(
    run_input=run_input,
    wait_secs=0  # Setting wait_secs to 0 makes the call fully asynchronous
)

print(f"Run started successfully. Run ID: {run['id']}")
print(f"Dataset ID: {run['defaultDatasetId']}")
Enter fullscreen mode Exit fullscreen mode

Why does the prefill property in the console input schema fail on API calls?

The prefill property in the Console UI input schema only populates values when a user interacts with the Apify Console website, and is completely ignored by direct API calls. Only the default properties in the Actor's schema are automatically applied during an API execution if a key is omitted. If your code relies on prefill values to shape your query, your API runs will execute with missing or empty inputs.

This behavior is documented in the platform specs: existing tasks and API calls do not inherit prefill settings. For instance, if you build a payload expecting the UI’s prefilled dates or country filters to apply themselves, the API will fall back to the schema defaults instead. For this Actor, the country field defaults to "ALL", and adActiveStatus defaults to "active".

To guarantee that your automated pipelines run with identical parameters to your manual tests, you must explicitly pass every input field in your JSON payload. Below is a complete, explicit input schema payload designed to prevent empty filter overrides when executing via the API.

{
  "searchTerms": ["SaaS", "software"],
  "country": "US",
  "countries": ["US"],
  "adActiveStatus": "active",
  "adType": "all",
  "mediaType": "all",
  "pageIds": [],
  "adIds": [],
  "bylines": [],
  "contentLanguages": ["en"],
  "startDate": "2026-01-01",
  "endDate": "2026-09-15",
  "sortBy": "relevance",
  "resultsPerSearch": 100
}
Enter fullscreen mode Exit fullscreen mode

How do residential proxy sessions change when a run exceeds 30 minutes?

When a run exceeds approximately 30 minutes, the residential proxy session assigned to the crawler is terminated and rotated by the platform. Since Facebook tracks active scraping sessions and flags rapid IP switching mid-query, this automatic rotation can break active pagination cursors. If your scraper is mid-way through extracting a large volume-set under a single search term, the IP rotation can trigger a login gate or an empty response.

Unlike datacenter proxy sessions, which persist for 26 hours, residential proxy sessions are short-lived. This behavior is critical to understand when configuring the resultsPerSearch parameter. If you set resultsPerSearch to its maximum limit across multiple searchTerms or pageIds, the run duration can easily push past the 30-minute mark, resulting in proxy rotation mid-job.

To mitigate session loss, you should break up massive runs into smaller, isolated batches. If you must run long-duration scrapers, you can supply your own session cookies via the cookies input field. This keeps the session authenticated even if the underlying residential IP rotates. Here is an example of passing Netscape-formatted or JSON-formatted cookies inside your scraper input payload:

{
  "searchTerms": ["competitor-brand"],
  "country": "GB",
  "resultsPerSearch": 200,
  "cookies": "[\n  {\n    \"name\": \"c_user\",\n    \"value\": \"100000000000000\",\n    \"domain\": \".facebook.com\",\n    \"path\": \"/\",\n    \"secure\": true\n  },\n  {\n    \"name\": \"xs\",\n    \"value\": \"secret_session_token_here\",\n    \"domain\": \".facebook.com\",\n    \"path\": \"/\",\n    \"secure\": true\n  }\n]"
}
Enter fullscreen mode Exit fullscreen mode

Structuring the Output and Detecting Gated Data

The payload returned by the facebook-ads-library-scraper dynamically alters its properties based on the type of ad being scraped. When extracting commercial ads, fields related to spend, reach, and impressions are omitted from the output entirely, rather than returning as null. This is because Facebook only publishes official spend, reach estimates, and "Paid for by" disclaimer labels for political, social issue, or EU-regulated ads.

Your parser must expect this schema variance. Checking for the existence of gated_type or specific regulatory keys is the only way to avoid key errors in strongly typed languages. Below is an example of the structured JSON output you can expect from a single scraped ad record.

{
  "ad_id": "283748293749283",
  "page_name": "Targeted Brand Campaign",
  "ad_text": "Get 20% off our cloud services today! Use code SPRING20.",
  "ad_snapshot_url": "https://www.facebook.com/ads/library/?id=283748293749283",
  "start_date": "2026-09-10",
  "end_date": null,
  "status": "active",
  "platforms": ["facebook", "instagram", "threads"],
  "media_type": "image",
  "media_url": "https://scontent.xx.fbcdn.net/v/...jpg",
  "cta_text": "SHOP_NOW",
  "link_url": "https://example.com/promo",
  "collation_count": 4,
  "page_like_count": 12500,
  "page_categories": ["Software Company", "Cloud Service"],
  "search_term": "cloud services",
  "scraped_at": "2026-09-15T14:30:22.123Z"
}
Enter fullscreen mode Exit fullscreen mode

If the ad were a political campaign, the output would contain additional top-level keys such as spend, impressions, reach_estimate, disclaimer_label, and byline. To safely parse this dataset in your data pipeline, use defensive key checking or schema validation. Below is a Python code snippet that parses raw output records, handling the absence of regulatory keys gracefully while logging commercial vs. political ads differently.

def process_scraped_ads(ad_records):
    for ad in ad_records:
        ad_id = ad.get("ad_id")
        page_name = ad.get("page_name", "Unknown Page")

        # Check if this is an EU-regulated or political ad containing transparency data
        is_gated = ad.get("gated_type") is not None

        if is_gated:
            spend = ad.get("spend", "No spend range provided")
            byline = ad.get("byline", "No disclaimer")
            print(f"[POLITICAL/EU] Ad {ad_id} by {page_name} - Spend: {spend} | Funded by: {byline}")
        else:
            # Commercial ads will omit spend and disclaimer keys entirely
            collation_count = ad.get("collation_count", 1)
            print(f"[COMMERCIAL] Ad {ad_id} by {page_name} - Variations (Collation Count): {collation_count}")

# Dummy payload processing example
process_scraped_ads([
    {
        "ad_id": "111", 
        "page_name": "Retailer A"
    },
    {
        "ad_id": "222", 
        "page_name": "Campaign B", 
        "gated_type": "political", 
        "spend": "100-499 USD", 
        "byline": "Committee for Change"
    }
])
Enter fullscreen mode Exit fullscreen mode

How do you handle regional regulation data in your parser?

When scraping ads in regulated markets, Meta inserts region-specific fields that reflect compliance flags like financial-services approvals or limited-delivery status. This regional regulation data is completely dynamic and appears only for ads running in jurisdictions with specific legal mandates, such as the United Kingdom or the European Union. If your extraction pipeline assumes a static, commercial-only schema, these unexpected nested objects will be skipped or cause mapping failures.

Because the scraper returns regional_regulation_data and contains_digital_created_media strictly when applicable, your ingestion script must inspect these properties conditionally. For example, if an ad has been flagged for using AI-generated content, contains_digital_created_media will be populated, indicating that Meta's digital disclosure rules were triggered.

The following Python parser isolates these dynamic compliance properties. It extracts region-specific regulatory flags and digital media indicators, allowing you to flag non-compliant or AI-generated creative variations automatically.

def extract_compliance_metadata(ad_record):
    ad_id = ad_record.get("ad_id")
    page_name = ad_record.get("page_name", "Unknown Page")

    # Extract optional digital media disclosure flags
    is_ai_generated = ad_record.get("contains_digital_created_media", False)

    # Extract nested regional regulatory details if present
    regional_data = ad_record.get("regional_regulation_data")

    metadata = {
        "ad_id": ad_id,
        "page_name": page_name,
        "is_ai_generated": is_ai_generated,
        "has_regulatory_flags": regional_data is not None,
        "regulatory_details": regional_data if regional_data else {}
    }

    if is_ai_generated:
        print(f"[COMPLIANCE WARNING] Ad {ad_id} by {page_name} contains AI-generated media.")

    if regional_data:
        print(f"[REGULATORY DATA FOUND] Ad {ad_id} subject to regional restrictions: {regional_data}")

    return metadata

# Test parsing of a record containing regional and AI flags
sample_ad = {
    "ad_id": "999888777",
    "page_name": "Financial Services Ltd",
    "contains_digital_created_media": True,
    "regional_regulation_data": {
        "status": "limited-delivery",
        "reason": "anti-scam verification pending"
    }
}

parsed_metadata = extract_compliance_metadata(sample_ad)
Enter fullscreen mode Exit fullscreen mode

Real Limits and Operational Caveats

While the scraper is robust, several server-side limits on Meta's end restrict its capability. Developers must design their workflows around these technical limitations:

  1. Multi-Country Search Limitations: The input schema exposes a countries array, but Facebook's server-side Ad Library API does not support querying multiple countries simultaneously. Consequently, only the first element in the countries array is applied by the scraper. If you need to search across several countries, you must trigger separate queries, or fall back to the single country string parameter set to "ALL".
  2. Best-Effort Date Filtering: The startDate and endDate fields act as a client-side or best-effort filter. Because Facebook's underlying endpoint does not reliably honor strict date ranges for keyword searches, the scraper may return ads outside your specified range. Your downstream pipeline must filter out unwanted dates on ingest.
  3. Storage Expiration Rules: On Apify's free tier, unnamed datasets and run histories expire. Only your 10 most recent runs are kept, and they are deleted after 4 months. If you are building an archive of political ads over time, you must assign explicit names to your storages or export the dataset to external storage upon run completion.
  4. Single-Consumer Request Queues: A request queue on Apify cannot be processed by multiple Actor runs at the same time. If you attempt to parallelize a massive Facebook search term list by starting multiple runs pointing to a single shared queue, the runs will block or conflict. Each Actor run must manage its own request queue.

How the Pay-Per-Event Billing Model Works for This Actor

Every charge this Actor makes is structured as a pay-per-event pricing model. There is no separate platform-compute charge, no flat subscription fee, and no background memory-duration rate. Checked against the Actor's input schema and Apify docs on 2026-09-15, the cost of running this scraper scales purely on the volume of events generated during execution.

The list of billable events is defined by two specific actions:

  • "Actor Start" (apify-actor-start): This event is charged once per run when the container starts up. The price is $0.01 per GB of memory allocated to the run. If you allocate 1 GB of memory, you are charged exactly this flat start cost.
  • "result" (apify-default-dataset-item): This event is charged for every single scraped ad record pushed to your default dataset. The base price is $0.002 per result under the FREE tier.

Volume-Tier Pricing for Results

Apify applies volume-tier discounts for the "result" event. As your total platform consumption increases across your account, the per-event price drops according to these exact tiers:

  • FREE: $0.002 per result
  • BRONZE: $0.00167 per result
  • SILVER: $0.00133 per result
  • GOLD: $0.001 per result
  • PLATINUM: $0.001 per result
  • DIAMOND: $0.001 per result

Because this is a pay-per-event model, your runtime duration does not influence your bill. Whether the Actor takes a short time or several minutes to extract ads, your charge is based solely on the memory size allocated at start, plus the number of results successfully returned.

Input Fields That Multiply Your Bill

The final cost of an execution is determined by the parameters you pass to the input schema. The primary multiplier is the length of your input arrays: searchTerms, pageIds, adIds, and bylines.

The scraper processes each item in these arrays separately. Therefore, your maximum result count is governed by the formula:

Max Results = (Number of Search Terms + Number of Page IDs + Number of Byline Queries) * resultsPerSearch

For example, if you input several search terms and set your results limit, the scraper can output a high volume of result events. If you pass direct adIds instead, each listed ID acts as a single target search, generating exactly one result event per valid ID.

Controlling Cost with Platform Caps and Run Budgeting

To prevent run budget runaways caused by search terms returning surprisingly high volumes, you should leverage the maxTotalChargeUsd platform parameter. This is a query parameter you can pass when starting a run via the Apify API. The platform exposes this to the scraper as the environment variable ACTOR_MAX_TOTAL_CHARGE_USD.

When the accrued cost of the run hits this cap, the platform terminates the run. While the run terminates, it keeps consuming resources for a brief moment to shut down gracefully, so it is not an instantaneous kill. However, it successfully prevents a runaway loop from consuming your entire account balance.

Below is an example of an asynchronous API call in Node.js that programmatically starts the scraper, configures it to fetch a high volume of ads, and sets a hard cost ceiling using maxTotalChargeUsd to safeguard your budget.

const { ApifyClient } = require('apify-client');

const client = new ApifyClient({
    token: 'APIFY_API_TOKEN_HERE',
});

async function runScraperWithBudget() {
    // Start the run with a hard spending cap
    const run = await client.actor('crawlerbros/facebook-ads-library-scraper').start({
        searchTerms: ['apparel', 'activewear'],
        country: 'US',
        resultsPerSearch: 250, // High potential result ceiling
        adActiveStatus: 'active'
    }, {
        // Platform parameter to enforce a spending cap on pay-per-event
        maxTotalChargeUsd: 1.00
    });

    console.log(`Scraper execution started. Run ID: ${run.id}`);
    console.log(`Max budget capped. Monitoring active.`);
}

runScraperWithBudget().catch(console.error);
Enter fullscreen mode Exit fullscreen mode

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Top comments (0)