DEV Community: Crawler Bros

Overcoming Cloudflare Blocks on Preply to Analyze Tutor Pricing Trends

Crawler Bros — Wed, 23 Sep 2026 12:01:37 +0000

Building a localized pricing index for online language tutoring requires structured, fresh market data. Analysts comparing tutoring marketplaces often need to evaluate average hourly rates, native speaker availability, and professional certifications across different languages and regions. However, extracting this data at scale from Preply presents immediate technical challenges. The platform uses Cloudflare protection to prevent automated collection, which frequently results in HTTP 503 errors or CAPTCHA challenges when using standard scraping libraries.

To bypass these blocks and gather clean structured datasets, developers can use the preply-scraper on the Apify platform. This tool is built specifically to navigate Preply's anti-scraping defenses while exposing clean parameters for filtering by subject, language, country, and price.

How to Query the Preply Public Directory

The scraper operates in two distinct execution paths defined by the mode parameter: directory search (searchTutors) and direct profile extraction (getTutor).

In searchTutors mode, the tool paginates through Preply's public index. It handles the underlying pagination logic automatically, returning up to 10 tutors per page. You can narrow your query using specific schema parameters to target exact market segments. For instance, if you need to analyze the premium market for certified German teachers, you can target specific sub-segments rather than pulling the entire directory.

Here is an example input payload for a targeted search:

{
  "mode": "searchTutors",
  "language": "german",
  "proOnly": true,
  "minRating": 4.8,
  "maxPrice": 50,
  "maxItems": 100,
  "useProxy": true
}

The useProxy boolean is set to true by default. When enabled, the execution automatically falls back to Playwright combined with residential proxy rotation if direct HTTP requests are blocked by Cloudflare. This ensures that the scraper mimics legitimate user traffic and maintains a high success rate over large runs.

Extracting Detailed Profiles with IDs

The directory search returns a high-level summary of each tutor, including their hourlyRate, rating, totalLessons, and responseRate. However, if your analysis requires deep academic or credential verification, you need the tutor's historical background.

By switching the mode parameter to getTutor, you can feed the scraper specific identifiers. This mode accepts either full profile links via the startUrls array or numeric IDs via the tutorIds array.

{
  "mode": "getTutor",
  "tutorIds": ["12345", "67890"]
}

When running in getTutor mode, the scraper extracts detailed nested structures that are omitted from the main search results page, specifically:

education: Academic degrees, institutions, and graduation years.
certificates: Certified teaching credentials and issuing bodies.

The output from this run is returned as a clean JSON object containing the scrapedAt timestamp, ensuring you can track data freshness for historical analysis.

{
  "tutorId": "12345",
  "name": "Maria S.",
  "country": "Spain",
  "nativeLanguage": "Spanish",
  "languagesTaught": ["Spanish"],
  "hourlyRate": 15.0,
  "currency": "USD",
  "rating": 4.9,
  "reviewCount": 150,
  "totalLessons": 1200,
  "responseRate": "98%",
  "isPro": true,
  "profileUrl": "https://preply.com/en/tutor/12345",
  "avatarUrl": "https://cdn.preply.com/avatar/12345.jpg",
  "recordType": "tutor",
  "scrapedAt": "2026-05-17T10:00:00+00:00"
}

Step-by-Step Implementation

To run this scraper programmatically, follow these steps:

Configure your Apify environment: Set up an account on the Apify platform to access the API and retrieve your API token.
Define your search parameters: Construct your JSON input. Specify the target language (such as "spanish" or "japanese") and apply filters like tutorCountry using ISO country codes (e.g., ES or MX,CO) to segment your search.
Execute the Actor: Run the preply-scraper with your payload. The platform handles the infrastructure, proxy rotation, and retries on rate limits.
Fetch the Dataset: Once the run completes, download the resulting data from the default dataset.

Predictable Event-Based Costs

Estimating the cost of data pipeline operations is straightforward because this scraper uses a transparent PAY_PER_EVENT pricing model. Unlike traditional scrapers that bill based on unpredictable server execution time, this tool charges strictly for successful data delivery and execution starts.

The cost structure consists of exactly two event types:

Actor Start (apify-actor-start): Charged at a flat rate of $0.005 per GB of memory allocated to the run. This event occurs once per run execution.
Result (apify-default-dataset-item): Charged at $0.005 per single tutor profile successfully returned in your dataset.

As your data requirements scale, the pricing for results decreases according to specific volume-tier prices:

FREE: $0.005 per result
BRONZE: $0.00433 per result
SILVER: $0.00367 per result
GOLD: $0.003 per result
PLATINUM: $0.003 per result
DIAMOND: $0.003 per result

For example, a run configured with maxItems: 1000 allocated with 1 GB of memory will cost exactly $0.005 for the run initiation, plus the cost of the results. Under the FREE tier rate, 1,000 extracted tutor profiles will cost $5.00, bringing the total run cost to $5.005.

Limitations of the Scraping Approach

This scraper is optimized for public directory extraction and individual profile retrieval, but it is not designed for real-time chat monitoring or scheduling updates. It cannot access private tutor-student messaging feeds, internal lesson bookings, or account-specific dashboards, as these require active user authentication. Additionally, while the scraper can filter by general availability using the availabilityDay parameter, it does not fetch the granular, real-time booking calendar slots for individual tutors.

The examples here were produced with Preply Tutor Profiles Scraper. Its README lists the output fields, so you can check a response against the schema before you build on it.

Extracting Track BPM and Key Data Past Beatport Search Limits

Crawler Bros — Wed, 23 Sep 2026 08:12:09 +0000

Building an automated ingestion pipeline for electronic music metadata often leads to Beatport. Music analytics platforms, DJ curation tools, and record labels rely on its catalog for critical track attributes: BPM, Camelot keys, catalog numbers, ISRC codes, and release dates.

However, querying Beatport directly presents engineering obstacles. The search endpoint behaves inconsistently, capping results at 10 items per page and looping result sets after one or two pages. Furthermore, the storefront utilizes anonymous token rotation that can reject requests with HTTP 401 status codes across certain IP ranges.

Using the Beatport Scraper actor bypasses these storefront quirks by wrapping the internal api.beatport.com/v4/catalog API, managing visitor session tokens, and offering targeted ingestion modes.

The Search Pagination Bottleneck

When sending free-text queries to Beatport's search endpoint via mode: "search", Beatport's API caps track results to 10 items per page regardless of request parameters. After the first one or two pages, the underlying API recycles results, making deep pagination via keyword search impossible. A free-text search typically tops out at 10 to 20 unique tracks before deduplication terminates the run.

If your pipeline requires systematic catalog coverage across genres or labels, keyword searching is the wrong ingestion path.

Instead, the catalog endpoints for genres, labels, releases, and charts do not suffer from this 10-item cap. Running a genre or release extraction allows consistent retrieval of full pages of tracks. For structured pipelines, shifting from mode: "search" to mode: "browseByGenre", mode: "browseReleases", or mode: "topCharts" provides stable, high-volume ingestion.

{
  "mode": "browseByGenre",
  "genre": "tech-house",
  "minBpm": 124,
  "maxBpm": 128,
  "key": "A Minor",
  "maxItems": 250
}

Structure of the Extracted Track Dataset

The actor emits clean records directly to the dataset, omitting empty fields rather than passing null values. A single track record includes identity, musical analysis, and media URLs:

{
  "id": 3731110,
  "title": "The Kid",
  "mixName": "Original Mix",
  "artists": ["Example Artist"],
  "primaryArtist": "Example Artist",
  "remixers": [],
  "label": "Example Records",
  "releaseId": 54321,
  "releaseName": "The Kid EP",
  "genre": {
    "id": 11,
    "name": "Tech House",
    "slug": "tech-house"
  },
  "bpm": 126,
  "key": "8A",
  "duration": "6:42",
  "durationMs": 402000,
  "price": {
    "value": 2.49,
    "currency": "USD",
    "display": "$2.49"
  },
  "isrc": "GB-AAA-24-00001",
  "catalogNumber": "EX001",
  "releaseDate": "2024-02-15",
  "previewUrl": "https://geo-samples.beatport.com/track/preview/3731110.mp3",
  "sampleStartMs": 0,
  "sampleEndMs": 60000,
  "artworkUrl": "https://geo-media.beatport.com/image_size/500x500/12345.jpg",
  "recordType": "track",
  "scrapedAt": "2025-02-23T12:00:00.000Z"
}

The payload delivers both standard musical keys and Camelot notations (such as 8A), making the output immediately usable in harmonic mixing engines or catalog tagging systems without custom mapping tables.

Filtering Parameters for Targeted Ingestion

To prevent downstream systems from discarding irrelevant records, the actor processes filter parameters before writing items to the dataset:

minBpm and maxBpm: Constrains the tempo range (valid between 40 and 200 BPM).
key: Restricts results to any of the 24 standard musical keys.
releasedAfter: Accepts an ISO date string (YYYY-MM-DD) to exclude historical back-catalogs during recurring monitoring runs.
containsKeyword: Performs case-insensitive matching on track titles.
includeReleaseTracks / includeChartTracks: When fetching parent objects in topCharts or browseReleases modes, setting these booleans to true extracts the nested individual track objects into the dataset alongside the parent record.

Running an Ingestion Workflow via Python

The Apify API client allows straightforward integration of the scraper into an ETL pipeline. The following script configures a release-monitoring extraction for a specific electronic music label.

import os
from apify_client import ApifyClient

client = ApifyClient(os.getenv("APIFY_TOKEN"))

run_input = {
    "mode": "browseReleases",
    "labelId": 20670,
    "releaseSort": "newest",
    "includeReleaseTracks": True,
    "maxItems": 100
}

# Run the actor and wait for completion
run = client.actor("crawlerbros/beatport-scraper").call(run_input=run_input)

# Fetch emitted dataset items
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

tracks = [item for item in dataset_items if item.get("recordType") == "track"]
print(f"Extracted {len(tracks)} tracks from label catalog.")

Step-by-Step Setup

Locate Target IDs: If targeting a specific artist or label, open their page on beatport.com. The ID is the integer at the end of the URL (e.g., beatport.com/artist/amy-dabbs/406088 yields artistId: 406088).
Select Ingestion Mode: Choose browseByGenre, browseByLabel, browseReleases, or topCharts for bulk ingestion. Reserve search strictly for targeted track name lookups.
Set Filters: Define releasedAfter to pull only new tracks since your last ingestion timestamp, and configure minBpm/maxBpm if your target taxonomy requires strict tempo boundaries.
Configure Proxy: If your runs encounter environment-level 401 blocks from Beatport's API, enable proxyConfiguration in the input payload to route catalog and token-minting requests through datacenter proxies.
Consume Dataset: Ingest the dataset items by streaming records from the default dataset into your analytical warehouse or music library database.

System Boundaries

This tool does not provide full-length audio downloads; it extracts only metadata and the public 30- to 60-second MP3 preview URLs hosted on Beatport's CDN. Additionally, because Beatport's API occasionally omits specific metadata fields (such as missing ISRC values on older releases), downstream schemas must handle missing fields rather than assuming every track record contains identical keys.

If you want to reproduce this, the Actor is Beatport Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

Calculating N M Distance Matrices Without API Key Quotas

Crawler Bros — Wed, 23 Sep 2026 04:04:12 +0000

Logistics optimization and vehicle routing algorithms require matrix inputs. When building a Travelling Salesman Problem (TSP) solver or evaluating warehouse distribution centers, sending thousands of coordinate pairs to commercial mapping APIs gets expensive fast. Most commercial routing engines enforce strict daily quotas and charge per request cell, making large-scale distance calculation cost-prohibitive during initial analysis phases.

The OSRM Route Planner Scraper solves this by tapping into the open Open Source Routing Machine engine backed by OpenStreetMap data. It converts latitude and longitude inputs into routing geometry, distance tables, and road network snapping records without requiring developer keys or dedicated proxy configurations.

Handling Coordinate Formatting and Direct API Queries

A common source of errors in routing pipelines is coordinate order. Standard mapping applications accept inputs as latitude followed by longitude (e.g., "51.5074,-0.1278" for London). However, underlying GeoJSON specs and the native OSRM API expect longitude followed by latitude (lon,lat).

When passing coordinates into this task, you supply standard "lat,lon" strings. The system normalizes the coordinate strings automatically before executing queries against the OpenStreetMap network.

The platform provides three core operational workflows:

{
  "mode": "route",
  "profile": "driving",
  "startCoordinate": "51.5074,-0.1278",
  "endCoordinate": "48.8566,2.3522",
  "waypoints": [
    "50.1109,8.6821"
  ]
}

If you already have pre-formatted routing URLs targeting an existing OSRM deployment, you can bypass input parameters using startUrls. Providing an array of direct endpoint URLs overrides mode, profile, and coordinate inputs entirely, passing those HTTP requests straight to the network handler.

Calculating Matrix Distances and Snapping Coordinates

Routing engines fail when coordinates fall on unroutable terrain, such as a building interior or water body. To prevent calculation errors, you can run a nearest-road lookup to snap points to the closest routable segment prior to route execution.

Nearest Road Lookup (`mode: "nearest"`)

By passing a single point to startCoordinate, the module retrieves the closest physical road name, precise snapped coordinates, and the radial offset in meters:

{
  "recordType": "nearest",
  "sourceUrl": "http://router.project-osrm.org/nearest/v1/driving/51.5074,-0.1278",
  "scrapedAt": "2026-01-01T00:00:00+00:00",
  "profile": "driving",
  "queryCoordinate": "51.5074,-0.1278",
  "nearestCoordinate": "51.507478,-0.127965",
  "nearestLatitude": 51.507478,
  "nearestLongitude": -0.127965,
  "nearestRoadName": "King Charles I Island",
  "distanceToRoadMeters": 14.37
}

Distance Matrix Generation (`mode: "matrix"`)

For fleet routing, calculating individual point-to-point routes scales exponentially ($O(N \times M)$). Matrix mode requests the entire cross-table in a single execution payload. You provide arrays to the origins and destinations parameters:

{
  "mode": "matrix",
  "profile": "driving",
  "origins": [
    "51.5074,-0.1278",
    "51.4545,-2.5879"
  ],
  "destinations": [
    "48.8566,2.3522",
    "50.6292,3.0573"
  ]
}

The output yields an array of matrixCells, containing durationSeconds and distanceMeters for every origin-destination pair combination:

{
  "recordType": "matrix",
  "sourceUrl": "http://router.project-osrm.org/table/v1/driving/...",
  "scrapedAt": "2026-01-01T00:00:00+00:00",
  "profile": "driving",
  "origins": [
    "51.5074,-0.1278"
  ],
  "destinations": [
    "48.8566,2.3522"
  ],
  "matrixCells": [
    {
      "origin": "51.5074,-0.1278",
      "destination": "48.8566,2.3522",
      "durationSeconds": 43265.6,
      "distanceMeters": 620363.4
    }
  ]
}

How to Set Up an OSRM Pipeline

Define the operation mode: Select route for turn-by-turn directions, matrix for distance tables, or nearest for geofence snapping.
Select a transport profile: Set profile to driving, cycling, or walking to adjust speed assumptions and access rules (e.g., pedestrian-only zones or highway restrictions).
Set coordinate inputs: Supply standard "lat,lon" values into startCoordinate and endCoordinate for simple paths, or populate origins and destinations arrays for matrix calculations.
Configure execution boundaries: Optionally set maxItems to restrict the total record output emitted to the default dataset.
Run the Task: Execute the module to generate output data directly into your dataset storage.

Cost Structure and Execution Mechanics

This module runs on a pure PAY_PER_EVENT pricing model. This means you are charged strictly for specific execution events rather than variable compute runtimes or subscription tiers.

The event charges are defined as follows:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run. This is a one-time start event fee per run execution.
Dataset Result (apify-default-dataset-item): $0.005 per event output emitted to the default dataset (with volume tiers dropping to $0.00433 on BRONZE, $0.00367 on SILVER, and $0.003 on GOLD, PLATINUM, and DIAMOND tiers).

For example, running a matrix calculation that yields 100 output dataset records using a 1 GB memory allocation incurs:

1 × Actor Start event at 1 GB = $0.005
100 × result events at base price ($0.005) = $0.50
Total run cost = $0.505

Because the routing compute uses the public demo server, you incur no extra third-party subscription charges or proxy bandwidth fees.

Practical Pipeline Limitations

While this approach works well for batch processing and network analysis, it does not account for real-time traffic congestion or live road closures. The underlying public OSRM demonstration server relies on static OpenStreetMap data dumps, making it unsuitable for live navigation applications that require live rerouting around temporary incidents.

The system adds controlled request spacing to respect public server resource limits. If your pipeline demands real-time dynamic re-routing or millions of sub-second matrix operations, you will need to point startUrls toward a privately self-hosted OSRM instance.

OSRM Route Planner Scraper is the Actor behind these examples. If a selector in your own version breaks, compare your output against the fields listed in its README first.

Why Exposing Facebook Comments to MCP Agents Times Out on 300 Second Runs

Crawler Bros — Wed, 23 Sep 2026 02:52:18 +0000

Exposing Facebook Comments to AI Agents via Model Context Protocol

When you expose a data extraction tool to an AI agent using the Model Context Protocol (MCP), the agent does not see a web interface or a simplified SDK wrapper. It sees a raw JSON Schema. If you expose facebook-comments-scraper (which extracts public comments from Facebook posts, Watch videos, and photo stories) directly to an LLM via the Apify MCP server, the agent must parse, understand, and accurately populate a highly nested input structure.

The integration relies on the Apify MCP server at https://mcp.apify.com, scoped using the ?tools=facebook-comments-scraper query parameter. This scoping isolates the toolset so the agent is not overwhelmed by thousands of unrelated utilities. However, once isolated, the agent is solely responsible for generating valid payloads that conform to the scraper's exact technical specifications.

When an agent processes a user request like "find out what people think about this post," it has to translate that intent into a tool call. If the input schema is too permissive or if the agent misunderstands the default properties, the execution fails before a single container starts. The most common point of failure is how the agent handles the transition from natural language requests to structured API payloads, particularly around URL formats and proxy configurations.

To expose facebook-comments-scraper to an AI agent, you can configure your MCP client to connect to the Apify MCP host with the specific actor tool scoped in the connection URL. This exposes only the target actor as a callable tool, reducing context window bloat and preventing the model from hallucinating input parameters.

Here is an example configuration for a Claude Desktop developer setup or any standard MCP host file that registers the scoped tool:

{
  "mcpServers": {
    "apify-facebook-comments": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-env",
        "https://mcp.apify.com?tools=facebook-comments-scraper"
      ],
      "env": {
        "APIFY_TOKEN": "YOUR_APIFY_API_TOKEN"
      }
    }
  }
}

When this configuration loads, the MCP server queries the actor's metadata and translates its input schema into a tool definition. The agent receives a tool called facebook-comments-scraper along with a structured JSON Schema describing what arguments it must generate.

What tool signature does the agent interpret?

The tool signature is an OpenAPI-like schema that maps the Actor's input fields into JSON Schema definitions for function calling. The AI model inspects field types, descriptions, and defaults to determine required properties and optional operational flags.

Checked against the Actor's input schema and Apify docs on 2026-09-23.

Here is the exact schema signature that the AI agent interprets when deciding how to invoke the tool:

{
  "name": "facebook-comments-scraper",
  "description": "Scrape public comments from Facebook posts, Watch videos, and photo stories. Extract comment text, author info, reactions, timestamps, and nested replies via GraphQL pagination.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "startUrls": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "url": {
              "type": "string",
              "format": "uri"
            }
          },
          "required": ["url"]
        },
        "description": "Start URLs | Facebook post, Reel, Watch, or photo URLs to scrape comments from. Supports /posts/, /reel/, /watch?v=, /photo?fbid=, /groups/.../permalink/, and /share/p/ URLs."
      },
      "maxItems": {
        "type": "integer",
        "default": 100,
        "description": "Max items per URL | Maximum number of comment rows to collect per start URL (not global across all URLs)."
      },
      "onlyCommentsNewerThan": {
        "type": "string",
        "description": "Only comments newer than | Limit results to comments posted after this date. Accepts YYYY-MM-DD, a full ISO 8601 timestamp (e.g. 2024-06-15T12:00:00Z), or a relative value like '1 day', '2 weeks', '3 months'"
      },
      "commentsMode": {
        "type": "string",
        "enum": ["ALL", "NEWEST", "MOST_RELEVANT"],
        "default": "ALL",
        "description": "Comments mode | Which comment view to scrape: ALL (Facebook's unfiltered 'All comments' view, in Facebook's own ranked ordering), NEWEST (newest first), or MOST_RELEVANT (Facebook's ranked 'Most relevant' view)."
      },
      "includeNestedComments": {
        "type": "boolean",
        "default": true,
        "description": "Include nested comments | If true, also scrape reply rows under top-level comments (with parentComment/parentReply metadata). If false, only top-level comments are collected (faster)."
      },
      "maxReplyDepth": {
        "type": "integer",
        "default": 1,
        "description": "Max reply nesting depth | How many levels of nested replies to follow when 'Include nested comments' is enabled. 1 = only direct replies to top-level comments (Facebook's most reliably available level)."
      },
      "proxy": {
        "type": "object",
        "default": {
          "useApifyProxy": true,
          "apifyProxyGroups": ["RESIDENTIAL"]
        },
        "description": "Proxy Configuration | Residential proxy recommended for reliable results, especially at higher volume or concurrency."
      }
    },
    "required": ["startUrls"]
  }
}

The agent uses this system description to construct its payload. If the user provides a raw URL string, the agent must wrap it inside the startUrls array of objects. Any failure to build this specific structure results in a validation error before the execution reaches the scraper.

Why does relying on UI prefill fields break programmatic agent calls?

The Apify API completely ignores schema prefill fields because they are console UI helpers rather than backend defaults. Programmatic calls and MCP drivers only apply schema default values, so relying on prefill hints causes runs to execute without intended configurations.

On the Apify platform, there is a distinct separation between how the Console UI is rendered and how the underlying API processes inputs. The prefill field in an actor's input schema controls what a human operator sees when they open the scraper inside their web browser. It is a user-experience design pattern to help humans get started quickly.

However, the Apify API does not apply prefill fields when executing runs via API endpoints or tool calls. Only true default values are merged into the payload if a key is omitted. This means if you program your agent to rely on UI-level hints or if the agent assumes certain complex objects (like proxy parameters) will automatically populate with human-centric prefilled data, the API will receive an incomplete structure.

To guarantee that your agent runs do not fail due to missing network routing parameters, the agent must be instructed to always construct and pass an explicit input payload containing all required keys. You cannot count on the platform to merge UI-level prefills behind the scenes.

How do you handle the 300 second synchronous run cap?

You handle the cap by initiating the Actor run asynchronously and polling its status before fetching dataset items. Synchronous endpoints cut connections after 300 seconds with an HTTP 408 error, making asynchronous polling mandatory for large thread extraction.

The synchronous run endpoint on the Apify platform has a strict physical cap of 300 seconds. If an Actor run exceeds this limit during a synchronous HTTP POST request, the platform terminates the HTTP connection and returns an HTTP 408 (Request Timeout) status.

When an AI agent triggers a run synchronously via an MCP tool call, it waits for the HTTP response. For threads with thousands of comments, deep nesting levels, or complex anti-scraping challenges, scraping can easily push past the 5-minute barrier.

To avoid this failure mode, any agent integration that handles large data sets must run the Actor asynchronously. Instead of waiting for the results in a single blocking request, the agent must POST to the asynchronous runs endpoint, receive a run ID, poll the run status, and then fetch the items from the default dataset once the run completes successfully.

Here is a Python implementation that demonstrates how to execute this asynchronous execution flow and handle potential timeouts gracefully:

import time
import requests

def run_facebook_scraper_async(api_token, start_url, max_items=100):
    actor_id = "facebook-comments-scraper"
    initiate_url = f"https://api.apify.com/v2/acts/{actor_id}/runs?token={api_token}"

    payload = {
        "startUrls": [{"url": start_url}],
        "maxItems": max_items,
        "commentsMode": "ALL",
        "includeNestedComments": True,
        "maxReplyDepth": 1,
        "proxy": {
            "useApifyProxy": True,
            "apifyProxyGroups": ["RESIDENTIAL"]
        }
    }

    response = requests.post(initiate_url, json=payload)
    if response.status_code != 201:
        raise RuntimeError(f"Failed to start Actor run: {response.text}")

    run_data = response.json().get("data", {})
    run_id = run_data.get("id")
    dataset_id = run_data.get("defaultDatasetId")

    status_url = f"https://api.apify.com/v2/actor-runs/{run_id}?token={api_token}"
    while True:
        status_response = requests.get(status_url)
        if status_response.status_code != 200:
            raise RuntimeError(f"Failed to fetch run status: {status_response.text}")

        status_data = status_response.json().get("data", {})
        status = status_data.get("status")

        if status == "SUCCEEDED":
            break
        elif status in ["FAILED", "ABORTED", "TIMED-OUT"]:
            raise RuntimeError(f"Actor run terminated with status: {status}")

        time.sleep(10)

    items_url = f"https://api.apify.com/v2/datasets/{dataset_id}/items?token={api_token}"
    items_response = requests.get(items_url)
    if items_response.status_code != 200:
        raise RuntimeError(f"Failed to fetch dataset items: {items_response.text}")

    return items_response.json()

How can an AI agent detect that the max total charge cap has been tripped?

An AI agent detects a tripped charge cap by inspecting run metadata for an ABORTED status combined with budget limit termination flags. Reading these fields lets the agent distinguish between empty post engagement and intentional cost enforcement shutdowns.

Apify allows you to pass a limit on how much a run can cost by using the maxTotalChargeUsd query parameter on the run endpoint. This maps internally to the environment variable ACTOR_MAX_TOTAL_CHARGE_USD inside the running container.

When the cost of the run reaches this threshold, the platform flags the run to abort. However, this is not an instantaneous hard kill. The container continues to run briefly while cleaning up resources, which means it may still consume minor compute overhead during its termination phase.

An AI agent processing the outputs of a run must check the termination metadata. If the run ended because it hit the budget ceiling, the agent must inform the user that the data is incomplete due to budget limits rather than assuming the post simply had no more comments.

Here is a JavaScript example showing how to inspect the run details and determine if a run was stopped by the cost safety cap:

async function checkRunBudgetStatus(apiToken, runId) {
    const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${apiToken}`;
    const response = await fetch(url);
    if (!response.ok) {
        throw new Error(`Failed to fetch run metadata: ${response.statusText}`);
    }

    const { data } = await response.json();
    const terminationReason = data.terminationReason || '';
    const wasAborted = data.status === 'ABORTED';

    if (wasAborted && terminationReason.includes('MAX_CHARGES_REACHED')) {
        return {
            complete: false,
            reason: "Budget limit reached. The output data represents a partial scrape.",
            chargedAmount: data.usageUsd
        };
    }

    return {
        complete: data.status === 'SUCCEEDED',
        reason: `Run completed with status: ${data.status}`,
        chargedAmount: data.usageUsd
    };
}

What are the limitations and caveats of this scraper?

When utilizing facebook-comments-scraper, several hard limitations exist in both the scraper itself and the Apify infrastructure that can cause an AI agent to fail if not handled in code.

Reels Require Login: The scraper explicitly states that Reels (/reel/ URLs) are not accessible without a Facebook session. Because this scraper operates without requiring authentication, passing Reel URLs will result in zero scraped items. You must use post or video URL formats instead.
Storage Rate Limits: Apify enforces strict storage rate limits: 60 requests per second for individual storage objects and 400 requests per second for dataset item pushes or request queue CRUD operations. High-concurrency agent workflows that exceed these rates will encounter throttling.
Request Queue Processing Restrictions: On the Apify platform, a request queue can only be processed by one Actor or task run at a time. Multiple concurrent runs cannot process a single shared queue in parallel; fan-out patterns must allocate separate queues.
Unnamed Storage Expiration: Unnamed storages on free Apify accounts retain only the 10 most recent runs for a maximum of 4 months before deletion. Integrations relying on persistent historical datasets must use named storages, which are permanently exempt from deletion rules.
Proxy Session Lifespans: While datacenter proxies persist for 26 hours, residential proxy sessions (recommended for Facebook scraping) last approximately 30 minutes. Extremely long runs that paginate through large threads may lose their session IP, causing minor connection pauses while establishing new proxy sessions.

What is the exact cost model for executing runs?

The facebook-comments-scraper Actor uses the PAY_PER_EVENT pricing model. You are charged solely for specific named events emitted during execution. There are no separate compute time fees or usage-based platform charges applied to the run.

The charged events and their exact costs are:

apify-actor-start: Charged once per run at $0.05 per GB of memory allocated to the run.
apify-default-dataset-item: Charged at $0.005 per event for every item written to the default dataset (representing a single scraped comment record).

This per-item event cost scales down based on Apify volume tiers:

FREE: $0.005 per event
BRONZE: $0.00367 per event
SILVER: $0.00233 per event
GOLD: $0.001 per event
PLATINUM: $0.001 per event
DIAMOND: $0.001 per event

The primary controls over total cost are the allocated container memory and the number of returned comments. Setting maxItems limits the total dataset item events generated per URL.

How do you process nested comments without double-charging?

Set includeNestedComments to false in your input configuration when you only need top-level comment text. Every nested reply emitted as a record incurs a distinct apify-default-dataset-item charge, so omitting reply rows directly reduces event volume.

When includeNestedComments is enabled (the default setting), the scraper collects reply rows under top-level comments and outputs them as individual records. Top-level records have a threadingDepth of 0, while replies have a threadingDepth of 1 or higher and include a replyToCommentId referencing their parent comment.

Because every item emitted to the default dataset generates a billed event, extracting deep reply threads for posts with heavy discussion increases event counts rapidly. If your agent only evaluates overall post sentiment or top-level engagement, setting includeNestedComments to false prevents the scraper from fetching replies, saving both runtime and event costs.

If you do collect nested comments, here is a Python helper to assemble the flat dataset items into a hierarchical tree in memory:

def structure_comment_tree(dataset_items):
    comments_by_id = {}
    top_level_comments = []

    for item in dataset_items:
        comment_id = item.get("commentId")
        if not comment_id:
            continue

        comments_by_id[comment_id] = {
            "commentId": comment_id,
            "text": item.get("text"),
            "author": item.get("profileName"),
            "likes": item.get("likesCount", 0),
            "date": item.get("date"),
            "threadingDepth": item.get("threadingDepth", 0),
            "replies": []
        }

    for item in dataset_items:
        comment_id = item.get("commentId")
        parent_id = item.get("replyToCommentId")

        if not comment_id or comment_id not in comments_by_id:
            continue

        node = comments_by_id[comment_id]

        if parent_id and parent_id in comments_by_id:
            comments_by_id[parent_id]["replies"].append(node)
        elif item.get("threadingDepth", 0) == 0:
            top_level_comments.append(node)

    return top_level_comments

How do you handle non-existent pages or authentication blocks?

You handle failure states by checking for an empty dataset output and searching the run log for diagnostic warning phrases. Inspecting logs for keywords like "block" or "invalid url" allows your application to differentiate blocked requests from quiet posts.

When scraping public Facebook URLs, invalid links, deleted posts, or restricted group content will result in empty datasets because no public comments can be rendered without login.

An AI agent must be able to tell whether zero output records signify a newly created post with no comments or a blocked extraction attempt. Parsing the container log output provides immediate context regarding network or access issues.

Here is a Node.js script using the official client that checks the dataset output, inspects run logs upon encountering an empty result, and raises distinct errors based on the output:

import { ApifyClient } from 'apify-client';

async function scrapeAndVerifyComments(apiToken, targetUrl) {
    const client = new ApifyClient({ token: apiToken });

    const run = await client.actor('facebook-comments-scraper').call({
        startUrls: [{ url: targetUrl }],
        maxItems: 50,
        commentsMode: "ALL",
        includeNestedComments: false
    });

    const { items } = await client.dataset(run.defaultDatasetId).listItems();

    if (items.length === 0) {
        const logText = await client.log(run.id).get();
        const lowerLog = (logText || '').toLowerCase();

        if (lowerLog.includes('block') || lowerLog.includes('captcha')) {
            throw new Error("Scraping was blocked by anti-bot protections. Ensure residential proxies are enabled.");
        } else if (lowerLog.includes('not found') || lowerLog.includes('invalid url')) {
            throw new Error(`Target URL is private, restricted, or non-existent: ${targetUrl}`);
        } else {
            return {
                status: "EMPTY",
                message: "Run succeeded, but no public comments were found on this post.",
                data: []
            };
        }
    }

    return {
        status: "SUCCESS",
        count: items.length,
        data: items
    };
}

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Tracking Per-Size ASOS Stock Across 10 Regional Stores

Crawler Bros — Wed, 23 Sep 2026 00:01:45 +0000

E-commerce pricing engines and retail analytics pipelines often fail when tracking inventory changes at the variant level. On fast-fashion platforms like ASOS, top-level product availability is misleading: a product page may list a dress as active, but the most common sizes (like S and M) are completely sold out, leaving only fringe sizes in stock. Furthermore, localized pricing, currency adjustments, and discount strategies vary across regional storefronts.

Building a scraper to track these localized changes directly against ASOS requires handling market-specific endpoints, localized currency headers, and variant-level stock flags. The asos-scraper actor standardizes extraction across 10 regional stores (US, GB, FR, DE, IT, ES, AU, NL, SE, RU) and returns structured product and stock metadata down to the individual SKU level.

Understanding Regional Differences and Per-Size Stock

ASOS uses independent catalogs and currency defaults for each country storefront. Scraped data from the US store (store: "US") does not necessarily mirror the stock levels or active promotions found on the United Kingdom (store: "GB") or German (store: "DE") sites.

When scraping top-level search results, ASOS provides summary metadata. However, granular sizing details—such as whether a specific size is low on inventory or completely out of stock—require fetching variant information from the underlying product display page (PDP) payloads.

The actor supports several modes to balance payload size and data depth:

search: Keyword queries over a target catalog.
byCategory: Browsing structured taxonomy using preset names or numeric categoryId values.
byProductIds and byUrls: Direct lookup for specific catalog items.
stockCheck: A lightweight variant designed specifically for price and inventory monitors tracking fixed SKU lists.

When extracting via search or byCategory, setting includeDetails: true instructs the actor to fetch the PDP data for every result, populating the sizes array with individual variant availability.

Execution Walkthrough

You can configure and trigger the actor via JSON payloads through the Apify API or platform interface.

Step 1: Define Target Store and Filter Constraints

To capture women's dresses on sale in the UK store within a specific price range, configure the mode, regional settings, and filter flags.

{
  "mode": "byCategory",
  "category": "women_dresses",
  "store": "GB",
  "currency": "GBP",
  "minPrice": 20,
  "maxPrice": 80,
  "onSaleOnly": true,
  "includeDetails": true,
  "maxItems": 100
}

Step 2: Extract Variant-Level Inventory

When includeDetails is enabled (or when using byProductIds), each emitted JSON record contains the sizes array. Each object in this array exposes the specific stock condition for that size.

Here is an example structure of a returned record:

{
  "productId": 210484705,
  "productCode": "1234567",
  "name": "Design Midi Dress in Floral Print",
  "brand": "ASOS DESIGN",
  "productType": "Dresses",
  "gender": "Women",
  "colour": "Multi",
  "currentPrice": 35.0,
  "currentPriceFormatted": "£35.00",
  "previousPrice": 50.0,
  "rrpPrice": 50.0,
  "currency": "GBP",
  "discountPercent": 30,
  "isOnSale": true,
  "isInStock": true,
  "isSellingFast": true,
  "isRestockingSoon": false,
  "sizes": [
    {
      "size": "UK 6",
      "brandSize": "US 2",
      "variantId": "1001",
      "sku": "SKU-1001",
      "isInStock": true,
      "isLowInStock": false
    },
    {
      "size": "UK 8",
      "brandSize": "US 4",
      "variantId": "1002",
      "sku": "SKU-1002",
      "isInStock": false,
      "isLowInStock": false
    },
    {
      "size": "UK 10",
      "brandSize": "US 6",
      "variantId": "1003",
      "sku": "SKU-1003",
      "isInStock": true,
      "isLowInStock": true
    }
  ],
  "mainImageUrl": "https://images.asos-media.com/products/...",
  "productUrl": "https://www.asos.com/prd/210484705",
  "scrapedAt": "2025-02-17T12:00:00.000Z"
}

Step 3: Parse and Ingest into a Data Pipeline

If you are running regular price checks on known products, switch to mode: "stockCheck". In this mode, the actor omits large text blocks, bullet points, and image arrays, outputting records with recordType: "stock" to optimize throughput.

You can process these records downstream using Python to flag items where popular sizes have sold out:

import json

def process_asos_record(record):
    product_id = record.get("productId")
    current_price = record.get("currentPrice")
    discount = record.get("discountPercent", 0)

    # Calculate stock availability across variants
    sizes = record.get("sizes", [])
    total_sizes = len(sizes)
    available_sizes = [s["size"] for s in sizes if s.get("isInStock")]
    low_stock_sizes = [s["size"] for s in sizes if s.get("isLowInStock")]

    return {
        "product_id": product_id,
        "price": current_price,
        "discount_pct": discount,
        "in_stock_ratio": len(available_sizes) / total_sizes if total_sizes else 0,
        "available_sizes": available_sizes,
        "low_stock_sizes": low_stock_sizes
    }

Pricing and Execution Event Structure

The actor operates on a pay-per-event pricing model. Charges are strictly tied to initialization and dataset ingestion events:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run, billed once at the start of execution.
Result Item (apify-default-dataset-item): $0.005 per result emitted to the default dataset.

Volume-tier prices apply directly to the result event based on usage tier:

FREE: $0.005 per result
BRONZE: $0.00433 per result
SILVER: $0.00367 per result
GOLD: $0.003 per result
PLATINUM: $0.003 per result
DIAMOND: $0.003 per result

There are no additional per-minute or infrastructure charges beyond these named events.

Limitations and Pipeline Boundaries

This actor does not interact with user accounts, cart sessions, or private checkout flows; it only reads publicly available catalog and inventory endpoints. If your workflow requires validating order limits, checking localized postal code delivery restrictions, or simulating an authenticated checkout session, this tool cannot perform those actions.

For monitoring regional markdowns across multiple territories, running mode: "stockCheck" on a defined list of productIds across the 10 supported regional codes (store) allows you to build historical pricing matrices without collecting unnecessary media assets.

Everything above runs on ASOS Scraper. Start with a small input and a low result limit before you widen the run -- the output shape is easier to check that way.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-23. Check the Actor page for the current rates.

Why Batching Instagram Profiles Reduces Your Actor Start Costs

Crawler Bros — Tue, 22 Sep 2026 14:18:07 +0000

Orchestrating Instagram Profile Data Workflows

Extracting public data from social media platforms like Instagram often feels like a cat-and-mouse game, complicated by rate limits, session management, and the sheer volume of data. For data engineers, the real challenge isn't just getting the data, but reliably wiring it into a broader pipeline without silent failures or unexpected costs. This article focuses on integrating the Instagram Profile Scraper into robust data workflows, emphasizing state management, efficient orchestration, and understanding the specific cost model to prevent double-processing on re-runs or overspending.

We're going to examine how to run the instagram-profile-scraper Actor to retrieve profile and post data, then process that output, all while managing the state to ensure idempotency when fetching new data or re-running a job. The focus is on the pipeline's mechanics: how to kick off a run, retrieve its results, and integrate them into downstream systems like n8n or a data warehouse, handling the critical aspects of data freshness and cost efficiency.

How do you avoid re-processing old Instagram data on subsequent runs?

You can avoid re-processing old data by maintaining a record of the last scraped_at timestamp for each profile or post. When initiating a new run, pass only the usernames that require updates or new posts since that timestamp. For posts, compare the pub_date from new results against previously stored data to identify only new entries, ensuring your pipeline only ingests fresh information.

The instagram-profile-scraper Actor generates distinct profile record and post record types, each carrying a scraped_at timestamp in ISO 8601 format. For profile records, this timestamp indicates when the profile metadata was last fetched. For post records, it marks when that specific post's data was collected.

To implement robust state management, you need a system that tracks what you've already processed. A simple approach is to store the scraped_at timestamp for the latest item (either profile or post) processed for a given username. Before initiating a new Actor run, your orchestration layer queries your destination data store (e.g., a data warehouse, a database, or even a local JSON file for simpler setups) to find the most recent scraped_at value for each username you're monitoring.

While the Actor doesn't directly support an onlyNewSince input parameter, you can achieve this by intelligently managing your input. For profiles, you would typically want the latest data, so you'd pass the username regardless. For posts, if you have stored the pub_date of the last post scraped for a given profile, you can then filter newly retrieved posts based on this pub_date before inserting them into your downstream system.

Here’s a basic Python example demonstrating how to retrieve data and then simulate state management for posts:

import os
import apify_client
from datetime import datetime

# Initialize the ApifyClient with your API token
# You can get your API token from Apify Console > Settings > API & Integrations
client = apify_client.ApifyClient(os.environ.get("APIFY_API_TOKEN"))

# Replace with the actual Actor ID from the Apify Store URL
ACTOR_ID = "crawlerbros/instagram-profile-scraper"

def run_scraper_and_process(usernames, max_posts_to_scrape, last_processed_dates):
    """
    Runs the Instagram Profile Scraper and processes its output,
    filtering for new posts based on a provided dictionary of last processed dates.
    """
    run_input = {
        "usernames": usernames,
        "maxPosts": max_posts_to_scrape,
    }

    print(f"Starting Actor run with input: {run_input}")
    run = client.actor(ACTOR_ID).call(run_input=run_input)

    dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

    new_posts = []
    updated_profiles = []

    for item in dataset_items:
        if "post_url" in item:  # This is a post record
            post_pub_date = datetime.fromisoformat(item["pub_date"])
            username = item["username"]

            # Check if this post is newer than the last processed date for this user
            if username in last_processed_dates and post_pub_date > last_processed_dates[username]:
                new_posts.append(item)
            elif username not in last_processed_dates: # If no previous date, it's new
                new_posts.append(item)

        elif "author_meta" in item: # This is a profile record
            # Profile records typically overwrite or update existing profile data
            updated_profiles.append(item)

    print(f"Found {len(new_posts)} new posts and {len(updated_profiles)} updated profiles.")
    return new_posts, updated_profiles

# Example usage:
# In a real scenario, last_processed_dates would come from your data warehouse
# For demonstration, let's assume we last processed up to May 1st, 2026 for 'instagram'
# and have no prior data for 'natgeo'
last_processed_dates_example = {
    "instagram": datetime.fromisoformat("2026-05-01T00:00:00"),
    "natgeo": datetime.min # Represents no previous processing
}

# Run the scraper
new_posts, updated_profiles = run_scraper_and_process(
    usernames=["instagram", "natgeo"],
    max_posts_to_scrape=50,
    last_processed_dates=last_processed_dates_example
)

# In a real pipeline, you would now push new_posts and updated_profiles
# to your downstream system (e.g., n8n, data warehouse)
print("\n--- New Posts ---")
for post in new_posts[:2]: # Print first 2 new posts as an example
    print(f"Post URL: {post['post_url']}, Pub Date: {post['pub_date']}")

print("\n--- Updated Profiles ---")
for profile in updated_profiles[:1]: # Print first updated profile as an example
    print(f"Username: {profile['author_meta']['username']}, Followers: {profile['author_meta']['followers_count']}")

This approach shifts the burden of deduplication from the scraper to your post-processing logic, which is generally more flexible and efficient. For each username, you'd update your last_processed_dates with the pub_date of the newest post ingested.

When should you use managed Instagram sessions instead of supplying your own cookies?

You should use managed Instagram sessions by leaving the cookies input field blank for most use cases, especially for larger-scale or continuous scraping. The Actor automatically handles session rotation and retry logic, significantly reducing the operational overhead of managing Instagram accounts, IP blocks, and expiring cookies yourself.

The instagram-profile-scraper offers two primary authentication methods: providing your own Instagram cookies or letting the Actor use a managed pool of shared sessions. For single, infrequent runs or highly specialized use cases where you need absolute control over the session identity, supplying your own cookies might seem appealing. However, this comes with significant drawbacks. Instagram frequently invalidates sessions, requiring you to constantly re-export and update cookies. This introduces manual intervention and fragility into automated pipelines.

The managed session pool, on the other hand, is designed for resilience. It leverages a large pool of automatically rotated sessions, proxy IP addresses, and built-in retry mechanisms to bypass common blocking challenges. This "set it and forget it" approach is particularly valuable for data engineers building robust, hands-off data pipelines. It streamlines maintenance and reduces the likelihood of runs failing due to authentication issues. Moreover, the cost model is volume-based, not duration-based, so you are not penalized for the overhead of managing these sessions; the cost scales with the number of result events (scraped items).

What is the most efficient way to fan out scraping multiple Instagram profiles?

The most efficient way to fan out scraping multiple Instagram profiles is to pass an array of usernames directly to a single instagram-profile-scraper Actor run. The Actor is designed for batch processing and will scrape each profile sequentially within that single run, handling internal rate limits and retries. Using separate runs for each profile would incur additional Actor Start charges and increase orchestration complexity without providing significant performance benefits.

The usernames input field accepts an array of strings. For example: ["cristiano", "nike", "natgeo"]. When you provide multiple usernames, the Actor processes them one after another. This internal batching is optimized for Instagram's rate limits and the Actor's operational logic.

Consider an alternative: initiating a separate Actor run for each username. This would require your orchestration system (e.g., n8n, a custom Python script) to loop through your list of usernames, making an API call for each one. Not only does this add complexity to your orchestrator, but it also results in multiple Actor Start charges. For example, scraping 100 profiles in a single run would incur one Actor Start charge. Scraping 100 profiles in 100 separate runs would incur 100 Actor Start charges. Given that the Actor Start charge is $0.008 per GB of memory allocated, this can add up.

Here’s an example of preparing input for a batch run:

{
  "usernames": [
    "instagram",
    "natgeo",
    "nike",
    "nasa",
    "therock",
    "cristiano",
    "leomessi",
    "arianagrande",
    "kimkardashian",
    "selenagomez"
  ],
  "maxPosts": 50
}

This single JSON input sent to the Actor's run endpoint will initiate a job that processes all ten profiles. The Actor's internal mechanisms manage the queuing and execution for each profile, ensuring efficient resource utilization and adherence to best practices for Instagram scraping.

How does the pricing model affect pipeline design for this Actor?

The pricing model for the instagram-profile-scraper is pay-per-event, specifically $0.005 per result (dataset item) and $0.008 per GB of memory allocated for an Actor Start. This means your pipeline design should prioritize minimizing redundant Actor Start events and only retrieving the necessary result items to optimize costs.

Understanding the pay-per-event model is crucial. Unlike duration-based billing, where longer runs cost more, here you pay for explicit actions:

"result" (apify-default-dataset-item): $0.005 per event. This is charged for every profile record and every post record pushed to the default dataset. If you scrape 10 profiles with maxPosts: 0, you get 10 profile records. If you scrape 1 profile with maxPosts: 100, you get 1 profile record and 100 post records.
- Volume tiers for this event: FREE $0.005, BRONZE $0.00433, SILVER $0.00367, GOLD $0.003, PLATINUM $0.003, DIAMOND $0.003. As you process more results across all your Apify Actors, your per-item cost decreases.
"Actor Start" (apify-actor-start): $0.008 per GB of memory allocated to the run. This is charged once when the Actor begins execution.
- Volume tiers for this event: FREE $0.008, BRONZE $0.00633, SILVER $0.00467, GOLD $0.003, PLATINUM $0.003, DIAMOND $0.003.

This model incentivizes batching and intelligent input parameters:

Batching usernames: As discussed, running multiple usernames in a single Actor run means you incur only one Actor Start charge for all of them, regardless of how many profiles are processed. This is significantly more cost-effective than initiating a new run for each username.
Using maxPosts wisely: Setting maxPosts to 0 allows you to scrape only profile metadata, avoiding the cost of post records if you don't need them. If you only need the latest few posts, keeping maxPosts at a low value like 12 (the default) or 50 will be much cheaper than fetching 500 posts per profile.
State Management: By implementing the state management strategy outlined earlier, you avoid re-ingesting (and thus re-paying for) data you already have. Your post-processing logic should filter out duplicate posts based on pub_date and shortcode, ensuring that only truly new data incurs cost for downstream storage and processing.

The Actor Start charge is small per GB, but it's a fixed overhead per run. For large-scale operations, optimizing this can lead to substantial savings. This pricing structure reinforces the engineering principle of doing more with less, especially with API calls and task invocations.

Checked against the Actor's input schema and Apify docs on 2026-09-22.

What happens if an Instagram profile does not exist or is private?

If an Instagram profile does not exist or is private, the instagram-profile-scraper Actor handles these scenarios gracefully without failing the entire run. For non-existent profiles, a dataset record with status: "error" will be pushed, detailing the failure message (e.g., Profile @username does not exist or is not available). For private accounts, the Actor will return the available profile metadata, and the is_private field in the author_meta object will be true.

This behavior is critical for robust data pipelines. Instead of a hard crash that halts your entire batch, you receive clear, actionable feedback within the dataset. Your downstream processing can then identify these records by checking the status field or the is_private flag and handle them appropriately. This might involve logging the error, flagging the profile for review, or simply omitting posts from private accounts. Posts cannot be scraped from private accounts, as the Actor respects Instagram's privacy settings. The output will reflect this limitation rather than attempting to bypass privacy.

Here's an json example of how you might check for these conditions in your output:

[
  {
    "author_meta": {
      "username": "nonexistent_profile_12345",
      "full_name": null,
      "biography": null,
      "profile_pic_url": null,
      "is_verified": false,
      "is_private": false,
      "is_business": false,
      "followers_count": null,
      "following_count": null,
      "posts_count": null,
      "profile_url": "https://www.instagram.com/nonexistent_profile_12345/",
      "external_urls": []
    },
    "status": "error",
    "error_message": "Profile @nonexistent_profile_12345 does not exist or is not available",
    "scraped_at": "2026-09-22T10:00:00.000Z"
  },
  {
    "author_meta": {
      "username": "private_account_example",
      "full_name": "Private User",
      "biography": "This is a private account.",
      "profile_pic_url": "https://...",
      "is_verified": false,
      "is_private": true,
      "is_business": false,
      "followers_count": 123,
      "following_count": 45,
      "posts_count": 0,
      "profile_url": "https://www.instagram.com/private_account_example/",
      "external_urls": []
    },
    "status": "success",
    "scraped_at": "2026-09-22T10:00:00.000Z"
  }
]

When iterating through the dataset items, you would typically check the status key. If status is "error", you have a problem profile. If it's "success" and author_meta.is_private is true, then you know posts won't be available, and you can adjust your downstream logic accordingly. This granular error reporting ensures your pipeline remains robust and provides specific insights into data collection failures or limitations.

How do you integrate Apify Actor results into n8n for further automation?

You integrate Apify Actor results into n8n using the Apify Trigger node, which fires upon Actor run completion, and then retrieve data from the associated dataset. This eliminates the need for manual polling and allows you to chain subsequent n8n nodes for data transformation, conditional logic, and pushing data to various destinations like webhooks, databases, or cloud storage.

n8n offers a seamless integration with Apify, making it a powerful tool for extending your data pipelines. The key is to leverage the Apify Trigger node, specifically configured for "On Run Finished" events.

Here's a high-level workflow:

Configure Apify Trigger Node: In n8n, add an Apify Trigger node. Select the "On Run Finished" trigger type and specify the instagram-profile-scraper Actor. You'll need to link your Apify API key.
Start the Actor (Optional, or separate flow): You can either manually start the Actor, schedule it on Apify, or even have a separate n8n workflow kick off the Actor run (using the "Apify" node set to "Execute Actor"). The trigger node only listens for completion, it doesn't start the run.
Retrieve Data with Apify Node: After the trigger fires, add another "Apify" node. Set its operation to "Get All Dataset Items" or "Get Dataset Items (Paginated)" and link it to the run["defaultDatasetId"] which will be available from the trigger node's output. This retrieves the scraped data.
Process and Transform: Use n8n's vast array of nodes (e.g., "Set", "Function", "Item Lists", "Split In Batches") to clean, filter, and transform the data. This is where you'd implement your state management logic if it's not handled upstream: checking scraped_at timestamps, deduplicating, or enriching data.
Downstream Actions: Finally, push the processed data to your desired destination. This could be a "Webhook" node to send data to another service, a "PostgreSQL" or "MySQL" node to insert into a database, an "S3" node to store files, or even a "Slack" node for notifications.

Using n8n is particularly useful for handling the variety of outputs from the instagram-profile-scraper. For example, you might want to send profile updates to one system and new post data to another. n8n's conditional logic (IF nodes) allows you to route profile record types differently from post record types based on the presence of fields like post_url or author_meta.posts_count.

# A hypothetical Python snippet that could be used in an n8n Function node
# or a custom script after fetching dataset items to filter for new posts.

def filter_new_posts(dataset_items, last_processed_dates):
    new_posts = []
    for item in dataset_items:
        if "post_url" in item: # Check if it's a post record
            post_pub_date_str = item.get("pub_date")
            username = item.get("username")

            if post_pub_date_str and username:
                post_pub_date = datetime.fromisoformat(post_pub_date_str.replace("Z", "+00:00")) # Handle 'Z' suffix

                # Assume last_processed_dates looks like {"username": "ISO_DATE_STRING"}
                last_date_for_user_str = last_processed_dates.get(username)

                if last_date_for_user_str:
                    last_date_for_user = datetime.fromisoformat(last_date_for_user_str.replace("Z", "+00:00"))
                    if post_pub_date > last_date_for_user:
                        new_posts.append(item)
                else: # No previous date, consider all posts new
                    new_posts.append(item)
    return new_posts

# Example usage within an n8n function node (data would come from previous nodes)
# let items = $json["datasetItems"]; // Replace with actual path to dataset items
# let last_dates = { "instagram": "2026-05-01T00:00:00Z" }; // This would come from your DB/state store
# return [{ json: filter_new_posts(items, last_dates) }];

The example above illustrates the logic; in n8n, you'd likely map the last_processed_dates from a database query node into the Function node, and the dataset_items would be the output of the Apify "Get Dataset Items" node. This modularity makes n8n a powerful orchestrator for complex data flows originating from Apify Actors.

When does Apify's synchronous run endpoint time out, and what should you use instead?

Apify's synchronous run endpoint hard-caps at 300 seconds (5 minutes) and returns an HTTP 408 "Request Timeout" error if the Actor run exceeds this duration. For runs expected to last longer, you should use the asynchronous run endpoint (POST /v2/acts/<actorId>/runs) and implement polling for status updates or configure a webhook to receive notifications upon run completion.

Many data scraping tasks, especially those involving multiple profiles or a high maxPosts count, will naturally exceed the 5-minute synchronous limit. Attempting to use the synchronous endpoint for such runs will lead to unreliable behavior and frequent timeouts.

The asynchronous approach involves:

Initiating the Run: Make a POST request to the /v2/acts/<actorId>/runs endpoint. This immediately returns a runId.
Monitoring the Run: You have two main options for monitoring:
- Polling: Periodically (GET /v2/acts/<actorId>/runs/<runId>) to check the status field. Continue polling until the status is SUCCEEDED, FAILED, ABORTED, or TIMED_OUT.
- Webhooks: The more efficient and recommended method. When you start the Actor run, you can specify a webhook URL in the request payload. Apify will then POST to this URL with the run details when it completes. This is ideal for integrating with services like n8n, which has a dedicated Apify Trigger node that acts as a webhook listener.
Retrieving Results: Once the run has completed, use the defaultDatasetId from the run details to fetch the scraped data from the dataset.

This asynchronous pattern is standard practice for long-running operations in distributed systems. It prevents your client from blocking indefinitely and provides a resilient mechanism for handling unpredictable execution times.

Here's a bash example for starting an asynchronous run and an example of its webhook payload:

curl -X POST \
  https://api.apify.com/v2/acts/crawlerbros~instagram-profile-scraper/runs?token=<YOUR_APIFY_API_TOKEN> \
  -H 'Content-Type: application/json' \
  -d '{
    "usernames": ["instagram", "natgeo"],
    "maxPosts": 50,
    "webhook": {
      "requestUrl": "https://your.n8n.webhook.url/webhook/endpoint",
      "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"],
      "payloadTemplate": "{\"runId\": \"{{run.id}}\", \"defaultDatasetId\": \"{{run.defaultDatasetId}}\", \"status\": \"{{run.status}}\"}"
    }
  }'

And an example of the JSON payload that Apify would send to your webhook URL upon run completion (as defined by payloadTemplate above):

{
  "runId": "YOUR_RUN_ID_ABCDEF",
  "defaultDatasetId": "YOUR_DATASET_ID_12345",
  "status": "SUCCEEDED"
}

This JSON payload provides the necessary identifiers to then fetch the results from the correct dataset, all without your client having to wait or poll.

Limitations and Caveats

While the instagram-profile-scraper is a powerful tool, it's crucial to be aware of its inherent limitations and specific platform behaviors to build a truly robust pipeline.

Firstly, the Actor operates on publicly available data. If an Instagram account is private, the Actor returns the available profile metadata but cannot scrape any posts. This is by design, respecting Instagram's privacy settings, and is reflected by the is_private field in the author_meta object. Your pipeline must account for these cases, perhaps by filtering out private profiles or marking them for manual review.

Secondly, specific business contact details like email, phone, and category are only included in the output when Instagram exposes them on the profile itself. The README clearly states that if Instagram's internal API errors for a particular account, these fields might be absent, even if other author_meta fields are present. This means you cannot reliably expect these fields for every business profile. Your data model and downstream applications should treat these fields as optional and nullable.

Thirdly, Instagram's profile API doesn't return complete data for every account. In a small number of cases, Instagram's backend may error, causing the Actor to fall back to the public profile page. While this still provides follower/following counts, bio, and verification status, it will result in the absence of business contact details (email, phone, category). This is an Instagram-side restriction with no workaround, meaning the Actor always returns the fullest data it can retrieve, but that "fullest" can vary.

Finally, while the Actor handles batching usernames efficiently within a single run, it has a maxPosts limit of 500 posts per profile. If you need more than 500 posts from a single profile, you would have to run the Actor multiple times, potentially using a more advanced pagination strategy that tracks previously scraped posts and filters new requests based on pub_date or shortcode to avoid duplicates and adhere to the state management principles discussed earlier. However, the schema does not natively expose a way to resume from a specific post cursor or date, so custom post-processing is essential for large historical scrapes.

Key takeaways

Building reliable data pipelines that integrate external scraping tools requires careful consideration of not just data extraction, but also orchestration, state management, and cost optimization. The instagram-profile-scraper Actor offers a robust way to collect public Instagram profile and post data, but its true power is unlocked when you wire it into a broader system that accounts for its specific input/output patterns and platform constraints.

By understanding the pay-per-event pricing model, leveraging managed sessions for operational simplicity, employing asynchronous run patterns for longer jobs, and diligently managing state to avoid reprocessing, data engineers can construct efficient and cost-effective pipelines. Handling non-existent or private profiles gracefully ensures pipeline resilience, while integrating with tools like n8n streamlines post-processing and downstream delivery. This holistic approach moves beyond mere data collection to building sustainable and intelligent data workflows.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Why Strict Filters Return Empty Results on Google Suggest Scraper

Crawler Bros — Tue, 22 Sep 2026 10:58:25 +0000

Data engineering, at its core, is about resilience. We build systems that expect failure, predict its shape, and gracefully recover. When working with external data sources, especially those that abstract away underlying complexities like proxy management and rate limiting, understanding specific failure modes is paramount. The documentation often hints at these, but it rarely spells out the defensive coding strategies needed to handle them.

Let's examine the Google Keywords Suggest Scraper Pro, an Actor on Apify for extracting autocomplete suggestions. Rather than a feature tour, we'll dissect its input schema, output shape, and platform constraints to identify implied failure modes and how to engineer against them.

What Input Constraints Mean for Run Failures

The google-keywords-suggest-scraper-pro Actor exposes a clear input schema, which, while helpful for defining valid parameters, also implicitly defines several ways a run can fail or return unexpected data. The keywords field, for instance, is an array of strings and is explicitly marked [required]. Submitting an empty array or omitting this field will cause an immediate validation error. However, a more subtle failure mode arises when the keywords provided are highly obscure or niche, particularly when combined with restrictive filters.

The minLength and maxLength integer fields, which drop suggestions outside a character range, are powerful but also dangerous. If minLength is set too high (e.g., minLength: 50) or maxLength too low (e.g., maxLength: 5), it's highly probable that many, if not all, legitimate suggestions will be filtered out. Similarly, the containsKeyword string filter, especially when inverted with !, can easily eliminate all results.

The Actor's README explicitly addresses this: "What happens when filters drop everything? The actor finishes without fake rows and sets a status message with the active filters so you can loosen them." This is a crucial piece of information. It means you won't get empty placeholder records, but you will get an empty dataset. Detecting this defensively requires inspecting the run's metadata or status message, not just the dataset's item count.

Here’s a Python example demonstrating a valid but potentially self-defeating input, which you’d submit to the Apify API:

import apify_client

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "keywords": ["an extremely obscure keyword that probably nobody searches"],
    "mode": "all",
    "country": "US",
    "language": "en",
    "minLength": 100,  # Highly restrictive, likely to drop everything
    "maxLength": 101,  # Equally restrictive
    "containsKeyword": "!commonword", # Invert filter, drop anything containing 'commonword'
    "maxItemsPerKeyword": 200,
    "outputFormat": "flat"
}

# Not executing the run here, just showing the input structure
print(run_input)

How Does a Synchronous Run Timeout Affect Data Extraction?

Synchronous API calls to an Actor, where you await completion, have a hard cap. Apify's synchronous run endpoint returns an HTTP 408 if the run exceeds 300 seconds (5 minutes). This means jobs expected to run longer than 5 minutes must use the asynchronous endpoint.

This is a critical limitation for larger extraction jobs. If you submit a request with a high number of keywords and mode set to alphabet (which generates 26 sub-queries per seed keyword), or if you set maxItemsPerKeyword to a high value like 5000, the run duration can easily exceed this 300-second threshold. When this cap is hit, your application will receive a timeout error, even if the Actor itself continues to run in the background. Your code might interpret this as a complete failure, leading to retries or incomplete data processing. To handle this, any job expected to run longer than 5 minutes must use the asynchronous POST /v2/acts/<actorId>/runs endpoint, followed by polling the run status or using webhooks.

import apify_client
import time

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

# Example input that could potentially exceed 300 seconds
run_input = {
    "keywords": [f"long tail keyword {i}" for i in range(500)], # Many keywords
    "mode": "alphabet", # 26 expansions per keyword
    "country": "US",
    "language": "en",
    "maxItemsPerKeyword": 200,
    "outputFormat": "flat"
}

# Start the Actor asynchronously
run = client.actor("crawlerbros/google-keywords-suggest-scraper-pro").call(run_input=run_input, wait_for_finish=0)
run_id = run["id"]
print(f"Actor run started with ID: {run_id}")

# Poll for run completion (simplified example, consider exponential backoff)
while True:
    run_status = client.run(run_id).get()
    if run_status["status"] == "SUCCEEDED":
        print("Run succeeded!")
        break
    elif run_status["status"] in ["FAILED", "ABORTED"]:
        print(f"Run failed or aborted: {run_status['statusMessage']}")
        break
    print(f"Run status: {run_status['status']}. Waiting...")
    time.sleep(30) # Wait 30 seconds before polling again

# At this point, you'd fetch the dataset items

How Does Actor Memory Affect Billing and Performance?

The "Actor Start" event in the pricing for the Google Keywords Suggest Scraper Pro is charged at $0.005 per GB of memory allocated to the run, with a minimum of one event. This means higher memory usage leads to increased "Actor Start" costs.

This is a subtle but important detail. While this Actor is unlikely to be extremely memory-intensive for typical use cases, it’s worth noting that if your input keywords array becomes extremely large, requiring significant internal processing or data structures, the system might allocate more memory, thus increasing the "Actor Start" cost. This isn't a direct failure mode but an implicit cost escalation you might not predict if you're only focused on the result events. Monitoring memory usage for your Actor runs, especially for large-scale operations, can help optimize costs.

What About Partial or Sentinel Values in the Output Shape?

The output schema for the Google Keywords Suggest Scraper Pro provides two formats: flat and tree. Understanding the nuances of these is crucial for robust data processing. In flat mode, each record is a single suggestion. Key fields include text, source_keyword, expansion_query, mode, country, language, and relevance. In tree mode, one record is emitted per source_keyword, containing a suggestions array.

Consider the relevance field. It's an integer, typically around 600, with higher values indicating more popularity. While this is a useful proxy for search volume, it's not the actual volume. Expecting search volume here would be a misinterpretation. The Actor explicitly states, "Volume requires Google Ads Keyword Planner (paid + OAuth). This actor returns Suggest's relevance score... which is a useful proxy." This means relevance is a sentinel value – a proxy that tells you something, but not the exact metric you might be accustomed to from other SEO tools.

Another important field is expansion_query. The README clarifies: "It's the generated query that was sent to Google Suggest for that batch of results... Google can still return nearby variants, so the suggestion text is not guaranteed to start with that exact string." This means you cannot assume text will always be a direct prefix of expansion_query. Your downstream processing should account for text being related but not necessarily strictly matching expansion_query in a prefix sense.

Here's an example of flat output, showing these fields:

{
  "recordType": "suggestion",
  "text": "python tutorial for beginners",
  "source_keyword": "python tutorial",
  "expansion_query": "python tutorial f",
  "mode": "alphabet",
  "country": "US",
  "language": "en",
  "relevance": 601,
  "suggestion_type": "QUERY",
  "sub_types": [512],
  "scrapedAt": "2026-04-29T10:30:00.000Z"
}

When consuming this data, you might want to filter or categorize based on relevance. For instance, dropping suggestions below a certain relevance score could be a valid data cleaning step, but it must be understood as filtering by a proxy, not an absolute volume.

Why Are Proxy Sessions Important for Consistent Data?

Proxy session management is crucial for other Apify Actors that utilize them, though not explicitly exposed in this Actor. While the Google Keywords Suggest Scraper Pro doesn't require proxies, understanding their behavior is vital for robust data pipelines using other Actors.

Datacenter proxies persist for around 26 hours, while residential proxies typically last only about 30 minutes. If you were to build a system that relies on persistent sessions for a different Actor, a residential proxy's short lifespan could lead to session drops and failed requests, even if the Actor itself attempts to manage retries. For this Actor, the lack of explicit proxy control means less configuration overhead but also less insight into potential transient network issues that Google's suggestion service might encounter globally. The Apify platform's ability to target specific US states with country-US_XX for proxies (though not exposed in this Actor's input) highlights the granularity available for geographic targeting. When a future Actor does expose proxy configuration, knowing these limits and capabilities prevents unexpected failures due to proxy expiration or geo-restriction issues.

How to Avoid Unnecessary Costs and Data Loss on the Free Tier?

Understanding the cost model and storage retention policies is crucial for managing data pipelines, especially on the Apify free plan. The Google Keywords Suggest Scraper Pro Actor, available at https://apify.com/crawlerbros/google-keywords-suggest-scraper-pro, charges per event: "$0.002 per event" for "result" (a single record in the default dataset) and "$0.005 per GB of memory allocated to the run" for "Actor Start".

Cost scales directly with the number of result records. A run with keywords=["python tutorial"], mode="alphabet", maxItemsPerKeyword=200 would generate up to 260 suggestions (10 * 26). If it produced 200 suggestions, that would be 200 "result" events. For a free-tier user, this would be 200 * $0.002 = $0.40, plus the "Actor Start" cost. For users on volume tiers, prices drop: BRONZE $0.00167, SILVER $0.00133, GOLD $0.001, PLATINUM $0.001, DIAMOND $0.001 per result event.

To avoid data loss on the free plan, remember that only the 10 most recent runs are retained for four months. After that, unnamed storages (like the default dataset) expire. If you need to persist data longer or across more than 10 runs, you must use named storages or export your data regularly. Named storages are exempt from deletion. This is a common pitfall for new users who assume all data is permanently saved.

import apify_client

client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

# Example of fetching dataset items after a run, and considering named storage
# Assuming 'my_named_dataset' is a dataset you created with a specific name
# If you didn't specify a named dataset in the Actor run, it uses the default (unnamed)
dataset_id = "my_named_dataset_id" # Replace with actual named dataset ID if used

try:
    dataset_items = client.dataset(dataset_id).list_items().items
    for item in dataset_items:
        print(item)
except apify_client.errors.ApifyApiError as e:
    print(f"Error fetching dataset items: {e}. Check if dataset exists or is expired.")

# To proactively save data to a named storage in a run:
# You'd typically set the datasetId in the ApifyClient.call() or Apify.main() options
# This example is illustrative of the concept.

Are Apify Schedules and Webhooks Reliable for Event-Driven Pipelines?

Yes, but with caveats. Apify schedules use a 6-field cron syntax (seconds optional), with a minimum interval of 10 seconds. However, new schedules are created DISABLED by default, and an Actor must have run at least once before it can be scheduled. These are critical setup steps that, if missed, will prevent your scheduled pipelines from triggering. Always enable the schedule and ensure a successful manual run has occurred.

Webhooks provide an event-driven primitive, POSTing to a URL on specific run events (like completion). This is the only direct integration for eventing. If you need to push data to Slack or S3, you must route it through a webhook to an intermediary service like n8n, Make, or Zapier, or a custom backend. There is no native S3 or Slack integration directly from Apify.

n8n offers a "Trigger node" that fires on run completion, eliminating the need for polling. This is a powerful feature for building low-latency, event-driven workflows. For n8n Cloud users, OAuth2 credentials are supported, while self-hosted n8n instances can use API keys.

{
  "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"],
  "requestUrl": "https://your-webhook-endpoint.com/receive-apify-data",
  "payloadTemplate": "{\"runId\": {{run.id}}, \"status\": \"{{run.status}}\", \"datasetId\": \"{{run.defaultDatasetId}}\"}"
}

This JSON represents a webhook configuration that would notify an external endpoint about run success or failure, including the run ID and default dataset ID.

What Are the Real Limitations and Caveats of the Google Keywords Suggest Scraper Pro?

The Google Keywords Suggest Scraper Pro Actor is highly effective for its stated purpose but has inherent limitations based on Google Suggest itself and the platform.

No Direct Search Volume Data: As noted, it returns a relevance score, not actual search volumes. If your primary use case requires precise volume metrics, you'll need to augment this data with a Google Ads Keyword Planner integration, which is a separate, paid service with OAuth requirements.
No Deep SERP Analysis: This Actor focuses solely on autocomplete suggestions. It doesn't scrape full search engine results pages (SERPs), nor does it provide features like "People Also Ask" boxes or related searches shown on the SERP. Those would require a different, more complex scraper.
Rate Limiting Handled Internally: While the Actor handles Google's rate limiting, you don't have explicit control over the pacing of requests. This is generally a benefit, but if you have very specific rate-limiting needs or want to integrate with a custom proxy solution, this Actor's abstraction limits that flexibility.
No Complex Query Logic: The mode field provides specific expansion patterns (questions, prepositions, alphabet, etc.). There's no way to pass arbitrary custom prefixes or suffixes beyond what the modes offer, nor complex boolean logic for keyword generation.
Single Request Queue Consumer: A crucial Apify platform detail is that a request queue can only be PROCESSED by one Actor or task run at a time. While multiple runs can add to a queue, you cannot fan out processing across a single shared queue. If you have a massive list of keywords to process in parallel, you would need to create multiple input queues or use different mechanisms for load distribution.

These aren't necessarily "failures" of the Actor, but rather design choices and platform constraints that define its scope and how it interacts with the broader Apify ecosystem. Understanding them prevents misaligned expectations and guides architectural decisions for integrating the data.

Checked against the Actor's input schema and Apify docs on 2026-09-22.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Scraping Missouri Business Entities Without Cold-Session Blocks

Crawler Bros — Tue, 22 Sep 2026 08:05:28 +0000

Automating corporate entity verification across Secretary of State portals presents distinct infrastructure challenges. In Missouri, the Business Entity Search system (bsd.sos.mo.gov) hosts public corporate records without requiring an account or a CAPTCHA. However, automated collection pipelines often fail when querying records directly because the underlying ASP.NET WebForms backend enforces session continuity. Fetching a detail page (BusinessEntityDetail.aspx) cold—without the session cookies established during the initial search query—triggers an immediate "Access Denied" response.

The Missouri Secretary of State Business Search Scraper handles session persistence across searches and detail lookups while managing Cloudflare IP-reputation denials via proxy session rotation.

The Cold-Session Bottleneck on Missouri SOS

Missouri's business entity portal uses classic ASP.NET WebForms architecture. When a user executes a search, the portal writes specific session state to the browser. If an automated script parses the search results grid and attempts to request individual entity detail pages using isolated, stateless HTTP clients, the server blocks those requests.

Furthermore, the portal sits behind a Cloudflare IP-reputation gate. Unlike JavaScript challenges or reCAPTCHA barriers, this gate responds with a flat HTTP "Access Denied" when a shared datacenter IP has accumulated too much traffic. A cold fetch to a detail page exacerbates this issue.

To scrape full entity details reliably, an automation system must:

Maintain session cookies from the initial search POST request through every subsequent detail-page fetch.
Catch "Access Denied" reputation blocks and rotate datacenter proxy IPs on demand.
Traverse the 20-row-per-page ASP.NET pagination state until hitting a user-defined record cap.

Search Axes and Core Input Parameters

The actor interfaces directly with the state's public search endpoints through two primary lookup modes and exact-to-broad matching rules.

{
  "mode": "byBusinessName",
  "searchTerm": "AMAZON CAPITAL SERVICES, INC.",
  "nameSearchMethod": "exactMatch",
  "activeOnly": true,
  "maxItems": 5
}

Key configuration properties in the input schema include:

mode: Accepts byBusinessName to query registered corporate entities or byRegisteredAgent to search against designated commercial or organizational agents.
searchTerm: The target string (for example, a company name or agent name).
nameSearchMethod: Maps directly to the underlying ASP.NET dropdown options: startingWith, allWords, anyWord, or exactMatch.
activeOnly: A boolean flag that restricts returned records to entities currently in Good Standing.
maxItems: An integer (ranging from 1 to 200) setting a hard ceiling on extracted records.

Execution Workflow

To run a search pipeline targeting active entities matching a specific naming pattern:

Configure the search payload defining the target string, search axis, and match strategy.
Initialize the run using the Apify Python Client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "mode": "byBusinessName",
    "searchTerm": "LOGISTICS",
    "nameSearchMethod": "startingWith",
    "activeOnly": True,
    "maxItems": 40,
}

run = client.actor("crawlerbros/missouri-sos-business-search-scraper").call(
    run_input=run_input
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(
        f"{item.get('charterNumber')}: {item.get('entityName')} ({item.get('status')})"
    )

The scraper accesses bsd.sos.mo.gov, parses the grid, captures the internal IDs, and fetches the detail view for each row under the same session.
If a detail page fails after internal retries, the actor preserves the high-level metadata (charter number, entity name, registration date, status) from the initial results grid rather than discarding the record.

Output Schema and Field Mapping

Each emitted dataset record contains full corporate identity data extracted from the detail page:

{
  "recordType": "entity",
  "entityName": "EXAMPLE LOGISTICS LLC",
  "charterNumber": "LC01429811",
  "entityType": "Domestic Limited Liability Company",
  "status": "Good Standing",
  "domesticity": "Domestic",
  "homeState": "Missouri",
  "dateFormed": "2018-04-12",
  "duration": "Perpetual",
  "principalAddress": "100 MAIN ST, KANSAS CITY, MO 64105",
  "registeredAgentName": "CORPORATE CREATIONS NETWORK INC.",
  "registeredAgentId": "RA0012948",
  "registeredAgentAddress": "2847 S INGRAM MILL RD STE A-100, SPRINGFIELD, MO 65804",
  "internalId": "12984711",
  "sourceUrl": "https://bsd.sos.mo.gov/BusinessEntity/BusinessEntityDetail.aspx?ID=12984711",
  "scrapedAt": "2025-02-17T12:00:00.000Z"
}

Optional attributes such as reportDue are emitted when present on the state's record and omitted when the field is empty on the source page.

Pricing and Cost Mechanics

Billing for this scraper uses a flat pay-per-event pricing model. Charges are tied directly to run events rather than execution duration:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run, billed once when execution begins (minimum 1 GB).
Result Item (apify-default-dataset-item): $0.005 per entity result emitted into the default dataset.

Under volume tiers, the per-result event price scales as follows:

FREE: $0.005
BRONZE: $0.00433
SILVER: $0.00367
GOLD: $0.003
PLATINUM: $0.003
DIAMOND: $0.003

A 1 GB run yielding 50 records at the base tier costs $0.005 (Actor Start) + 50 × $0.005 (results) = $0.255.

Architecture Boundaries

This scraper does not parse historical document filings or retrieve PDF copies of annual reports from the state archive. Additionally, for registered agent queries, the tool pulls the representative entity returned by Missouri's primary search grid rather than expanding the nested multi-entity index associated with large corporate agents. Large runs targeting high-volume registered agents require configuring higher run timeouts to accommodate IP rotation delays across repeated detail lookups.

Runs in this article used Missouri Secretary of State Business Search Scraper. Its README is the reference for input fields and output structure; this post is only one path through them.

Handling Imperva Cookie Handshakes for Massachusetts Entity Data

Crawler Bros — Tue, 22 Sep 2026 04:01:44 +0000

Automating business verification for Massachusetts entities often runs into a specific infrastructural barrier. The Massachusetts Corporations Division search (corp.sec.state.ma.us) hosts registry data—entity type, registered agent name, principal office address, and active officers. However, the search form sits behind Imperva/Incapsula protection.

Unlike registry portals that deploy full JavaScript execution walls (such as Nevada's SilverFlume portal, which returns a static 1 KB _Incapsula_Resource iframe on raw HTTP requests), the Massachusetts registry operates protection primarily at the cookie level. Standard headless browser setups add unnecessary overhead when pure HTTP session warming can solve the access problem directly.

The massachusetts-sos-scraper actor bypasses this barrier over HTTP, collecting registry data and structuring it for downstream data pipelines.

Bypassing Cookie Challenges Over Plain HTTP

When a standard HTTP client hits the Massachusetts search portal for the first time, Imperva returns an interstitial response. If your scraper does not maintain the session state and evaluate cookie headers, subsequent search requests fail or get dropped.

The scraper handles this protocol interaction automatically:

It sends an initial warm-up HTTP request to capture the seeded visid_incap_* and incap_ses_* session cookies.
It re-uses that initialized session for subsequent form submissions against corp.sec.state.ma.us.
It queries the registry search endpoints and parses each entity summary page (CorpSummary.aspx) without requiring a headless browser.

Because every search match requires an independent detail-page request to extract complete records (such as officer tables and merger history), the actor uses deliberate delays between requests and falls back to free datacenter (AUTO) proxies if the target endpoint applies temporary rate limits.

Search Axes and Filtering Modes

The scraper accepts four operational modes via the mode parameter: byEntityName, byIndividualName, byIdentificationNumber, and byFilingNumber.

{
  "mode": "byEntityName",
  "entityName": "Acme Logistics",
  "entityMatchMode": "M",
  "maxItems": 10
}

Entity Name Matching

When querying with mode: "byEntityName", the entityMatchMode parameter controls how the Massachusetts backend filters company records:

B (Begins with): Returns entities starting with the search string.
M (Exact match): Restricts results to exact legal name matches.
F (Full text): Finds the query substring anywhere in the registered legal name.
S (Soundex): Matches phonetically similar company names.

Individual Lookups for Officers and Agents

When auditing individuals or mapping corporate networks, setting mode: "byIndividualName" queries the registry's officer and registered agent index. The search requires lastName, with optional narrowing via firstName and middleName.

{
  "mode": "byIndividualName",
  "lastName": "SMITH",
  "firstName": "JOHN",
  "individualMatchMode": "M",
  "maxItems": 25
}

Because individual search results return an entry for every distinct appointment across all registered entities, common surnames yield dozens of separate corporate records. Using individualMatchMode: "M" prevents unbounded result lists.

Derived Inactivity Status and Data Structure

The Massachusetts portal does not provide a standard status: "Active" string on its summary page. Instead, inactive corporations display an explicit inactivity label (such as Date of Involuntary Dissolution by Court Order or by the SOC) alongside a dissolution date, while active entities leave this section empty.

The scraper normalizes this quirk into structured fields:

status: Outputs either "Active" or "Inactive".
statusReason: Contains the exact dissolution or revocation string provided by the state when inactive.
statusDate: Formatted ISO date (YYYY-MM-DD) when inactive.
officers: Array of objects containing { title, name, address }.
mergedWith: Array of absorbed entities ({ entityName, date, sourceUrl }), linking directly to the merged entity's historical summary page.
previousNames: Historical record of legal name changes ({ previousName, date }).

Empty fields are omitted from the resulting dataset items to keep payloads compact.

Running the Scraper via the Apify Python Client

You can run this extraction pipeline programmatically using Python to pull data directly into your ETL flow.

Install the client library:

pip install apify-client

Initialize the client and run a targeted search:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "mode": "byIdentificationNumber",
    "identificationNumber": "000445089",
    "maxItems": 1
}

run = client.actor("crawlerbros/massachusetts-sos-scraper").call(run_input=run_input)
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for entity in dataset_items:
    print(f"Entity: {entity.get('entityName')}")
    print(f"Status: {entity.get('status')} ({entity.get('statusReason', 'N/A')})")
    print(f"Agent: {entity.get('registeredAgentName')}")
    for officer in entity.get("officers", []):
        print(f"  - {officer.get('title')}: {officer.get('name')}")

Pricing and Cost Calculation

The actor uses a pay-per-event pricing model based strictly on executed events.

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run (billed once upon run initialization).
Result (apify-default-dataset-item): Billed per record written to the dataset.
- FREE: $0.005 per result
- BRONZE: $0.00433 per result
- SILVER: $0.00367 per result
- GOLD: $0.003 per result
- PLATINUM: $0.003 per result
- DIAMOND: $0.003 per result

For example, on the FREE tier, running a batch job that initializes with 1 GB of memory ($0.005) and extracts 100 entity records ($0.50) costs a total of $0.505.

Operational Constraints

This scraper does not download original PDF filings or annual report documents, as raw document retrieval requires the Secretary of the Commonwealth's paid certificate portal. Furthermore, the maxItems property is strictly capped at 100 items per run to prevent registry endpoint blocks during multi-page entity traversals. For large corporate directories, batch your inputs by distinct entity numbers or narrow prefixes rather than executing broad open-ended queries.

Massachusetts Secretary of State Business Search Scraper is the Actor behind these examples. If a selector in your own version breaks, compare your output against the fields listed in its README first.

Extracting B2B Software Metadata via Shared Catalog Records

Crawler Bros — Tue, 22 Sep 2026 00:00:54 +0000

Building B2B competitive intelligence pipelines requires extracting consistent software catalog metrics across pricing models, platform dependencies, and core features. Bypassing front-end scraper protection on product aggregators directly often results in ip blocks or inconsistent DOM parsing. The Capterra Software Scraper handles this by querying the underlying Gartner Digital Markets infrastructure behind Capterra and GetApp, returning structured JSON metadata across product catalogs.

Resolving Capterra Products Without Direct HTML Scraping

Extracting metadata directly from Capterra product pages presents scraping challenges due to aggressive bot mitigation controls. The catalog data powering Capterra is shared across the Gartner Digital Markets network, including GetApp. Instead of parsing heavily protected Capterra HTML directly, this actor targets the shared catalog backend through the GetApp stack while returning URL attributes pointing back to original Capterra listings.

When executing queries, the actor uses a Google SERP proxy to resolve public product pages. For individual product detail extraction, it routes traffic through residential proxy rotation to ensure high retrieval rates without triggering rate limits.

The extracted output includes high-value fields such as startingPrice, pricingPlans, licensingModel, features, integrations, pros, cons, and vendor background data like vendorFoundedYear and vendorWebsite. Empty fields are omitted entirely rather than returned as null placeholders.

Configuring Modes for Direct URLs vs Search Queries

The actor operates in two distinct operational modes determined by the mode parameter: direct URL ingestion (byProductUrls) and discovery search (bySearchQueries).

Direct Extraction with `byProductUrls`

When specific software entities are already identified, pass explicit Capterra product URLs. This approach bypasses search resolution entirely, moving directly to catalog data extraction.

{
  "mode": "byProductUrls",
  "productUrls": [
    "https://www.capterra.com/p/135003/Slack/",
    "https://www.capterra.com/p/228385/Notion/"
  ],
  "maxItems": 2
}

Keyword Resolution with `bySearchQueries`

For exploratory collection or dynamic catalog building, supply targeted search strings. The actor resolves these broad queries against public product pages before pulling catalog details.

{
  "mode": "bySearchQueries",
  "searchQueries": [
    "Slack",
    "project management software"
  ],
  "maxItems": 10
}

The maxItems integer restricts the total number of processed products across the run execution, preventing unexpected dataset inflation.

Executing the Scraper Programmatically

You can invoke the scraper using the Apify Python SDK. Install the client library first:

pip install apify-client

This Python script initializes the client, configures search queries to capture project management tools, and extracts the results into a pandas DataFrame for analysis:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_KEY")

run_input = {
    "mode": "bySearchQueries",
    "searchQueries": ["project management software"],
    "maxItems": 50,
}

# Run the actor and wait for completion
run = client.actor("crawlerbros/capterra-software-scraper").call(
    run_input=run_input
)

# Fetch results from the run's default dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    name = item.get("name")
    rating = item.get("rating")
    starting_price = item.get("startingPrice")
    features = item.get("features", [])

    print(
        f"Product: {name} | Rating: {rating} | Starts at: {starting_price}"
    )
    print(f"Features count: {len(features)}\n")

Structure of the Extracted Catalog Schema

A successfully processed product record returns structured JSON containing company, technical, and commercial parameters:

{
  "productId": "135003",
  "slug": "Slack",
  "name": "Slack",
  "url": "https://www.capterra.com/p/135003/Slack/",
  "tagline": "Where work happens",
  "rating": 4.7,
  "reviewCount": 23000,
  "startingPrice": "8.75",
  "pricingModel": "Per User",
  "priceCurrency": "USD",
  "freeTrial": true,
  "freeVersion": true,
  "primaryCategory": "Collaboration Software",
  "categories": ["Team Communication", "Collaboration"],
  "features": ["Chat/Messaging", "File Sharing", "Video Conferencing"],
  "featureCount": 42,
  "integrationsCount": 2400,
  "pros": ["Ease of use", "Extensive integration library"],
  "cons": ["Search indexing limits on free plan"],
  "vendorName": "Slack Technologies",
  "vendorCountry": "United States"
}

If a product fails to resolve during the search phase, the actor skips emitting the record entirely. It does not generate synthetic or fallback placeholder items.

Cost Structure Analysis

Pricing for this actor operates exclusively under a Pay-Per-Event billing model. You pay strictly for runtime initialization and successfully dataset-emitted items.

Actor Start Event (apify-actor-start): Flat $0.005 per GB of memory allocated to the run.
Dataset Result Event (apify-default-dataset-item): $0.005 per emitted item at the base Free tier.

Volume discounts apply to result events for higher platform usage tiers:

BRONZE: $0.00433 per item
SILVER: $0.00367 per item
GOLD / PLATINUM / DIAMOND: $0.003 per item

For example, a execution run configured with 1 GB of memory ($0.005) that extracts 100 resolved software products on the base tier ($0.005 x 100 = $0.50) incurs a total run cost of $0.505.

It is important to note that this actor does not scrape individual end-user review text strings or paginated category listing pages; it focuses exclusively on product-level catalog metadata.

Capterra Software Scraper is the Actor behind these examples. If a selector in your own version breaks, compare your output against the fields listed in its README first.

Normalizing Luxury Resale Attributes Across 7 Inconsistent Departments

Crawler Bros — Mon, 21 Sep 2026 20:01:39 +0000

Secondary luxury resale markets do not follow uniform product schemas. If you write scrapers for authenticated resale platforms like The RealReal, you immediately hit schema fragmentation across departments. Clothing uses standard apparel sizes (CLOTHING_SIZE or FOREIGN_SIZE), footwear uses shoe sizing (SHOE_SIZE), jewelry stores dimensions as RING_SIZE alongside METAL_TYPE, and watches store physical dimensions under CASE_DIAMETER.

Extracting these disparate properties into a clean analytics pipeline usually requires writing custom normalization logic for each product category. If you try to track resale margins or build pricing models across multiple luxury departments, raw API responses leave critical fields empty unless your scraper resolves department-specific attribute mappings.

The therealreal-scraper Actor solves this by consolidating department-specific attributes across seven top-level departments (women, men, jewelry, watches, art, home, and kids) into unified output fields like size and material, while surfacing specialized luxury metadata when present.

Resolving Department-Specific Attributes

When crawling authenticated listings, general-purpose scrapers often drop department-level metadata. The RealReal's internal facet architecture separates fields based on category context. For example, a collector tracking luxury timepieces needs movement mechanisms and reference numbers, while a jewelry analyst needs clarity and gemstone details.

The scraper maps these underlying attributes into concrete top-level fields:

Watches: Populates watchModel, referenceNumber, watchCollection, movementType, caseMaterial, bandMaterial, watchStyles[], and complications[].
Jewelry: Populates caratWeight (summed total carat weight across stones of that type), clarityGrade, colorGrade, gemstone, stoneType, and stoneShapes[].
Apparel: Populates category-specific attributes such as dressSilhouette (e.g., "Wrap Dress", "Evening Gown") for dresses and chestSize (e.g., "42", "48 +") for men's garments.
Condition: Separates the structured condition grade (As Is, Fair, Good, Very Good, Excellent, Pristine) from the free-text conditionNotes field available when fetching complete detail.

Because empty fields are omitted from emitted records, items only contain the attributes disclosed by the source listing.

Discovering Live Facets Before Extraction

Targeting specific categories or designers often fails when scrapers rely on hardcoded URL slugs. If a brand changes its catalog naming or a taxonomy node shifts, scrapers return zero results.

To prevent blind querying, the Actor provides two discovery modes:

discoverCategories: Reads The RealReal's live category tree straight from facet data, returning records with name, department, categoryPath, depth, and current productCount.
discoverDesigners: Emits up to 200 designer names for a department or category, returning designerName, designerSlug, designerId, and productCount.

Using discovery runs first ensures that downstream extraction runs target valid categoryPath and designerSlug inputs with active inventory.

Execution Walkthrough: Building a Filtered Watch Dataset

To collect listings for luxury watches matching specific physical attributes, configure the scraper to target the watches department with precise attribute filtering.

1. Configure the Run Input

Define the run parameters using the Actor's filtering fields. The configuration below queries the watches department, applies price boundaries, filters for automatic movements in stainless steel cases, and caps the volume using maxItems.

{
  "mode": "byCategory",
  "gender": "watches",
  "minPrice": 2000,
  "maxPrice": 15000,
  "movementType": "Automatic",
  "caseMaterial": "Stainless Steel",
  "condition": ["Very Good", "Excellent", "Pristine"],
  "sortBy": "price_asc",
  "maxItems": 100
}

2. Execute via the Apify Python SDK

Run the Actor and process the dataset records programmatically:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "mode": "byCategory",
    "gender": "watches",
    "minPrice": 2000,
    "maxPrice": 15000,
    "movementType": "Automatic",
    "caseMaterial": "Stainless Steel",
    "condition": ["Very Good", "Excellent", "Pristine"],
    "sortBy": "price_asc",
    "maxItems": 100,
}

run = client.actor("crawlerbros/therealreal-scraper").call(run_input=run_input)

dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    print(
        f"{item.get('brand')} {item.get('watchModel')} "
        f"| Ref: {item.get('referenceNumber')} "
        f"| Price: ${item.get('priceUsd')} "
        f"| Condition: {item.get('condition')}"
    )

3. Handle Product Detail Payloads

When running in byUrl mode with explicit productUrls, the returned dataset items contain deep product data, including measurements[], authenticationInfo, carbonSavedKg, waterSavedLiters, and valuationReportUrl:

{
  "recordType": "product",
  "productId": "10984211",
  "sku": "WCH12345",
  "brand": "Rolex",
  "name": "Submariner Date 116610LN",
  "department": "watches",
  "condition": "Excellent",
  "conditionNotes": "Minor scratches throughout metal.",
  "movementType": "Automatic",
  "caseMaterial": "Stainless Steel",
  "priceUsd": 11500,
  "estRetailPriceUsd": 10250,
  "productUrl": "https://www.therealreal.com/products/watches/rolex-submariner-date-116610ln",
  "scrapedAt": "2026-07-27T10:15:30.000Z"
}

Actor Billing Mechanics

This Actor operates on a pay-per-event pricing model rather than standard compute runtimes. Charges are calculated directly from two event types:

Actor Start (apify-actor-start): A flat charge of $0.005 per GB of memory allocated to the run, incurred once when execution begins.
Dataset Result (apify-default-dataset-item): Billed per emitted item in the default dataset. Pricing follows volume tiers:
- FREE: $0.005 per event
- BRONZE: $0.00433 per event
- SILVER: $0.00367 per event
- GOLD: $0.003 per event
- PLATINUM: $0.003 per event
- DIAMOND: $0.003 per event

For example, on the standard FREE tier, a run allocated 1 GB of memory that extracts 500 watch listings incurs $0.005 for the start event and $2.50 for the 500 emitted results ($0.005 × 500).

Pipeline Boundaries

This Actor is designed for extraction and facet discovery; it does not monitor real-time shopping cart state or automate the checkout process. Additionally, while byUrl mode retrieves full condition descriptions and measurement arrays, markdown indicators like discountPercent are rarely present because the underlying browse listing API does not reliably expose markdown signals. Filtering pipelines requiring strict markdown verification must calculate discounts downstream by comparing priceUsd against originalPriceUsd or estRetailPriceUsd.

The RealReal Scraper is the Actor behind these examples. If a selector in your own version breaks, compare your output against the fields listed in its README first.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-21. Check the Actor page for the current rates.

Scrape 1,000 LinkedIn Post Reactions per URL Using the Voyager API

Crawler Bros — Mon, 21 Sep 2026 16:04:07 +0000

Extracting audience engagement data from social platforms often presents a common engineering hurdle: UI limitations. On LinkedIn, when a post accumulates thousands of reactions, clicking the reaction count only reveals a partial modal. The interface caps the visible list of users, preventing manual or basic DOM-scraping tools from capturing the full picture.

For data engineers building lead generation pipelines or sentiment analysis models, this restriction is a critical bottleneck. Safely extracting this data requires interacting with the platform's internal APIs while managing session states and rate limits.

The LinkedIn Post Reactions Scraper solves this by querying the internal Voyager API—the same data layer that powers the official web application. This allows developers to retrieve structured profiles of users who engaged with any specific post.

Parsing LinkedIn Reactor Data Structures

When querying the reactions surface, the scraper bypasses the heavy web UI and directly extracts structured JSON. The output normalizes varied user interaction states into a clean, predictable schema.

For each reactor, the tool returns a dataset containing the following fields:

reactorName: The full name of the user.
reactorProfileUrl: The direct link to their LinkedIn profile.
reactorHeadline: The user's current headline or professional title.
reactionType: The specific interaction type, mapped from raw API codes to human-readable strings (like, celebrate, love, support, insightful, funny, curious).
postId: The unique numerical activity ID associated with the update.
scrapedAt: An ISO 8601 UTC timestamp recording when the extraction occurred.

This structured format makes it easy to pipe data directly into downstream database tables or CRM systems without complex post-processing regex.

Here is an example of the structured JSON payload returned by the actor:

{
  "reactorName": "Jane Doe",
  "reactorProfileUrl": "https://www.linkedin.com/in/janedoe",
  "reactorHeadline": "Senior Engineer at Acme Corp",
  "reactionType": "like",
  "postUrl": "https://www.linkedin.com/feed/update/urn:li:activity:7234567890123456789/",
  "postId": "7234567890123456789",
  "inputPostUrl": "https://www.linkedin.com/feed/update/urn:li:activity:7234567890123456789/",
  "scrapedAt": "2025-01-15T10:30:00+00:00"
}

Configuring the Input Schema for Targeted Scraping

To execute a run, the scraper requires specific configuration parameters to navigate authentication and scope the request.

The Session Cookie (`cookie`)

Because the LinkedIn reaction modal is not publicly accessible without authentication, you must provide a valid session cookie. Specifically, the scraper requires the li_at cookie value, which authenticates requests to the Voyager API. You can extract this from your browser's Developer Tools under the Application tab, or export your full cookies array using a browser extension.

Target Posts (`postUrls`)

This array accepts complete LinkedIn URLs or bare activity IDs. The scraper natively parses standard formats, including:

Activity URNs: urn:li:activity:7234567890123456789
Bare IDs: 7234567890123456789
Feed update URLs: https://www.linkedin.com/feed/update/urn:li:activity:7234567890123456789/

Scrape Volume and Filtering (`maxReactorsPerPost` and `reactionTypeFilter`)

LinkedIn strictly limits the visible list of reactors to approximately 1,000 per post, regardless of whether the post shows a higher reaction count in the UI. To optimize your run and control data volume, set the maxReactorsPerPost parameter (defaults to 100, with a maximum cap of 5,000 to accommodate edge-case API behavior).

If you only want to extract high-intent leads—such as users who left an "insightful" or "curious" reaction—you can configure the reactionTypeFilter array. Passing specific values like insightful or love instructs the scraper to drop other reaction types before writing to the dataset.

Step-by-Step Extraction Workflow

Retrieve your session cookie: Log into LinkedIn, open your browser's Developer Tools, navigate to the Cookies storage section for www.linkedin.com, and copy the value of the li_at cookie.
Collect target URLs: Copy the URLs of the LinkedIn posts you want to analyze.
Configure the input JSON: Construct your payload with the required variables. Below is a standard configuration payload:

{
  "postUrls": [
    "https://www.linkedin.com/feed/update/urn:li:activity:7234567890123456789/"
  ],
  "cookie": "YOUR_LI_AT_COOKIE_VALUE_HERE",
  "maxReactorsPerPost": 500,
  "reactionTypeFilter": ["insightful", "love"],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "groups": ["RESIDENTIAL"]
  }
}

Run the Actor: Execute the run on the Apify platform. The scraper will automatically handle rate limits using exponential backoff, mimicking human pacing to avoid session invalidation.

Calculating Run Costs on the Pay-Per-Event Model

This scraper operates under a strict Pay-Per-Event pricing model, meaning you are only billed for specific successful actions rather than execution time or arbitrary platform units.

The costs are calculated using two distinct event types:

"Actor Start" (apify-actor-start): Charged once per run at a flat rate of $0.005 per GB of memory allocated to the run.
"result" (apify-default-dataset-item): Charged per individual reactor profile successfully written to the default dataset. This event uses volume-tier pricing:
- FREE: $0.005 per result
- BRONZE: $0.00433 per result
- SILVER: $0.00367 per result
- GOLD: $0.003 per result
- PLATINUM: $0.003 per result
- DIAMOND: $0.003 per result

For example, running a single-GB execution to extract 500 reactors under the FREE tier would cost exactly $0.005 for the start event and $2.50 for the 500 result events, bringing the total run cost to $2.505.

Knowing When to Use a Different Tool

While this scraper is highly efficient for targeted engagement analysis, it is not a general-purpose search tool. It cannot search for posts globally by keyword or hashtag; it requires specific, pre-determined post URLs or activity IDs to begin extraction. If your pipeline requires finding high-performing posts before analyzing who reacted to them, you would need to feed this actor with URLs extracted from a separate keyword-search scraper first.

Source for the runs in this article: LinkedIn Post Reactions Scraper. The input schema there is authoritative; treat anything in this post that contradicts it as out of date.

DEV Community: Crawler Bros

Overcoming Cloudflare Blocks on Preply to Analyze Tutor Pricing Trends

How to Query the Preply Public Directory

Extracting Detailed Profiles with IDs

Step-by-Step Implementation

Predictable Event-Based Costs

Limitations of the Scraping Approach

Extracting Track BPM and Key Data Past Beatport Search Limits

The Search Pagination Bottleneck

Structure of the Extracted Track Dataset

Filtering Parameters for Targeted Ingestion

Running an Ingestion Workflow via Python

Step-by-Step Setup

System Boundaries

Calculating N M Distance Matrices Without API Key Quotas

Handling Coordinate Formatting and Direct API Queries

Calculating Matrix Distances and Snapping Coordinates

Nearest Road Lookup (mode: "nearest")

Distance Matrix Generation (mode: "matrix")

How to Set Up an OSRM Pipeline

Cost Structure and Execution Mechanics

Practical Pipeline Limitations

Why Exposing Facebook Comments to MCP Agents Times Out on 300 Second Runs

Exposing Facebook Comments to AI Agents via Model Context Protocol

What tool signature does the agent interpret?

Why does relying on UI prefill fields break programmatic agent calls?

How do you handle the 300 second synchronous run cap?

How can an AI agent detect that the max total charge cap has been tripped?

What are the limitations and caveats of this scraper?

What is the exact cost model for executing runs?

How do you process nested comments without double-charging?

How do you handle non-existent pages or authentication blocks?

Tracking Per-Size ASOS Stock Across 10 Regional Stores

Understanding Regional Differences and Per-Size Stock

Execution Walkthrough

Step 1: Define Target Store and Filter Constraints

Step 2: Extract Variant-Level Inventory

Step 3: Parse and Ingest into a Data Pipeline

Pricing and Execution Event Structure

Limitations and Pipeline Boundaries

Why Batching Instagram Profiles Reduces Your Actor Start Costs

Orchestrating Instagram Profile Data Workflows

How do you avoid re-processing old Instagram data on subsequent runs?

When should you use managed Instagram sessions instead of supplying your own cookies?

What is the most efficient way to fan out scraping multiple Instagram profiles?

How does the pricing model affect pipeline design for this Actor?

What happens if an Instagram profile does not exist or is private?

How do you integrate Apify Actor results into n8n for further automation?

When does Apify's synchronous run endpoint time out, and what should you use instead?

Limitations and Caveats

Key takeaways

Why Strict Filters Return Empty Results on Google Suggest Scraper

What Input Constraints Mean for Run Failures

How Does a Synchronous Run Timeout Affect Data Extraction?

How Does Actor Memory Affect Billing and Performance?

What About Partial or Sentinel Values in the Output Shape?

Why Are Proxy Sessions Important for Consistent Data?

How to Avoid Unnecessary Costs and Data Loss on the Free Tier?

Are Apify Schedules and Webhooks Reliable for Event-Driven Pipelines?

What Are the Real Limitations and Caveats of the Google Keywords Suggest Scraper Pro?

Scraping Missouri Business Entities Without Cold-Session Blocks

The Cold-Session Bottleneck on Missouri SOS

Search Axes and Core Input Parameters

Execution Workflow

Output Schema and Field Mapping

Pricing and Cost Mechanics

Architecture Boundaries

Handling Imperva Cookie Handshakes for Massachusetts Entity Data

Bypassing Cookie Challenges Over Plain HTTP

Search Axes and Filtering Modes

Entity Name Matching

Individual Lookups for Officers and Agents

Derived Inactivity Status and Data Structure

Running the Scraper via the Apify Python Client

Pricing and Cost Calculation

Operational Constraints

Extracting B2B Software Metadata via Shared Catalog Records

Resolving Capterra Products Without Direct HTML Scraping

Configuring Modes for Direct URLs vs Search Queries

Direct Extraction with byProductUrls

Nearest Road Lookup (`mode: "nearest"`)

Distance Matrix Generation (`mode: "matrix"`)

Direct Extraction with `byProductUrls`

Keyword Resolution with `bySearchQueries`

The Session Cookie (`cookie`)

Target Posts (`postUrls`)

Scrape Volume and Filtering (`maxReactorsPerPost` and `reactionTypeFilter`)