Checked against the Actor's input schema and Apify docs on 2026-09-18.
Connecting an autonomous agent or programmatic pipeline to live Instagram discussion threads requires balancing browser automation with remote protocol limits. The Model Context Protocol (MCP) allows language models and local clients to invoke remote tools directly. However, running an extraction tool over thousands of nested comments can cause clients to drop connections long before Instagram finishes paginating through replies.
When using the Instagram Comment Scraper, long jobs must be handled correctly within tool execution protocols, accounting for proxy rotation, session handling, and platform event caps.
How do you expose an Apify Actor as an MCP tool?
Configure your client to point to the Apify MCP server URL with an explicit tool filter. Passing ?tools=crawlerbros/instagram-comment-scraper to the server URL restricts the protocol context strictly to this Actor, preventing catalog token bloat.
The public Apify MCP server at https://mcp.apify.com exposes four tools unauthenticated: search-actors, fetch-actor-details, search-apify-docs, and fetch-apify-docs. Actually running an Actor always requires an Apify API token. If you expose the root endpoint without filtering, the server feeds your client dozens of generic discovery tools.
To scope the client exclusively to Instagram comment retrieval, configure the MCP client configuration file:
{
"mcpServers": {
"instagram-comments": {
"command": "npx",
"args": [
"-y",
"@modelcontextprotocol/server-sse",
"https://mcp.apify.com/sse?tools=crawlerbros/instagram-comment-scraper"
],
"env": {
"APIFY_TOKEN": "apify_api_yourActualTokenHere"
}
}
}
}
When scoped, the MCP server parses the Actor definition directly into a single callable function declaration. This prevents the LLM from attempting hallucinated multi-step discovery searches across the entire store directory.
How does the Actor input schema translate into a tool signature?
The schema translates directly into JSON Schema parameter blocks within the MCP tool declaration. Required fields like postUrls become mandatory properties, while operational flags carry default integer and boolean types.
The Actor expects parameters controlling crawl depth, nested reply expansion, and session cookies. Here is the translated JSON Schema representing the input tool signature exposed to an MCP client:
{
"name": "crawlerbros_instagram-comment-scraper",
"description": "Extract comments from Instagram posts and reels with complete metadata including replies, likes, and author details.",
"parameters": {
"type": "object",
"properties": {
"postUrls": {
"type": "array",
"items": { "type": "string" },
"description": "List of Instagram post or reel URLs to scrape comments from. Supports formats: /p/, /reel/, /tv/, /share/, or direct shortcodes."
},
"maxCommentsPerPost": {
"type": "integer",
"default": 100,
"description": "Maximum number of comments to scrape per post, up to 10000. Higher values may take longer."
},
"includeReplies": {
"type": "boolean",
"default": true,
"description": "Include comment replies (nested comments). Enabling this will expand reply threads."
},
"maxRepliesPerComment": {
"type": "integer",
"default": 0,
"description": "Maximum number of replies to fetch per individual comment thread (when Include Replies is enabled). Set to 0 for unlimited."
},
"cookies": {
"type": "string",
"description": "Instagram authentication cookies in JSON format. Optional, leave blank to use the managed session pool."
},
"sessionName": {
"type": "string",
"description": "Saved session name in key-value store if cookies were stored previously."
}
},
"required": ["postUrls"]
}
}
A critical platform rule applies when executing Actors programmatically: schema prefill values configured in the Apify Console UI are ignored entirely during API and MCP invocation. Only the explicit default properties defined in the schema persist if omitted by the caller. Your client must pass explicit parameters if it requires values different from the bare defaults. For instance, if an automated agent does not supply maxCommentsPerPost, the Actor falls back to its schema default of 100, regardless of what might appear visually in the console form.
Managing long running MCP tool calls with progress notifications
When an MCP client invokes a tool that extracts thousands of comments, standard synchronous execution models often break at the client layer. Most client runtimes default to short read timeouts (such as 30 to 60 seconds). If the remote scraper takes several minutes expanding nested replies across multiple posts, the client closes the transport, causing the parent task to fail.
To handle long executions within the Model Context Protocol without abandoning tool abstraction, the protocol supports progress notifications (notifications/progress) on active tool calls. When the client issues a tools/call request with a progressToken in its _meta field, the server can emit continuous intermediate updates.
Here is an example showing how an MCP client manages a long-running extraction by establishing a progress token and handling progress notifications to keep transport sessions alive:
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def run_scraper_with_progress():
server_params = StdioServerParameters(
command="npx",
args=["-y", "@modelcontextprotocol/server-sse", "https://mcp.apify.com/sse?tools=crawlerbros/instagram-comment-scraper"],
env={"APIFY_TOKEN": "your_apify_token"}
)
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Handler keeps transport active and logs pagination progress
async def handle_progress(params):
print(f"Scraper progress: {params.progress}/{params.total} comments fetched")
session.on_notification("notifications/progress", handle_progress)
# Invoke tool with an explicit progress token
result = await session.call_tool(
"crawlerbros_instagram-comment-scraper",
arguments={
"postUrls": ["https://www.instagram.com/p/C_example_code/"],
"maxCommentsPerPost": 500,
"includeReplies": True,
"maxRepliesPerComment": 5
},
_meta={"progressToken": "req-scrape-001"}
)
return result
By providing progress heartbeats across the protocol channel, the client avoids local socket read timeouts. This enables the scraper to complete deep thread traversals without requiring the client to bail out of MCP into external ad-hoc polling loops.
Why do comment scrapers fail during cursor pagination?
Instagram comments do not load in a static sequence. The platform relies on GraphQL cursors to paginate top-level comments and child replies. During large extractions, several scraping-specific failure modes occur.
First, residential proxy sessions have an expected persistence window of roughly 30 minutes, whereas datacenter sessions persist for around 26 hours. Because Instagram actively flags datacenter IP ranges for logged-in endpoints, scrapers often route through residential proxies. If pagination across a viral post with nested replies exceeds 30 minutes, the residential proxy session rotates. Instagram frequently binds active GraphQL pagination cursors and CSRF tokens to the originating IP address and TLS fingerprint. When the proxy switches mid-crawl, subsequent cursor queries return HTTP 400 or empty node arrays.
Second, active threads suffer cursor invalidation. When hundreds of users post comments concurrently on a viral reel, the internal pagination pointer shifts. If the scraper attempts to paginate backward or fetch the next cursor page after a delay, Instagram's API can invalidate the cursor pointer, terminating pagination prematurely.
Third, aggressive reply expansion triggers Instagram's internal rate limits. While fetching top-level comments requires one request per page, fetching nested replies requires separate requests for each parent comment that contains child threads. When includeReplies: true is combined with maxRepliesPerComment: 0, the scraper must issue requests for every single thread. Under rapid rotation, Instagram detects the repetitive GraphQL query hash and returns rate-limit challenges.
To mitigate pagination failure on large jobs:
- Cap child threads using
maxRepliesPerComment(e.g., set to 5 or 10 rather than 0). - Provide dedicated authenticated session cookies via
cookiesorsessionNameto reduce session rotation friction. - Keep individual runs bounded to specific batches using
maxCommentsPerPost.
Output structure and defensive field parsing
Every item written to the default dataset represents an individual comment or reply. Downstream processing pipelines must parse these records defensively because reply relationships and media attachments modify the available properties.
Here is an example output record returned by the scraper:
{
"commentId": "17843670504683182",
"text": "Congrats! 🎉 @friendname check this out",
"commentType": "text",
"isGif": false,
"authorUsername": "user123",
"authorId": "427553890",
"authorIsVerified": false,
"authorProfilePic": "https://scontent.cdninstagram.com/profile.jpg",
"timestamp": "2025-05-01T14:32:10",
"likesCount": 12,
"replyCount": 3,
"isReply": false,
"isEdited": false,
"mentions": ["friendname"],
"hashtags": [],
"postUrl": "https://www.instagram.com/p/ABC123xyz/",
"postShortcode": "ABC123xyz",
"commentUrl": "https://www.instagram.com/p/ABC123xyz/c/17843670504683182/",
"scrapedAt": "2025-12-04T13:06:03.499460"
}
When building an ingestion pipeline, downstream logic must account for fields present only on replies: parentCommentId and parentCommentAuthor. If isReply is false, those keys are absent.
Additionally, items contain a commentType field indicating the content format: text, image, video, reel, photo_share, album_share, gif, or sticker. If an automated pipeline extracts text without evaluating commentType, it will ingest placeholder strings such as [Photo unavailable] or [GIF:id].
Here is a defensive Python processing loop that cleans raw dataset records and tags media attachments appropriately:
def process_scraped_comments(items: list) -> list:
processed_records = []
for item in items:
c_type = item.get("commentType", "text")
raw_text = item.get("text", "")
# Detect Instagram web API placeholders
is_placeholder = raw_text.startswith("[") and raw_text.endswith("]")
clean_record = {
"id": item.get("commentId"),
"author": item.get("authorUsername"),
"is_reply": item.get("isReply", False),
"parent_id": item.get("parentCommentId") if item.get("isReply") else None,
"likes": item.get("likesCount", 0),
"media_type": c_type,
"has_direct_media": bool(item.get("mediaUrl")),
"media_url": item.get("mediaUrl"),
"text": "" if is_placeholder else raw_text,
"placeholder_token": raw_text if is_placeholder else None,
}
processed_records.append(clean_record)
return processed_records
Real limitations and caveats
The scraper cannot view private accounts without explicit credentials, caps extraction at 10000 comments per post, and cannot extract direct media URLs for non-GIF attachments.
First, private account posts cannot be scraped using the managed session pool. If you submit a post URL from a private profile, the Actor returns zero comments unless valid JSON session cookies belonging to an account that follows that target profile are supplied via cookies or sessionName.
Second, Instagram's web API withholds CDN URLs for attached images, videos, and shared reels, returning only placeholder strings like [Photo unavailable] or [Reel unavailable]. Direct media links (mediaUrl) are only returned when Instagram includes them natively, which currently applies strictly to GIF comments.
Third, the scraper imposes a platform ceiling of 10000 comments per post via maxCommentsPerPost. Attempting to scrape threads larger than this threshold will result in truncation at the 10000th item.
Storage lifecycle rules also dictate retention. Unnamed datasets expire automatically. On the free platform tier, data is retained for only the 10 most recent runs over a 4-month span. Named storages are exempt from automatic deletion. Pipelines querying past runs must store their data in named storages or ingest the records immediately.
Additionally, developers attempting to parallelize extraction across multiple workers must understand platform concurrency boundaries: a single request queue can only be processed by one Actor run at a time. While multiple runs may add requests to a queue, fan-out processing across a single shared queue does not work.
Defending against pagination traps with maxTotalChargeUsd
Agents given broad prompts will often trigger unbounded runs by requesting thousands of comments on viral posts. This can cause unexpected charges.
Apify provides the maxTotalChargeUsd parameter to enforce strict spend ceilings. This value is passed into the run configuration and exposed to the container environment as ACTOR_MAX_TOTAL_CHARGE_USD. When the accrued event charges hit this threshold, the platform initiates run termination. Note that this termination is not an instant process: the Actor container receives a termination signal, giving it a brief window to stop, during which minimal additional events could still be recorded.
You can configure spend limits and timeouts directly via the Apify JavaScript SDK:
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({
token: process.env.APIFY_TOKEN,
});
async function safeCommentScrape(targetPostUrl) {
const run = await client.actor('crawlerbros/instagram-comment-scraper').call(
{
postUrls: [targetPostUrl],
maxCommentsPerPost: 500,
includeReplies: true,
maxRepliesPerComment: 5,
},
{
// Abort if cost hits $0.50 USD
maxTotalChargeUsd: 0.50,
timeoutSecs: 300,
}
);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
return items;
}
Setting maxTotalChargeUsd guarantees that deep pagination loops terminate before consuming unexpected balance.
Cost breakdown for pay-per-event execution
The Instagram Comment Scraper uses a PAY_PER_EVENT pricing model. Every charge this Actor makes is one of two published named events, with no separate platform runtime fees.
The charged events are:
-
Actor Start (
apify-actor-start): Charged once when the Actor starts running at $0.01 per GB of memory allocated to the run. Across volume tiers, this base start event costs:- FREE: $0.01
- BRONZE: $0.00767
- SILVER: $0.00533
- GOLD: $0.003
- PLATINUM: $0.003
- DIAMOND: $0.003
-
Result (
apify-default-dataset-item): Charged at $0.005 per event for every single result stored in the default dataset. Across volume tiers, this result event costs:- FREE: $0.005
- BRONZE: $0.00433
- SILVER: $0.00367
- GOLD: $0.003
- PLATINUM: $0.003
- DIAMOND: $0.003
Cost scales with the number of events a run emits, which corresponds directly to the total number of result records saved. The event multiplier is governed by your input parameters:
total_cost = actor_start_cost + (total_dataset_items * result_event_price)
The number of output records is multiplied by maxCommentsPerPost, the total number of entries in postUrls, and the threading flags: includeReplies and maxRepliesPerComment. Setting includeReplies: true with maxRepliesPerComment: 0 removes the reply ceiling, allowing viral comment threads to emit thousands of items and directly scaling the total event count.
To keep runs economical, set includeReplies: false when high-level sentiment suffices, or set maxRepliesPerComment to a small non-zero integer to sample conversations without capturing entire reply cascades.
How do you route scraped comments into external webhooks?
Configure a platform webhook that fires an HTTP POST payload to your endpoint upon run completion. Apify webhooks support exactly one action: POSTing an event payload to a target URL.
Rather than running constant polling loops inside long-lived microservices, you can attach an event-driven webhook directly when initiating the crawl. The platform sends run metadata directly to your external orchestration API, serverless function, or n8n workflow.
Here is an example curl command demonstrating how to start an Actor run and attach an event-driven webhook that triggers immediately on run completion:
curl -X POST "https://api.apify.com/v2/acts/crawlerbros~instagram-comment-scraper/runs?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"postUrls": ["https://www.instagram.com/p/ABC123xyz/"],
"maxCommentsPerPost": 50,
"includeReplies": false
}' \
--data-urlencode 'webhooks=[{
"eventTypes": ["ACTOR.RUN.SUCCEEDED"],
"requestUrl": "https://api.yourdomain.com/webhooks/instagram-comments"
}]'
When the scraping job succeeds, the platform POSTs a JSON payload containing the run ID and default dataset ID to https://api.yourdomain.com/webhooks/instagram-comments. Downstream microservices can then ingest the dataset items without maintaining idle open connections.
If you are automating through n8n, note that n8n provides a dedicated Apify Trigger node that listens for Actor completions natively, eliminating the need to write custom webhook receivers. However, there are no native AWS S3 or Slack integrations on the platform; any export into cloud buckets or notifications must be routed through these webhooks or an automation platform.
By isolating schema mechanics, handling MCP progress notifications, avoiding proxy pagination traps, and parsing payload placeholders defensively, developers can maintain reliable extraction pipelines across Instagram posts and reels.
The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com
Top comments (0)