Input Schema Quirks and Silent Failures
The instagram-keyword-search-scraper Actor provides a powerful interface for extracting Instagram post data, but understanding the nuances of its input schema is crucial for preventing silent failures or unexpected behavior. While the keywords array is explicitly required, other fields like cookies and sessionName carry implicit failure modes if not handled defensively.
For instance, the cookies field, if left blank, defaults to a managed pool of shared Instagram sessions. This is convenient, but residential proxy sessions on the Apify platform typically persist for around 30 minutes. If your run exceeds this duration, especially for a maxPosts value near the 10000 limit across many keywords, you might see a degradation in search results or even a complete halt for subsequent keywords due to expired sessions, without an explicit error that points to cookie failure. The Actor documentation implies this by stating that the managed pool is recommended for most users, suggesting that complex or very long running operations benefit from user-provided cookies.
Moreover, the sessionName field, designed for saving and loading cookies between runs, is only effective when providing your own cookies. If you rely on the managed pool and also set a sessionName, the field will do nothing, yet the Actor will still accept it. This isn't a failure, but it's a wasted input that might lead to confusion when debugging session persistence issues. Always be explicit in your code about whether you're managing sessions or relying on the platform's defaults. Additionally, note that Apify input schema prefill values only show in the Console UI and are not applied to API calls; only default values are. Always pass your inputs explicitly.
{
"keywords": ["travel", "streetfood"],
"maxPosts": 1500,
"cookies": "[{\"name\":\"sessionid\",\"value\":\"YOUR_SESSION_ID\",\"domain\":\".instagram.com\"}]",
"sessionName": "my_instagram_session_prod"
}
This input JSON explicitly uses user-provided cookies and a session name, mitigating the reliance on the managed pool's session lifetime for a potentially long-running scrape.
How do you detect partial output data?
To detect partial output data, inspect the status field for values other than success, and check for missing keys in optional fields like like_count or comment_count. When an author hides likes, the like_count key is omitted entirely from the JSON payload rather than being set to null or zero. Your processing pipeline must explicitly check for these missing keys before handling.
A robust data pipeline needs to check for the presence of these optional fields before attempting to process them. Instead of assuming their existence, defensively check for the key. If the key is missing, it signifies that the information was either unavailable or intentionally suppressed by Instagram, as per the likes_hidden flag. This is a critical distinction from a null or zero value, which would imply the data was present but zero.
For instance, if is_paid_partnership is missing, it doesn't mean the post is not a paid partnership, it means the Actor could not determine or confirm it. The presence of sponsor_tags is a stronger indicator for paid content. Relying solely on is_paid_partnership could lead to undercounting sponsored content.
import json
def process_instagram_post(post_data):
# Always present fields
input_keyword = post_data.get("input_keyword")
post_url = post_data.get("post_url")
status = post_data.get("status")
# Defensively check for optional fields
like_count = post_data.get("like_count")
comment_count = post_data.get("comment_count")
is_paid_partnership = post_data.get("is_paid_partnership")
location = post_data.get("location")
if status != "success":
print(f"Warning: Post {post_url} has status: {status}")
if like_count is None and not post_data.get("likes_hidden", False):
print(f"Anomaly: Like count missing for {post_url} but likes not hidden.")
if is_paid_partnership:
print(f"Sponsored content detected for {post_url}")
elif post_data.get("sponsor_tags"):
print(f"Potential sponsored content (via sponsor tags) for {post_url}")
# Process location if present
if location:
print(f"Post from {location.get('name')} at {location.get('lat')}, {location.get('lng')}")
# Example usage with a sample output item
sample_output_item = {
"input_keyword": "travel",
"post_url": "https://www.instagram.com/p/ABC123/",
"shortcode": "ABC123",
"post_id": "3711296250294340356",
"username": "john_travels",
"author_meta": {
"id": "123456789",
"username": "john_travels",
"full_name": "John Smith",
"is_verified": False,
"profile_pic_url": "https://scontent.cdninstagram.com/v/example.jpg",
"profile_url": "https://www.instagram.com/john_travels/",
"followers": 12500,
"post_count": 340
},
"caption": "Exploring the streets of Lisbon #travel #portugal",
"hashtags": ["travel", "portugal"],
"mentions": [],
"pub_date": "2024-11-15T14:23:10+00:00",
"media_type": "Photo",
"media_count": 1,
"thumbnail_url": "https://scontent.cdninstagram.com/v/example-thumb.jpg",
"media_items": [
{
"index": 0,
"type": "Photo",
"url": "https://scontent.cdninstagram.com/v/example-full.jpg",
"width": 1080,
"height": 1350
}
],
"like_count": 1842,
"comment_count": 34,
"likes_hidden": False,
"comments_disabled": False,
"tagged_users": [],
"collaborators": [],
"sponsor_tags": [],
"is_paid_partnership": False,
"is_ad": False,
"status": "success",
"scraped_at": "2026-07-03T10:00:00+00:00"
}
process_instagram_post(sample_output_item)
How to handle Actor timeouts and proxy limits?
To handle Actor timeouts, switch to asynchronous runs and poll the run endpoint or use webhooks. The Apify synchronous endpoint hard-caps runs at 300 seconds, throwing an HTTP 408 error past that threshold. To manage proxy limits and rotation issues during long scrapes, provide your own browser cookies via the input schema to reduce reliance on the platform's shared session pool.
If your Actor run is expected to exceed 5 minutes, as a comprehensive keyword search for maxPosts: 10000 likely would, you must initiate the run asynchronously via the /v2/acts/<actor>/runs endpoint and then poll for its completion or use webhooks. Attempting a synchronous run for a task that naturally exceeds 300 seconds will cause an opaque failure.
Proxy limits relate to session longevity. Residential proxy sessions, often used in managed proxy pools, typically persist for about 30 minutes. If you are using the Actor's default managed session pool and scraping for extended periods, you might find that after approximately half an hour, the effectiveness of the search step decreases as underlying proxy sessions expire. While the Actor has built-in retry logic and automatic managed session rotation, it is not foolproof against all forms of session degradation. For critical, long-duration runs, providing your own robust cookie set via the cookies input field, especially if linked to a stable, dedicated proxy setup, can significantly improve reliability.
# Asynchronous run initiation for tasks expected to exceed 300 seconds
# Replace ACTOR_ID and API_TOKEN with your actual values
curl -X POST \
https://api.apify.com/v2/acts/crawlerbros/instagram-keyword-search-scraper/runs?token=YOUR_API_TOKEN \
-H 'Content-Type: application/json' \
-d '{
"keywords": ["travel", "photography", "foodie"],
"maxPosts": 5000
}'
This curl command initiates an asynchronous run, returning a run ID immediately. You would then poll the run status or configure a webhook to be notified upon completion.
What happens if the input maxPosts is excessive?
If you provide a maxPosts value exceeding the maximum limit of 10000, the Actor's input schema constraint restricts processing to the documented ceiling of 10000 per keyword. Alternatively, setting maxPosts to 0 triggers a full crawl using your own cookies. This instructs the scraper to run indefinitely and paginate through all available search results until no new posts are found.
It is reasonable to infer that the Actor will either cap the effective maxPosts at 10000 internally or, if validated by the platform, reject the input schema. The safest assumption for robust code is that it will respect the upper bound. However, even within the 10000 limit, setting a high maxPosts can have other implications.
For example, if you set maxPosts to 0 with your own cookies for a "full crawl," as shown in the README example, it signifies that you intend to retrieve all available posts for that keyword. This effectively overrides the 10000 maximum. The Actor will then continue paginating until no more unique posts are found or until an external limit, such as a maximum total charge, is hit.
This means that maxPosts is not just a simple cap; it interacts with the Actor's pagination logic and the concept of a full crawl. If you intend to get a comprehensive dataset, setting it to 0 with robust authentication is the correct approach, acknowledging that this can lead to significantly longer run times and higher resource consumption.
{
"keywords": ["architecture"],
"maxPosts": 0,
"cookies": "[{\"name\":\"sessionid\",\"value\":\"YOUR_SESSION_ID\",\"domain\":\".instagram.com\"}]",
"sessionName": "my_instagram_full_crawl"
}
This specific maxPosts: 0 input tells the Actor to scrape all posts for "architecture" using your provided session.
What limitations does this Actor have?
This Actor cannot access posts from private profiles or view age-restricted posts without proper account-level authentication. If Instagram applies rate limits or blocking, the Actor may produce error or blocked statuses despite its built-in session rotation. It is also limited to what Instagram's search results expose, meaning it cannot scrape content hidden by search algorithms.
The Instagram Keyword Search Scraper is powerful, but it has inherent limitations tied to Instagram's public data availability and the platform's constraints. It explicitly states it will not work on private accounts, meaning any posts from private profiles appearing in search results are filtered out or inaccessible for scraping. This is not an Actor defect but a fundamental limitation of scraping public web content. Consequently, your data will always represent a subset of all Instagram content for a keyword.
Furthermore, the output schema mentions status values like age_restricted, error, blocked, and invalid_url. These are direct indicators of limitations. Age-restricted posts are detected but not fully enriched, highlighting that even for public content, some data may remain inaccessible. Error and blocked statuses indicate that despite the Actor's built-in retry logic and automatic managed session rotation, failures can still occur, often due to aggressive bot detection or rate limiting by Instagram, or issues with the underlying proxy infrastructure. These are not silent failures, as the status field clearly marks them, but they represent boundaries of the Actor's capabilities in the face of external resistance.
The Actor also relies on Instagram's search results. If Instagram's search algorithm de-prioritizes certain types of content or applies content moderation that affects visibility, the Actor's output will reflect these biases. It cannot surface content that Instagram itself chooses not to display in search.
How can I monitor costs effectively?
To monitor costs effectively, track your Compute Unit (CU) consumption and residential proxy usage, which are billed based on your Apify pricing tier. One CU represents 1024MB memory running for one hour. You can protect your budget by passing the maxTotalChargeUsd query parameter with your API calls, which terminates the run when your specified dollar threshold is approached.
Apify charges primarily based on Compute Units and proxy usage. One CU is defined as 1024MB of memory for one hour. The rate per CU varies by pricing tier: $0.20 on Free and Starter plans, $0.16 on Scale, and $0.13 on Business. Residential proxy usage is charged separately: $8/GB for Free/Starter, $7.50/GB for Scale, and $7/GB for Business.
For an Actor like this, cost scales directly with maxPosts and the number of keywords. More posts and keywords mean more network requests, more data processing, and longer run durations, all contributing to higher CU and proxy consumption. A critical defensive strategy is to use the maxTotalChargeUsd query parameter when starting a run, which is also exposed to Actor code as ACTOR_MAX_TOTAL_CHARGE_USD. Setting this will automatically terminate the run when the specified dollar limit is approached, preventing unexpected bills. While termination is not instantaneous and some resources may be consumed briefly after the cap is tripped, it is an essential safeguard.
Additionally, be mindful of unnamed storages on the Free plan, where only the 10 most recent runs are retained for 4 months. If you need to preserve data from every run for cost analysis or auditing, ensure you use named storages, which are exempt from deletion.
from apify_client import ApifyClient
apify_client = ApifyClient("YOUR_APIFY_TOKEN")
# Run the Actor with a maximum cost limit of $5 (example for Free/Starter tier)
run = apify_client.actor("crawlerbros/instagram-keyword-search-scraper").call(
run_input={
"keywords": ["travel", "architecture"],
"maxPosts": 1000
},
max_total_charge_usd=5
)
print(f"Actor run started: {run['id']}")
print(f"Current CU consumption: {run['stats']['computeUnits']:.2f} CU")
This Python snippet demonstrates how to initiate a run with a max_total_charge_usd parameter, a crucial mechanism for cost control.
Detecting failed post enrichment and data integrity issues
The status field in the output record is your primary line of defense against data integrity issues. It is a sentinel value that explicitly communicates the outcome of the post enrichment process. Instead of assuming all returned records are fully valid, always inspect status. A status of success indicates full enrichment. Any other value signals a partial or failed enrichment.
For example, a status of not_found means the post was deleted or private, even if it initially appeared in search results. age_restricted means the post could not be fully processed due to account-level age gating. If your downstream analytics expect complete like_count or comment_count data, you must filter out records where likes_hidden or comments_disabled are true, or where the status is not success, as these fields will be omitted or unavailable.
The scraped_at timestamp is also crucial for data freshness. If you are performing continuous monitoring, comparing scraped_at with a previous run's pub_date can help identify stale or duplicated data. The always present fields like input_keyword, post_url, and username act as robust identifiers even if enrichment fails, allowing you to trace back problematic records.
{
"input_keyword": "example_keyword",
"post_url": "https://www.instagram.com/p/SOME_INVALID_SHORTCODE/",
"shortcode": "SOME_INVALID_SHORTCODE",
"post_id": "0",
"username": "unknown",
"author_meta": {},
"caption": null,
"hashtags": [],
"mentions": [],
"pub_date": null,
"media_type": null,
"media_count": 0,
"thumbnail_url": null,
"media_items": [],
"likes_hidden": true,
"comments_disabled": true,
"tagged_users": [],
"collaborators": [],
"sponsor_tags": [],
"status": "not_found",
"scraped_at": "2026-07-03T10:05:00+00:00"
}
This example output shows a record where the status indicates not_found, meaning most enrichment data is absent or set to default/null values. Your processing logic should immediately flag this record for exclusion or specific handling.
What about scheduled runs and state management?
To manage scheduled runs, create your schedules, enable them manually, and run the Actor at least once beforehand. For state management, monitor the status field for cookie expiration and handle programmatic rotation if using custom cookies. Ensure concurrent scheduled runs do not attempt to process the same request queue, as a queue can only be processed by one run at a time.
When working with this Actor in a continuous integration or monitoring context, scheduled runs become essential. However, Apify's scheduler has specific behaviors to consider. New schedules are created disabled by default, meaning you must explicitly enable them after creation. Furthermore, an Actor must have run at least once before it can be scheduled. This means your deployment pipeline needs an initial manual or programmatic run before scheduling automation can take over.
State management, particularly for cookies and sessionName, becomes more complex with schedules. If you are using your own cookies, they might expire. The README explicitly warns that if your cookies expire mid-run, you must re-export them from your browser and restart the Actor. For scheduled runs, this implies a need for a robust mechanism to detect cookie expiration, perhaps by monitoring run status or output for repeated authentication errors, and a process to refresh them programmatically or manually. Relying on the managed session pool avoids this specific cookie management overhead but introduces the proxy session longevity challenge described earlier.
Crucially, a request queue can only be processed by one Actor or task run at a time. If your scheduled runs are part of a larger workflow involving shared request queues, ensure that concurrent scheduled runs do not attempt to process the same queue simultaneously, as this will lead to race conditions or idle runs. Fan-out architectures need separate queues per concurrent consumer.
Checked against the Actor's input schema and Apify docs on 2026-09-09.
Written with AI assistance and checked against the Actor's published input schema, README and Apify's platform documentation before publishing. Figures quoted here come from those sources, not from a benchmark we ran.
Top comments (1)
You've done a great job highlighting the critical importance of defensive programming when dealing with input schemas like the one in the
instagram-keyword-search-scraper. The nuances you pointed out about session management and the implications of using defaults versus user-provided cookies are essential for ensuring robust scraping operations. It might be beneficial to consider implementing logging for session expirations to provide clearer feedback during long-running processes. If you're looking for an extra set of hands to help enhance this aspect of the project, I’d be glad to explore a paid collaboration. How are you planning to handle session management in scenarios where the scrape needs to run for extended periods?