`maxItems` Can Truncate Results Despite Individual Limits

The reddit-keywords Actor, available on Apify, is a powerful tool for scraping Reddit posts, comments, communities, and users based on search terms. While its README outlines many capabilities, the true depth of its utility, and its potential pitfalls, lies in understanding the nuances of its input schema, its output structure, and how these interact with the underlying Apify platform. As senior data engineers, our goal isn't just to make a tool run, but to make it run reliably, efficiently, and predictably, especially when dealing with the edge cases the documentation implies but does not explicitly detail. This means proactively addressing the failure modes that can lead to partial results, unexpected costs, or run timeouts.

How do you avoid empty output records from reddit-keywords?

Avoid empty or near-empty records by using the Actor's extensive filtering parameters in the input schema to refine your search before the run begins. This minimizes processing of irrelevant data and prevents runs from returning nothing if core keywords fail to match content.

The reddit-keywords Actor offers a rich set of pre-run filtering options. For instance, if you are only interested in posts with a significant number of interactions, setting minComments or minScore will prevent the Actor from collecting and charging for low-engagement content. Similarly, onlyWithFlair: true will drop posts lacking flairs, ensuring that if flair data is critical for your analysis, you will not receive records that are unusable for your purposes. The crucial point here is that "empty fields are omitted from every record" and "the output never contains null or blank values" (from the README), meaning that if your filtering is too aggressive, you might get no records for a search term, rather than records with null values. This is why knowing the output schema's partiality is important.

Consider a scenario where you are looking for posts about "machine learning" but only those that are original content and have been highly upvoted.

{
  "keywords": ["machine learning"],
  "searchPosts": true,
  "onlyOriginalContent": true,
  "minScore": 100,
  "resultLimit": 500
}

If this run returns zero results, it is not a failure of the scraper itself, but an indication that no content matches all of your criteria. Conversely, if you collected all machine learning posts and then filtered them in your downstream ETL for original content with high scores, you would have paid for the collection of data you ultimately discarded.

How does maxItems interact with individual result limits?

maxItems acts as a hard cap on the total number of records returned across the entire run (all keywords and all result types combined). Individual limits like resultLimit, maxComments, maxCommunities, and maxUsers control results per keyword. The run stops as soon as the maxItems budget is spent, which means it can truncate results for some keywords even if their individual resultLimit has not been reached.

If you specify maxItems: 10000 and have 10 keywords, each with resultLimit: 1000, the Actor will stop after collecting 10,000 items in total, regardless of how many items each keyword contributed. If one keyword produces many more results early in the run, it could consume a disproportionate share of the maxItems budget, leaving other keywords with fewer or even no results.

For example, if you are searching for "python" and "rust" with resultLimit: 500 for each, and maxItems: 600, but "python" generates 550 posts before "rust" generates any, your "rust" search will only yield 50 posts before the maxItems cap is hit. This scenario underlines the importance of setting maxItems generously enough to accommodate the expected output from all your specified keywords and search types, or dynamically adjusting it based on the number of keywords.

How to detect session limits and proxy failures defensively?

The Actor uses proxies, with datacenter sessions persisting ~26 hours and residential ~30 minutes. While the Actor handles rotation and retries, persistent proxy issues can reduce expected results. No explicit "proxy_failure" field exists, so observe output volume and completeness for detection.

There is no explicit "proxy_failure" field in the output schema. Instead, you need to observe the volume and completeness of your output. If you are consistently getting fewer results than expected for certain keywords, or if the crawled_at timestamp for a keyword suddenly stops updating while the run is still active, it might indicate underlying proxy issues. Defensive coding involves analyzing the output dataset for patterns that suggest network or session problems. For instance, if you expect 100 results per keyword (resultLimit: 100), but consistently receive significantly less (e.g., 50-100 items per keyword) without other apparent filtering reasons, it is a strong indicator.

You could implement a post-run check comparing the expected resultLimit against the actual count of items per search_term in the dataset.

import apify_client

client = apify_client.ApifyClient("YOUR_API_TOKEN")

run_id = "YOUR_ACTOR_RUN_ID"
run = client.actor(actor_id="crawlerbros/reddit-keywords").call(run_id=run_id)

# Get the dataset associated with the run
dataset_id = run["defaultDatasetId"]
dataset_client = client.dataset(dataset_id)

# Iterate over all items in the dataset
total_items = dataset_client.get_info()["itemCount"]
items_by_keyword = {}

for item in dataset_client.iterate_items():
    search_term = item.get("search_term", "unknown_keyword")
    items_by_keyword.setdefault(search_term, 0)
    items_by_keyword[search_term] += 1

print(f"Total items collected: {total_items}")
print("Items per keyword:")
for keyword, count in items_by_keyword.items():
    print(f"- '{keyword}': {count}")

# Example defensive check: if any keyword collected less than 50% of its expected limit
expected_limit_per_keyword = 100 # Assuming resultLimit was 100 for all keywords
for keyword, count in items_by_keyword.items():
    if count < expected_limit_per_keyword * 0.5:
        print(f"WARNING: Keyword '{keyword}' collected significantly fewer items ({count}) than expected ({expected_limit_per_keyword}).")

This post-processing allows you to flag potentially under-scraped keywords, which could then trigger a rerun with adjusted proxy settings (e.g., forcing residential proxies if datacenter ones seem to be struggling, or rotating proxy groups more frequently if using a custom setup).

What are the implications of prefill versus default values?

Prefill values appear in the Console UI but are not applied to API calls or existing Actor tasks; only default values are programmatically applied when an input field is omitted via API. When using the reddit-keywords Actor's API, you must always explicitly pass an input dictionary with all required fields and any optional fields where you don't want the Actor's published default value.

For example, reddit-keywords has a keywords field that is explicitly [required]. If you omit keywords in an API call, it will fail, even if a prefill value might appear in the Console UI. However, for a field like searchPosts which has default=true, omitting it from your API input will correctly result in posts being searched.

Always construct your API run_input JSON explicitly, rather than assuming UI defaults will carry over.

{
  "keywords": ["developer jobs", "data engineering"],
  "searchPosts": true,
  "searchComments": false,
  "resultLimit": 200,
  "timeFilter": "month",
  "sort": "new"
}

If you omitted searchPosts here, it would default to true. If you omitted resultLimit, it would default to 100. But if you omitted keywords, the run would fail.

How to manage run duration and synchronous run limits?

Apify imposes a 300-second (5-minute) hard cap for synchronous Actor runs, returning an HTTP 408 error past that. For reddit-keywords, especially with many keywords or high resultLimit values, runs often exceed this. Longer runs require using the asynchronous POST method and polling run status or configuring webhooks.

Attempting to force a long run synchronously will result in an unresponsive client and a potential loss of immediate feedback. When you anticipate a run duration beyond 5 minutes, you must change your interaction pattern.

Consider this Python example for asynchronous execution:

import apify_client
import time

client = apify_client.ApifyClient("YOUR_API_TOKEN")

actor_id = "crawlerbros/reddit-keywords"

# Input for the Reddit Keywords Actor
run_input = {
  "keywords": ["AI ethics", "LLM bias"],
  "searchPosts": True,
  "resultLimit": 500,
  "maxItems": 5000,
  "timeFilter": "year"
}

print("Starting asynchronous Actor run...")
run = client.actor(actor_id=actor_id).call(run_input=run_input, wait_for_finish=0) # wait_for_finish=0 makes it async
run_id = run["id"]
print(f"Actor run started with ID: {run_id}")

# Poll for run completion
while True:
    run_status = client.run(run_id=run_id).get()
    print(f"Current run status: {run_status['status']}")
    if run_status["status"] in ["SUCCEEDED", "FAILED", "ABORTED", "TIMED_OUT"]:
        print(f"Run finished with status: {run_status['status']}")
        break
    time.sleep(30) # Wait 30 seconds before polling again

if run_status["status"] == "SUCCEEDED":
    print("Run successful. Fetching results from default dataset.")
    dataset_id = run_status["defaultDatasetId"]
    dataset_client = client.dataset(dataset_id)
    print(f"Total items in dataset: {dataset_client.get_info()['itemCount']}")
    # Process results further
else:
    print(f"Run failed or was interrupted: {run_status.get('statusMessage', 'No message provided')}")

This wait_for_finish=0 parameter is key to decoupling your client from the Actor's execution time, allowing for robust handling of potentially long-running scraping jobs.

What happens when the maxTotalChargeUsd cap is reached?

The maxTotalChargeUsd parameter terminates the Actor run when a predefined dollar amount is reached. The Actor continues consuming resources briefly during graceful shutdown, potentially causing the final charge to slightly exceed the cap. This small overshoot is a consideration for extremely tight budgets.

The maxTotalChargeUsd parameter is exposed to the Actor code as ACTOR_MAX_TOTAL_CHARGE_USD, allowing for internal logic that could, for example, prioritize critical searches or perform early shutdowns if the budget is running low. For practical purposes, as a data engineer, you would set this parameter at the run invocation level.

Consider a scenario where you want to ensure a run never exceeds a specific budget:

import apify_client

client = apify_client.ApifyClient("YOUR_API_TOKEN")

actor_id = "crawlerbros/reddit-keywords"
run_input = {
  "keywords": ["web scraping", "data pipeline"],
  "searchPosts": True,
  "resultLimit": 1000,
  "maxItems": 200000 # Potentially high, but capped by maxTotalChargeUsd
}

# Running the Actor with a maximum charge limit
run = client.actor(actor_id=actor_id).call(
    run_input=run_input,
    max_total_charge_usd=0.005 # Placeholder for your budget
)

print(f"Actor run initiated. ID: {run['id']}. Max charge set.")

This ensures that even if maxItems is set very high, the financial ceiling acts as a final safeguard. Monitoring the ACTOR_MAX_TOTAL_CHARGE_USD environment variable within a custom Actor, or checking the final run cost after completion via the Apify API, is key to understanding its effect.

What limitations does reddit-keywords have?

The reddit-keywords Actor does not support searching for posts or comments based on negative karma/score values. It also lacks an explicit way to target specific Reddit users beyond matching keywords in usernames, and cannot search based on the number of crossposts. Furthermore, maxUsers has a Reddit-imposed hard ceiling of 100 items per keyword, and timeFilter only applies to the "Top" sort.

A subtle but important limitation is the behavior of the maxUsers parameter: "Reddit serves a single results page for account search (no further pagination), so 100 is the reliable ceiling." This is a hard technical limit imposed by Reddit's API, not the Actor, meaning that if you expect more than 100 user results for a single keyword, you will need a different approach or accept the truncation. The timeFilter parameter specifically notes it "Applies to the Top sort (as in Reddit's search UI); other sorts ignore it", which can be a silent data quality issue if a time-bound search is combined with a sort order like new or relevance. Finally, while the Actor provides minAwards, it does not offer a corresponding maxAwards filter. This prevents filtering for posts with a low number of awards (e.g., to find non-viral content), meaning all posts with at least minAwards will be returned.

Understanding the cost of reddit-keywords runs

The reddit-keywords Actor uses a PAY_PER_EVENT pricing model, meaning your cost scales directly with the number of specific events emitted during the run, not its duration or the amount of compute used directly. There are no separate platform-usage charges or subscription-plan rates on top of the published event prices.

The charged events are:

• "result" (apify-default-dataset-item): $0.005 per event. This is charged for every single record saved to the default dataset. This event has volume-tier prices: FREE $0.005, BRONZE $0.00367, SILVER $0.00233, GOLD $0.001, PLATINUM $0.001, DIAMOND $0.001. The number of items collected directly impacts this cost.
• "Actor Start" (apify-actor-start): $0.05 per GB of memory allocated to the run, charged once per run. The minimum is one event, so even if an Actor uses less than 1GB, you are charged $0.05.

The primary driver of cost for most reddit-keywords runs will be the "result" events. Each keyword you add, each type of result you enable (searchPosts, searchComments, searchCommunities, searchUsers), and the limits you set (resultLimit, maxComments, maxCommunities, maxUsers, maxItems) all directly multiply the number of records collected, and thus the number of "result" events. A run with 10 keywords and resultLimit: 100 (searching only posts) could theoretically yield 1,000 "result" events, costing $0.005 per event at the FREE tier, plus the "Actor Start" charge.

To manage costs effectively, it is crucial to:

1. Be precise with keywords: Avoid overly broad keywords that return massive, irrelevant datasets.
2. Utilize pre-run filters: As discussed earlier, minScore, minComments, onlyWithFlair, and others prevent collecting data you do not need, directly reducing "result" events.
3. Set maxItems judiciously: This is your hard budget cap for results. If you know you only need a maximum of 10,000 items in total, set maxItems: 10000 to prevent over-collection.
4. Balance individual limits: If you have many keywords, consider if resultLimit: 1000 for each is truly necessary, or if a lower limit (e.g., 100-200) would suffice given your maxItems budget.

# Example demonstrating cost impact of input parameters
# This is NOT a live run, but illustrates parameter effects
run_input_high_cost_potential = {
  "keywords": ["programming", "coding", "software engineering", "data science", "machine learning", "web development", "mobile development", "cloud computing", "devops", "cybersecurity"], # 10 keywords
  "searchPosts": True,
  "searchComments": True, # Doubling potential results
  "searchCommunities": True,
  "searchUsers": True,
  "resultLimit": 1000,    # Max posts per keyword
  "maxComments": 1000,    # Max comments per keyword
  "maxCommunities": 100,  # Max communities per keyword
  "maxUsers": 100,        # Max users per keyword
  "maxItems": 1000000     # Highest possible cap, allowing for maximum collection
}

run_input_lower_cost_example = {
  "keywords": ["python automation"], # Single keyword
  "searchPosts": True,
  "searchComments": False, # Only posts
  "resultLimit": 50,      # Low limit
  "maxItems": 100        # Strict total cap
}

# The first example, if it hit all its limits, could theoretically generate
# (10 * 1000 [posts]) + (10 * 1000 [comments]) + (10 * 100 [communities]) + (10 * 100 [users])
# = 10000 + 10000 + 1000 + 1000 = 22,000 records, potentially costing $0.005 per record (FREE tier) + $0.05 for Actor Start.
# The second example aims for 50 records (if searchPosts is True), costing $0.005 per record + $0.05.
# This illustrates the multiplier effect of input parameters on the "result" charge event.

This demonstrates how thoughtful input parameter selection is essential for controlling your reddit-keywords costs.

Checked against the Actor's input schema and Apify docs on 2026-09-14.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]