Scrape Reddit Comments with Python

Integrating third-party scrapers into production data pipelines requires moving past the simple workflows shown in basic documentation. When retrieving deep comment threads from Reddit, you deal with deeply nested trees, user-deleted accounts, and platform-level rate limits.

We can build a resilient integration using the reddit-comment-scraper Actor on Apify. By analyzing the input schema, output shapes, and platform behaviors, we can design defensive code to handle raw API limitations, partial payloads, and platform-enforced boundaries.

Checked against the Actor's input schema and Apify docs on 2026-09-09.

What are the silent failure modes of Reddit comment scraping?

Reddit comment threads fail silently when client-side structures change, nested depths exceed parser limits, or target comments disappear due to user deletion or moderator action. When a comment is deleted or removed, the scraper returns modified schema structures that can break downstream processing if not handled defensively.

To prevent your data pipeline from crashing when encountering unexpected payloads, you must anticipate several specific edge cases:

• Deleted and Removed Sentinel States: If a user deletes their comment or a moderator removes it, the scraper still returns the record to preserve thread structure. However, the author field becomes the literal string "[deleted]", the author_id field is entirely omitted from the JSON, and the body contains "[deleted]" or "[removed]".
• Missing Community Flair Attributes: Community-specific fields like author_flair, author_flair_css_class, or author_flair_template_id are omitted from the output object when a user does not have flair set. Relying on strict key access instead of safe dictionary retrieval will trigger runtime exceptions.
• Schema Polymorphism: When the includePost boolean input is enabled, the resulting dataset is no longer homogeneous. The first record emitted for each thread represents the parent post, while subsequent records represent comments. They share a dataType discriminator field ("post" vs "comment") but have radically different schemas.

Here is how you can defensively parse these records in Python:

def process_scraper_record(record: dict) -> None:
    # Handle schema polymorphism explicitly
    data_type = record.get("dataType")

    if data_type == "post":
        handle_parent_post(record)
        return

    if data_type != "comment":
        # Ignore or log unexpected record types
        return

    # Defensive handling of deleted or removed comments
    author = record.get("author")
    author_id = record.get("author_id")  # Safe get; omitted when deleted
    body = record.get("body", "")

    is_deleted = (
        author == "[deleted]" 
        or author_id is None 
        or body in ["[deleted]", "[removed]"]
    )

    # Safely handle community flairs which are omitted when absent
    flair_text = record.get("author_flair")
    flair_css = record.get("author_flair_css_class")

    # Access metadata with safe fallbacks
    score = record.get("score", 0)
    depth = record.get("depth", 0)

    print(f"Comment {record.get('comment_id')}: Deleted={is_deleted}, Score={score}, Depth={depth}")

def handle_parent_post(post_record: dict) -> None:
    # Process parent post record metadata safely
    post_id = post_record.get("post_id")
    title = post_record.get("title", "")
    print(f"Parent Post {post_id}: {title}")

How to handle the 300 second synchronous timeout?

To handle the 300 second synchronous timeout, you must decouple the execution trigger from the data retrieval step by starting the Actor run asynchronously. Instead of holding an open connection that will be forcefully severed with an HTTP 408 error after 5 minutes, you POST to the runs endpoint to receive a run ID immediately and poll the status at regular intervals.

This asynchronous execution pattern is mandatory for reliable production pipelines. Running a synchronous request against any thread with thousands of comments will inevitably hit the timeout.

Here is a complete, runnable Python script that starts the run asynchronously, polls for its status, and downloads the final dataset:

import time
import requests

API_TOKEN = "YOUR_APIFY_API_TOKEN"
ACTOR_ID = "crawlerbros/reddit-comment-scraper"

# Input configuration mapping directly to the Actor input schema
run_input = {
    "postUrls": ["https://www.reddit.com/r/programming/comments/1abc123/some_title/"],
    "maxComments": 100,
    "commentSort": "confidence",
    "includePost": True
}

# Start the Actor run asynchronously
run_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs"
headers = {"Authorization": f"Bearer {API_TOKEN}"}

# We POST to start the run and get the run metadata instantly
response = requests.post(run_url, json=run_input, headers=headers)
response.raise_for_status()
run_data = response.json()["data"]
run_id = run_data["id"]
default_dataset_id = run_data["defaultDatasetId"]

print(f"Run started successfully. Run ID: {run_id}")

# Poll the run status safely without holding an open connection
status_url = f"https://api.apify.com/v2/actor-runs/{run_id}"
while True:
    status_response = requests.get(status_url, headers=headers)
    status_response.raise_for_status()
    run_status = status_response.json()["data"]["status"]

    print(f"Current run status: {run_status}")

    if run_status == "SUCCEEDED":
        break
    elif run_status in ["FAILED", "ABORTED", "TIMED-OUT"]:
        raise RuntimeError(f"Actor run ended with terminal status: {run_status}")

    # Wait before polling again to avoid hitting rate limits
    time.sleep(15)

# Retrieve the items from the default dataset
dataset_url = f"https://api.apify.com/v2/datasets/{default_dataset_id}/items"
dataset_response = requests.get(dataset_url, headers=headers)
dataset_response.raise_for_status()
items = dataset_response.json()

print(f"Successfully retrieved {len(items)} items from the dataset.")

Why do programmatic API calls ignore console prefill values?

Programmatic API calls ignore console prefill values because those values are purely cosmetic templates defined in the console UI, whereas the API itself only honors hardcoded schema defaults. When initiating an Actor run via direct HTTP requests, any field without a hardcoded default will be treated as empty or undefined unless you explicitly pass it in your JSON body.

Because the scraper is designed to skip malformed or empty items in postUrls rather than throwing an error, initiating a run via the API with an empty payload will succeed but result in zero scraped records. You must always pass an explicit dictionary in your code.

Here is a comprehensive JSON payload that defines all the key filtering parameters explicitly to ensure your programmatic runs produce consistent outputs:

{
  "postUrls": [
    "https://www.reddit.com/r/programming/comments/1abc123/some_title/"
  ],
  "maxComments": 100,
  "commentSort": "confidence",
  "includePost": true,
  "postedAfter": "2026-01-01",
  "minDepth": 0,
  "maxDepth": 5,
  "minCommentScore": 10,
  "excludeDeletedRemoved": true,
  "excludeAuthors": ["AutoModerator"]
}

How to trigger the Reddit Comment Scraper with curl?

You can trigger the scraper directly using a shell command with the standard command-line tool curl. This method is highly effective for testing your configuration payload without writing any wrapper code.

By sending a POST request directly to the Apify API run endpoint, you bypass the UI completely. Make sure to specify your API token as a query parameter and set the content type to JSON.

Here is a ready-to-run terminal command that triggers a run using the input schema options:

curl --request POST \
  --url "https://api.apify.com/v2/acts/crawlerbros/reddit-comment-scraper/runs?token=YOUR_APIFY_API_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{
    "postUrls": ["https://www.reddit.com/r/programming/comments/1abc123/some_title/"],
    "maxComments": 100,
    "commentSort": "confidence",
    "includePost": true,
    "postedAfter": "2026-01-01",
    "minDepth": 0,
    "maxDepth": 5,
    "minCommentScore": 10,
    "excludeDeletedRemoved": true,
    "excludeAuthors": ["AutoModerator"]
  }'

How to parse the polymorphic comment and post dataset?

When you enable the includePost parameter, the dataset returned by the Actor contains two different object types. This requires a robust parsing pattern in your processing application.

The parent post object contains fields like title, subreddit_subscribers, and upvote_ratio, whereas comment objects contain fields like body, depth, and parent_id. They share the discriminator field dataType.

The following JSON block illustrates the exact shape of a comment record in the output dataset:

{
  "dataType": "comment",
  "comment_id": "k1a2b3c",
  "comment_name": "t1_k1a2b3c",
  "post_id": "1abc123",
  "post_url": "https://www.reddit.com/r/programming/comments/1abc123/some_title/",
  "post_title": "Some Title",
  "link_id": "t3_1abc123",
  "permalink": "/r/programming/comments/1abc123/some_title/k1a2b3c/",
  "author": "example_user",
  "author_id": "t2_xyz987",
  "body": "This is a comment body in markdown.",
  "body_html": "<p>This is a comment body in markdown.</p>",
  "score": 42,
  "ups": 42,
  "downs": 0,
  "score_hidden": false,
  "controversiality": 0,
  "subreddit": "programming",
  "subreddit_prefixed": "r/programming",
  "subreddit_id": "t5_2fwo",
  "subreddit_type": "public",
  "created_utc": 1788888888,
  "created_at": "2026-09-09T12:00:00.000Z",
  "depth": 0,
  "parent_id": "t3_1abc123",
  "parent_kind": "post",
  "is_op": false,
  "is_stickied": false,
  "is_locked": false,
  "archived": false,
  "collapsed": false,
  "total_awards_received": 0,
  "crawled_at": "2026-09-09T12:05:00.000Z",
  "source": "json"
}

What are the system limits and crawl caveats of this Actor?

When designing an ingestion engine around this Actor, there are structural constraints in both the Reddit platform design and the Apify architecture that you must account for.

First, the platform limits storage traffic to safeguard database integrity. Apify imposes a rate limit of 60 requests per second per storage object, and 400 requests per second for dataset item pushes or request queue CRUD operations. If you attempt to execute hundreds of concurrent instances of this Actor writing to the same dataset, your runs will encounter write throttle errors.

Second, if you attempt to scale your scraping operations by running multiple tasks or Actors in parallel, you cannot share a single request queue. A request queue on Apify can only be processed by one Actor or task run at a time. While multiple processes can add items to a queue, fanning out across a single shared queue to parallelize the crawling of multiple subreddits simultaneously will fail.

Third, look out for storage retention behaviors on the Apify platform. If you run your scrapers on the Free plan, unnamed datasets and run histories expire rapidly. Only the 10 most recent runs are retained, and they are deleted entirely after 4 months. To prevent the loss of historical data, you must explicitly name your datasets upon creation or configure an external database write-back pipeline. Named storages are completely exempt from this deletion policy.

Finally, we must look at session limits. The reddit-comment-scraper relies on proxies to fetch comment threads without requiring authentication. Datacenter proxy sessions persist for up to 26 hours, but residential proxy sessions expire and rotate roughly every 30 minutes. If you are scraping exceptionally deep comment threads that run for hours, session expiration during the run can force a connection reset or trigger Reddit's rate-limit walls.

How much does running this Actor actually cost?

To estimate and control your budget, you must calculate the costs associated with both Apify Compute Units (CUs) and proxy usage. Apify compute consumption is calculated using the formula:

CU = (Memory in MB / 1024) * Duration in Hours

The billing rate for these CUs is determined by your subscription tier:

• Free: $0.20 per CU
• Starter: $19/month, CUs billed at $0.20 per CU
• Scale: $199/month, CUs billed at $0.16 per CU
• Business: $999/month, CUs billed at $0.13 per CU

The second variable is proxy consumption. The Actor utilizes proxies to fetch data. If you configure the run to use residential proxies, the bandwidth consumed is billed per gigabyte. The rates for residential proxies scale down as your subscription tier increases:

• Free and Starter Tiers: $8 per GB
• Scale Tier: $7.50 per GB
• Business Tier: $7 per GB

Because scraping heavy media posts or recursive comment segments increases both the execution time (consuming more CUs) and the network payload size (consuming more residential proxy bandwidth), your costs scale with the depth of the threads you crawl.

To prevent run-away billing scenarios on massive threads, you should defensively set the maxTotalChargeUsd query parameter on your run requests. This parameter is exposed to the Actor code as ACTOR_MAX_TOTAL_CHARGE_USD. When the calculated cost of the run hits this cap, the platform initiates a termination sequence.

Note that this is not an instant hard-kill of the process container. The Actor will continue to consume resources briefly during its cleanup and graceful shutdown window, so your actual billing may slightly exceed the set threshold. Always configure your maxComments parameter defensively to keep run sizes predictable.

How to schedule recurring comment scraper runs?

To monitor discussions on specific Reddit threads or subreddits over time, you can schedule the Actor to run at regular intervals using Apify's native scheduler.

The scheduler uses a 6-field cron format, and you can configure schedules to execute as frequently as every 10 seconds. However, before a schedule can be active, the Actor must have been run at least once manually. Additionally, when you create a new schedule, it is disabled by default to prevent accidental charges, so you must explicitly enable it in your dashboard.

Since there is no native AWS S3 or Slack integration built directly into the Actor, you should configure a webhook to POST to an external integration platform like Make, Zapier, or n8n when the run completes. If you use n8n, you can employ the built-in Apify Trigger node, which fires automatically upon run completion and eliminates the need for manual API polling.

Using webhooks, you can automate your ETL pipeline, routing data directly into your analytical databases or alert systems when specific trigger keywords are detected.

How to handle the limits of the target comment focus mode?

To handle the limits of the target comment focus mode, you must understand that specifying a direct comment permalink behaves differently than scraping an entire post thread. When focusOnTargetComment is enabled, the Actor changes its behavior: it avoids pulling the entire thread and instead isolates that specific comment plus its direct ancestor chain.

The depth of this ancestor chain is controlled by the commentContext integer parameter, which defaults to 3 and is restricted to a range of 0 to 8. This mirrors the behavior of Reddit's own comment permalink page, which displays the target comment in context with its immediate parents.

If you attempt to use focusOnTargetComment on a URL that is a top-level post link rather than a comment permalink, the parameter is ignored, and the Actor defaults to scanning the entire post.

Here is a Python example illustrating how to configure the Actor to focus on a single deep comment, limiting the parent context to exactly 2 levels above:

import requests

API_TOKEN = "YOUR_APIFY_API_TOKEN"
ACTOR_ID = "crawlerbros/reddit-comment-scraper"

# Input configuration targeting a specific comment permalink
focus_input = {
    "postUrls": ["https://www.reddit.com/r/programming/comments/1abc123/some_title/comment/xyz987/"],
    "focusOnTargetComment": True,
    "commentContext": 2,
    "maxComments": 10
}

run_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs?token={API_TOKEN}"
response = requests.post(run_url, json=focus_input)
response.raise_for_status()

print("Focus-mode run started successfully.")

How to bypass common filtering issues when retrieving raw bodies?

When extracting comment bodies for natural language processing or sentiment analysis, developers often encounter a large volume of low-quality, automated, or empty responses that pollute their datasets. The scraper provides built-in, low-level filtering options to address this before saving or exporting the data.

You can combine minCommentLength and maxCommentLength to filter out boilerplate noise, emojis, or massive text blocks. For instance, setting minCommentLength to 20 characters ensures you skip single-word replies like "Thanks" or "This", while setting a maximum threshold prevents loading overly verbose essays that skew length distributions.

Additionally, using excludeAuthors to filter out common bots (such as AutoModerator) and enabling excludeDeletedRemoved keeps your records clean. This reduces post-processing overhead in Python since your downstream systems will only ingest records that contain real human text.

Here is an example Python configuration demonstrating how to execute the Actor using these combined quality-control filters:

import requests

API_TOKEN = "YOUR_APIFY_API_TOKEN"
ACTOR_ID = "crawlerbros/reddit-comment-scraper"

clean_run_input = {
    "postUrls": ["https://www.reddit.com/r/programming/comments/1abc123/some_title/"],
    "maxComments": 200,
    "excludeDeletedRemoved": True,
    "excludeStickied": True,
    "excludeCollapsed": True,
    "minCommentLength": 30,
    "maxCommentLength": 500,
    "excludeAuthors": ["AutoModerator", "SnapshotImageBot"]
}

run_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs?token={API_TOKEN}"
response = requests.post(run_url, json=clean_run_input)
response.raise_for_status()

print("Data cleaning run successfully started.")

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]