Reddit Scraping Costs Drop 80 Percent by Fetching Rules Over Posts

Tracking target subreddits for community governance changes, brand enforcement, or niche interest shifts traditionally requires hitting Reddit's API under strict rate limits or maintaining custom headless browser setups. If you need structural community metadata—such as posting rules, moderator lists, and wiki pages—pulling full post feeds is an unnecessary compute expense.

The reddit-community-scraper splits subreddit discovery, community metadata extraction, and post collection into toggleable input fields. You can capture community profiles, active user ratios, rules, and wiki documentation without needing a logged-in session or an official API key.

Extracting Community Architecture Without Post Payloads

Data pipelines monitoring community health often only need high-level metadata and rules to detect policy changes or track subscriber growth. When you only need community configuration, fetching post lists inflates the output dataset size and increases run costs.

Every item written to the actor's default dataset incurs a charge based on its event type. Under the PAY_PER_EVENT model, each dataset item output costs $0.005 on the standard FREE tier (scaling down to $0.001 per event on the GOLD, PLATINUM, and DIAMOND volume tiers). Additionally, the actor charges a flat "$Actor Start" event fee of $0.005 per GB of memory allocated to the run.

If you scrape 100 subreddits to fetch their posting rules and active user metrics, you generate exactly 100 dataset items. At $0.005 per result item plus $0.005 for a 1 GB memory startup, the total cost is $0.505. If you also scrape 25 posts per subreddit, you generate 2,500 additional post records (dataType: "post"), increasing the item cost alone to $12.50. Disabling post ingestion when analyzing community structures yields an immediate 96% reduction in item generation fees.

Structuring a Metadata-Only Payload

To collect subreddits by topic and extract their core structural data, pass a search query to discoverQuery and set includePosts to false.

{
  "discoverQuery": "dataengineering",
  "discoverMode": "search",
  "discoverSort": "activity",
  "maxDiscoverResults": 15,
  "includeRules": true,
  "includeWiki": true,
  "maxWikiPages": 5,
  "includeWeeklyStats": true,
  "includeModerators": true,
  "includePosts": false
}

The resulting output produces one JSON record per subreddit with dataType: "community".

{
  "dataType": "community",
  "subreddit": "dataengineering",
  "subreddit_id": "t5_2txpp",
  "url": "https://www.reddit.com/r/dataengineering/",
  "title": "Data Engineering",
  "subscribers": 150000,
  "active_user_count": 420,
  "weekly_active_users": 12500,
  "weekly_contributions": 850,
  "rules": [
    {
      "short_name": "No low-effort self-promotion",
      "description": "Links to personal blogs or tools must include context.",
      "kind": "link",
      "priority": 0
    }
  ],
  "wiki_pages": [
    {
      "name": "index",
      "content_md": "# Welcome to the Data Engineering Wiki\n...",
      "revision_id": "abc123xyz"
    }
  ],
  "crawled_at": "2026-03-30T10:00:00.000Z",
  "source": "reddit"
}

If Reddit does not return a specific property for a given community—such as optional banner dimensions or empty wiki lists—the scraper omits those fields from the final JSON payload rather than outputting null keys.

Downstream Filtering Options for Post Scraping

When post extraction is necessary, filtering at the crawler stage avoids paying for items that your ingestion pipeline will drop later. The actor supports over 20 post-level execution filters directly in the input configuration.

Setting parameters like minScore, minComments, and postedAfter truncates low-engagement noise before dataset records are written.

{
  "subreddits": ["python", "learnpython"],
  "includePosts": true,
  "maxPosts": 50,
  "sort": "top",
  "timeFilter": "month",
  "minScore": 50,
  "minComments": 10,
  "excludeStickied": true,
  "excludeRemoved": true,
  "onlyOriginalContent": true,
  "postedAfter": "2026-01-01"
}

Server-Side Filtering Properties

Key schema properties for controlling post output volume include:

• minScore (integer): Filters out posts below a specified net score (upvotes minus downvotes).
• minComments (integer): Drops post records that fail to meet a conversation depth threshold.
• excludeStickied (boolean): Drops pinned announcement posts that skew score distribution trends.
• excludeRemoved (boolean): Filters out posts deleted by users or removed by subreddit moderators.
• postType (string): Constrains results strictly to self, link, image, video, gallery, or poll.
• titleContains / contentContains (string): Applies case-insensitive keyword matches against the post title or body text.

By setting minScore: 50 and excludeRemoved: true, a subreddit query that evaluates 50 recent submissions might only emit 8 high-value items to the dataset, capping item charges to exactly those 8 records.

Step-by-Step: Running a Community Discovery and Extraction Task

You can execute a discovery task directly through the Apify API using Python.

Step 1: Install the Apify Client

pip install apify-client

Step 2: Configure and Execute the Run

This script searches for subreddits matching the query "machinelearning", retrieves up to 5 matching communities ranked by activity, grabs their rules and weekly stats, and collects up to 10 top posts per community from the past week that have at least 20 upvotes.

import os
from apify_client import ApifyClient

client = ApifyClient(os.getenv("APIFY_TOKEN"))

run_input = {
    "discoverQuery": "machinelearning",
    "discoverMode": "search",
    "discoverSort": "activity",
    "maxDiscoverResults": 5,
    "includeRules": True,
    "includeWeeklyStats": True,
    "includePosts": True,
    "maxPosts": 10,
    "sort": "top",
    "timeFilter": "week",
    "minScore": 20,
    "excludeStickied": True,
}

run = client.actor("crawlerbros/reddit-community-scraper").call(
    run_input=run_input
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("dataType") == "community":
        print(
            f"Community: {item['subreddit']} | Subs: {item['subscribers']} | Weekly Actives: {item.get('weekly_active_users')}"
        )
    elif item.get("dataType") == "post":
        print(
            f"Post: [{item['subreddit']}] {item['title']} (Score: {item['score']})"
        )

Data Schema Differences: Community vs. Post

The dataset outputs two distinct JSON structures depending on the item type:

Field Category	dataType: "community" Output	dataType: "post" Output
Identifiers	subreddit, subreddit_id, url	post_id, post_name, permalink
Engagement	subscribers, active_user_count, weekly_active_users	score, ups, downs, upvote_ratio, num_comments
Governance	rules[], moderators[], restrict_posting	removed_by_category, is_stickied, is_locked
Media/Content	banner_img, community_icon, description	url_overridden_by_dest, gallery_images[], poll_data

Downstream processing pipelines must route incoming records based on the dataType key to avoid schema parsing errors in strictly typed stores like BigQuery or Postgres.

This approach does NOT bypass private or banned subreddits; any community restricted behind private permission settings will fail to return complete rule or post payloads.

---

Reddit Community Scraper is what these steps drive. The README covers the inputs this article skipped, including the ones that change how much a run costs.