DEV Community

Charbel Daccache
Charbel Daccache

Posted on

How to Scrape Reddit Posts and Comments Without the Reddit API (with Python)

#webscraping #api #python #reddit

To scrape Reddit without the official API, read Reddit's public pages the way a logged-out visitor does: subreddit feeds, search results, user profiles and post threads. This guide shows how to do that with one API call and get posts, full comment threads, communities and users as clean JSON, with no API key, no OAuth app and no developer account.

This is part 4 of "How my fun projects turned into APIs". Each post takes one side project, explains how it works, and ends with something you can call from your own code. Part 1 covered finding TikTok creator emails, part 2 video transcripts, and part 3 recipes from cooking videos.

Table of contents

  1. Why scraping Reddit got harder
  2. How the Reddit scraper works
  3. How to scrape Reddit in 4 steps
  4. How to scrape Reddit with Python
  5. How to scrape Reddit with JavaScript
  6. What data you get from Reddit
  7. How to find SaaS ideas on Reddit with AI tags
  8. How much it costs
  9. FAQ

Why scraping Reddit got harder

Reddit used to be the easiest site on the internet to get data from: add .json to a URL, or register a free API app. That changed:

  • Reddit's Data API now needs an approved app, has strict rate limits, and charges for commercial use.
  • Self-service API keys were closed in 2025.
  • Most old Reddit scraping tutorials and libraries rely on access that no longer works.

The project that started it

After three side projects, I had a different problem: I didn't know what to build next. The best ideas usually come from people describing a real problem in their own words, and the place they do that most is Reddit. Posts like "is there a tool that...", "alternative to X" and "I'd pay for something that...".

My plan was simple: search Reddit for those phrases across a few niches, read the threads, and make a list of problems people would pay to solve. Doing that by hand took an evening per niche, and the official API route meant applying for access and waiting.

So I wrote a scraper that reads the same public pages any logged-out visitor sees. First subreddit feeds, then keyword search, then full comment threads, because the best details are always in the replies. Then I added keyword filters to cut the noise, and finally AI tags that mark each post with its pain point, whether the person asks for a recommendation, and whether they say they'd pay.

The list of ideas was useful. The scraper was more useful, so I made it a public API.

How the Reddit scraper works

The tool is an Apify Actor called Reddit Scraper. It reads Reddit as a logged-out visitor, so it needs no login, no Reddit API key and no OAuth app.

It accepts four kinds of input:

Input Example What you get
Subreddit r/startups, reddit.com/r/startups/top/?t=month Posts in Hot, New, Top, Rising or Controversial order
Keyword search best crm for startups Matching posts from all of Reddit or one subreddit
User profile u/spez The user's posts
Post URL reddit.com/r/x/comments/abc123/... One post with its full comment thread

Keyword searches can also return comments, communities and users. Every post can bring its comments, with reply depth and parent ID, so you can rebuild the whole thread.

How to scrape Reddit in 4 steps

  1. Create a free Apify account. Runs on the free plan work within the monthly credit.
  2. Open Reddit Scraper and click Try for free.
  3. Paste this input (switch the input editor to JSON): 
{
  "searchTerms": ["best crm for small business"],
  "searchSort": "top",
  "time": "year",
  "searchFullPosts": true,
  "maxPostsPerSource": 50,
  "maxCommentsPerPost": 30
}
Enter fullscreen mode Exit fullscreen mode
  1. Click Start. Export the results as JSON, CSV or Excel, or read them through the API.

To get the top posts of a subreddit instead:

{
  "startUrls": ["https://www.reddit.com/r/startups/top/?t=month"],
  "maxPostsPerSource": 100,
  "skipStickied": true
}
Enter fullscreen mode Exit fullscreen mode

How to scrape Reddit with Python

Install the client with pip install apify-client, then:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("scrape-lads/reddit-scraper").call(run_input={
    "searchTerms": ["notion", "obsidian"],
    "searchSort": "new",
    "postedAfter": "7 days",
    "maxPostsPerSource": 100,
    "maxCommentsPerPost": 20,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["dataType"] == "post":
        print(f"[{row['score']}] r/{row['subreddit']}: {row['title']}")
    elif row["dataType"] == "comment":
        print("   ", row["body"][:100])
Enter fullscreen mode Exit fullscreen mode

Your API token is in Apify Console under Settings → API & Integrations.

How to scrape Reddit with JavaScript

Install the client with npm install apify-client, then:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('scrape-lads/reddit-scraper').call({
  startUrls: ['https://www.reddit.com/r/SaaS/top/?t=month'],
  maxPostsPerSource: 50,
  maxCommentsPerPost: 10,
  commentSort: 'top',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const posts = items.filter((r) => r.dataType === 'post');
posts.forEach((p) => console.log(p.score, p.numComments, p.title));
Enter fullscreen mode Exit fullscreen mode

What data you get from Reddit

Each row has a dataType: post, comment, community or user. A real post:

{
  "dataType": "post",
  "id": "1x4jwxm",
  "url": "https://www.reddit.com/r/tifu/comments/1x4jwxm/tifu_by_asking_my_coworker_about_her_kindle_that/",
  "title": "TIFU by asking my coworker about her Kindle that died",
  "author": "v-fee",
  "subreddit": "tifu",
  "createdAt": "2026-10-13T01:35:57.185Z",
  "score": 7296,
  "upvoteRatio": 0.98,
  "numComments": 273,
  "postType": "text"
}
Enter fullscreen mode Exit fullscreen mode

A real comment:

{
  "dataType": "comment",
  "postId": "1qqxvlw",
  "parentId": "t3_1qqxvlw",
  "depth": 0,
  "author": "arlaneenalra",
  "body": "It's called a \"perishable skill\" you have to use it or you lose it.",
  "score": 1479,
  "createdAt": "2026-01-30T06:36:11.512Z"
}
Enter fullscreen mode Exit fullscreen mode
Group Fields
Posts title, body, author, subreddit, score, upvoteRatio, numComments, flair, postType, link, images, videoUrl, createdAt
Comments body, author, score, postId, parentId, depth, createdAt
Communities name, description, weeklyVisitors, weeklyContributions
Users username, description, karma

How to find SaaS ideas on Reddit with AI tags

This is the part I built the scraper for. Search for phrases people use when they need a tool, search comments too, and turn on AI tags:

{
  "searchTerms": ["looking for a tool", "is there an app that", "alternative to quickbooks"],
  "searchComments": true,
  "aiTagging": true,
  "maxPostsPerSource": 100
}
Enter fullscreen mode Exit fullscreen mode

Every post and comment then gets tags like these (a real example):

{
  "aiTags": {
    "painPoint": "Auto reconciliation with QuickBooks was horrendous, manual fixes",
    "asksForRecommendation": false,
    "willingToPay": false,
    "featureRequest": "AI agent resolving transaction issues",
    "productsMentioned": ["Alternative Payments", "QuickBooks Premier", "FlexPoint"],
    "sentiment": "mixed"
  }
}
Enter fullscreen mode Exit fullscreen mode

Filter the dataset on aiTags.willingToPay or aiTags.asksForRecommendation, and you have a list of people asking for what you might build. The tags are for sorting large lists, so spot-check the rows you act on.

How much it costs

  • $1 per 1,000 posts
  • $0.50 per 1,000 comments
  • $1 per 1,000 communities or users
  • +$1 per 1,000 for full text on search results, and +$1 per 1,000 for AI tags
  • $0.003 per run
  • Rows removed by your filters, and duplicates, are never charged

Example: 100 posts with 50 comments each (5,000 comments) cost about $2.60, or about $7.70 with AI tags.

FAQ

Can I scrape Reddit without an API key?

Yes. The scraper reads the public pages any logged-out visitor sees, so it needs no API key, OAuth app or developer account.

Does Reddit have an official API? Why not use it?

Reddit's Data API needs an approved app, has strict rate limits, and charges for commercial use. Self-service keys were closed in 2025. Scraping public pages avoids the approval process, and you pay only for the posts and comments you collect.

How many posts can I get from a subreddit?

Reddit lists up to about 1,000 posts per feed or search. To go further, combine sorts and time ranges (Top of the year, Top of the month, New), or schedule runs to collect new posts over time.

Can I scrape Reddit comments by keyword?

Yes. Set searchComments to true with your keywords. Comment search results include the full comment text and the post they belong to.

How do I monitor Reddit for my brand?

Search your brand name with searchSort: "new", turn on comment search, set postedAfter to 1 day, and schedule the run daily. Send new rows to Slack or Google Sheets with an Apify integration.

Is it legal to scrape Reddit?

The scraper collects only public posts and comments that any logged-out visitor can see. You are responsible for using the data in line with Reddit's terms and privacy laws such as GDPR.

Try it

Next in the series: remember the pet gear site from part 1? It needed live Amazon prices. Part 5 is the Amazon product scraper that keeps it up to date.

Top comments (0)