DEV Community

Coco
Coco

Posted on

How to scrape Reddit in 2026?

#python #webscraping #tutorial #api

Three Reddit scraping jobs in Python: a single post with its comment tree, a subreddit listing, and search results. Every request below ran against the Chocodata API on 27 July 2026 with Python 3.13.7 and requests 2.34.2, and every output block is what my terminal actually printed.

Why is it hard to scrape Reddit?

Reddit blocks datacenter IPs on the public JSON views, renders the logged-out site from JavaScript so the HTML you download holds no scores, and serves the same content through several front ends that do not all expose vote counts. The one that costs the most time is comment pagination, because a thread returns a slice of its discussion by default and a partial tree is indistinguishable from a complete one unless you check the count.

Prerequisites

  1. A free Chocodata API key. Sign up, confirm the email, copy the key from the dashboard. Free, no card.

Copying a free API key from the dashboard

  1. Python 3.9+ and requests. 
python -m venv .venv && source .venv/bin/activate
pip install requests
Enter fullscreen mode Exit fullscreen mode
  1. The Reddit URL you want to scrape, copied from the address bar.

Tested with Python 3.13.7 and requests 2.34.2 in July 2026.

How to scrape Reddit posts?

Scraping a Reddit post takes the thread URL and returns the post plus its comment tree in one call.

1. GET the post endpoint with the thread URL

Pass the post URL as url and read the fields off the top level of the response, because there is no envelope around them.

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
POST_URL = "https://www.reddit.com/r/webscraping/comments/p64rqq/linkedin_scraping/"


def get_json(path, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}", params=params, timeout=60)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


data = get_json("reddit/post", url=POST_URL)
print(list(data.keys()))
print(data["post"]["title"])
Enter fullscreen mode Exit fullscreen mode 
['post', 'comments', 'comments_returned', '_meta']
LinkedIn Scraping
Enter fullscreen mode Exit fullscreen mode

HTTP 200 in 5.04s. Four top-level keys, and post is where the thread lives.

Terminal showing the post request returning the four top-level keys

2. Pull the post object off the top level

The post object carries score, ratio, comment count, author and body, so one request answers the engagement questions.

post = data["post"]

print(f"{post['score']} points at {post['upvote_ratio']:.2f} ratio")
print(f"{post['num_comments']} comments | u/{post['author']['username']} | {post['created'][:10]}")
print(f"{post['domain']} | locked={post['is_locked']} | body {len(post['body'])} chars")
Enter fullscreen mode Exit fullscreen mode 
12 points at 0.93 ratio
17 comments | u/pizzihut | 2021-08-17
self.webscraping | locked=None | body 224 chars
Enter fullscreen mode Exit fullscreen mode

author is a nested object, not a string. That trips code written against the subreddit endpoint, where the same field is a plain username.

Parsed post fields printed from the post object

3. Flatten the nested reply tree

Comments come back as a tree where each comment holds its own replies list, so recursion turns it into rows you can store.

def flatten(comments, depth=0, out=None):
    out = [] if out is None else out
    for c in comments:
        out.append({"depth": depth, "author": c["author"]["username"],
                    "score": c["score"], "body": c["body"]})
        flatten(c["replies"], depth + 1, out)
    return out


rows = flatten(data["comments"])
for c in rows[:4]:
    print("  " * c["depth"], f"[{c['score']:>2}] {c['author']}: {c['body'][:46]}")

print(f"{len(rows)} flattened / {data['comments_returned']} returned "
      f"/ {post['num_comments']} reported")
print("truncated:", data["_meta"]["truncated"])
Enter fullscreen mode Exit fullscreen mode 
 [ 3] [deleted]: I need to scan profiles of generel information
   [ 3] pizzihut: I need to scan profiles of generel information
 [ 2] nubela: Try Proxycurl! It has a dedicated Linkedin Scr
   [ 1] pizzihut: Hi! Just tried the free subscription, but it s
8 flattened / 8 returned / 17 reported
truncated: True
Enter fullscreen mode Exit fullscreen mode

Four top-level threads flattened to 8 comments against the 17 Reddit reports, and _meta.truncated says so explicitly. Assert on comments_returned before you treat a thread as complete. Listings behave differently, because there the pagination handle is a cursor.

Comment tree flattened with depth and score

How to scrape Reddit subreddits?

Scraping a subreddit returns the community listing as an array of posts plus a cursor for the next batch.

1. Fetch the first 25 posts

Send the subreddit name and the response returns the listing under posts with the sort it used.

sub = get_json("reddit/subreddit", subreddit="webscraping")

print(sub["sort"], "|", sub["total_results"], "posts |", sub["after_cursor"])
for p in sub["posts"][:2]:
    print(f"{p['score']:>4} {p['num_comments']:>4} {p['upvote_ratio']:.2f}  {p['title'][:44]}")
Enter fullscreen mode Exit fullscreen mode 
hot | 25 posts | dDNfMXV3YWxodg==
   7   12 1.00  I own a wood factory and want to use webscra
   2    8 0.67  Vinted cloudfare detection
Enter fullscreen mode Exit fullscreen mode

HTTP 200 in 1.87s. Each post carries 13 fields including awards, domain, external_url and permalink. Adding sort="top" with t="week" returned 16 posts for the week, led by an 83-point thread.

Terminal showing the subreddit listing with sort and cursor

2. Page with after_cursor

Feed after_cursor back as after to walk deeper, and stop when the cursor comes back empty.

import time


def fetch_pages(subreddit, n_pages=2, delay=1.0):
    cursor, seen = None, []
    for _ in range(n_pages):
        params = {"subreddit": subreddit}
        if cursor:
            params["after"] = cursor
        page = get_json("reddit/subreddit", **params)
        seen.extend(page["posts"])
        cursor = page["after_cursor"]
        if not cursor:
            break
        time.sleep(delay)
    return seen


allp = fetch_pages("webscraping", 2)
print(len(allp), "posts |", len({p["id"] for p in allp}), "unique ids")
Enter fullscreen mode Exit fullscreen mode 
50 posts | 50 unique ids
Enter fullscreen mode Exit fullscreen mode

Fifty rows and fifty distinct ids, so the cursor advances cleanly with no overlap between pages. The id set is the check worth keeping, because a cursor that stalls silently returns duplicates rather than an error.

Two pages fetched with zero duplicate post ids

3. Write the listing to CSV

Select the fields you need and write rows, since keeping the whole payload per post buys nothing for trend work.

import csv

FIELDS = ["id", "title", "score", "num_comments", "upvote_ratio",
          "author", "created", "permalink"]

with open("subreddit_posts.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=FIELDS)
    w.writeheader()
    for p in allp:
        w.writerow({k: p.get(k) for k in FIELDS})

print(f"wrote {len(allp)} rows to subreddit_posts.csv")
print("mean score:", round(sum(p["score"] for p in allp) / len(allp), 1))
Enter fullscreen mode Exit fullscreen mode 
wrote 50 rows to subreddit_posts.csv
mean score: 11.3
Enter fullscreen mode Exit fullscreen mode

A mean of 11.3 across 50 posts is the baseline that makes a 83-point thread readable as an outlier. Search covers the same content across communities rather than inside one.

CSV written with the selected listing fields

How to scrape Reddit search results?

Scraping Reddit search results returns a ranked mix of communities and posts for a keyword.

1. Query the search endpoint

Send the query and the response returns positioned results with the type of each hit.

search = get_json("reddit/search", q="web scraping")

print(search["query"], "|", search["sort"], "|", search["total_results"], "results")
for r in search["results"][:4]:
    print(f"{r['position']:>3}  {r['result_type']:<9} {r['title'][:44]}")
Enter fullscreen mode Exit fullscreen mode 
web scraping | relevance | 25 results
  1  subreddit webscraping
  2  subreddit WebScrapingInsider
  3  subreddit Scraping the web
  4  post      The Complete Web Scraping & Anti-Bot Bypass
Enter fullscreen mode Exit fullscreen mode

HTTP 200 in 2.55s under the default relevance sort.

Search results with position and result type

2. Filter on result_type

Split the results by result_type so community hits and post hits go into different pipelines.

from collections import Counter

posts = [r for r in search["results"] if r["result_type"] == "post"]
subs = [r for r in search["results"] if r["result_type"] == "subreddit"]

print(len(subs), "communities |", len(posts), "posts")
for name, n in Counter(p["subreddit"] for p in posts).most_common(5):
    print(f"r/{name:<20} {n}")
Enter fullscreen mode Exit fullscreen mode 
3 communities | 22 posts
r/webscraping          2
r/scrapingtheweb       2
r/programming          2
r/automation           1
r/PythonLearning       1
Enter fullscreen mode Exit fullscreen mode

Three communities and 22 posts, with the topic spread thin past the top three subreddits.

Results filtered by result type and counted per subreddit

3. Sort by new for monitoring

Pass sort="new" when the job is monitoring rather than research, because relevance ordering hides anything posted this morning.

newest = get_json("reddit/search", q="web scraping", sort="new")
fresh = [r for r in newest["results"] if r["result_type"] == "post"]

for r in fresh[:4]:
    print(r["created"][:16], "|", f"r/{r['subreddit']}", "|", r["title"][:40])
Enter fullscreen mode Exit fullscreen mode 
2026-07-27T12:00 | r/LocalLLaMA | XYZAILab/XYZ-Aquila-mini · Hugging Face
2026-07-27T10:23 | r/jobsearch | Roast my resume
2026-07-27T09:14 | r/BotNation | What's the best way to learn web scrapin
2026-07-27T08:38 | r/hireforgigs | [for hire] I'm available: Web Designer/D
Enter fullscreen mode Exit fullscreen mode

Same-day results, so a cron job on this call is a working keyword alert.

Search results sorted by new showing same-day posts

The part that breaks

There is no data envelope. Every field sits at the top level, so the reflex from other APIs fails immediately:

KeyError: 'data'
Enter fullscreen mode Exit fullscreen mode

author changes shape between endpoints. On a post it is an object, on a listing it is a string:

AttributeError: 'dict' object has no attribute 'lower'
Enter fullscreen mode Exit fullscreen mode

Normalise it once at the parse boundary with a["username"] if isinstance(a, dict) else a.

Vote counts are not on every surface. Search results can return score as null, and sorting on it raises:

TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
Enter fullscreen mode Exit fullscreen mode

Use key=lambda r: r["score"] or 0 or filter the nulls out before sorting.

Comment trees arrive truncated by default. comments_returned was 8 against a num_comments of 17, with _meta.truncated set to True. Treat a thread as partial unless those two numbers agree.

Full script 

"""Scrape Reddit posts, subreddit listings and search results.
Python 3.9+, requests. Tested 27 July 2026 with 3.13.7 / requests 2.34.2.
"""
import csv
import time
from collections import Counter

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def get_json(path, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}", params=params, timeout=60)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


def author_name(a):
    return a["username"] if isinstance(a, dict) else a


def scrape_post(post_url):
    data = get_json("reddit/post", url=post_url)
    post = data["post"]

    def flatten(comments, depth=0, out=None):
        out = [] if out is None else out
        for c in comments:
            out.append({"depth": depth, "author": author_name(c["author"]),
                        "score": c["score"], "body": c["body"]})
            flatten(c["replies"], depth + 1, out)
        return out

    comments = flatten(data["comments"])
    if data["comments_returned"] < post["num_comments"]:
        print(f"[warn] {data['comments_returned']}/{post['num_comments']} comments returned")
    return post, comments


def scrape_subreddit(subreddit, n_pages=2, delay=1.0, path="subreddit_posts.csv"):
    cursor, seen = None, []
    for _ in range(n_pages):
        params = {"subreddit": subreddit}
        if cursor:
            params["after"] = cursor
        page = get_json("reddit/subreddit", **params)
        seen.extend(page["posts"])
        cursor = page["after_cursor"]
        if not cursor:
            break
        time.sleep(delay)

    fields = ["id", "title", "score", "num_comments", "upvote_ratio",
              "author", "created", "permalink"]
    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=fields)
        w.writeheader()
        for p in seen:
            w.writerow({k: p.get(k) for k in fields})
    return seen


def search_reddit(query, sort=None):
    params = {"q": query}
    if sort:
        params["sort"] = sort
    results = get_json("reddit/search", **params)["results"]
    posts = [r for r in results if r["result_type"] == "post"]
    return posts, Counter(p["subreddit"] for p in posts)


if __name__ == "__main__":
    post, comments = scrape_post(
        "https://www.reddit.com/r/webscraping/comments/p64rqq/linkedin_scraping/")
    print(post["title"], "|", post["score"], "points |", len(comments), "comments")

    listing = scrape_subreddit("webscraping", n_pages=2)
    print(len(listing), "posts written |",
          "mean score", round(sum(p["score"] for p in listing) / len(listing), 1))

    posts, where = search_reddit("web scraping")
    print(len(posts), "search posts |", where.most_common(3))
Enter fullscreen mode Exit fullscreen mode

Summary

One request each covers the three jobs: a thread URL returns the post and its comment tree in about five seconds, a subreddit name returns 25 posts with a cursor that pages cleanly to 50 unique ids, and a keyword returns 25 ranked results split across communities and posts. Any language with an HTTP client works the same way, so Node, Go and Ruby need no special handling. The thing to build in from the start is the comments_returned check, because a truncated thread parses perfectly and quietly loses most of the discussion.

FAQ

Do I need a Reddit account or OAuth token?

No, the requests above authenticate with a Chocodata key and never touch Reddit credentials or an app registration.

Why is score null on some Reddit results?

Not every Reddit surface exposes vote counts, so search results return identity, subreddit and timestamps with score left null rather than guessed.

Is scraping Reddit legal?

Public pages are not automatically illegal to scrape, but Reddit's user terms restrict automated collection, which makes it a contract question rather than a criminal one, and this is not legal advice.

Top comments (0)