DEV Community

Web Obsidians
Web Obsidians

Posted on

How to Get Every Post of a Substack Newsletter with Python (and Track New Posts)

Substack newsletters are a goldmine for research: what topics get the most reactions, which posts start the longest comment threads, how often a writer publishes. You do not need RSS (which only holds the latest posts) or an account to get the whole list. Every Substack newsletter, on name.substack.com or on its own domain, answers a public archive API.

Option 1: the archive API in Python

The endpoint is /api/v1/archive?sort=new&limit=25. It returns up to 25 posts and a nextCursor. Older code used offset for paging; that now answers 400, so page with the cursor instead.

import json
import urllib.parse
import urllib.request


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (newsletter-research-script)"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)


def archive(site, pages=2):
    posts, cursor = [], None
    for _ in range(pages):
        url = f"https://{site}/api/v1/archive?sort=new&limit=25"
        if cursor:
            url += "&cursor=" + urllib.parse.quote(cursor)
        data = get(url)
        for p in data["posts"]:
            posts.append({
                "date": p["post_date"],
                "title": p["title"],
                "free": p["audience"] == "everyone",
                "reactions": p.get("reaction_count"),
                "comments": p.get("comment_count"),
                "words": p.get("wordcount"),
                "url": p["canonical_url"],
            })
        cursor = data.get("nextCursor")
        if not cursor:
            break
    return posts


posts = archive("astralcodexten.substack.com")
print(len(posts), "posts")
for p in sorted(posts, key=lambda p: p["reactions"] or 0, reverse=True)[:3]:
    print(p["reactions"], p["comments"], p["title"])
Enter fullscreen mode Exit fullscreen mode

What to know before you build on it:

  • Free or paid: audience is everyone for free posts and something else (such as only_paid) for paid ones. The archive lists paid posts too, with their numbers, but not their full text.
  • Full text is not in the archive. You need one more request per post (/api/v1/posts/<slug>), and only free posts return the whole body.
  • Search works on the same endpoint with search=<words>, but results are ranked by relevance, not by date, so read them all before filtering by date.
  • Custom domains (for example www.lennysnewsletter.com) use the same paths.
  • Be polite: pause between pages; Substack answers bursts with 429.

Option 2: a ready-made Actor with search, full text and monitoring

I packaged that work as an Apify Actor: Substack Scraper. You paste newsletter names or addresses and get one clean row per post. A real item from a run (shortened):

{
  "publication": "www.lennysnewsletter.com",
  "title": "State of the tech job market: late 2026",
  "url": "https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late",
  "date": "2026-11-10T15:04:03Z",
  "audience": "paid",
  "authors": ["Lenny Rachitsky"],
  "tags": ["Career"],
  "reactions": 404,
  "comments": 42,
  "restacks": 37,
  "wordcount": 2183
}
Enter fullscreen mode Exit fullscreen mode

What it adds:

  • Authors, tags, restacks, section, podcast link and cover image in one format
  • Search inside a newsletter, a date range and free posts only
  • Include the full text of every free post as clean plain text
  • Return only new posts: schedule it and each run returns only the posts that appeared since the last run
  • Export to JSON, CSV or Excel, or send new posts to Slack, Google Sheets or a webhook

From code:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("webobsidians/substack-scraper").call(
    run_input={"publications": ["astralcodexten", "https://www.lennysnewsletter.com"], "maxPostsPerPublication": 50}
)
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(post["publication"], post["date"], post["reactions"], post["title"])
Enter fullscreen mode Exit fullscreen mode

It costs $1.00 per 1,000 posts plus $0.00005 per run start, with platform usage included (the full text costs nothing extra), and Apify's free plan includes monthly credit to try it.

Which one should you use?

  • One newsletter's post list, once: the Python script.
  • Many newsletters, full text, search, or a feed of new posts: the Actor.

Either way, you are reading what writers publish publicly, so respect paywalls and keep request rates reasonable.

Top comments (0)