DEV Community

Signal Wharf
Signal Wharf

Posted on Originally published at signalwharf.hashnode.dev

How to Get Substack Newsletter Data: Posts, Comments and Subscriber Counts

#substack #scraping #python #data

Substack hosts many of the most-read independent newsletters, and the data around them is useful. Sponsors want to know how big a newsletter really is. Researchers want the full text of posts. Media teams track what competitors publish and how readers react.

Substack has no official public API for this. The data is visible in the browser, though: post pages, comment threads, the archive, and the "Over 466,000 subscribers" line on a publication's homepage. This guide shows how to collect it in a structured way without logging in, using the Substack Scraper on Apify. The output works as JSON, CSV or Excel, or from Python.

What data you can get from Substack

Posts (one item per post):

  • title, subtitle, URL, publish date, authors, section and tags
  • isPaid and audience (free or subscriber-only)
  • word count, likes (reactions), comment count and restacks
  • full body text, optionally the HTML, and the links in the post
  • podcast URL and duration for audio posts

Publication profiles (one per newsletter):

  • the subscriber estimate as shown publicly ("Over 466,000 subscribers" → 466000)
  • the paid-subscriber tier text, and the Substack bestseller tier
  • monthly, yearly and founding-member prices
  • posting cadence (posts per week), the share of posts that are paid, and average likes, comments and restacks

Comments (optional): author, date, body, likes, reply count and thread depth.

What it does not do: it doesn't log in, bypass paywalls or collect subscriber emails. For subscriber-only posts you get the metadata plus the free preview that Substack shows every visitor, flagged with isFullTextComplete: false. Substack Notes aren't covered.

Step 1: List the newsletters

Use whatever identifier you have. The scraper accepts:

  • a subdomain: noahpinion
  • a Substack URL: https://noahpinion.substack.com
  • a custom domain: https://www.astralcodexten.com or www.noahpinion.blog
  • a profile link: https://substack.com/@noahpinion

Step 2: Run the scraper

  1. Open Substack Scraper and click Try for free. You need a free Apify account.
  2. Add your newsletters under Publications.
  3. Set Max posts per publication. Use 20–50 for a quick look, or more for a full archive.
  4. Keep Include full text on if you want the article bodies.
  5. Turn on Include comments if you need them. You can also set Max comments per post.
  6. Optional: add a date range or keywords.
  7. Click Start.

Our test input (Nov 11, 2026):

{
  "publications": ["noahpinion", "https://www.astralcodexten.com"],
  "maxPostsPerPublication": 20,
  "includeFullText": true,
  "includePublicationProfile": true
}
Enter fullscreen mode Exit fullscreen mode

It finished with 40 posts from 2 of 2 publications, plus a profile for each.

Step 3: Read the output

Every item has an itemType: post, publication or comment. Here are real items from our test runs, trimmed.

A publication profile:

{
  "itemType": "publication",
  "publicationName": "Noahpinion",
  "publicationUrl": "https://www.noahpinion.blog",
  "authorName": "Noah Smith",
  "createdAt": "2020-03-28T03:32:51.086Z",
  "subscriberCountText": "Over 466,000 subscribers",
  "subscriberCountApprox": 466000,
  "paidSubscriberTierText": "Tens of thousands of paid subscribers",
  "bestsellerTier": 10000,
  "monthlyPrice": 10.0,
  "yearlyPrice": 100.0,
  "foundingMemberPrice": 420.0,
  "postsPerWeek": 7.0,
  "avgReactions": 190.3,
  "avgComments": 12.7
}
Enter fullscreen mode Exit fullscreen mode

A post:

{
  "itemType": "post",
  "publicationName": "Noahpinion",
  "title": "India is quietly doing great",
  "subtitle": "The world’s biggest country continues to defy the skeptics.",
  "url": "https://www.noahpinion.blog/p/india-is-quietly-doing-great",
  "postDate": "2026-11-11T08:51:56.921Z",
  "audience": "everyone",
  "isPaid": false,
  "wordcount": 1596,
  "reactionCount": 404,
  "commentCount": 25,
  "restacks": 47,
  "bodyText": "My last post was about how for the first time, I’m becoming bearish on China’s ..."
}
Enter fullscreen mode Exit fullscreen mode

A comment (with comments enabled):

{
  "itemType": "comment",
  "postTitle": "India is quietly doing great",
  "authorName": "Matthew",
  "date": "2026-11-11T09:29:04.810Z",
  "reactionCount": 35,
  "replyCount": 3,
  "depth": 0,
  "body": "Great piece. I wish you had gone into some of the politics of this, ..."
}
Enter fullscreen mode Exit fullscreen mode

How accurate are the subscriber counts?

They're as accurate as what Substack shows publicly, and that's a rounded figure. "Over 466,000 subscribers" becomes subscriberCountApprox: 466000. The real number is at least that high, and it includes free subscribers. Paid subscribers only appear as a tier ("Tens of thousands of paid subscribers"), never as an exact count. Some newsletters hide the count entirely, and then the field is null. The averages (avgReactions, postsPerWeek) are computed from the posts scraped in that run. The field statsBasedOnPosts tells you how many posts that was, so scrape more posts if you want steadier averages.

Step 4: Export to CSV, Excel or Google Sheets

The Output tab has views for each item type:

View Best for
Overview one row per post: date, title, paid flag, word count, likes, comments, restacks
Content full text, authors, tags and links, for reading or analysis
Publications subscriber estimates, prices, bestseller tier, cadence and engagement
Comments comment text, author, likes and depth

Choose a view, then Export as CSV, Excel or JSON. To send results to Google Sheets automatically, use Apify's Google Sheets integration on the Actor or on a scheduled task.

Step 5: Use it from Python 

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("signalwharf/substack-scraper").call(run_input={
    "publications": ["noahpinion", "https://www.astralcodexten.com"],
    "maxPostsPerPublication": 20,
})

items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
pubs = [i for i in items if i["itemType"] == "publication"]
posts = [i for i in items if i["itemType"] == "post"]

for p in pubs:
    print(p["publicationName"], p["subscriberCountApprox"], p["monthlyPrice"])

# Most-liked posts first
for post in sorted(posts, key=lambda x: x["reactionCount"] or 0, reverse=True)[:5]:
    print(post["reactionCount"], post["title"])
Enter fullscreen mode Exit fullscreen mode

With pandas, pd.DataFrame(posts) gives you a table ready to filter or plot.

Step 6: Track new posts automatically

Set onlyNewPosts: true and create a schedule, for example hourly or daily. The scraper remembers the newest post it saw for each publication and returns only newer ones on the next run. A run that finds nothing new costs nothing. This is a simple way to build a newsletter monitor that feeds Slack, email or a spreadsheet.

Practical use cases

  • Sponsorship research: compare subscriber estimates, posting cadence and engagement across newsletters before you buy an ad slot.
  • Competitive content tracking: see what competitors publish, how long their posts are, and which topics get the most likes and restacks.
  • Research and NLP datasets: collect full text from free posts for topic modeling, summarization or sentiment analysis.
  • Pricing benchmarks: see the monthly and yearly prices that newsletters in your niche charge.

How much does it cost?

Pay per event, no subscription:

Event Price
Post (full text included) $0.0015 ($1.50 per 1,000)
Publication profile $0.003
Comment (only if enabled) $0.0003

Our 40-post test with 2 profiles cost 40 × $0.0015 + 2 × $0.003 = $0.066. A single newsletter with 100 posts and a profile costs about $0.15. You can cap the cost per run in the run options. Apify's free plan includes a monthly platform credit, which is enough to try it out. Check Apify's pricing page for the current amount.

Troubleshooting

  • A custom domain isn't recognized: paste the full homepage URL, e.g. https://www.astralcodexten.com.
  • Paid post with short text: expected. Only the public preview is available without a subscription.
  • Subscriber fields are null: that publication doesn't show a public count.
  • Comment counts differ from the comments returned: check maxCommentsPerPost.

Is it legal?

The scraper reads only public pages and public JSON that any visitor's browser loads. It doesn't log in, bypass paywalls, or collect emails or private subscriber data. Post text is copyrighted by its authors, so use it for analysis and research, and don't republish it. Comments are personal data under the GDPR. Ask a lawyer if you're unsure.

Related tools

Built and maintained by Signal Wharf. Substack is a trademark of its owner; this tool is not affiliated with or endorsed by Substack. Example newsletters are shown only to illustrate public data.

Top comments (0)