Substack Newsletter Scraper: paywall status isn't a boolean, it's inferred

Quick answer

Substack's public post API doesn't ship a paywall boolean. It ships an audience string, and every "is this post free or paid" answer you've ever seen from a Substack scraper is someone else's inference from that string, same as ours. The Substack Newsletter Scraper makes the inference explicit — is_paywalled is true whenever audience != "everyone" — and returns 17 typed fields per post at $0.0025 per row plus a $0.005 run-start charge, $2.505 for 1,000 posts.

There's no is_paywalled field in the raw API 🔒

Query Substack's /api/v1/posts endpoint yourself and look for a boolean flag that says "this post is behind the paywall." You won't find one. What you'll find is audience, a string that reads "everyone" for free posts and something else — "only_paid" today — for gated ones.

We derive is_paywalled from that comparison rather than hardcoding a fixed set of "paid" values, and we deliberately keep audience itself as a raw passthrough string on every row instead of coercing it into an enum. That's not indecision — Substack has introduced new audience tiers before, and an enum that only knows about today's two values would either reject or silently mis-bucket a value it's never seen. A string that's always present and a derived boolean that's always correct beats a brittle enum that might not survive the next tier Substack ships.

The practical effect on body_html: it's populated only when include_body=true and the post is free. A paywalled post always returns body_html: null, regardless of your includeBody setting — there's no partial-body leak, and no attempt to guess at gated content from preview fragments.

The pagination asymmetry between posted_after and posted_before ⏱️

Substack's archive API returns posts newest-first. That ordering is what makes one date filter behave completely differently from the other, and it's worth understanding before you build a date-ranged pull.

posted_after stops pagination entirely. Because the archive walks newest-to-oldest, the moment a post's date falls before your posted_after cutoff, every post after that point in the archive is also older — so the scraper stops fetching that publication's pages right there. This is a meaningful performance property: a narrow recent-date pull on a publication with a 10-year archive doesn't walk the whole thing.

posted_before only skips individual posts, it doesn't stop anything. A post newer than posted_before gets excluded from the output, but pagination keeps going — because older, in-range posts are still further back in the newest-first order and haven't been reached yet. If you set only posted_before with no posted_after, expect the full archive to be walked regardless of how far back your cutoff sits.

Combine both and you get the tight range you'd expect: pagination stops once you cross posted_after, and anything newer than posted_before along the way gets filtered out.

Reaction counts are a sum, not a single number 🧮

reaction_count isn't a single "likes" field. Substack's reactions object is a dict of reaction-type-to-count (hearts, and whatever else a publication's reaction set includes), and we sum every value in it. Only when that dict is absent entirely do we fall back to a flat reaction_count field on the raw post, defaulting to 0 if neither is present. If you're benchmarking engagement across publications, know that the number reflects total reaction volume across all reaction types, not one specific emoji.

What we handle for you 🛡️

• We rotate browser fingerprints — curl-cffi impersonation across Chrome, Firefox, and Safari TLS profiles, randomized per run, so requests present as a browser rather than a Python client.
• We retry with exponential backoff on 408 / 429 / 5xx, up to 5 attempts per page, honoring Retry-After.
• We isolate publication failures. If one publication in a multi-publication batch fails outright, it's logged and skipped — the run continues through the rest of your list rather than aborting the whole batch.
• We handle unparseable dates without dropping the row. A malformed post_date is logged and exempted from your posted_after/posted_before filters rather than silently truncating an otherwise-complete archive.
• We keep the dataset clean — Pydantic-validated rows, extra="forbid", ISO-8601 timestamps, JSON / CSV / Excel export straight from the Apify Console.
• You pay only for results that hit your dataset. No data, no charge, beyond the small run-start warm-up fee.

Full output schema 📦

Seventeen fields per post:

Field	Type	Notes
publication	string	Base URL of the source publication
post_id	int	Substack internal numeric post ID
title	string	Post title
subtitle	string | null	Post subtitle / deck
slug	string	URL slug — builds the /p/<slug> path
url	string	Canonical post URL
post_date	string	ISO-8601
post_type	string	e.g. newsletter, podcast, thread
authors	list[string]	Byline names from publishedBylines
audience	string	Raw passthrough — everyone or only_paid today
is_paywalled	bool	Derived: true when audience != "everyone"
reaction_count	int	Summed across all reaction types
comment_count	int	Public comment count
word_count	int | null	When provided by the API
cover_image	string | null	Cover image URL
body_html	string \	null
scraped_at	string	ISO-8601

Who this is for

Newsletter intelligence — track a competitor publication's cadence, topic mix, and engagement trend over time using post_date, reaction_count, and comment_count together.

Content research — aggregate posts across several newsletters in one niche to find recurring themes and coverage gaps.

Paywall mapping — understand the free-vs-paid split for any publication with is_paywalled and audience, useful before pitching a guest post or gauging a competitor's monetization strategy.

Archive backfill — pull a full post history into a warehouse for analysis or ML training, max_posts_per_publication: 0 for the complete archive.

Frequently asked questions

How does the Actor know a post is paywalled if Substack doesn't say so directly?
It compares audience against "everyone". Anything else is treated as paywalled. audience itself ships unchanged on every row so you're never locked into our interpretation.

If I set postedBefore but not postedAfter, will it scan the whole archive?
Yes. postedBefore filters individual posts out of the output but doesn't stop pagination — only postedAfter does that, because the archive is walked newest-first.

What does 10,000 posts cost?
$25.005 — 10,000 × $0.0025, plus the $0.005 run-start charge.

Does it work on custom domains, not just *.substack.com?
Yes — both native Substack subdomains and custom domains hit the same /api/v1/posts path.

Can I get comments or subscriber counts?
Comment count ships on every row (comment_count); individual comment text and subscriber counts aren't part of the public post API this Actor reads.

Try it

Live on the Apify Store: Substack Newsletter Scraper.

Point it at one or more publication URLs and get back typed, paywall-labeled rows — full archive or a date-bounded slice. Pay-per-event, no subscription.

---

Built by Devil Scrapes — we build scrapers for the targets that fight back.