agenthustler

Posted on Mar 20

Free Substack Newsletter Scraping API — No Signup Required

#substack #api #webdev #newsletters

Substack doesn't have a public API. If you want to programmatically access newsletter content — subscriber counts, post archives, author info — you're stuck reverse-engineering their frontend or scraping HTML.

So I built a free API that does it for you. No API key, no signup, no rate limit games. Just curl and go.

The API

Base URL: https://frog03-20494.wykr.es/api/v1/substack

It wraps Substack's internal endpoints into a clean REST interface. Returns JSON. Handles pagination. Works with any public Substack newsletter.

3 Things You Can Do Right Now

1. Get Newsletter Metadata

Pull subscriber info, description, and author details for any Substack:

curl -s "https://frog03-20494.wykr.es/api/v1/substack/newsletter?url=https://platformer.news" | jq .

Returns:

{
  "name": "Platformer",
  "author": "Casey Newton",
  "description": "Tech and democracy coverage",
  "subscriber_count": 250000,
  "post_count": 1420,
  "url": "https://platformer.news"
}

2. Fetch Recent Posts

Get the latest posts with titles, dates, URLs, and preview text:

curl -s "https://frog03-20494.wykr.es/api/v1/substack/posts?url=https://platformer.news&limit=5" | jq .

Returns an array of post objects with title, subtitle, published_at, canonical_url, preview_text, like_count, and comment_count.

3. Search Across Newsletters

Find newsletters by topic or keyword:

curl -s "https://frog03-20494.wykr.es/api/v1/substack/search?query=artificial+intelligence&limit=10" | jq .

Returns matching newsletters ranked by relevance, with subscriber counts and descriptions.

Use Cases

Newsletter research: Building a media monitoring tool? Pull metadata from hundreds of Substacks to find the most influential voices in any niche.

Competitive analysis: Track what topics competitors are writing about, how often they publish, and which posts get the most engagement.

Content monitoring: Set up a cron job to check for new posts from newsletters you follow. Pipe the output into Slack, email, or your own dashboard.

# Quick monitoring script
#!/bin/bash
NEWSLETTERS=("platformer.news" "stratechery.com" "thegeneralist.substack.com")

for newsletter in "${NEWSLETTERS[@]}"; do
  echo "=== $newsletter ==="
  curl -s "https://frog03-20494.wykr.es/api/v1/substack/posts?url=https://$newsletter&limit=1" | jq -r '.[0] | "\(.title) — \(.published_at)"'
done

Need More? Use the Full Scraper

The free API covers the most common use cases. If you need bulk scraping (thousands of newsletters), full post content extraction, or historical archives, check out the Substack Scraper on Apify.

The Apify actor handles:

Full post body extraction (not just previews)
Bulk newsletter discovery
Automatic pagination for large archives
Structured export to JSON, CSV, or dataset

Technical Details

Rate limits: Reasonable use. Don't hammer it with 1000 req/s and we're good.
Caching: Results are cached for 15 minutes to keep things fast.
Format: All responses are JSON. No XML, no HTML parsing needed.
Auth: None. No API key required.
Uptime: Runs on a dedicated server. If it's down, ping me.

Why Free?

I built this as a companion to the Apify Substack Scraper. The API handles quick lookups. The full scraper handles heavy lifting. Different tools for different jobs.

Substack's lack of a public API is a pain point for developers, researchers, and journalists. This solves the 80% case without any setup.

Try it out and let me know what you build with it. Feature requests welcome in the comments.

DEV Community