DEV Community

Cedarfetch Labs
Cedarfetch Labs

Posted on AI-assisted

How to pull a Substack newsletter's posts as Markdown

You want a newsletter's archive as Markdown files, for offline reading, a search index or a knowledge base. Every Substack publication serves its archive and its posts as JSON, on a name.substack.com address and on custom domains, and free posts need no login.

The archive endpoint

GET https://<publication>/api/v1/archive?sort=new&limit=25 returns {"posts": [...], "nextCursor": "..."}. Each post has id, slug, title, subtitle, post_date, audience, type, wordcount, canonical_url and reaction and comment counts. The body is not in the listing: body_html is null there.

Two quirks. limit=25 works and limit=50 returned HTTP 400 today. offset is rejected with 400 as well, so you page with the opaque nextCursor string, passed back as &cursor=. Replay it as given. The last page has nextCursor: null.

import json, re, time, urllib.request
from html.parser import HTMLParser

UA = {"User-Agent": "substack-export-example/1.0 (personal script)"}

def get(url):
    with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r:
        return json.load(r)

def archive(pub):
    """Yield post metadata, newest first, following the server's cursor."""
    cursor = None
    while True:
        url = f"https://{pub}/api/v1/archive?sort=new&limit=25"
        if cursor:
            url += "&cursor=" + cursor
        page = get(url)
        yield from page["posts"]
        cursor = page.get("nextCursor")
        if not cursor:
            return
        time.sleep(1)           # stay near one request per second

for i, post in enumerate(archive("importai.substack.com")):
    if i < 3:
        print(post["post_date"][:10], post["audience"], post["wordcount"], post["slug"])
print(i + 1, "posts")
Enter fullscreen mode Exit fullscreen mode

Output:

2026-11-09 everyone 2622 import-ai-480-self-sustaining-space
2026-11-02 everyone 2425 import-ai-479-spacewar-low-latency
2026-10-26 everyone 2197 import-ai-478-math-for-alignment
163 posts
Enter fullscreen mode Exit fullscreen mode

Fetch a post and convert it

GET https://<publication>/api/v1/posts/<slug> returns one post with body_html. Substack's HTML includes subscribe forms, SVG icons and buttons, which you want to drop. This converter handles headings, emphasis, links, images, lists, quotes, code and fenced code, using only the standard library. Save it as to_markdown.py.

import re
from html.parser import HTMLParser

SKIP = {"form", "svg", "button", "script", "style"}      # subscribe widgets, icons
START = {"strong": "**", "b": "**", "em": "_", "i": "_", "li": "\n- ", "blockquote": "\n\n> ",
         "hr": "\n\n---\n\n", "pre": "\n\n```

\n", **{f"h{n}": "\n\n" + "#" * n + " " for n in range(1, 7)}}
END = {"strong": "**", "b": "**", "em": "_", "i": "_", "p": "\n\n", "pre": "\n

```\n\n",
       **{f"h{n}": "\n\n" for n in range(1, 7)}}

class Markdown(HTMLParser):
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.out, self.skip, self.pre, self.href = [], 0, False, None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        self.skip += tag in SKIP
        if self.skip:
            return
        self.pre = self.pre or tag == "pre"
        if tag == "a":
            self.href = a.get("href"); self.out.append("[")
        elif tag == "img" and a.get("src"):
            self.out.append(f'![{a.get("alt") or ""}]({a["src"]})')
        elif tag == "code" and not self.pre:
            self.out.append("`")
        else:
            self.out.append(START.get(tag, ""))

    def handle_endtag(self, tag):
        self.skip -= tag in SKIP
        if self.skip:
            return
        if tag == "pre":
            self.pre = False
        self.out.append(f"]({self.href})" if tag == "a" else
                        "`" if tag == "code" and not self.pre else END.get(tag, ""))

    def handle_data(self, data):
        if not self.skip:
            self.out.append(data if self.pre else re.sub(r"\s+", " ", data))

def to_markdown(html):
    md = Markdown(); md.feed(html)
    return re.sub(r"\n{3,}", "\n\n", "".join(md.out)).strip()
Enter fullscreen mode Exit fullscreen mode

Then a small script to fetch one post:

import json, sys, urllib.request
from to_markdown import to_markdown

pub, slug = sys.argv[1], sys.argv[2]          # e.g. importai.substack.com import-ai-480-self-sustaining-space
req = urllib.request.Request(f"https://{pub}/api/v1/posts/{slug}",
                             headers={"User-Agent": "substack-export-example/1.0 (personal script)"})
post = json.load(urllib.request.urlopen(req, timeout=30))

text = to_markdown(post["body_html"] or "")
print(f'{post["title"]} | audience={post["audience"]} | wordcount={post["wordcount"]}')
print(text[:500])
Enter fullscreen mode Exit fullscreen mode
$ python get_post.py importai.substack.com import-ai-480-self-sustaining-space
Import AI 480: Self-sustaining space economies; AI for science progress, ... | audience=everyone | wordcount=2622
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. ...

[Subscribe now](https://importai.substack.com/subscribe?)
Enter fullscreen mode Exit fullscreen mode

The converter is deliberately small. Ordered lists become bullets, <br> is dropped, and a quote with several paragraphs only gets > on its first line. A library such as markdownify or html2text will do better if you need fidelity.

Paywalled posts

audience is everyone for free posts. Anything else, such as only_subscribers or only_paid, is gated. For a gated post the endpoint returns only the public preview. I fetched a paid post from a newsletter that reports wordcount=6286; the API returned roughly 2,850 words of it, the part Substack shows to anyone.

... | audience=only_paid | wordcount=6286
Enter fullscreen mode Exit fullscreen mode

Paid content is not accessible through this endpoint, and you should not try to get it another way: no borrowed cookies, no logging in on someone else's behalf. Check audience == "everyone" before you treat a body as a complete post, or a search index will fill up with half-articles. These are other people's writing, so keep archives for your own use unless the author has agreed to more.

Limits and gotchas

  • Rate limit. Roughly two requests a second sustained is too fast. During my testing, two parallel workers with a half-second delay drew a burst of HTTP 429 after about 100 requests. One request per second was clean, including the 7 archive pages above, so the loop sleeps a second and you should back off on a 429.
  • Hidden posts. Publishers can hide sections, and often their podcast, from the main archive. /api/v1/archive?section_id=<id> lists a section, and /sitemap.xml lists every public post URL with a lastmod. On one large newsletter the main archive held 485 posts and the sitemap 1,296. Compare the sitemap count with what you pulled.
  • Not everything on a custom domain is still Substack. A site that moved away answers with a plain 404, not a Substack error.
  • Some endpoints refuse you. /api/v1/publication returned 403 on every newsletter I tried, and the global substack.com search and category pages do not answer anonymous API clients.
  • Public data only. Skip comments and anything about readers. Keep the request rate low and follow the publication's terms.

If you don't want to maintain this yourself

Disclosure: I'm the developer of an Apify Actor, Substack Scraper, published under Cedarfetch Labs. It does the paging and conversion above and adds Markdown with footnotes and embeds handled, rows marked isPaywalled and contentAvailable: "preview" so previews never pass for full posts, an option to include hidden sections and sitemap-only posts, and a new-posts feed through change tracking. It never logs in and never gets past a paywall.

It is pay per event: $0.001 per post row, body included, plus $0.00005 per run start, as shown on the listing on 2026-11-11. Check the Pricing tab for current figures. Listing: https://apify.com/cedarfetch/substack-scraper

Runnable Python, Node.js and cURL examples for the Actor: https://github.com/cedarfetch/substack-scraper-examples

Not affiliated with Substack.

Top comments (0)