How to export Substack posts to clean Markdown for RAG (no browser needed)

#rag #llm #webscraping #tutorial

If you have tried to feed a Substack newsletter into an LLM or a RAG pipeline, you have probably hit the same three problems I did:

The post HTML is full of subscribe buttons, share widgets and nested image wrappers that pollute your chunks.
Substack's archive endpoint no longer accepts offset. It returns HTTP 400, and you have to follow a cursor instead.
The archive list does not include the post body, so you need a second request per post.

I got tired of rewriting this, so I built a small tool for it and published it on Apify. Disclosure: I am the author.

Substack Scraper - Markdown Posts, Search & Comments

It uses plain HTTP requests against Substack's public JSON API. No headless browser and no residential proxies, so runs are fast and cheap.

1. Give it a publication

Open the Actor, paste this input and press Start:

{
    "publications": ["astralcodexten"],
    "maxPosts": 20,
    "sinceDate": "30 days"
}

publications accepts a name, a name.substack.com URL or a custom domain. You can also set searchQuery to find publications about a topic.

2. Get Markdown back

Each post is one dataset item. The markdown field starts with the title as an H1, so every document is self-contained when you chunk it:

{
    "title": "AI Company Employees Should Unionize",
    "url": "https://www.astralcodexten.com/p/ai-company-employees-should-unionize",
    "publishedAt": "2026-11-12T23:09:15.486Z",
    "authors": [{ "name": "Scott Alexander", "handle": "astralcodexten" }],
    "wordCount": 1273,
    "markdown": "# AI Company Employees Should Unionize\n\n**I.**\n\nShould safety-conscious employees at AI companies quit in protest, or stay and fight? ..."
}

Headings, lists, code blocks, images with captions and footnotes are kept as standard Markdown. Subscribe and share widgets are removed.

3. Call it from Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("gunpark/substack-scraper").call(
    run_input={"publications": ["astralcodexten"], "maxPosts": 20}
)

for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(post["title"], len(post["markdown"]))

From here you can pass post["markdown"] straight to your text splitter and embedding model.

Keeping an index up to date

Turn on incremental and schedule the Actor. The first run sets a baseline; later runs return only posts published since the previous run, so you never embed the same post twice.

Good to know

Paywalled posts only expose a public preview. They are skipped by default, or delivered flagged as incomplete if you ask for previews.
Comments are optional (includeComments).
Pricing is pay per delivered post: $1.20 per 1,000 posts. Failed requests, duplicates and skipped paywalled posts are not charged.
The posts belong to their authors. Use the output in ways their terms and copyright allow.

If something breaks or you need a field that is missing, open an issue on the Actor page and I will look at it.

DEV Community