Build a RAG knowledge base from Substack newsletters (clean Markdown, no paywall tricks)

#ai #javascript #llm #rag

Newsletters are some of the best expert content on the internet — and a great source for a RAG chatbot or an AI
research agent. The problem: Substack's HTML is full of "Subscribe now" buttons, share widgets and image wrappers
that pollute your embeddings. Here's a clean pipeline.

1. List posts

Every publication — including custom domains like lennysnewsletter.com — serves its archive as JSON:

curl "https://www.lennysnewsletter.com/api/v1/archive?sort=new&offset=0&limit=50"

Page with offset until you get []. Each item has title, slug, post_date, audience, wordcount, reaction_count, publishedBylines, postTags… but no body.

2. Fetch the body

curl "https://www.lennysnewsletter.com/api/v1/posts/{slug}"

body_html is there. Check audience: everyone is free; for paid posts the API only returns a preview.
Please respect that — use metadata only for paid posts.

3. Convert to clean Markdown

turndown does the conversion. Two lessons learned:

Use addRule to drop junk, not remove() — turndown's built-in paragraph rule matches <p class="button-wrapper"> first, so remove() never fires.
Every image is wrapped in <a class="image-link">, which turns into an ugly [![](img)](img). Unwrap it.

td.addRule('junk', {
  filter: (n) => ['FORM', 'BUTTON', 'SVG'].includes(n.nodeName.toUpperCase())
    || /subscription-widget|button-wrapper|captioned-button|icon-container/.test(n.getAttribute?.('class') || ''),
  replacement: () => '',
});
td.addRule('imageLink', {
  filter: (n) => n.nodeName === 'A' && /image-link/.test(n.getAttribute('class') || ''),
  replacement: (content) => `\n\n${content.trim()}\n\n`,
});

4. Chunk it

Split on blank lines, carry the most recent # heading into each chunk, cap at ~2,000 characters with ~200 characters
of overlap, and store url, title, postDate, heading as metadata so your bot can cite sources.

5. Keep it fresh

The archive is newest-first, so a daily job only needs to page until it meets a post ID it has already seen.

Reference & ready-made tool

Endpoint notes and full code: github.com/Mersetti/substack-api-guide
No-code version (Markdown/text/HTML, RAG chunks, filters, new-post monitoring, paywall-safe): Substack to Markdown on Apify

What are you building on top of newsletters? Curious to hear in the comments.

DEV Community