DEV Community

Viktor
Viktor

Posted on

Substack Scraper in Node.js: Post Archives, Likes, Comments, and the Walls You'll Hit (2026)

#webscraping #javascript #node #tutorial

Substack has quietly become one of the more useful datasets on the web. If you do competitor research for a newsletter, track a niche, or want to know which topics actually get people to hit the heart button, the numbers are sitting right there on every publication.

In this post I'll build a small Node.js script that:

  1. Pages through any publication's post archive
  2. Exports title, date, author, paywall status, word count, likes, comments, and restacks to CSV
  3. Prints a quick summary (paywalled share, top posts by likes)
  4. Pulls the comment thread for a single post

No headless browser and no dependencies: Node 18+ has fetch built in. Everything below was tested against live publications in October 2026, and I'll point out the quirks I ran into along the way.

Before you start: stick to public data, keep request rates low, and read Substack's terms. The endpoints below are the ones Substack's own web app uses. They're not a documented public API, so they can change without notice.

Your options, briefly

  • RSS. Every publication has /feed. Great for "what's new", but you only get the most recent posts and no engagement numbers.
  • HTML pages. Works, but you'd be parsing markup for numbers that are already available as JSON.
  • The archive JSON (this post). The archive page loads its data from /api/v1/archive. It returns clean JSON with engagement counts, and it paginates.

Step 1: Find the archive endpoint

Open any publication's /archive page, scroll, and watch the network tab. You'll see requests like this:

https://www.lennysnewsletter.com/api/v1/archive?sort=new&offset=0&limit=12
Enter fullscreen mode Exit fullscreen mode

Each item is a post object with a lot of fields. The ones I actually use:

Field What it is
title, slug, canonical_url The basics
post_date ISO timestamp
audience everyone, only_paid, or founding
type newsletter, podcast, thread
wordcount Full length, even for paywalled posts
reaction_count, comment_count, restacks Engagement
publishedBylines Authors (guest posts have more than one)

sort=top also works and gives you the most-liked posts first, which is handy if you only care about what performed.

Step 2: Page through it politely

Here's the whole scraper. Save it as scrape.mjs:

// usage: node scrape.mjs https://www.lennysnewsletter.com [maxPosts]
import { writeFile } from "node:fs/promises";

const base = new URL(process.argv[2] ?? "https://www.lennysnewsletter.com").origin;
const maxPosts = Number(process.argv[3] ?? 60);
const PAGE = 12; // larger values return odd page sizes, see below
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function getJSON(url, tries = 4) {
  for (let i = 0; i < tries; i++) {
    const res = await fetch(url, {
      headers: { "user-agent": "substack-research-script/1.0 (contact: you@example.com)" },
      redirect: "follow",
    });
    if (res.ok) return res.json();
    if (res.status === 429 || res.status >= 500) {
      const wait = Number(res.headers.get("retry-after")) * 1000 || 2 ** i * 2000;
      console.warn(`${res.status} on ${url}, waiting ${wait}ms`);
      await sleep(wait);
      continue;
    }
    throw new Error(`${res.status} ${res.statusText} for ${url}`);
  }
  throw new Error(`gave up on ${url}`);
}

async function fetchArchive(sort = "new") {
  const posts = [];
  for (let offset = 0; posts.length < maxPosts; offset += PAGE) {
    const url = `${base}/api/v1/archive?sort=${sort}&offset=${offset}&limit=${PAGE}`;
    const page = await getJSON(url);
    if (!Array.isArray(page) || page.length === 0) break;
    posts.push(...page);
    await sleep(1000); // one request per second
  }
  return posts.slice(0, maxPosts);
}
Enter fullscreen mode Exit fullscreen mode

The retry loop backs off on 429 and 5xx, and respects Retry-After if Substack sends one. I didn't hit a rate limit at one request per second over a few dozen pages, but I wouldn't run this in parallel across hundreds of publications without a lot more care.

Step 3: Flatten and export to CSV 

const toRow = (p) => ({
  id: p.id,
  date: p.post_date?.slice(0, 10),
  title: p.title,
  type: p.type,
  audience: p.audience,
  words: p.wordcount,
  likes: p.reaction_count,
  comments: p.comment_count,
  restacks: p.restacks,
  author: p.publishedBylines?.map((b) => b.name).join(" & "),
  url: p.canonical_url,
});

const csvCell = (v) => {
  const s = v == null ? "" : String(v);
  return /[",\n]/.test(s) ? `"${s.replaceAll('"', '""')}"` : s;
};

const posts = (await fetchArchive()).map(toRow);
const cols = Object.keys(posts[0]);
const csv = [cols.join(","), ...posts.map((r) => cols.map((c) => csvCell(r[c])).join(","))].join("\n");
await writeFile("posts.csv", csv + "\n");
Enter fullscreen mode Exit fullscreen mode

Titles contain commas and quotes all the time, so the csvCell escaping isn't optional. Spreadsheet apps will happily shift every column otherwise.

Step 4: A quick summary 

const paid = posts.filter((p) => p.audience !== "everyone");
const avg = (xs) => Math.round(xs.reduce((a, b) => a + b, 0) / (xs.length || 1));

console.log(`paywalled: ${paid.length} (${Math.round((100 * paid.length) / posts.length)}%)`);
console.log(`avg likes free: ${avg(posts.filter((p) => p.audience === "everyone").map((p) => p.likes))}, paid: ${avg(paid.map((p) => p.likes))}`);
for (const p of [...posts].sort((a, b) => b.likes - a.likes).slice(0, 5))
  console.log(`  ${p.likes} likes, ${p.comments} comments  ${p.date}  ${p.title}`);
Enter fullscreen mode Exit fullscreen mode

Running it against two very different publications shows why this is interesting:

$ node scrape.mjs https://www.lennysnewsletter.com 36
paywalled: 29 (81%)
avg likes free: 335, paid: 496

$ node scrape.mjs https://astralcodexten.substack.com 24
paywalled: 2 (8%)
...
Enter fullscreen mode Exit fullscreen mode

One is mostly paywalled with comparatively few comments per post; the other is almost entirely free, and its top posts had hundreds of comments each. Same platform, completely different playbooks, and you can see it from 30 lines of output.

Step 5: Comments for a single post

Comments live on a separate endpoint keyed by the post's numeric id:

async function fetchComments(postId) {
  const url = `${base}/api/v1/post/${postId}/comments?all_comments=true&sort=best_first`;
  const { comments = [] } = await getJSON(url);
  const flat = [];
  const walk = (list, depth = 0) => {
    for (const c of list) {
      flat.push({ depth, name: c.name, likes: c.reaction_count, date: c.date, body: c.body });
      walk(c.children ?? [], depth + 1);
    }
  };
  walk(comments);
  return flat;
}
Enter fullscreen mode Exit fullscreen mode

Replies are nested under children, so a small recursive walk flattens them into rows with a depth column. Threads go deep: one popular post in my test nested 28 levels. Deleted comments stay in the tree with deleted: true and a null body (so their replies keep their place), and the number you collect won't exactly match the post's comment_count. Filter deleted rows before any text analysis.

Where this approach hits a wall

  1. The limit parameter is weird. Up to 20 gave me exactly what I asked for. 25 to 49 all returned 23 posts, and limit=100 returned a single post. Keep pages small (I use 12, which is what the web app uses) and paginate with offset.
  2. Paywalled posts are previews. For an only_paid post, /api/v1/posts/<slug> returns a body_html that stops at the paywall. On one 4,153-word post I got about 2,300 words. wordcount still reports the full length, which is useful on its own.
  3. Custom domains redirect. astralcodexten.substack.com answers with a 301 to www.astralcodexten.com. fetch follows it, but if you dedupe publications by hostname, normalize after the redirect.
  4. Cross-publication search is a different beast. Everything above works per publication. Finding publications by keyword means hitting substack.com's own internal search endpoints, which are even less stable.
  5. It's undocumented. If a field disappears, your CSV quietly fills with blanks. Check that reaction_count is a number on the first page and fail loudly if it isn't.

When you'd rather not maintain it

If you're tracking a lot of publications, or you need search, notes, and writer profiles too, a scraping API can take over the fetching. I use ScrapeAtlas* for this. Its /v1/substack/posts endpoint takes the publication url plus optional limit (1 to 100), offset, and sort (new or top), and returns a posts array with the same field names you saw above, plus a nextOffset for the next page:

async function apiPosts(pubUrl, offset = 0) {
  const q = new URLSearchParams({ url: pubUrl, limit: "50", offset: String(offset), sort: "new" });
  const res = await fetch(`https://api.scrapeatlas.com/v1/substack/posts?${q}`, {
    headers: { "x-api-key": process.env.SCRAPEATLAS_API_KEY },
  });
  const data = await res.json();
  // data.status: "success" or "partial"; data.coverage.warnings explains gaps
  return { posts: data.posts ?? [], next: data.nextOffset };
}
Enter fullscreen mode Exit fullscreen mode

Because the field names match, toRow and the CSV code work unchanged. There are also /v1/substack/publication, /v1/substack/post/comments, and /v1/substack/search endpoints if you need publication details, comment threads, or keyword search.

Wrapping up

  • A few publications, checked weekly: the archive endpoint and the script above are all you need, and it costs nothing.
  • Keep pages small and paginate with offset; don't trust large limit values.
  • Treat paywalled bodies as previews, and lean on wordcount and engagement fields instead.
  • Many publications, search, or anything you need to rely on: that's where maintaining it yourself stops being fun.

If the archive endpoint starts returning something unexpected, leave a comment and I'll update the post.

*I build it.

Top comments (0)