<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Alejandro Robledo</title>
    <description>The latest articles on DEV Community by Alejandro Robledo (@alejandro_robledo_e838933).</description>
    <link>https://dev.to/alejandro_robledo_e838933</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4246655%2Ffddb4e82-70aa-4ec5-ad6b-3639cd174938.png</url>
      <title>DEV Community: Alejandro Robledo</title>
      <link>https://dev.to/alejandro_robledo_e838933</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/alejandro_robledo_e838933"/>
    <language>en</language>
    <item>
      <title>How to export Substack posts to clean Markdown for RAG (no browser needed)</title>
      <dc:creator>Alejandro Robledo</dc:creator>
      <pubDate>Sun, 15 Nov 2026 23:28:44 +0000</pubDate>
      <link>https://dev.to/alejandro_robledo_e838933/how-to-export-substack-posts-to-clean-markdown-for-rag-no-browser-needed-4f5g</link>
      <guid>https://dev.to/alejandro_robledo_e838933/how-to-export-substack-posts-to-clean-markdown-for-rag-no-browser-needed-4f5g</guid>
      <description>&lt;p&gt;If you have tried to feed a Substack newsletter into an LLM or a RAG pipeline, you have probably hit the same three problems I did:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The post HTML is full of subscribe buttons, share widgets and nested image wrappers that pollute your chunks.&lt;/li&gt;
&lt;li&gt;Substack's archive endpoint no longer accepts &lt;code&gt;offset&lt;/code&gt;. It returns HTTP 400, and you have to follow a cursor instead.&lt;/li&gt;
&lt;li&gt;The archive list does not include the post body, so you need a second request per post.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I got tired of rewriting this, so I built a small tool for it and published it on Apify. Disclosure: I am the author.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://apify.com/gunpark/substack-scraper" rel="noopener noreferrer"&gt;Substack Scraper - Markdown Posts, Search &amp;amp; Comments&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It uses plain HTTP requests against Substack's public JSON API. No headless browser and no residential proxies, so runs are fast and cheap.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Give it a publication
&lt;/h2&gt;

&lt;p&gt;Open the Actor, paste this input and press Start:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"publications"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"astralcodexten"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"maxPosts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"sinceDate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"30 days"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;publications&lt;/code&gt; accepts a name, a &lt;code&gt;name.substack.com&lt;/code&gt; URL or a custom domain. You can also set &lt;code&gt;searchQuery&lt;/code&gt; to find publications about a topic.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Get Markdown back
&lt;/h2&gt;

&lt;p&gt;Each post is one dataset item. The &lt;code&gt;markdown&lt;/code&gt; field starts with the title as an H1, so every document is self-contained when you chunk it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"AI Company Employees Should Unionize"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.astralcodexten.com/p/ai-company-employees-should-unionize"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"publishedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-12T23:09:15.486Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"authors"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Scott Alexander"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"handle"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"astralcodexten"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"wordCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1273&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"# AI Company Employees Should Unionize\n\n**I.**\n\nShould safety-conscious employees at AI companies quit in protest, or stay and fight? ..."&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Headings, lists, code blocks, images with captions and footnotes are kept as standard Markdown. Subscribe and share widgets are removed.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Call it from Python
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"gunpark/substack-scraper"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"publications"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"astralcodexten"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"maxPosts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;post&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;post&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;post&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From here you can pass &lt;code&gt;post["markdown"]&lt;/code&gt; straight to your text splitter and embedding model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping an index up to date
&lt;/h2&gt;

&lt;p&gt;Turn on &lt;code&gt;incremental&lt;/code&gt; and schedule the Actor. The first run sets a baseline; later runs return only posts published since the previous run, so you never embed the same post twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Good to know
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Paywalled posts only expose a public preview. They are skipped by default, or delivered flagged as incomplete if you ask for previews.&lt;/li&gt;
&lt;li&gt;Comments are optional (&lt;code&gt;includeComments&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Pricing is pay per delivered post: $1.20 per 1,000 posts. Failed requests, duplicates and skipped paywalled posts are not charged.&lt;/li&gt;
&lt;li&gt;The posts belong to their authors. Use the output in ways their terms and copyright allow.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If something breaks or you need a field that is missing, open an issue on the Actor page and I will look at it.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>webscraping</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
