DEV Community: Delta Tools

How I Built a Substack New-Post Monitor (and the Dedup Problem Nobody Warns You About)

Delta Tools — Sun, 04 Oct 2026 00:33:16 +0000

TL;DR: Fetching a Substack publication's latest posts took an afternoon. Making the monitor not alert twice on the same post — across runs, crashes, and machine moves — was the actual project. Here's the naive version, the three ways it broke, and the dedup design that finally held. The finished tool is our Substack New-Post Monitor on the Apify Store.

The naive version (afternoon one)

The job sounded trivial: watch a Substack publication, tell me when a new post appears. Substack even hands you the data — every publication has an RSS feed at <publication>.substack.com/feed. Parse it, compare against last time, alert on the difference.

import feedparser

feed = feedparser.parse("https://example.substack.com/feed")
latest = feed.entries[0]
print(f"Latest: {latest.title} — {latest.link}")

Done, right? It printed the latest post. I wired it to a Slack webhook, set a 15-minute cron, and felt extremely productive for about a day.

Then the double alerts started.

Break #1: "What did I already see?" has no obvious home

The first version kept seen_ids.json next to the script. It worked until I moved the script to a different machine and every post re-alerted — the state file didn't come along. Then I forgot the file on a deploy and got a fun burst of "NEW POST" messages for articles from 2023.

The lesson: the state file is the product, and it needs to live where the compute lives. On Apify, that meant a named key-value store shared across scheduled runs — state travels with the actor, not with my laptop. Anywhere else, it means a real database or at minimum a state file on the same always-on host as the cron job, backed up.

Break #2: The crash between "detect" and "save"

This one was subtler. The original loop was:

Fetch feed
Diff against seen IDs
Send alerts for new posts
Save updated seen IDs

If the process died between 3 and 4 — machine reboot, OOM kill, deploy — the next run re-detected the same posts and re-alerted. I "fixed" it by saving state before sending alerts, which flipped the failure mode: crash between save and alert meant silently missing posts, which is worse. A missed alert is invisible; a duplicate is just annoying.

The correct order is a tiny transaction: fetch → diff → persist the new seen-set first → send alerts → if alerting fails, the next run sees the posts as already-seen... no wait, that drops them again.

What actually works: persist seen IDs and a pending-alerts queue atomically, then drain the queue. On startup, drain any leftover queue before doing new work. It's a miniature outbox pattern, and yes, it's overkill for a side project — until the one time your 3am deploy eats the alert for the post you were actually waiting for.

For the Apify version, the platform's dataset + key-value store gives atomic-ish primitives that make this straightforward. The principle is the same anywhere: never let "I noticed it" and "I told someone" be separable without a recovery path.

Break #3: Identity is harder than it looks

What identifies "the same post" across polls? My first attempt used the post URL. Fine — until Substack's feed briefly served a post with a different canonical URL during an update, and it re-alerted. Then I tried titles; Substack authors edit titles, and every edit re-alerted.

RSS entries have GUIDs (entry.id in feedparser), which are supposed to be stable identifiers. They mostly are. My prototype scheme: GUID primary, URL as fallback, and never the title. Two identifiers agreeing beats one identifier you trust.

The shipped version simplified this further: Substack's archive API hands you stable post IDs, so the dedup key is just the post ID. Same principle, less machinery.

What the finished actor does

The Substack New-Post Monitor on the Apify Store is this design, packaged:

You give it a publication URL and a schedule. It fetches via Substack's public archive API, dedups on Substack's stable post IDs, persists its snapshot in a named key-value store (shared across scheduled runs — the lesson from Break #1, applied), and emits only genuinely new posts to the output dataset.
New posts land in the dataset ready to wire to alerts via Apify webhooks; a check that finds nothing costs a fraction of a cent.
It passed 7/7 of my stress tests: duplicate suppression across runs, crash recovery, multi-publication state isolation, feed hiccups.

I'm not claiming it's the only way to do this. The 20-line script from my alerts tutorial plus a seen_ids.json on a Raspberry Pi is a completely legitimate setup — I ran exactly that for weeks. The actor is for when you'd rather not be the ops team for your own alerts.

The real lesson

Every "monitor X for changes" tool is two tools: a fetcher (afternoon) and a rememberer (the actual project). The fetcher is the demo. The rememberer is the product. If you're building any kind of change-detection — price trackers, job alert bots, listing monitors — budget your time accordingly: 20% fetching, 80% state, identity, and crash recovery.

FAQ

Why not just use Substack's email subscriptions?
They work, but they're inbox-bound and all-or-nothing per publication. A monitor routes to Slack/Discord/webhooks and watches without subscribing.

Does it handle paid publications?
Only public/free posts via the public feed — same limitation as any RSS-based approach. Paywalled content needs an authenticated session, which is a different tool.

How much does it cost to run?
On Apify's pricing: a scheduled check that finds nothing costs a fraction of a cent. You pay meaningfully only when there's actually a new post to report. (Exact numbers depend on Apify's current compute pricing.)

Could I build this myself?
Absolutely — the tutorial version is 20 lines plus a state file. Build it if you enjoy the ops; use the actor if you don't.

How to Get Notified When a Substack Publication Posts Something New

Delta Tools — Sun, 04 Oct 2026 00:30:36 +0000

TL;DR: Every Substack publication has a hidden RSS feed at publicationname.substack.com/feed. You can watch it with an RSS reader, poll it with a 20-line Python script, or — if you want alerts without running infrastructure — use a scheduled monitor like our Substack New-Post Monitor. Below are all three methods, working code included.

I follow about a dozen Substack publications. For months my system was "remember to check them," which is to say I missed half of everything. Substack has email subscriptions, but I didn't want a dozen newsletters clogging my inbox — I wanted a ping only when something new went live, routed where I actually look.

There are three solid ways to do this. They go from simplest to most automated.

Method 1: The hidden RSS feed + a reader (2 minutes, free)

This is the thing most people don't know: every Substack publication exposes an RSS feed at:

https://<publication>.substack.com/feed

So if you follow stratechery.substack.com, the feed is https://stratechery.substack.com/feed. It contains the latest posts with titles, links, and timestamps. It just works — no API key, no scraping.

Point any RSS reader at it (Feedly, Inoreader, NetNewsWire, even an old-school reader). You'll see new posts alongside everything else you follow. Zero code, zero cost.

Downside: it's pull, not push. You see new posts when you open the reader, not the moment they go live. If "the moment it happens" matters — job alerts, deal alerts, time-sensitive analysis — keep reading.

Method 2: Poll the feed with a script (20 lines of Python)

If you want actual notifications (Slack, Discord, email, SMS), poll the feed on a schedule and diff against what you've already seen. Here's the whole thing:

import feedparser, json, os, time

FEED_URL = "https://example.substack.com/feed"
SEEN_FILE = "seen_ids.json"

seen = set(json.load(open(SEEN_FILE))) if os.path.exists(SEEN_FILE) else set()

feed = feedparser.parse(FEED_URL)
new_posts = [e for e in feed.entries if e.id not in seen]

for post in new_posts:
    print(f"NEW: {post.title} — {post.link}")
    # send_to_slack(f"New post: {post.title}\n{post.link}")

seen.update(e.id for e in feed.entries)
json.dump(sorted(seen), open(SEEN_FILE, "w"))

Run it on a cron every 15 minutes. The seen_ids.json file is the entire "database" — and it's also the entire point. Dedup is the whole product. Without remembering what you've already seen, every poll re-alerts on the same posts. With it, you get exactly one alert per new post.

This works fine for one or two publications on a machine that's always on (a Raspberry Pi, a cheap VPS, or just your laptop if you don't mind it sleeping through alerts).

Downsides at scale: run it for ten publications and you're managing ten cron jobs and ten state files. Move machines and the state file doesn't come with you — hello, duplicate alerts for everything. And if the script crashes between "detect" and "save state," you either miss posts or double-alert. I learned all of this the annoying way, which is why method 3 exists.

Method 3: A scheduled monitor that handles the boring parts (no infrastructure)

This is what I ended up building: a small Apify actor that watches a Substack publication and reports new posts. You give it the publication URL and a schedule. New posts land in the output dataset, which you can wire to Slack, email, or webhooks through Apify's integrations — no cron jobs or state files to babysit. A check that finds nothing new costs a fraction of a cent.

The Substack New-Post Monitor is the one I run myself. One job per tool, priced per result — a scheduled check that finds nothing new costs almost nothing.

I'm not going to pretend it's the only option. If you like running your own infra, method 2 is genuinely fine and the code above is yours to keep. The actor exists for people who'd rather not babysit cron jobs and state files.

Which method should you pick?

Just want to read new posts in one place? Method 1. RSS reader, done in two minutes.
Want push alerts and don't mind a tiny bit of ops? Method 2. The script above, a cron job, somewhere always-on.
Want alerts with zero infrastructure? Method 3. Set it once, get pinged.

FAQ

Does this work for paid Substack publications?
The public /feed only includes free posts. Paid-only content behind the paywall isn't in the public feed — for that you'd need to be subscribed and logged in, which is a different (harder) problem.

How fast are the alerts?
Method 1 is whenever you open your reader. Method 2 is your cron interval (15 minutes is polite; don't hammer it every 60 seconds). Method 3 runs on whatever schedule you set.

Will Substack block polling?
At reasonable intervals (every 10–15 minutes), no. It's a standard RSS feed designed to be polled. Don't poll every minute from five IPs — that's how you get rate-limited anywhere.

Can I watch multiple publications at once?
Method 1: add multiple feeds to the reader. Method 2: loop over a list of feed URLs (one state file per publication, or namespace the IDs). Method 3: the actor's input takes a list of publications in a single run.

What about Substack's own email notifications?
They work, but they're all-or-nothing per publication and they land in your inbox mixed with everything else. These methods give you routing control — Slack, Discord, webhook, SMS — and let you watch publications without subscribing to their emails.