TL;DR: Fetching a Substack publication's latest posts took an afternoon. Making the monitor not alert twice on the same post — across runs, crashes, and machine moves — was the actual project. Here's the naive version, the three ways it broke, and the dedup design that finally held. The finished tool is our Substack New-Post Monitor on the Apify Store.
The naive version (afternoon one)
The job sounded trivial: watch a Substack publication, tell me when a new post appears. Substack even hands you the data — every publication has an RSS feed at <publication>.substack.com/feed. Parse it, compare against last time, alert on the difference.
import feedparser
feed = feedparser.parse("https://example.substack.com/feed")
latest = feed.entries[0]
print(f"Latest: {latest.title} — {latest.link}")
Done, right? It printed the latest post. I wired it to a Slack webhook, set a 15-minute cron, and felt extremely productive for about a day.
Then the double alerts started.
Break #1: "What did I already see?" has no obvious home
The first version kept seen_ids.json next to the script. It worked until I moved the script to a different machine and every post re-alerted — the state file didn't come along. Then I forgot the file on a deploy and got a fun burst of "NEW POST" messages for articles from 2023.
The lesson: the state file is the product, and it needs to live where the compute lives. On Apify, that meant a named key-value store shared across scheduled runs — state travels with the actor, not with my laptop. Anywhere else, it means a real database or at minimum a state file on the same always-on host as the cron job, backed up.
Break #2: The crash between "detect" and "save"
This one was subtler. The original loop was:
- Fetch feed
- Diff against seen IDs
- Send alerts for new posts
- Save updated seen IDs
If the process died between 3 and 4 — machine reboot, OOM kill, deploy — the next run re-detected the same posts and re-alerted. I "fixed" it by saving state before sending alerts, which flipped the failure mode: crash between save and alert meant silently missing posts, which is worse. A missed alert is invisible; a duplicate is just annoying.
The correct order is a tiny transaction: fetch → diff → persist the new seen-set first → send alerts → if alerting fails, the next run sees the posts as already-seen... no wait, that drops them again.
What actually works: persist seen IDs and a pending-alerts queue atomically, then drain the queue. On startup, drain any leftover queue before doing new work. It's a miniature outbox pattern, and yes, it's overkill for a side project — until the one time your 3am deploy eats the alert for the post you were actually waiting for.
For the Apify version, the platform's dataset + key-value store gives atomic-ish primitives that make this straightforward. The principle is the same anywhere: never let "I noticed it" and "I told someone" be separable without a recovery path.
Break #3: Identity is harder than it looks
What identifies "the same post" across polls? My first attempt used the post URL. Fine — until Substack's feed briefly served a post with a different canonical URL during an update, and it re-alerted. Then I tried titles; Substack authors edit titles, and every edit re-alerted.
RSS entries have GUIDs (entry.id in feedparser), which are supposed to be stable identifiers. They mostly are. My prototype scheme: GUID primary, URL as fallback, and never the title. Two identifiers agreeing beats one identifier you trust.
The shipped version simplified this further: Substack's archive API hands you stable post IDs, so the dedup key is just the post ID. Same principle, less machinery.
What the finished actor does
The Substack New-Post Monitor on the Apify Store is this design, packaged:
- You give it a publication URL and a schedule. It fetches via Substack's public archive API, dedups on Substack's stable post IDs, persists its snapshot in a named key-value store (shared across scheduled runs — the lesson from Break #1, applied), and emits only genuinely new posts to the output dataset.
- New posts land in the dataset ready to wire to alerts via Apify webhooks; a check that finds nothing costs a fraction of a cent.
- It passed 7/7 of my stress tests: duplicate suppression across runs, crash recovery, multi-publication state isolation, feed hiccups.
I'm not claiming it's the only way to do this. The 20-line script from my alerts tutorial plus a seen_ids.json on a Raspberry Pi is a completely legitimate setup — I ran exactly that for weeks. The actor is for when you'd rather not be the ops team for your own alerts.
The real lesson
Every "monitor X for changes" tool is two tools: a fetcher (afternoon) and a rememberer (the actual project). The fetcher is the demo. The rememberer is the product. If you're building any kind of change-detection — price trackers, job alert bots, listing monitors — budget your time accordingly: 20% fetching, 80% state, identity, and crash recovery.
FAQ
Why not just use Substack's email subscriptions?
They work, but they're inbox-bound and all-or-nothing per publication. A monitor routes to Slack/Discord/webhooks and watches without subscribing.
Does it handle paid publications?
Only public/free posts via the public feed — same limitation as any RSS-based approach. Paywalled content needs an authenticated session, which is a different tool.
How much does it cost to run?
On Apify's pricing: a scheduled check that finds nothing costs a fraction of a cent. You pay meaningfully only when there's actually a new post to report. (Exact numbers depend on Apify's current compute pricing.)
Could I build this myself?
Absolutely — the tutorial version is 20 lines plus a state file. Build it if you enjoy the ops; use the actor if you don't.
Top comments (2)
The detour in Break #2 ("no wait, that drops them again") is the most honest part of the post. That's exactly how it goes in real life.
One gap the outbox still leaves: draining the queue is at-least-once. If the process dies after Slack accepts the message but before the item is marked sent, the next run posts it again. Two ways to narrow it:
How does the actor treat a post that Substack republishes with a new date? Same ID, so silent, or an "updated" event?
You're right about the at-least-once drain — that's a real gap I glossed over. The chat.postMessage + ts trick is a good one; persisting the ts right after the API call (before any retry logic) means a crash between accept and store becomes recoverable instead of a guaranteed dup. The Discord ?wait=true equivalent is news to me, thanks.
One honest caveat from the actor side: the shipped version doesn't run its own outbox drain for alerts. New posts go into the dataset, and alerts fan out through Apify webhooks, so delivery retry semantics are Apify's, not mine. The outbox story in the post is really aimed at the DIY version — with the actor, the parts you own are fewer.
On republishes: same post ID, so it stays silent. The actor keys on Substack's post ID with no content hashing, so a post that gets a new date (or a heavy edit) doesn't produce an "updated" event. That's a deliberate call — it answers "what's new," not "what changed." If you need change detection on existing posts, that's a different tool.