My Own Content Pipeline Didn’t Have My Own Content. Here’s the Backfill That Fixed It.
Three platforms, three different integration battles — and a parser bug that silently ate three of my articles.
Three platforms, three different ways to hand you back your own history. | Generated with ChatGPT
Not a member? Use this link.
60 articles lived in my analytics dashboard. 0 of them had actual content.
I built Article Forge to manage everything I publish — drafts, section-scoped AI edits, cross-posting from Medium to dev.to, analytics from all three platforms in one place.
The whole point is that my content lives in my own tool. But for months, only the posts I’d drafted inside Article Forge had their body text stored. Everything written before the tool existed was just a row in article_analytics with a URL and zero context.
One evening, I sat down to backfill. I expected one script. I wrote three.
Architecture diagram showing Article Forge | Generated with Claude
Dev.to — the platform that just works
Dev.to has a clean REST API. /api/articles/me/published paginates through everything you've ever shipped. The catch: the list endpoint returns metadata only. To get body_markdown, you fetch each article individually via /api/articles/{id}.
The script is twenty lines of “list, then loop and fetch detail.” I wrote a dedupe by publishedUrl so re-runs are safe, and linked to each new articles row back to the corresponding article_analytics row by externalId.
Twenty articles inserted, twenty analytics linked, zero failures. Five minutes of actual work.
This is the version of “content backfill” that fits in a tweet. Then there’s Substack and Medium.
Substack — the platform that wants your cookie
Substack doesn’t have a public published-list API for owners. Or rather, it does, but it’s behind cookie auth. The dashboard hits an endpoint called post_management/published that returns a paginated list of your posts, including drafts. It accepts your substack.sid cookie as the auth token.
I already had this plumbing — the analytics sync uses the same endpoint to read view/open counts. The backfill enumeration was free. What was missing was the post body. For that, you call https://{your-subdomain}.substack.com/api/v1/posts/{slug}, again with the cookie, and you get back full body_html, cover image, subtitle.
Ten new articles inserted, nine linked to existing analytics rows, seven skipped because they already existed in the DB.
The auth model is honest. You don’t pretend a cookie is an API key. It is what it is — a session, scoped to your account, that you paste into your tool’s settings page. Once you accept that, the API surface is decent.
Side-by-side comparison of three platform integration approaches | Generated with Claude
If you like reading about the small tools I build for my own workflow, I turned one of them into The Claude Code Memory Starter — a free email series you can follow along with.
Medium — the platform that hands you a zip file
Medium has no API for reading your own historical articles. You can pull the latest ten from the public RSS feed, and that’s it. If you want everything, you go to Settings → Security and apps → Download your information, wait for an email, click a link, and download a zip of HTML files.
I downloaded the zip. Eighty-eight HTML files. Three drafts (filenames start with draft_), about thirty short "responses" (Medium's name for comments — stored as posts in the export), and the rest were real articles.
I added node-html-parser as a dependency and wrote a parser. The Medium export structure is consistent — every file is an <article class="h-entry"> with <h1 class="p-name"> for the title, <section data-field="subtitle"> for the subtitle, <section data-field="body"> for the body, and <time class="dt-published"> for the date.
Dry-run reported: 53 articles to insert, 29 short responses skipped, 3 unparseable.
Three? On a stable, well-formed export format? With a parser I’d just verified worked on the other 53?
The silent bug
I opened one of the three files. The HTML was fine. Tag balance was fine. <article class="h-entry"> was right where I expected it. I dropped into a one-line node REPL:
const root = parse(html);
console.log(root.querySelectorAll('article').length); // 0
console.log(root.querySelectorAll('h1.p-name').length); // 1
Zero articles. One title.
The title element is inside the article element. If the parser found one, it should find the other. Unless the parser thinks the article closed before the title.
I scanned for what might trip a permissive HTML parser. The three “broken” files all had something in common: code blocks. Long **<pre>** blocks containing JavaScript with template literals, comparison operators, JSX-ish snippets — all the things that produce raw < characters inside what should be opaque text.
node-html-parser doesn't fully escape < inside <pre>. When it scans, it sees what looks like a tag, applies its auto-closing rules, and silently decides that the wrapping **<article>** ended way earlier than it actually did. Outer selectors miss. Inner selectors that don't depend on the wrapping element still work.
The fix is one line. Instead of:
const article = root.querySelector("article.h-entry");
const title = article?.querySelector("h1.p-name");
Do:
const title = root.querySelector("h1.p-name");
Query from the document root. Don’t assume the wrapping element is reliably reachable just because the source HTML is valid.
Re-ran the script. Three “unparseable” files dropped to zero. One was new content (got inserted), two were already in Article Forge as drafts and matched my title-fallback dedupe.
Simulated terminal output showing the dry-run results | Generated with Claude
The deduplication logic that saved me twice
I mentioned dedupe by publishedUrl earlier. That's the primary key — if a URL already exists in the articles table, skip it. Works for dev.to and Substack, where every published post has a stable URL.
Medium broke this assumption. I had eight articles already drafted in Article Forge before the backfill. They had no publishedUrl yet because they were still drafts. The ZIP export didn't know about my internal drafts, and my internal drafts didn't have Medium URLs.
The fallback: normalized title matching. Strip punctuation, lowercase, compare. It caught all eight. Not elegant, but it prevented eight duplicate rows that would have confused every downstream query.
If you’re building any kind of content sync, plan for the case where the same content exists on both sides with no shared identifier. URL-only dedupe works until it doesn’t.
What the unified data actually unlocked
Eighty-something rows added across three platforms in a few hours. The row count isn’t the point. Every analytics row now has a corresponding article row with the actual content sitting next to the view count.
The “related articles” feature in my AI-powered optimizer can finally see my entire archive instead of just the things I drafted in-tool. When I’m writing about background jobs, it knows I already published a deep dive on Inngest and can suggest an internal link.
The Medium → dev.to cross-post sync now has the full Medium body to compare against, not a 200-character RSS snippet. Drift detection between platforms actually works.
And the simplest win: I can search my own articles by content, not just by title. “Which article mentioned Promise.allSettled?" is now a database query, not a manual hunt across three browser tabs.
Three things I’ll keep doing
1. APIs aren’t the only data source. A zip of HTML is a perfectly good integration if it’s the only thing the platform offers. Write a parser, handle edge cases, and move on.
2. Dedupe by URL with a title fallback. URL-only dedupe missed eight Medium drafts that hadn’t been published yet. A normalized-title fallback caught them all. Two layers of matching cost almost nothing and prevent real problems.
3. When a permissive parser disagrees with the source, look at what’s inside the elements. Code blocks with raw < will lie to your parser long before the parser admits it's confused.
The next thing on the list: backfilling Medium’s external_url mismatch so analytics rows actually link correctly. But that's a different evening.
External Sources
- node-html-parser — the fast HTML parser that tripped on raw < in pre blocks (GitHub)
- Forem API v1 — the dev.to REST API used for the cleanest backfill
- Medium data export — official guide for downloading your information
If you want more of this in your inbox, The Claude Code Memory Starter is a short, free email series where I walk through setups like this one.
Top comments (0)