DEV Community

Ben Barazandeh
Ben Barazandeh

Posted on

Reading Medium Articles by Tag, Author and Publication With RSS (and Where It Stops)

#python #rss #webscraping #javascript

Disclosure: I'm Ben, and I built the Apify Actor mentioned at the end of this post. I charge per result for it. Everything in the tutorial part works with free tools too.

Medium has no public API for reading articles. What it does have is a set of RSS feeds that work without a login or an API key. If you want to track a topic, follow a few writers, or watch a publication, they get you most of the way there, as long as you know what's missing.

The three feeds 

https://medium.com/feed/tag/{tag}            # latest articles in a tag
https://medium.com/feed/@{username}          # one author's latest stories
https://medium.com/feed/{publication-slug}   # one publication's latest posts
Enter fullscreen mode Exit fullscreen mode

The tag is the slug from medium.com/tag/... (for example programming or artificial-intelligence). The publication slug is the part after medium.com/, such as better-programming. Publications with a custom domain still answer on the medium.com/feed/{slug} URL.

Each <item> gives you the title, link, a guid of the form https://medium.com/p/{postId}, the author in dc:creator, pubDate, atom:updated and the tags as <category> elements.

What each feed actually contains

I pulled one of each on 31 Oct 2026:

Feed Items Items with full text
tag/programming 10 0
@ev 10 10
better-programming 10 4

That table covers the three things you need to know:

  1. Every feed returns 10 items. There's no pagination: adding ?page=2 just gives you the same 10 again.
  2. Tag feeds never include the body. You get a medium-feed-snippet paragraph in <description> (about 140 characters, cut off with "…") and a "Continue reading on {Publication}" link. Use that link text to get the publication name.
  3. Author and publication feeds include the full HTML in content:encoded, except for member-only stories. Those come through with only the snippet, which is why 6 of the 10 Better Programming items had no body. In practice, "author or publication item without content:encoded" is a good member-only signal.

A few smaller details:

  • An unknown username returns HTTP 404, so you can tell a typo apart from an empty feed.
  • Links carry a ?source=rss-... tracking parameter. Strip it before you dedupe.
  • Full-text bodies include a 1×1 tracking image (medium.com/_/stat?...). Skip it when you pick a cover image.
  • Feed items have no clap or response counts. Those are only on the article page and Medium's internal GraphQL endpoint, and both sit behind Cloudflare bot protection. From a server, I got 403 "Just a moment..." on every request from datacenter IPs and only occasional success through residential proxies. Treat claps as best effort.

A parser you can run today (Python, standard library only) 

import re, html, urllib.request
import xml.etree.ElementTree as ET

NS = {"dc": "http://purl.org/dc/elements/1.1/", "content": "http://purl.org/rss/1.0/modules/content/"}
UA = {"User-Agent": "Mozilla/5.0 (compatible; medium-feed-reader/0.1)"}

def feed_url(kind, name):
    return {"tag": f"https://medium.com/feed/tag/{name}",
            "author": f"https://medium.com/feed/@{name.lstrip('@')}",
            "publication": f"https://medium.com/feed/{name}"}[kind]

def text_of(h):
    h = re.sub(r"<(figure|script|style)[\s\S]*?</\1>", " ", h or "")
    return re.sub(r"\s+", " ", html.unescape(re.sub(r"<[^>]+>", " ", h))).strip()

def read_feed(kind, name):
    req = urllib.request.Request(feed_url(kind, name), headers=UA)
    root = ET.fromstring(urllib.request.urlopen(req, timeout=30).read())
    for item in root.iter("item"):
        body = item.findtext("content:encoded", default="", namespaces=NS)
        desc = item.findtext("description", default="")
        snippet = re.search(r'medium-feed-snippet">(.*?)</p>', desc, re.S)
        text = text_of(body)
        yield {
            "id": item.findtext("guid", "").rsplit("/p/", 1)[-1],
            "title": item.findtext("title"),
            "url": item.findtext("link", "").split("?source=")[0],
            "author": item.findtext("dc:creator", namespaces=NS),
            "published": item.findtext("pubDate"),
            "tags": [c.text for c in item.findall("category")],
            "summary": text_of(snippet.group(1)) if snippet else text[:280],
            "words": len(text.split()) if text else None,
        }

if __name__ == "__main__":
    for kind, name in [("tag", "programming"), ("author", "ev"), ("publication", "better-programming")]:
        rows = list(read_feed(kind, name))
        print(kind, name, len(rows), sum(1 for r in rows if r["words"]), rows[0]["title"])
Enter fullscreen mode Exit fullscreen mode

Output on 31 Oct 2026:

tag programming 10 0 7 Python Automation Projects That Save Hours Every Week
author ev 10 10 Making “Social” Social Again
publication better-programming 10 4 Let a thousand programming publications bloom.
Enter fullscreen mode Exit fullscreen mode

For a rough reading time, divide words by 265 (an average adult reading speed in words per minute) and round up.

The same idea in JavaScript (Node 18+) 

const res = await fetch('https://medium.com/feed/@ev', { headers: { 'User-Agent': 'medium-feed-reader/0.1' } });
const xml = await res.text();
const pick = (s, t) => (s.match(new RegExp(`<${t}[^>]*>([\\s\\S]*?)</${t}>`)) || [])[1]?.replace(/<!\[CDATA\[|\]\]>/g, '').trim();
const items = xml.split('<item>').slice(1).map((it) => ({
  title: pick(it, 'title'),
  url: pick(it, 'link')?.split('?source=')[0],
  author: pick(it, 'dc:creator'),
  tags: [...it.matchAll(/<category><!\[CDATA\[(.*?)\]\]><\/category>/g)].map((m) => m[1]),
  hasFullText: it.includes('<content:encoded>'),
}));
console.log(items.length, items[0]);
Enter fullscreen mode Exit fullscreen mode

Getting more than 10 articles

Since the feeds don't page, there are two honest options:

  • Widen: more tags, authors and publications per run. The same story often shows up in several, so dedupe on the post ID from guid.
  • Poll: run it on a schedule (hourly for busy tags, daily for authors) and keep every ID you've seen. A history builds up quickly.

Be polite: one request every second or so is plenty, back off on 429 and 5xx, and send a real User-Agent.

If you don't want to maintain it

I wrapped this in an Apify Actor, Medium Scraper: Articles by Tag, Author and Publication. It takes tags, usernames, publication slugs and article URLs, dedupes across them, and outputs one clean record per article: title, URL, author, publication, dates, tags, summary or full text, word count, reading time and a member-only flag. Claps and responses are an opt-in, best-effort extra. It costs $0.001 per article, and you can set a maximum cost per run. My QA run (one tag, one author, one publication) returned 30 articles in about 7 seconds. It has the same 10-per-feed limit described above, because that's Medium's limit.

Run-ready link: open it in Apify Console with an example input already filled in ({"tags": ["artificial-intelligence"], "maxItems": 30}). Click Start, or change the input first.

What do you track on Medium: topics, writers, or competitors' publications? Tell me in the comments.

Top comments (1)

Collapse
 
junfengzhou01 profile image
Junfeng Zhou •

With only 10 items per feed, polling often matters. Sending ETag or If-Modified-Since and deduping by guid keeps those frequent polls cheap.