Extract Plain Text from Medium Posts for RAG and Search Indexes

Sebastian Casvean — Sat, 30 May 2026 09:15:17 +0000

Chunk clean article content for embeddings, summarization, and full-text search—skip nav, clap bars, and scripts.

Extract Plain Text from Medium Posts for RAG and Search Indexes

HTML embeds are for humans; plain text is for chunking, embeddings, and summarization. One call should return body text without nav, clap bars, or script tags.

Tool outcome: ingest-medium-article.ts → chunked documents in your vector DB.

Pipeline

Discover ids via user feed or search.
GET /article/{id}/content → plain text.
Optional: GET /article/{id} for title, tags, author metadata.
Chunk → embed → upsert vector store.
Query in your chat UI or internal search.

Ingest script

const API = 'https://api.zenndra.com';
const headers = { Authorization: `Bearer ${process.env.ZENNDRA_API_KEY}` };

export async function fetchArticleText(articleId) {
  const [contentRes, metaRes] = await Promise.all([
    fetch(`${API}/article/${articleId}/content`, { headers }),
    fetch(`${API}/article/${articleId}`, { headers }),
  ]);

  const { content } = await contentRes.json();
  const meta = await metaRes.json();

  return {
    id: articleId,
    title: meta.title,
    tags: meta.tags,
    text: content,
  };
}

export function chunkText(text, { size = 800, overlap = 100 } = {}) {
  const words = text.split(/\s+/);
  const chunks = [];
  for (let i = 0; i < words.length; i += size - overlap) {
    chunks.push(words.slice(i, i + size).join(' '));
  }
  return chunks.filter(Boolean);
}

Wire chunkText to OpenAI embeddings, Ollama, or your host’s model—swap the vector client, keep the ingest shape.

Chunking tips

Include title + tags in the embedding preamble for better retrieval.
Store article_id and chunk_index in metadata for citations.
Deduplicate re-ingest with content hash if posts are edited rarely.

Compliance (non-optional)

Respect Medium’s Terms of Service and author rights.
Many teams only index their own posts or licensed partners.
Do not expose paywalled or member-only content through public bots without permission.

For human-readable syndication, see embed articles—different threat model than LLM training.

Keywords

medium plain text api, medium rag pipeline, medium embeddings, medium article content extraction, llm medium.

Medium API in 2026: Scraping vs Official API vs Zenndra (Honest Comparison)

Sebastian Casvean — Fri, 29 May 2026 19:24:27 +0000

If you've ever tried to pull data from Medium programmatically, you already know the pain.
Medium retired their official API years ago. What's left is a graveyard of Stack Overflow threads, half-working scrapers, and a handful of third-party services with questionable reliability.
This article breaks down your three real options in 2026 — honestly, with code — so you can pick what actually fits your use case.

The Three Options

DIY scraping — build and maintain your own scraper
Medium's official API — limited, but it exists for authenticated flows
Zenndra — a third-party REST API built after Medium killed theirs

Let's go through each one.

Option 1: DIY Scraping
This is where most developers start. You fire up a headless browser, parse the HTML, and call it a day.
The appeal: free, no dependency on a third party, full control.
The reality: Medium's frontend changes constantly. Your scraper will break — not if, when. You'll spend more time maintaining the scraper than building the product you actually care about.
A typical flow looks like this:
bash# You'd need a headless browser or parse the raw HTML

Medium serves most content client-side, so curl alone won't cut it

curl "https://medium.com/@username" \
-H "User-Agent: Mozilla/5.0"

Good luck parsing that response reliably

The HTML you get back is a mess of React-rendered markup with no stable selectors. And if Medium detects scraping patterns, you get rate-limited or blocked entirely.
When to choose this: Almost never, unless you have very specific needs that no API covers and you're prepared to maintain it long-term.

Option 2: Medium's Official API
Medium does have an official API — but it's extremely limited.
What it can do:

Create posts (via OAuth)
Get your own profile info
Publish to publications you're a contributor to

What it cannot do:

Fetch any arbitrary user's articles
Search content
Get follower graphs
Read publications you don't own
Access tags, trending feeds, or article stats

bash# The official API — only works for authenticated write operations
curl "https://api.medium.com/v1/me" \
-H "Authorization: Bearer YOUR_MEDIUM_TOKEN"

Returns YOUR profile only — not anyone else's

If you're building anything read-heavy — a dashboard, a content aggregator, an AI pipeline, a research tool — the official API is essentially useless for you.
When to choose this: Only if you need to publish content to Medium programmatically and nothing else.

Option 3: Zenndra
Zenndra is an unofficial REST API built specifically to fill the gap Medium left. 42 endpoints covering articles, users, publications, tags, feeds, and search — all returning clean, stable JSON.
bash# Fetch any user's profile
curl "https://api.zenndra.com/v1/users/@username" \
-H "Authorization: Bearer YOUR_ZENNDRA_KEY"

Response: 200 OK in ~142ms

{

"username": "username",

"name": "...",

"followers": 4201,

"bio": "..."

}

bash# Get a user's articles
curl "https://api.zenndra.com/v1/user/{user_id}/articles" \
-H "Authorization: Bearer YOUR_ZENNDRA_KEY"
bash# Search articles
curl "https://api.zenndra.com/v1/search/articles?query=machine+learning" \
-H "Authorization: Bearer YOUR_ZENNDRA_KEY"
bash# Trending feed for a tag
curl "https://api.zenndra.com/v1/recommended_feed/python" \
-H "Authorization: Bearer YOUR_ZENNDRA_KEY"
The free tier gets you started with no credit card required. Paid plans start at €3.90 for 1 million requests — which is about as cheap as this category gets.
When to choose this: Any time you need read access to Medium content at scale — dashboards, newsletters, AI/LLM pipelines, research tools, content aggregators.

Side-by-Side Comparison
DIY ScrapingOfficial APIZenndraSetup timeDays / weeksMinutesMinutesRead any user✅ (fragile)❌✅Search content✅ (fragile)❌✅Stable endpoints❌✅✅LatencyUnpredictableFast~142ms P50Maintenance burdenHighNoneNoneCostYour timeFreeFrom €3.90/moRate limitsDependsStrictGenerousProduction-readyRiskyLimited useYes

The Verdict
If you're building anything that needs to read Medium content — just use Zenndra. The scraping path looks attractive until you've rebuilt your parser for the third time after a Medium frontend update.
The official API is fine for publishing workflows, but it's not a real option for read-heavy use cases.
Zenndra has a free tier with a playground where you can test every endpoint before writing a single line of integration code. Worth 5 minutes of your time: zenndra.com

Have a use case this doesn't cover? Drop it in the comments — happy to help.

DEV Community: Sebastian Casvean

Extract Plain Text from Medium Posts for RAG and Search Indexes

Extract Plain Text from Medium Posts for RAG and Search Indexes

Pipeline

Ingest script

Chunking tips

Compliance (non-optional)

Keywords

Further reading

Medium API in 2026: Scraping vs Official API vs Zenndra (Honest Comparison)

Medium serves most content client-side, so curl alone won't cut it

Good luck parsing that response reliably

Returns YOUR profile only — not anyone else's

Response: 200 OK in ~142ms

{

"username": "username",

"name": "...",

"followers": 4201,

"bio": "..."

}