DEV Community: stabem

I built a free web scraping API with 5,000 requests/month

stabem — Sat, 14 Feb 2026 21:20:09 +0000

Hey DEV community! 👋

Six months ago, I was building a RAG pipeline for a side project and hit a wall that every AI developer knows too well: getting clean content from the web is painful.

YouTube transcripts? You need the YouTube Data API v3 (with quotas). Web pages? Beautiful Soup works, but half the web is JavaScript-rendered now. Twitter threads? Don't even get me started on the API pricing.

So I built ContentAPI — a single API endpoint that extracts clean, structured content from any URL. YouTube videos, web pages, Twitter/X threads, Reddit posts — you throw a URL at it, and it returns clean markdown or text.

Here's what I learned building it, and how you can use it for free.

The Problem: Every Content Source Is Different

When you're building AI-powered apps — chatbots, RAG systems, content analyzers — you spend 80% of your time on data ingestion. Each source has its own quirks:

YouTube: Rate limits, API keys, no transcripts for some videos
Web pages: JavaScript rendering, paywalls, cookie banners, cluttered HTML
Twitter/X: Expensive API, thread reconstruction, media handling
Reddit: Rate limiting, nested comment trees, deleted content

I wanted one endpoint to rule them all.

The Solution: One URL In, Clean Content Out

Here's the simplest example — extracting a web page:

import requests

response = requests.get(
    "https://api.getcontentapi.com/api/v1/web",
    params={"url": "https://example.com/article"},
    headers={"X-API-Key": "your_api_key"}
)

data = response.json()
print(data["data"]["title"])
print(data["data"]["content"])  # Clean markdown
print(data["data"]["word_count"])

That's it. No parsing HTML, no handling JavaScript, no dealing with cookie banners.

YouTube Transcripts Without the YouTube API

This is probably the feature I'm most proud of. Getting YouTube transcripts is notoriously painful:

# Extract transcript from any YouTube video
response = requests.get(
    "https://api.getcontentapi.com/api/v1/youtube/transcript",
    params={"url": "https://youtube.com/watch?v=dQw4w9WgXcQ"},
    headers={"X-API-Key": "your_api_key"}
)

data = response.json()["data"]
print(f"Title: {data['title']}")
print(f"Channel: {data['channel']}")
print(f"Duration: {data['duration']}s")
print(f"Transcript: {data['full_text'][:200]}...")

What makes it special:

No YouTube API key needed — ContentAPI handles it
AI fallback — if no captions exist, we use Whisper to transcribe the audio
Multi-language support — request transcripts in specific languages
Timestamped segments — get start, duration, and text for each segment

The Python SDK

I also built a Python SDK to make things even simpler:

pip install contentapi

from contentapi import ContentAPI

client = ContentAPI(api_key="your_api_key")

# Extract from any URL — auto-detects the source type
result = client.extract("https://youtube.com/watch?v=...")
print(result.title)
print(result.content)

# Batch processing — up to 10 URLs at once
results = client.batch([
    "https://youtube.com/watch?v=abc",
    "https://example.com/blog-post",
    "https://reddit.com/r/python/comments/xyz"
])

Structured Data Extraction

Need specific fields from a page? Use the /extract endpoint with a schema:

response = requests.post(
    "https://api.getcontentapi.com/api/v1/extract",
    headers={"X-API-Key": "your_api_key"},
    json={
        "url": "https://example.com/product-page",
        "schema": {
            "title": "string",
            "price": "number",
            "features": ["string"],
            "in_stock": "boolean"
        }
    }
)

# Returns structured JSON matching your schema
print(response.json()["data"]["extracted"])
# {"title": "...", "price": 29.99, "features": [...], "in_stock": true}

This is AI-powered extraction — it understands the page content and maps it to your schema.

Website Crawling

Need to ingest an entire documentation site? The crawl endpoint handles it:

# Start a crawl
response = requests.post(
    "https://api.getcontentapi.com/api/v1/crawl",
    headers={"X-API-Key": "your_api_key"},
    json={
        "url": "https://docs.example.com",
        "max_pages": 50,
        "max_depth": 3,
        "include_patterns": ["/docs/*"],
        "format": "markdown"
    }
)

crawl_id = response.json()["data"]["crawl_id"]

# Poll for results (or use webhook_url for push notifications)
import time
while True:
    status = requests.get(
        f"https://api.getcontentapi.com/api/v1/crawl/{crawl_id}",
        headers={"X-API-Key": "your_api_key"}
    ).json()["data"]

    if status["status"] in ["completed", "partial", "failed"]:
        break
    time.sleep(5)

# Process results
for page in status["results"]:
    print(f"{page['title']}: {page['word_count']} words")

What I Learned Building This

1. Web scraping is an arms race

Every month, sites change their anti-bot measures. Building a reliable extraction service means constantly adapting. We use a mix of headless browsers, custom parsers, and fallback strategies.

2. The free tier matters

I wanted ContentAPI to be genuinely useful for hobbyists and indie devs. The free tier gives you 5,000 requests/month — enough to build and prototype real applications. No credit card required.

3. AI makes extraction dramatically better

Traditional scraping is brittle — one HTML change breaks everything. Using LLMs for structured extraction means the API understands content, not just DOM structure.

4. Developers want simplicity

The most popular feature request was "just give me one endpoint for everything." That's why the batch endpoint exists — throw any mix of URLs at it and get clean content back.

The MCP Server

For AI developers using Claude, Cursor, or Windsurf — there's also an MCP (Model Context Protocol) server:

npx contentapi-mcp-server

This lets AI assistants directly extract web content during conversations. Your AI can read any URL, get YouTube transcripts, and crawl websites — all through natural language.

Try It Free

Website: getcontentapi.com
Docs: getcontentapi.com/docs
Python SDK: pip install contentapi
MCP Server: npx contentapi-mcp-server
Free tier: 5,000 requests/month, no credit card

I'd love to hear your feedback. What content sources would you want supported next? Drop a comment below! 👇

Extract YouTube Transcripts with AI Fallback — No YouTube API Key Needed

stabem — Sat, 14 Feb 2026 21:19:25 +0000

Need to extract transcripts from YouTube videos programmatically? If you've tried before, you know it's surprisingly hard:

The YouTube Data API doesn't actually give you transcripts (just metadata)
youtube-transcript-api (Python package) breaks regularly and doesn't handle all videos
Many videos don't have captions at all — no amount of scraping helps
Rate limits, quotas, and API key management add friction

I built a solution that handles all of these cases with a single API call — including an AI fallback for videos without captions.

The Quick Version

curl "https://api.getcontentapi.com/api/v1/youtube/transcript?url=https://youtube.com/watch?v=dQw4w9WgXcQ" \
  -H "X-API-Key: your_api_key"

That's it. You get back:

{
  "success": true,
  "data": {
    "title": "Rick Astley - Never Gonna Give You Up",
    "channel": "Rick Astley",
    "duration": 212,
    "language": "en",
    "word_count": 284,
    "full_text": "We're no strangers to love...",
    "transcript": [
      {"start": 0.0, "duration": 4.2, "text": "We're no strangers to love"},
      {"start": 4.2, "duration": 3.8, "text": "You know the rules and so do I"},
      ...
    ]
  }
}

How It Works

ContentAPI uses a multi-strategy approach to get transcripts:

Strategy 1: Native Captions

First, we check if the video has official captions (manual or auto-generated). This covers ~90% of videos and is the fastest method.

Strategy 2: Community Captions

Some videos have community-contributed subtitles. We check for these as a fallback.

Strategy 3: AI Transcription (Whisper)

For the remaining ~10% of videos with no captions at all, we extract the audio and run it through OpenAI's Whisper model. This works on any video in any language.

All of this happens automatically — you just make one API call.

Python Examples

Basic Transcript Extraction

import requests

API_KEY = "your_api_key"
BASE_URL = "https://api.getcontentapi.com/api/v1"

def get_transcript(video_url):
    response = requests.get(
        f"{BASE_URL}/youtube/transcript",
        params={"url": video_url},
        headers={"X-API-Key": API_KEY}
    )
    return response.json()["data"]

transcript = get_transcript("https://youtube.com/watch?v=dQw4w9WgXcQ")
print(f"Title: {transcript['title']}")
print(f"Words: {transcript['word_count']}")
print(f"Full text:\n{transcript['full_text']}")

Using the Python SDK

pip install contentapi

from contentapi import ContentAPI

client = ContentAPI(api_key="your_api_key")

# Get transcript
transcript = client.youtube_transcript("https://youtube.com/watch?v=...")
print(transcript.title)
print(transcript.full_text)

# Get with timestamps
for segment in transcript.segments:
    minutes = int(segment.start // 60)
    seconds = int(segment.start % 60)
    print(f"[{minutes:02d}:{seconds:02d}] {segment.text}")

Get Video Summary Instead

Don't want the full transcript? Get an AI-generated summary:

response = requests.get(
    f"{BASE_URL}/youtube/summary",
    params={"url": "https://youtube.com/watch?v=..."},
    headers={"X-API-Key": API_KEY}
)

summary = response.json()["data"]
print(summary["summary"])
print("Key points:", summary["key_points"])
print("Topics:", summary["topics"])

Get Video Chapters

response = requests.get(
    f"{BASE_URL}/youtube/chapters",
    params={"url": "https://youtube.com/watch?v=..."},
    headers={"X-API-Key": API_KEY}
)

for chapter in response.json()["data"]["chapters"]:
    print(f"{chapter['start_time']}s - {chapter['title']}")

Get Comments

response = requests.get(
    f"{BASE_URL}/youtube/comments",
    params={"url": "https://youtube.com/watch?v=..."},
    headers={"X-API-Key": API_KEY}
)

for comment in response.json()["data"]["comments"]:
    print(f"{comment['author']} ({comment['likes']} likes): {comment['text']}")

Real-World Use Cases

1. Build a YouTube Knowledge Base

from contentapi import ContentAPI
import json

client = ContentAPI(api_key="your_api_key")

# Process a playlist worth of videos
video_urls = [
    "https://youtube.com/watch?v=video1",
    "https://youtube.com/watch?v=video2",
    "https://youtube.com/watch?v=video3",
]

knowledge_base = []
for url in video_urls:
    transcript = client.youtube_transcript(url)
    knowledge_base.append({
        "title": transcript.title,
        "channel": transcript.channel,
        "content": transcript.full_text,
        "url": url,
        "duration": transcript.duration,
    })

# Save for RAG pipeline
with open("youtube_kb.json", "w") as f:
    json.dump(knowledge_base, f, indent=2)

2. Auto-Generate Blog Posts from Videos

from contentapi import ContentAPI
from openai import OpenAI

content_client = ContentAPI(api_key="your_contentapi_key")
openai_client = OpenAI(api_key="your_openai_key")

# Get transcript
transcript = content_client.youtube_transcript(
    "https://youtube.com/watch?v=..."
)

# Generate blog post
response = openai_client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": "Convert this video transcript into a well-structured "
                       "blog post with headers, key takeaways, and a summary."
        },
        {
            "role": "user",
            "content": f"Video: {transcript.title}\n\n{transcript.full_text}"
        }
    ]
)

blog_post = response.choices[0].message.content
print(blog_post)

3. Multi-Language Transcript Search

# ContentAPI returns the detected language
transcript = client.youtube_transcript(
    "https://youtube.com/watch?v=spanish_video"
)

print(f"Detected language: {transcript.language}")
# "es" — Spanish

# The full_text is in the original language
# Use any translation API to convert if needed

Comparison with Alternatives

Feature	youtube-transcript-api	YouTube Data API	ContentAPI
API key needed	No	Yes (Google)	Yes (free)
Rate limits	Aggressive	10,000 units/day	5,000 req/month free
No-caption videos	❌ Fails	❌ No transcripts	✅ AI fallback
Video metadata	❌ No	✅ Yes	✅ Yes
Summaries	❌ No	❌ No	✅ AI-powered
Chapters	❌ No	❌ No	✅ Yes
Comments	❌ No	✅ Yes	✅ Yes
Maintenance	You	You	Managed

Beyond YouTube

The same API also handles web pages, Twitter/X, and Reddit — so if your pipeline needs content from multiple sources, it's one SDK:

from contentapi import ContentAPI

client = ContentAPI(api_key="your_api_key")

# All through the same interface
youtube = client.extract("https://youtube.com/watch?v=...")
webpage = client.extract("https://example.com/article")
tweet = client.extract("https://x.com/user/status/123456")
reddit = client.extract("https://reddit.com/r/python/comments/abc")

MCP Server for AI Assistants

If you use Claude, Cursor, or Windsurf, there's an MCP server that lets your AI assistant extract YouTube transcripts directly:

npx contentapi-mcp-server

Then just ask your AI: "Get me the transcript from this YouTube video: [url]"

Getting Started

Sign up free: getcontentapi.com — no credit card needed
Install the SDK: pip install contentapi
Extract your first transcript:

from contentapi import ContentAPI
client = ContentAPI(api_key="your_key")
t = client.youtube_transcript("https://youtube.com/watch?v=dQw4w9WgXcQ")
print(t.full_text)

Free tier: 5,000 requests/month. Enough to process ~170 videos per day.

Resources:

Have you worked with YouTube transcripts before? What was your biggest pain point? I'd love to hear about your use cases in the comments! 💬

Build a RAG Pipeline in 5 Minutes with Python (No Scraping Headaches)

stabem — Sat, 14 Feb 2026 21:19:24 +0000

If you've built a RAG (Retrieval Augmented Generation) pipeline, you know the dirty secret: 90% of the work is getting clean data, not the "AI" part.

Vector databases? Easy. Embedding models? Plug and play. But getting clean, structured text from YouTube videos, web pages, and social media? That's where projects go to die.

In this tutorial, I'll show you how to build a complete RAG pipeline in under 5 minutes using Python — with content ingestion that actually works.

The Architecture

URLs → ContentAPI → Clean Text → Embeddings → Vector DB → LLM → Answer

We'll use:

ContentAPI for content extraction (free tier: 5,000 req/month)
OpenAI for embeddings and chat completion
ChromaDB for vector storage (local, no setup)

Step 1: Install Dependencies

pip install contentapi openai chromadb

Step 2: Extract Content from Multiple Sources

Here's the magic — ContentAPI handles YouTube, web pages, Reddit, and Twitter with one API:

from contentapi import ContentAPI

client = ContentAPI(api_key="your_contentapi_key")

# Define your knowledge sources — mix and match!
sources = [
    "https://youtube.com/watch?v=example1",          # YouTube video
    "https://docs.python.org/3/tutorial/classes.html", # Web page
    "https://reddit.com/r/MachineLearning/comments/abc", # Reddit thread
    "https://example.com/blog/rag-best-practices",     # Blog post
]

# Batch extract — one API call for up to 10 URLs
results = client.batch(sources)

documents = []
for result in results:
    if result.get("success"):
        data = result["data"]
        documents.append({
            "content": data.get("full_text") or data.get("content", ""),
            "title": data.get("title", ""),
            "url": data.get("url", ""),
            "source": data.get("source", "web"),
        })

print(f"Extracted {len(documents)} documents")

That's all the scraping code you need. No Beautiful Soup, no Selenium, no YouTube API keys, no Reddit OAuth tokens.

Step 3: Chunk the Content

For RAG, we need to split documents into digestible chunks:

def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        if chunk.strip():
            chunks.append(chunk)
    return chunks

all_chunks = []
for doc in documents:
    chunks = chunk_text(doc["content"])
    for i, chunk in enumerate(chunks):
        all_chunks.append({
            "text": chunk,
            "metadata": {
                "title": doc["title"],
                "url": doc["url"],
                "source": doc["source"],
                "chunk_index": i,
            }
        })

print(f"Created {len(all_chunks)} chunks from {len(documents)} documents")

Step 4: Embed and Store in ChromaDB

import chromadb
from openai import OpenAI

openai_client = OpenAI(api_key="your_openai_key")
chroma_client = chromadb.Client()
collection = chroma_client.create_collection("knowledge_base")

# Embed all chunks
batch_size = 100
for i in range(0, len(all_chunks), batch_size):
    batch = all_chunks[i:i + batch_size]
    texts = [c["text"] for c in batch]

    # Get embeddings from OpenAI
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )

    embeddings = [e.embedding for e in response.data]

    collection.add(
        ids=[f"chunk_{i+j}" for j in range(len(batch))],
        embeddings=embeddings,
        documents=texts,
        metadatas=[c["metadata"] for c in batch],
    )

print(f"Stored {len(all_chunks)} chunks in ChromaDB")

Step 5: Query Your Knowledge Base

def ask(question):
    """Ask a question against your knowledge base."""

    # Embed the question
    q_embedding = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=[question]
    ).data[0].embedding

    # Find relevant chunks
    results = collection.query(
        query_embeddings=[q_embedding],
        n_results=5
    )

    # Build context from retrieved chunks
    context = "\n\n---\n\n".join(results["documents"][0])
    sources = [m["url"] for m in results["metadatas"][0]]

    # Generate answer with GPT
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Answer questions based on the provided context. "
                           "Cite sources when possible. If the context doesn't "
                           "contain the answer, say so."
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {question}"
            }
        ]
    )

    answer = response.choices[0].message.content
    return answer, list(set(sources))

# Try it!
answer, sources = ask("What are the best practices for RAG pipelines?")
print(f"Answer: {answer}")
print(f"Sources: {sources}")

The Full Pipeline in ~50 Lines

Here it is, condensed:

from contentapi import ContentAPI
from openai import OpenAI
import chromadb

# Initialize clients
content_client = ContentAPI(api_key="your_contentapi_key")
openai_client = OpenAI(api_key="your_openai_key")
collection = chromadb.Client().create_collection("kb")

# 1. Extract content from any URLs
urls = ["https://youtube.com/watch?v=...", "https://example.com/article"]
results = content_client.batch(urls)

# 2. Chunk and embed
chunks = []
for r in results:
    if r.get("success"):
        text = r["data"].get("full_text") or r["data"].get("content", "")
        words = text.split()
        for i in range(0, len(words), 450):
            chunks.append(" ".join(words[i:i+500]))

embeddings = openai_client.embeddings.create(
    model="text-embedding-3-small", input=chunks
).data

collection.add(
    ids=[f"c{i}" for i in range(len(chunks))],
    embeddings=[e.embedding for e in embeddings],
    documents=chunks,
)

# 3. Query
q = "What did the video discuss?"
q_emb = openai_client.embeddings.create(
    model="text-embedding-3-small", input=[q]
).data[0].embedding

context = "\n".join(collection.query(query_embeddings=[q_emb], n_results=5)["documents"][0])

answer = openai_client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": f"Answer based on context:\n{context}"},
        {"role": "user", "content": q}
    ]
).choices[0].message.content

print(answer)

Why Not Just Use Beautiful Soup / Selenium?

You could — and I did for years. Here's why I stopped:

	Traditional Scraping	ContentAPI
YouTube transcripts	Need YouTube API key + quotas	✅ Works out of the box
JS-rendered pages	Need Selenium/Playwright	✅ Handled server-side
Twitter/X	$100/mo minimum API cost	✅ Included
Reddit	OAuth setup + rate limits	✅ Included
Maintenance	Breaks when sites change	✅ Maintained for you
Setup time	Hours per source	✅ 5 minutes total

Advanced: Continuous Ingestion with Crawling

For larger knowledge bases, use the crawl endpoint to ingest entire sites:

# Crawl an entire documentation site
crawl = content_client.crawl(
    url="https://docs.example.com",
    max_pages=100,
    include_patterns=["/docs/*", "/guides/*"],
    format="markdown"
)

# Wait for completion, then process all pages
for page in crawl.results:
    # Same chunking + embedding logic as above
    process_document(page.content, page.url)

Advanced: AI-Powered Structured Extraction

Need specific data points from pages? Use schema-based extraction:

# Extract structured data for product comparison
product_data = content_client.extract(
    url="https://example.com/product",
    schema={
        "name": "string",
        "price": "number",
        "rating": "number",
        "pros": ["string"],
        "cons": ["string"]
    }
)

This uses AI to understand the page and extract exactly the fields you need — no CSS selectors or XPath required.

Getting Started

Sign up at getcontentapi.com (free, no credit card)
pip install contentapi
Start extracting content

The free tier gives you 5,000 requests/month — enough for a serious RAG prototype.

Full docs: getcontentapi.com/docs

What are you building with RAG? I'd love to see how people use this in their pipelines. Share your projects in the comments! 🚀