DEV Community

BulkTranscripts youtube
BulkTranscripts youtube

Posted on Fully Autonomous

youtube-transcript-api works locally but gets blocked on your server: why, and 3 fixes

You build something with youtube-transcript-api. It works perfectly on your laptop. You deploy it to AWS, Render, Railway, Vercel or a VPS, and every call now fails with this:

youtube_transcript_api._errors.RequestBlocked: Could not retrieve a transcript for the video ...
Enter fullscreen mode Exit fullscreen mode

Sometimes it shows up as IpBlocked instead. Your code didn't change and neither did the video, so what happened?

Why it happens

YouTube blocks most IP addresses that belong to cloud providers. The library's own README says so: requests from AWS, Google Cloud, Azure and similar networks are very likely to be refused. A home IP can get blocked too if it sends a lot of requests in a short time.

So the problem is where your request comes from, not your code. That's also why switching libraries rarely helps. yt-dlp on the same server usually hits the same wall, as a "Sign in to confirm you're not a bot" error.

There are three ways around it.

Fix 1: Rotating residential proxies

This is the fix the library supports itself. Residential proxies send your request through ordinary home connections, which YouTube treats like normal viewers.

from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api.proxies import WebshareProxyConfig

ytt_api = YouTubeTranscriptApi(
    proxy_config=WebshareProxyConfig(
        proxy_username="<proxy-username>",
        proxy_password="<proxy-password>",
    )
)

transcript = ytt_api.fetch("dQw4w9WgXcQ")
print(" ".join(s.text for s in transcript))
Enter fullscreen mode Exit fullscreen mode

Any provider works through GenericProxyConfig:

from youtube_transcript_api.proxies import GenericProxyConfig

ytt_api = YouTubeTranscriptApi(
    proxy_config=GenericProxyConfig(
        http_url="http://user:pass@proxy.example.com:8000",
        https_url="http://user:pass@proxy.example.com:8000",
    )
)
Enter fullscreen mode Exit fullscreen mode

Things to know:

  • Buy rotating residential proxies. Static or datacenter proxies get banned after a while.
  • You pay per GB, and the price varies a lot between providers. Transcripts are small, but YouTube pages around them are not.
  • Some requests will still fail, so add retries with backoff.
  • Cache every transcript you fetch. Captions rarely change, and a cache hit costs no proxy traffic.
import json, pathlib, time
from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api._errors import RequestBlocked, IpBlocked

CACHE = pathlib.Path("transcripts")
CACHE.mkdir(exist_ok=True)

def get_transcript(api: YouTubeTranscriptApi, video_id: str, tries: int = 4):
    path = CACHE / f"{video_id}.json"
    if path.exists():
        return json.loads(path.read_text())
    for attempt in range(tries):
        try:
            snippets = [
                {"text": s.text, "start": s.start, "duration": s.duration}
                for s in api.fetch(video_id)
            ]
            path.write_text(json.dumps(snippets))
            return snippets
        except (RequestBlocked, IpBlocked):
            time.sleep(2 ** attempt)  # rotating proxies give a new IP on retry
    raise RuntimeError(f"still blocked after {tries} tries: {video_id}")
Enter fullscreen mode Exit fullscreen mode

Good fit: you want full control, you fetch a lot every day, and you're fine maintaining it.

Fix 2: Fetch from a machine that isn't in a data center

If your volume is small, run the fetching part somewhere with a normal home IP: an old laptop, a Raspberry Pi, or your dev machine. Your cloud app puts video IDs in a queue (a database table is enough), and the home worker fetches the transcripts and writes them back.

It's free, but it's another machine to keep running. If you send too many requests, your home IP can get blocked too. Keep it slow, a few requests a minute, and cache everything.

Good fit: side projects, a few dozen videos a day.

Fix 3: Use a hosted transcript API

The third option is to make the IP problem somebody else's. Several hosted APIs do this: Supadata, a few Apify actors, and the one I build, BulkTranscripts. They run the proxies and retries, and your server makes a plain HTTPS call that YouTube never sees.

Disclosure: I build BulkTranscripts, so here's what it looks like:

import requests

r = requests.get(
    "https://bulktranscripts.co/api/v1/transcript",
    params={"video": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"},
    headers={"Authorization": "Bearer bt_ak_YOUR_KEY"},
    timeout=60,
)
r.raise_for_status()
data = r.json()
print(data["title"], data["word_count"])
print(data["text"][:500])          # full transcript as one string
print(data["segments"][0])         # {"text": ..., "start": ..., "duration": ...}
Enter fullscreen mode Exit fullscreen mode

Some details, since you'd want them before choosing:

  • 30 free credits to try it, no card. After that, packs start at $4.99 for 200 transcripts, and credits don't expire.
  • A failed fetch isn't charged, and re-reading a transcript you already fetched is free.
  • There are endpoints for a whole channel or playlist (/api/v1/channel/videos, /api/v1/playlist/videos), plus an MCP server if you want Claude or ChatGPT to fetch transcripts directly.
  • Docs here.

Whichever hosted API you pick, check three things: what a failed request costs, whether re-fetches are billed, and whether it can do channels and playlists in bulk.

Good fit: you want to ship your actual product and stop thinking about proxies.

Which one to pick

Your situation Pick
Hobby project, under ~50 videos a day Fix 2: a home worker
High volume, you want control and can maintain it Fix 1: rotating residential proxies
You want it working today and don't want to maintain it Fix 3: a hosted API

Whatever you choose, cache transcripts. It's the one thing that makes all three cheaper and more reliable.

If you've found another way around RequestBlocked, I'd like to hear it in the comments.

Top comments (0)