How to get YouTube transcripts in bulk without getting IP-blocked

#python #youtube #webscraping #api

If you've used youtube-transcript-api from a laptop, it works. Put the same script on a server or run it over a few hundred videos and you start seeing RequestBlocked or IpBlocked. YouTube rate-limits caption requests per IP, and cloud IPs are blocked almost on sight.

You have three ways out:

Rotate residential proxies yourself. Works, but now you maintain a proxy pool, retries and a queue.
The official YouTube Data API. It only returns captions for videos you own, so it doesn't help here.
Use a hosted scraper that already handles the proxies. That's what this post shows.

Disclosure: I build scrapers on Apify, including the YouTube Transcript Scraper used below.

Getting transcripts for a list of videos

I'll use the YouTube Transcript Scraper on Apify. It takes any number of video URLs (or a channel URL) in one run and returns one row per video: the full transcript, timestamped segments, language and stats. No YouTube API key.

Install the client:

pip install apify-client

Then:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("scrape-lads/youtube-transcript-scraper").call(run_input={
    "urls": [
        "https://www.youtube.com/watch?v=aircAruvnKk",
        "https://www.youtube.com/watch?v=arj7oStGLkU",
        "https://youtu.be/dQw4w9WgXcQ",
    ],
    "includeSegments": True,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["status"] == "success":
        print(row["title"], row["wordCount"], row["language"])
        print(row["transcript"][:200], "...")

Each row looks like this (trimmed):

{
  "url": "https://www.youtube.com/watch?v=aircAruvnKk",
  "status": "success",
  "title": "But what is a neural network? | Deep learning chapter 1",
  "language": "en",
  "transcriptSource": "subtitles",
  "wordCount": 3357,
  "transcript": "This is a 3. It's sloppily written ...",
  "segments": [{ "start": 4.2, "end": 5.4, "text": "This is a 3." }]
}

A whole channel

Swap urls for profiles and set how many of the newest videos you want:

run_input = {
    "profiles": ["https://www.youtube.com/@veritasium"],
    "maxVideosPerProfile": 25,
    "includeSegments": False,
}

Add "postedWithinDays": 7 and put it on a weekly schedule if you only want new uploads.

Subtitle files

"outputFormats": ["srt", "vtt"] adds ready-made SRT and WebVTT text to each row, which is handy for editors and translation tools.

What it doesn't do

It reads YouTube's own captions (creator-uploaded first, then auto-generated). Videos with no captions at all can only be transcribed when they're very short, because YouTube serves full audio only to signed-in players. Those rows come back as failed and aren't charged.
Private, members-only and deleted videos can't be read.

Cost

Pay per transcript, a fraction of a cent each, and failed or empty rows are free. The Apify free plan's monthly credit covers a few thousand transcripts, so you can test it on real data before paying anything.

If you're building a RAG pipeline or summarizing videos with an LLM, setting includeSegments to False gives you clean text that you can chunk straight away.

DEV Community