Mustapha

Posted on Oct 28 AI-assisted

How to get YouTube transcripts in bulk with Python (no API key, works on cloud servers)

#python #ai #youtube #tutorial

Disclosure: I built the Apify actor used in this post. It is a paid tool (pay per transcript), and I say below where it doesn't fit.

If you have tried to get YouTube transcripts from code, you have probably hit one of these walls:

The official YouTube Data API doesn't help. Its captions.download endpoint only works for videos you own. For anyone else's video you get a 403.
youtube-transcript-api works on your laptop and then dies on a server. YouTube answers requests from cloud IP ranges (AWS, GCP, Hetzner, most VPSes) with a bot check, so the same code that ran fine locally starts throwing RequestBlocked / IpBlocked in production. Developers on the library's issue tracker discuss exactly this.
There is no "whole channel" button. You need to list the channel's videos first, then fetch each transcript, then handle the videos that have no captions.

This post shows a way around all three from Python, in about 20 lines.

The short version: one HTTP call

YouTube Transcript Scraper runs on Apify's infrastructure. It tries a datacenter proxy first and automatically retries through a residential IP when YouTube shows a bot check, so you don't manage proxies yourself. You send video, playlist or channel URLs and get one JSON row per video.

You need a free Apify account and your API token (Console → Settings → Integrations). Then:

import requests

TOKEN = "YOUR_APIFY_TOKEN"
url = ("https://api.apify.com/v2/acts/groupoject~youtube-transcript-scraper"
       f"/run-sync-get-dataset-items?token={TOKEN}")

rows = requests.post(url, json={
    "urls": ["https://www.youtube.com/watch?v=iG9CE55wbtY"],
    "outputFormats": ["text"],
}, timeout=320).json()

for row in rows:
    print(row["title"], "-", row["wordCount"], "words")
    print(row["text"][:300])

run-sync-get-dataset-items starts the run, waits for it to finish and returns the rows in one response. Apify caps synchronous calls at about 5 minutes, so use it for a handful of videos. For a whole channel, use the client library below.

Bulk: a whole channel or playlist

pip install apify-client, then:

from pathlib import Path
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("groupoject/youtube-transcript-scraper").call(run_input={
    "urls": [
        "@lexfridman",                                          # a channel handle
        "https://www.youtube.com/playlist?list=PLxxxxxxxxxxxx",  # a playlist
    ],
    "maxVideosPerSource": 50,          # newest 50 per channel / playlist
    "channelContentTypes": ["videos"], # or add "shorts", "streams"
    "languages": ["en"],               # preference order, e.g. ["es", "en"]
    "outputFormats": ["text", "segments", "srt"],
})

out = Path("transcripts")
out.mkdir(exist_ok=True)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    (out / f"{item['videoId']}.txt").write_text(item["text"], encoding="utf-8")
    (out / f"{item['videoId']}.srt").write_text(item["srt"], encoding="utf-8")
    print(item["videoId"], item["languageCode"],
          "auto" if item["isAutoGenerated"] else "manual", item["title"])

A few input details worth knowing (these are the real field names from the actor's input schema):

urls accepts watch?v= links, youtu.be links, Shorts, /live/ URLs, bare 11-character IDs, playlist URLs, channel URLs and @handles, all mixed in one list.
maxVideosPerSource caps each playlist and each channel content type (newest first, up to 5,000). Single video URLs are not affected.
captionType is any (default: prefer creator-uploaded, fall back to auto-generated), manual or auto.
fallbackToAnyLanguage (default true) returns the original-language captions when none of your languages exist. Set it to false to skip those videos instead.
includeVideoMetadata (default true) adds title, channel, publish date, views, likes, duration, description and tags to each row.

Which videos had no transcript?

Videos without captions are not pushed to the dataset and are not charged. They are listed in the run's key-value store:

store = client.key_value_store(run["defaultKeyValueStoreId"])
missing = (store.get_record("NO_TRANSCRIPT") or {}).get("value") or []
for m in missing:
    print(m["videoId"], m["reason"], m.get("availableLanguages"))

reason is NO_CAPTIONS, LANGUAGE_NOT_AVAILABLE or EMPTY_CAPTIONS. Private, deleted or region-blocked videos go to the ERRORS record instead.

Feeding transcripts to an LLM or a RAG pipeline

For summarising, the text field is all you need. For retrieval with citations, use segments: each segment has start, duration and text in seconds, so you can chunk by time and keep a deep link back to the moment in the video.

def chunks(item, seconds=90):
    buf, t0 = [], None
    for seg in item["transcript"]:
        t0 = seg["start"] if t0 is None else t0
        buf.append(seg["text"])
        if seg["start"] - t0 >= seconds:
            yield {"text": " ".join(buf),
                   "source": f"{item['url']}&t={int(t0)}s",
                   "title": item["title"]}
            buf, t0 = [], None
    if buf:
        yield {"text": " ".join(buf), "source": f"{item['url']}&t={int(t0)}s",
               "title": item["title"]}

Each chunk can go straight into your vector store with source as metadata, so answers can link to the exact timestamp.

No-code: n8n, Make, Zapier

If you'd rather not run Python, the actor works with Apify's integrations. In n8n, add the Apify node, choose Run an Actor and Get Dataset, set the actor to groupoject~youtube-transcript-scraper and paste the JSON input above. Connect a Google Sheets or Notion node after it. In Make and Zapier the Apify app has the same "run actor" and "get dataset items" steps. You can also call it from Claude, Cursor or another MCP client through https://mcp.apify.com?tools=groupoject/youtube-transcript-scraper.

Monitor a channel for new transcripts

Turn on onlyNewVideos and give the monitor a stateKey (any name, e.g. ai-podcasts). Each run then saves only videos it hasn't saved before for that key, so you're never charged twice for the same video. Schedule it daily in Apify (Schedules → Create) and connect a Slack, email or Google Sheets integration to get each new transcript as it lands:

run = client.actor("groupoject/youtube-transcript-scraper").call(run_input={
    "urls": ["https://www.youtube.com/@lexfridman"],
    "onlyNewVideos": True,
    "stateKey": "lex-podcast",
    "publishedAfter": "7 days",   # optional: ignore older uploads on the first run
})

Brand-new uploads sometimes get auto-captions a few hours late; the actor re-checks those for 24 hours instead of skipping them.

What it costs

$2.50 per 1,000 transcripts on the free plan ($0.0025 per video), with proxy traffic included. Paid Apify plans get a lower per-result rate automatically. Videos with no captions, and failed, private or deleted videos, are not charged. A 100-video playlist where every video has captions costs $0.25. You can set a maximum charge per run, and the actor stops cleanly before it goes over.

Honest limits

It returns YouTube's own captions. It does not transcribe audio. If a video has no captions at all (music clips, some Shorts), you get nothing for it. For those you need Whisper or a speech-to-text API.
Auto-generated captions are only as good as YouTube's speech recognition. Names and jargon are often wrong. isAutoGenerated tells you which kind you got.
Translation (translateTo) is best effort. YouTube rate-limits its auto-translate endpoint heavily. When it refuses, you still get the original-language transcript and translationError explains why.
No login, by design. Age-restricted, members-only and private videos are reported in ERRORS, not scraped.
If you need a few videos a day from your laptop, you don't need this. youtube-transcript-api or yt-dlp --write-auto-subs is free and works fine from a home connection. This tool is for when you need volume, channels, or a server.
Respect copyright. Transcripts are the creators' work. Summarising for research is one thing. Republishing them is another.

DEV Community