I needed transcripts for a few hundred videos from a couple of channels for a RAG project. The YouTube Data API doesn't give you captions for videos you don't own, and the usual Python transcript library broke on me twice in one month, so I ended up with the setup below. Posting it because the question comes up a lot.
Two steps: list the videos of a channel, then fetch transcripts for that list. I'm using two Apify actors I wrote for this (they read the public pages and the innertube endpoints, no API key, no quota). The client library is pip install apify-client.
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
# 1. all videos of a channel
run = client.actor("pulsedata/youtube-channel-videos-scraper").call(run_input={
"channelUrls": ["https://www.youtube.com/@3blue1brown"],
"channelTabs": ["videos"],
"maxVideosPerChannel": 0, # 0 = no limit
})
videos = [i for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i["type"] == "video"]
print(len(videos), "videos")
# 2. transcripts
run = client.actor("pulsedata/youtube-transcript-scraper").call(run_input={
"videoUrls": [v["url"] for v in videos],
"languages": ["en"],
"fallbackToAnyLanguage": True,
"outputFormats": ["text", "segments"],
"includeMetadata": True,
})
docs = [d for d in client.dataset(run["defaultDatasetId"]).iterate_items() if not d.get("error")]
Each item in docs has text (whole transcript), segments (list of {start, duration, text}) and metadata: title, channelName, durationSeconds, publishDate, url. Items with an error field are videos without captions, I just drop those.
Chunking
I first chunked by a fixed number of tokens and it was fine, but chunking along the transcript segments works a bit better because the segment boundaries follow the speaker's pauses, and you get a start time for free. This groups segments until roughly 300 words:
def chunks(doc, max_words=300):
buf, start, n = [], None, 0
for s in doc["segments"]:
if start is None:
start = s["start"]
buf.append(s["text"])
n += len(s["text"].split())
if n >= max_words:
yield {"text": " ".join(buf), "start": start, "title": doc["title"],
"channel": doc["channelName"], "url": f'{doc["url"]}&t={int(start)}'}
buf, start, n = [], None, 0
if buf:
yield {"text": " ".join(buf), "start": start, "title": doc["title"],
"channel": doc["channelName"], "url": f'{doc["url"]}&t={int(start)}'}
The &t= on the URL is the useful part. When the model answers it can link to the exact spot in the video, which people like a lot more than a link to the video.
Cost, and things that bit me
- Listing is $1 per 1,000 videos, transcripts $3 per 1,000. A 500 video channel is about $2 total. Apify's free tier ($5/month) covers the first few channels.
- Auto-generated captions are fine for retrieval. For quoting them verbatim they're not great, punctuation is missing and names get mangled. The
isAutoGeneratedflag tells you which is which. -
translateTo: "en"gives you English text for non-English channels (YouTube's own translation, quality varies). - If you already have the video URLs, skip step 1. The transcript actor also takes playlist URLs directly.
- Shorts have transcripts too but they're usually a few sentences, I filter them out with
isShortfrom step 1.
Actors: https://apify.com/pulsedata/youtube-channel-videos-scraper and https://apify.com/pulsedata/youtube-transcript-scraper. If one of them fails on a particular channel, tell me which one, that's the quickest way for me to find out.
Top comments (0)