DEV Community

Fabian
Fabian

Posted on

YouTube transcripts for RAG without the Data API (and without maintaining a scraper)

I needed transcripts for a few hundred videos from a couple of channels for a RAG project. The YouTube Data API doesn't give you captions for videos you don't own, and the usual Python transcript library broke on me twice in one month, so I ended up with the setup below. Posting it because the question comes up a lot.

Two steps: list the videos of a channel, then fetch transcripts for that list. I'm using two Apify actors I wrote for this (they read the public pages and the innertube endpoints, no API key, no quota). The client library is pip install apify-client.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# 1. all videos of a channel
run = client.actor("pulsedata/youtube-channel-videos-scraper").call(run_input={
    "channelUrls": ["https://www.youtube.com/@3blue1brown"],
    "channelTabs": ["videos"],
    "maxVideosPerChannel": 0,   # 0 = no limit
})
videos = [i for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i["type"] == "video"]
print(len(videos), "videos")

# 2. transcripts
run = client.actor("pulsedata/youtube-transcript-scraper").call(run_input={
    "videoUrls": [v["url"] for v in videos],
    "languages": ["en"],
    "fallbackToAnyLanguage": True,
    "outputFormats": ["text", "segments"],
    "includeMetadata": True,
})
docs = [d for d in client.dataset(run["defaultDatasetId"]).iterate_items() if not d.get("error")]
Enter fullscreen mode Exit fullscreen mode

Each item in docs has text (whole transcript), segments (list of {start, duration, text}) and metadata: title, channelName, durationSeconds, publishDate, url. Items with an error field are videos without captions, I just drop those.

Chunking

I first chunked by a fixed number of tokens and it was fine, but chunking along the transcript segments works a bit better because the segment boundaries follow the speaker's pauses, and you get a start time for free. This groups segments until roughly 300 words:

def chunks(doc, max_words=300):
    buf, start, n = [], None, 0
    for s in doc["segments"]:
        if start is None:
            start = s["start"]
        buf.append(s["text"])
        n += len(s["text"].split())
        if n >= max_words:
            yield {"text": " ".join(buf), "start": start, "title": doc["title"],
                   "channel": doc["channelName"], "url": f'{doc["url"]}&t={int(start)}'}
            buf, start, n = [], None, 0
    if buf:
        yield {"text": " ".join(buf), "start": start, "title": doc["title"],
               "channel": doc["channelName"], "url": f'{doc["url"]}&t={int(start)}'}
Enter fullscreen mode Exit fullscreen mode

The &t= on the URL is the useful part. When the model answers it can link to the exact spot in the video, which people like a lot more than a link to the video.

Cost, and things that bit me

  • Listing is $1 per 1,000 videos, transcripts $3 per 1,000. A 500 video channel is about $2 total. Apify's free tier ($5/month) covers the first few channels.
  • Auto-generated captions are fine for retrieval. For quoting them verbatim they're not great, punctuation is missing and names get mangled. The isAutoGenerated flag tells you which is which.
  • translateTo: "en" gives you English text for non-English channels (YouTube's own translation, quality varies).
  • If you already have the video URLs, skip step 1. The transcript actor also takes playlist URLs directly.
  • Shorts have transcripts too but they're usually a few sentences, I filter them out with isShort from step 1.

Actors: https://apify.com/pulsedata/youtube-channel-videos-scraper and https://apify.com/pulsedata/youtube-transcript-scraper. If one of them fails on a particular channel, tell me which one, that's the quickest way for me to find out.

Top comments (0)