DEV Community

novosai0day-a11y
novosai0day-a11y

Posted on

Get YouTube transcripts for your RAG pipeline with a few lines of Python

#ai #rag #youtube #python

If you are building a RAG app, a summarizer or a "chat with this video" tool, the first problem is boring but real: getting clean transcripts at scale. The official YouTube Data API doesn't give you caption text for videos you don't own, and running youtube-transcript-api from a cloud server quickly hits RequestBlocked because YouTube blocks datacenter IPs.

Here's the setup I use now.

1. Get the transcripts

I published a small Apify Actor that wraps the transcript endpoint and automatically falls back to residential proxies only when YouTube blocks the request: YouTube Transcript & Subtitles Scraper.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("fujiwaranosai/youtube-transcript-scraper").call(run_input={
    "videos": ["https://www.youtube.com/watch?v=aircAruvnKk", "https://youtu.be/8jPQjjsBbIc"],
    "languages": ["en"],
    "outputFormat": "both",
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["word_count"])
Enter fullscreen mode Exit fullscreen mode

Each item has the full transcript text, segments with timestamps, the language, whether it was auto-generated, the title and the channel. You pay per transcript (videos without captions are not charged).

2. Chunk by time, not by characters

Timestamped segments make much better chunks than fixed character windows, because you can link every answer back to the exact second of the video:

def chunks(segments, seconds=60):
    buf, start = [], segments[0]["start"]
    for s in segments:
        buf.append(s["text"])
        if s["start"] - start >= seconds:
            yield {"start": start, "text": " ".join(buf)}
            buf, start = [], s["start"]
    if buf:
        yield {"start": start, "text": " ".join(buf)}
Enter fullscreen mode Exit fullscreen mode

Store start as metadata and build a link like https://youtu.be/{video_id}?t={int(start)} for citations.

3. Translate when needed

Set "translateTo": "en" and Spanish, Portuguese or Japanese videos come back in English (YouTube's own translation), so your embeddings stay in one language.

Notes

  • Prefer human subtitles (preferManual: true): they are punctuated and much better for LLMs.
  • Respect creators: transcripts are their content. Summarize and cite; don't republish verbatim.

If you need playlists or channels as input, leave a comment — that's next on my list.

More ready-to-run examples: https://github.com/novosai0day-a11y/public-data-apis

Top comments (0)