sblabs

Posted on Oct 18 AI-assisted

How to get YouTube transcripts for RAG and AI pipelines (with timestamps)

#rag #python #youtube #ai

If you're building anything with LLMs and video (search over a channel, a "chat with this podcast" bot, meeting-style summaries of talks), the first problem is boring: getting clean transcripts out of YouTube in bulk.

The usual options all have a catch:

The official YouTube Data API only gives caption files for videos you own.
youtube-transcript-api and similar libraries work great on your laptop, then start failing once you run them from a server, because YouTube blocks most datacenter IPs for transcript requests.
Many tools pick the wrong language. Popular videos list volunteer translations first, so you ask for a TED talk and get the Arabic subtitles.

I hit all three, so I built a small tool that handles them and runs in the cloud. This post shows how to use it from Python, and what the output looks like for a RAG pipeline.

Disclosure: I'm the author of the tool below. It's pay-per-use on Apify, and you only pay for transcripts it actually delivers.

The tool

YouTube Transcript Scraper on Apify. You give it video links or IDs, and for each video it returns:

text: the whole transcript as one string
chunks: the transcript split into ~1,000-character pieces, each with start and end seconds, ready for a vector database
segments: every caption line with its timestamp
optional srt / vtt subtitle files
video metadata (title, channel, duration, views) at no extra cost

It reads the captions YouTube already has (creator-uploaded or auto-generated). In my release tests on 49 varied videos (talks, news, podcasts, Shorts, seven languages), 81 of 82 caption fetches succeeded across two runs. Videos with no captions come back as a no_captions row and aren't charged.

Getting transcripts from Python

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")  # free account: Settings → API & Integrations

run = client.actor("sblabs/youtube-transcript-scraper").call(run_input={
    "videos": [
        "https://www.youtube.com/watch?v=5MuIMqhT8DM",
        "https://youtu.be/arj7oStGLkU",
    ],
    "chunkSize": 1000,       # characters per chunk; 0 turns chunks off
    "includeSrt": False,
})

for video in client.dataset(run["defaultDatasetId"]).iterate_items():
    if video["status"] != "ok":
        print("skipped", video["url"], video["error"])
        continue
    print(video["title"], "-", len(video["chunks"]), "chunks")

Putting the chunks into a vector store

Because every chunk keeps its timestamps, you can link answers back to the exact moment in the video:

docs = []
for video in client.dataset(run["defaultDatasetId"]).iterate_items():
    if video["status"] != "ok":
        continue
    for c in video["chunks"]:
        docs.append({
            "text": c["text"],
            "metadata": {
                "title": video["title"],
                "channel": video["channel"],
                "url": f'{video["url"]}&t={int(c["start"])}s',  # deep link to the moment
            },
        })

# then embed `docs` with your library of choice (OpenAI, Voyage, sentence-transformers…)
# and upsert into Chroma, pgvector, Pinecone, etc.

When your bot answers, it can cite "TED, 4:31" with a clickable link instead of a vague "somewhere in this video".

Whole playlists and channels

Update: added after this post first went out. You can now pass a playlist or channel link instead of single videos. The Actor lists the videos for free and you only pay for the transcripts:

run_input = {
    "videos": [
        "https://www.youtube.com/@TED",                       # newest uploads first
        "https://www.youtube.com/playlist?list=PLlT0ph_Ig5Rc8xVBw47aZfLsOGw-lxRWb",
    ],
    "maxVideosPerSource": 20,      # per playlist or channel; 0 = all
    "channelContent": "videos",    # or "shorts", "streams", "all"
}

Each result says which playlist or channel it came from (sourceType, sourceTitle, sourcePosition).

Picking a language

By default it returns the captions in the video's spoken language, not whatever translation happens to be listed first. To prefer another language when the video has it:

run_input = {"videos": [...], "language": "de"}

No-code option

You can also run it from the Apify web UI (paste links, click Start, download CSV or JSON). It also works from n8n, Make or Zapier, or as a tool for AI agents through the Apify MCP server (https://mcp.apify.com). Then you can just ask Claude or Cursor: "get the transcript of this video and summarize it".

Limits worth knowing

It only reads existing captions. Music videos and very short clips often have none.
Respect creators: use transcripts for search, analysis and summaries, not to republish someone's content as your own.

If you try it and something breaks, open an issue on the Actor's page and I'll fix it.

DEV Community