DEV Community

TORIX
TORIX

Posted on

YouTube Transcripts for RAG Without the YouTube Data API (Python, 30 Lines)

#webscraping #python #ai #langchain

YouTube is one of the best knowledge sources you can put behind an LLM: conference talks, tutorials, product demos,
earnings calls, podcasts. The problem is getting the text out.

The official route doesn't work for this. The YouTube Data API's captions.download endpoint needs OAuth and only
works for videos you own (or where the owner allows third-party contributions). For "index this competitor's channel"
or "build a RAG bot over 300 conference talks", it's a dead end. The usual Python workaround libraries break whenever
YouTube changes its internals and get your server IP blocked after a few hundred requests.

In this guide we'll take a hosted transcript scraper, turn transcripts into timestamped chunks, and load them into a vector
store. Each chunk links back to the exact second in the video.

What we're building 

YouTube URLs / @channel / playlist / search
        │
        ▼
YouTube Transcript Scraper (Apify)  →  transcript + timestamped segments + metadata
        │
        ▼
chunker (300 words, 50 overlap, keeps start time)
        │
        ▼
chunks.jsonl  →  Chroma / pgvector / Pinecone / Qdrant
Enter fullscreen mode Exit fullscreen mode

Step 1 — get transcripts

Create a free Apify account, copy your API token, and install the client:

pip install apify-client
export APIFY_TOKEN=<your token>
Enter fullscreen mode Exit fullscreen mode

The YouTube Transcript Scraper accepts video links, Shorts,
@channels, playlists or plain search keywords. It returns one row per video with the full text and timestamped segments:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("murch1k/youtube-transcript-scraper").call(run_input={
    "urls": ["https://www.youtube.com/@AndrejKarpathy"],
    "maxVideosPerSource": 20,
    "languages": ["en"],          # ordered preference; manual captions beat auto-generated
    "includeSegments": True,
})
videos = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(len(videos), videos[0]["title"], videos[0]["wordCount"])
Enter fullscreen mode Exit fullscreen mode

A row looks like this (shortened, values illustrative):

{
  "videoId": "zjkBMFhNj_g",
  "title": "[1hr Talk] Intro to Large Language Models",
  "channelName": "Andrej Karpathy",
  "url": "https://www.youtube.com/watch?v=zjkBMFhNj_g",
  "language": "en",
  "isAutoGenerated": false,
  "wordCount": 11873,
  "transcript": "hi everyone so recently I gave a 30-minute talk on large language models …",
  "segments": [{ "start": 0.16, "duration": 4.32, "text": "hi everyone so recently I gave a" }]
}
Enter fullscreen mode Exit fullscreen mode

Pricing is per transcript ($3 per 1,000). Videos without captions, private or removed videos cost nothing. Apify's Free plan
monthly credits cover a few hundred videos, which is enough to prototype.

Step 2 — chunk with timestamps

Don't embed a one-hour transcript as a single vector. Split it into ~300-word windows with overlap, and keep the start
time of each window. A &t=123s deep link makes answers verifiable:

CHUNK_WORDS, OVERLAP = 300, 50

def chunks(video):
    buf, start = [], None
    for seg in video.get("segments") or []:
        start = seg["start"] if start is None else start
        buf.extend(seg["text"].split())
        if len(buf) >= CHUNK_WORDS:
            yield {"text": " ".join(buf), "url": f"{video['url']}&t={int(start)}s", "title": video["title"]}
            buf, start = buf[-OVERLAP:], seg["start"]
    if buf:
        yield {"text": " ".join(buf), "url": f"{video['url']}&t={int(start or 0)}s", "title": video["title"]}

docs = [c for v in videos for c in chunks(v)]
Enter fullscreen mode Exit fullscreen mode

Step 3 — load into a vector store

Any store works. Here's Chroma, because it needs zero setup:

import chromadb

col = chromadb.PersistentClient("yt_index").get_or_create_collection("talks")
col.add(
    ids=[f"{i}" for i in range(len(docs))],
    documents=[d["text"] for d in docs],
    metadatas=[{"url": d["url"], "title": d["title"]} for d in docs],
)
hits = col.query(query_texts=["why do LLMs hallucinate?"], n_results=3)
for meta in hits["metadatas"][0]:
    print(meta["title"], "→", meta["url"])
Enter fullscreen mode Exit fullscreen mode

Pass the hits to your model with an instruction like "answer only from the excerpts and cite the URLs" and you have
a cited, timestamp-accurate RAG bot over any channel.

Bonus — let the agent fetch transcripts itself (MCP / LangChain)

Instead of pre-indexing, you can give an agent the scraper as a tool. With the hosted Apify MCP server, add this to
Claude Desktop, Cursor or VS Code:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=murch1k/youtube-transcript-scraper",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Then ask: "Summarise the last 5 videos on @mkbhd and list every product he recommends."

In LangChain it is one line: ApifyActorsTool("murch1k/youtube-transcript-scraper") from the langchain-apify package.

Practical tips

  • Languages: pass "languages": ["en", "es"] in order of preference; set translateTo to get everything in one language.
  • Whole channels: maxVideosPerSource caps how far back you go; channelTab: "shorts" switches to Shorts.
  • Cost cap: maxResults puts a hard limit on any run, and your Apify spending limit is respected.
  • Freshness: schedule the run daily on Apify and only embed new videoIds.

The full script is in the Apify Scrapers Pack repo
(python/youtube_transcripts_rag.py), next to Node.js versions and examples for other scrapers.

Respect creators' copyright: use transcripts for analysis, search and summaries, not for republishing their content.

The toolkit used in this series

All five are pay-per-result Actors on Apify (no subscription; the Apify free plan's monthly credits cover a test run):

  1. YouTube Transcript Scraper: transcripts with timestamps from videos, channels and playlists.
  2. Career Site Jobs Scraper: open roles from Greenhouse, Lever, Ashby, Workday and 4 more ATSs.
  3. Google Trends Scraper: interest over time, related queries and topics.
  4. Substack Scraper: posts and newsletter metadata as clean text for LLMs.
  5. Website Contact Details Extractor: emails, phones and socials, one row per domain.

Python, Node.js, LangChain and MCP examples for each one are in apify-scrapers-pack on GitHub.
A star there helps other people find it.

Top comments (0)