Panix AI Lab

Posted on Oct 26 Originally published at dev.to

Turn any YouTube channel into RAG-ready transcripts (no YouTube API key)

#ai #n8n #python #youtube

If you've tried to build a "chat with this YouTube channel" bot, you know the boring part isn't the LLM. It's getting the transcripts:

The official YouTube Data API doesn't give you caption text for videos you don't own.
Many scrapers take one video URL at a time.
Some silently return an English auto-translation for Spanish, Hindi or Arabic videos instead of what was actually said.
You still have to clean ' codes, split the text into chunks and keep timestamps for citations.

I built an Apify Actor that does all of that: YouTube Transcript API & Scraper. This post shows the three ways to use it: the web console, Python, and a scheduled n8n workflow that keeps a vector database up to date.

What you get

Give it any mix of video URLs, Shorts, playlists or channels (@handle). For each video you get one row:

{
  "recordType": "transcript",
  "dedupeKey": "youtube:video:jNQXAC9IVRw",
  "videoId": "jNQXAC9IVRw",
  "title": "Me at the zoo",
  "channelName": "jawed",
  "language": "en",
  "transcript": "All right, so here we are, in front of the elephants...",
  "segments": [{ "start": 1.2, "end": 3.36, "text": "All right, so here we are, in front of the elephants" }],
  "chunks": [{ "chunkIndex": 0, "start": 1.2, "end": 18.9, "url": "https://www.youtube.com/watch?v=jNQXAC9IVRw&t=1s", "text": "..." }]
}

Channel runs also include each video's publish date. Optional formats: SRT, WebVTT, Markdown and RAG chunks. Chunks never cross a chapter boundary, and each one has a deep link to the exact second, which makes citations easy.

Does the language actually come out right?

I tested it on 20 videos in 20 languages in one run: English, German, French, Spanish, Hindi, Arabic, Urdu, Japanese, Italian, Filipino, Polish, Romanian, Dutch, Persian, Portuguese, Bengali, Tamil, Telugu, Ukrainian and Marathi.

20/20 came back in the language actually spoken, in about 30 seconds. Two of the videos (Telugu and Ukrainian) also had an English auto-generated track, and the Actor still picked the spoken language. Arabic, Persian and Urdu are marked right-to-left, and Urdu, which YouTube writes in Hindi script, is converted to Urdu script.

Videos without any captions return a clear NO_CAPTIONS row and aren't charged.

Option 1: try it in the browser (2 minutes)

Open the Actor page and click Try for free.
Paste a channel, for example https://www.youtube.com/@TED, into YouTube channel URLs or @handles.
Set Maximum discovered videos to 10 and turn on RAG-ready chunks.
Click Start, then download the results as JSON, CSV or Excel.

Option 2: Python

Install the client with pip install apify-client, then:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("panix/youtube-transcript-scraper").call(run_input={
    "channelUrls": ["https://www.youtube.com/@TED"],
    "channelMode": "latest",
    "maxVideos": 10,
    "includeChunks": True,
})

for row in client.dataset(run.default_dataset_id).iterate_items():
    if row["recordType"] != "transcript":
        continue  # free error rows: no captions, private video, and so on
    for chunk in row["chunks"]:
        print(row["title"], chunk["url"], chunk["text"][:80])

Swap print for your embedding call and an upsert into Pinecone, Qdrant, Supabase pgvector or Chroma. Use dedupeKey plus chunkIndex as the record ID so reruns update instead of duplicating.

Option 3: a daily n8n workflow

Three nodes: Schedule → HTTP Request → Code, then your vector store node.

HTTP Request node

Method: POST
URL: https://api.apify.com/v2/acts/panix~youtube-transcript-scraper/run-sync-get-dataset-items
Authentication: Header Auth, name Authorization, value Bearer <YOUR_APIFY_TOKEN>
Body (JSON):

{
  "channelUrls": ["https://www.youtube.com/@TED"],
  "channelMode": "latest",
  "maxVideos": 15,
  "includeChunks": true
}

channelMode: "latest" reads the channel's newest uploads and Shorts with exact publish dates. Add "publishedAfter": "2026-10-01" if you only want new videos.

Code node (turns each video into vector-store records):

const rows = $input.all().map(i => i.json);
return rows
  .filter(r => r.recordType === 'transcript')
  .flatMap(r => (r.chunks || []).map(c => ({
    json: {
      id: `${r.dedupeKey}:${c.chunkIndex}`,
      text: c.text,
      metadata: {
        videoId: r.videoId, title: r.title, channelId: r.channelId,
        publishedDate: r.publishedDate, url: c.url,
        start: c.start, end: c.end, chapter: c.chapter,
      },
    },
  })));

Connect the output to your vector store node, run it once by hand, then turn on the schedule.

The synchronous endpoint waits up to 300 seconds, so keep maxVideos modest for scheduled runs and use the normal asynchronous API for big back-catalogue imports.

What it costs

It's pay per result:

$0.003 per delivered transcript (lower on paid Apify plans), with every format included.
A $0.003 start fee per run.
Failed videos, duplicates and retries are free.

Apify's free plan includes monthly credits, enough for well over a thousand transcripts to try it out.

Limitations

It reads existing captions (human-made or YouTube's auto-generated ones). It doesn't run speech-to-text, so a video with no captions returns NO_CAPTIONS.
Private, members-only and age-restricted videos aren't supported.
You're responsible for how you use the content, including YouTube's terms and copyright.

If you build something with it, or a video comes back wrong, let me know in the comments or open an issue on the Actor's Issues tab. I read them all.

DEV Community