DEV Community

SureSignal
SureSignal

Posted on

Get YouTube transcripts for every new upload from the channels you follow (for RAG, summaries and research)

#ai #rag #python #automation

Get YouTube transcripts for every new upload from the channels you follow (for RAG, summaries and research)

YouTube transcripts are some of the most useful text on the internet for AI work: conference talks, earnings-call recaps, tutorials, interviews. Getting them reliably at scale is a well-known headache. Search autocomplete for "youtube transcript api" suggests "rate limit" and "ip blocked" right away.

Is there an official API?

YouTube's official Data API lets you download captions only for videos you own. For public videos, tools read the publicly displayed captions, the same text you see under Show transcript. The popular open-source youtube-transcript-api Python library does this well for small jobs. The usual pain points at volume are IP blocks from cloud servers, and keeping track of which videos you've already processed.

Step 1: One video, quickly

The YouTube Transcript Scraper Actor on Apify accepts watch, youtu.be, Shorts and live URLs:

{
  "videos": ["https://www.youtube.com/watch?v=dQw4w9WgXcQ"],
  "languages": ["en"],
  "preferManualCaptions": true,
  "includeTimestamps": true
}
Enter fullscreen mode Exit fullscreen mode

You get transcriptText, timestamped segments (start, duration, text), language, whether captions were auto-generated, word count, and video metadata (title, channel, views, duration, publish date, tags). It also retries and rotates proxies when YouTube rate-limits.

Step 2: Monitor channels and only pay for new videos 

{
  "channels": ["@somechannel", "@anotherchannel"],
  "deliveryMode": "newOnly",
  "monitorName": "ai-talks",
  "languages": ["en"],
  "maxTranscripts": 200
}
Enter fullscreen mode Exit fullscreen mode

Schedule it daily. Each run checks the channels' latest uploads (up to 15 per channel from the public feed) and delivers only videos this monitor hasn't delivered before.

  • Money: $0.008 per transcript ($8 per 1,000). Already-delivered, unavailable and no-caption videos are free. maxTranscripts caps each run.
  • Time: no "have I processed this one?" bookkeeping, and no re-embedding duplicates into your vector store.
  • Mental effort: your summary digest only contains new videos.

Need to catch caption corrections? Use New + changed mode. It re-delivers a video only when its captions changed, with transcriptChanged: true.

Step 3: Feed it into RAG or a summary 

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("SureSignal/youtube-transcript-scraper").call(run_input={
    "channels": ["@somechannel"],
    "deliveryMode": "newOnly",
    "monitorName": "ai-talks",
})
for v in client.dataset(run["defaultDatasetId"]).iterate_items():
    if v.get("transcriptText"):
        # chunk on segment boundaries so every chunk keeps a timestamp you can cite
        chunks = [(s["start"], s["text"]) for s in v["segments"]]
        # embed and upsert into your vector store, keyed by videoId + start time
Enter fullscreen mode Exit fullscreen mode

Chunking on segment boundaries lets every retrieved passage link back to https://youtu.be/<videoId>?t=<start>, so answers can cite the exact moment.

Honest limits

  • No captions means no transcript. You get a free error row; the tool doesn't do speech-to-text.
  • Channel monitoring covers the latest 15 uploads per run, so it builds the archive going forward. Paste older video URLs directly.
  • Respect YouTube's Terms and copyright: use transcripts for summaries, research and accessibility, and don't republish creators' content wholesale.

A decision is only as good as the information behind it. If your research agent is summarizing last month's videos twice, you're paying twice for less.

SureSignal: only pay for new data.

Try it: youtube-transcript-scraper on the Apify Store. You only pay for new data: runs on a schedule skip what you've already received.

Top comments (0)