DEV Community: Hammer Nexon

How I Handle Edge Cases in a YouTube Transcript Tool

Hammer Nexon — Mon, 02 Feb 2026 21:35:04 +0000

Building ScripTube (scriptube.me) taught me that the "happy path" is about 20% of the engineering work. The other 80% is edge cases.

Here's a tour of the edge cases I've encountered and how I handle them.

Edge Case 1: URL Variations

Users paste URLs in every imaginable format:

https://www.youtube.com/watch?v=dQw4w9WgXcQ
https://youtu.be/dQw4w9WgXcQ
https://youtube.com/watch?v=dQw4w9WgXcQ
http://www.youtube.com/watch?v=dQw4w9WgXcQ
https://www.youtube.com/watch?v=dQw4w9WgXcQ&t=120
https://www.youtube.com/embed/dQw4w9WgXcQ
https://m.youtube.com/watch?v=dQw4w9WgXcQ
youtube.com/watch?v=dQw4w9WgXcQ
dQw4w9WgXcQ
www.youtube.com/watch?v=dQw4w9WgXcQ&list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf

Some users paste the video ID directly. Some paste URLs with tracking parameters. Some paste playlist URLs where the video ID is buried in query params.

Solution: A robust URL parser that handles all of these, plus validation that the extracted ID is exactly 11 characters matching [a-zA-Z0-9_-].

Edge Case 2: No Transcript Available

Not all videos have transcripts. Reasons:

Creator disabled captions
Video is too new (auto-generation hasn't run yet)
Language not supported by auto-captioning
Music/instrumental content with no speech
Live streams (sometimes)

Solution: Clear, specific error messages. Not just "Error" — tell the user WHY and WHAT to do:

"This video doesn't have captions available. 
The creator may have disabled them, or auto-generation 
isn't available for this video's language."

Edge Case 3: Age-Restricted Videos

YouTube requires sign-in for age-restricted content. The transcript API call fails because it can't authenticate.

Solution: Detect this specific error and tell the user clearly. I considered implementing cookie-based auth to bypass this, but decided against it for privacy reasons.

Edge Case 4: Auto-Generated vs. Manual Captions

Some videos have both. The quality difference is significant — manual captions are accurate, auto-generated ones have errors.

Solution: Prefer manual captions when available. Show the user which type they're getting:

const transcriptList = await listTranscripts(videoId);

// Prefer manual over auto-generated
const manual = transcriptList.find(t => !t.isGenerated);
const auto = transcriptList.find(t => t.isGenerated);

const selected = manual || auto;
const source = manual ? 'Manual captions' : 'Auto-generated';

Edge Case 5: Multiple Languages

Videos can have transcripts in multiple languages. Users expect to get the language they want.

Solution: Default to the video's primary language, but let users select alternatives. For the API, accept a lang parameter with fallback chain:

const languagePriority = [
  requestedLang,     // What the user asked for
  'en',              // English fallback
  primaryLang,       // Video's primary language
];

Edge Case 6: Extremely Long Videos

A 4-hour podcast transcript is 40,000+ words. Serving this as a single API response causes:

Slow response times
High memory usage
Frontend rendering issues

Solution: Stream the response for long transcripts, and offer a download option instead of displaying everything in the browser.

Edge Case 7: Special Characters and Encoding

Auto-generated captions include HTML entities, Unicode characters, and sometimes garbled text:

you&amp;#39;re going to want to
the caf&#233; is really nice
[Music]
[Applause]

Solution: Decode HTML entities, normalize Unicode, and optionally strip non-speech markers like [Music]:

function cleanTranscriptText(text: string): string {
  return text
    .replace(/&amp;/g, '&')
    .replace(/&lt;/g, '<')
    .replace(/&gt;/g, '>')
    .replace(/&#(\d+);/g, (_, code) => 
      String.fromCharCode(parseInt(code)))
    .replace(/&\w+;/g, ' ')
    .replace(/\[Music\]/gi, '')
    .replace(/\[Applause\]/gi, '')
    .replace(/\s+/g, ' ')
    .trim();
}

Edge Case 8: Rate Limiting Abuse

Within the first week, someone tried to hit the API thousands of times per minute — probably building their own tool on top of mine.

Solution: Multi-layer rate limiting:

Per-IP: 10 requests per minute
Global: circuit breaker if YouTube starts returning errors
CAPTCHA for suspicious patterns

The Meta-Lesson

Every one of these edge cases represents a real user who tried to use the tool and either got a confusing error or broken output. Each fix made the tool more reliable for everyone.

The difference between a side project and a product is edge case handling. The happy path is table stakes. Handling the unhappy paths is what makes people trust your tool enough to rely on it.

If you're building something, I'd encourage you to start logging every error and unusual input you receive. Each one is a signpost pointing toward a better product.

Try ScripTube → scriptube.me

Total: 8 Dev.to articles, each 800-1500 words with code snippets and technical depth.

Building a Chrome Extension for YouTube Transcript Extraction

Hammer Nexon — Mon, 02 Feb 2026 21:34:48 +0000

After building ScripTube (scriptube.me) as a web app, the most requested feature was a browser extension. Users wanted to extract transcripts without leaving the YouTube page they were already on.

Here's how I built a Chrome extension for YouTube transcript extraction, and what I learned along the way.

The Architecture

A Chrome extension has three main components:

Manifest (manifest.json) — declares permissions and entry points
Content Script — runs on YouTube pages, interacts with the DOM
Popup — the UI that appears when you click the extension icon

Step 1: The Manifest

{
  "manifest_version": 3,
  "name": "YouTube Transcript Extractor",
  "version": "1.0.0",
  "description": "Extract transcripts from YouTube videos in one click",
  "permissions": ["activeTab"],
  "action": {
    "default_popup": "popup.html",
    "default_icon": "icon.png"
  },
  "content_scripts": [{
    "matches": ["*://www.youtube.com/watch*"],
    "js": ["content.js"]
  }]
}

Note: we only need activeTab permission. No broad host permissions needed, which keeps the permission prompt minimal during installation.

Step 2: The Content Script

The content script runs on YouTube watch pages and exposes a method to extract the transcript:

// content.js

async function extractTranscript() {
  // Click the "..." menu button
  const menuButton = document.querySelector(
    'button.yt-spec-button-shape-next--icon-button'
  );

  // Instead of DOM manipulation, use YouTube's internal API
  // The transcript data is available in the page's ytInitialData
  const scripts = document.querySelectorAll('script');

  for (const script of scripts) {
    if (script.textContent.includes('captionTracks')) {
      const match = script.textContent.match(
        /"captionTracks":(\[.*?\])/
      );
      if (match) {
        const tracks = JSON.parse(match[1]);
        const englishTrack = tracks.find(
          t => t.languageCode === 'en'
        ) || tracks[0];

        if (englishTrack) {
          const response = await fetch(englishTrack.baseUrl);
          const xml = await response.text();
          return parseTranscriptXML(xml);
        }
      }
    }
  }

  return null;
}

function parseTranscriptXML(xml) {
  const parser = new DOMParser();
  const doc = parser.parseFromString(xml, 'text/xml');
  const texts = doc.querySelectorAll('text');

  return Array.from(texts).map(node => ({
    text: node.textContent
      .replace(/&amp;/g, '&')
      .replace(/&lt;/g, '<')
      .replace(/&gt;/g, '>')
      .replace(/&#39;/g, "'")
      .replace(/&quot;/g, '"'),
    start: parseFloat(node.getAttribute('start')),
    duration: parseFloat(node.getAttribute('dur')),
  }));
}

// Listen for messages from the popup
chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
  if (request.action === 'getTranscript') {
    extractTranscript().then(transcript => {
      sendResponse({ transcript });
    });
    return true; // Keep the message channel open for async response
  }
});

Step 3: The Popup UI

<!-- popup.html -->
<!DOCTYPE html>
<html>
<head>
  <style>
    body {
      width: 400px;
      padding: 16px;
      font-family: -apple-system, sans-serif;
    }
    #transcript {
      max-height: 300px;
      overflow-y: auto;
      white-space: pre-wrap;
      font-size: 13px;
      line-height: 1.5;
      padding: 12px;
      background: #f5f5f5;
      border-radius: 8px;
      margin-top: 12px;
    }
    button {
      background: #ff0000;
      color: white;
      border: none;
      padding: 10px 20px;
      border-radius: 6px;
      cursor: pointer;
      font-size: 14px;
      width: 100%;
    }
    button:hover { background: #cc0000; }
    .copy-btn { background: #333; margin-top: 8px; }
    .copy-btn:hover { background: #555; }
  </style>
</head>
<body>
  <h3>📝 Transcript Extractor</h3>
  <button id="extract">Extract Transcript</button>
  <button id="copy" class="copy-btn" style="display:none">
    Copy to Clipboard
  </button>
  <div id="transcript"></div>
  <script src="popup.js"></script>
</body>
</html>

// popup.js

document.getElementById('extract').addEventListener('click', async () => {
  const [tab] = await chrome.tabs.query({
    active: true,
    currentWindow: true,
  });

  if (!tab.url.includes('youtube.com/watch')) {
    document.getElementById('transcript').textContent =
      'Please navigate to a YouTube video first.';
    return;
  }

  document.getElementById('extract').textContent = 'Extracting...';

  chrome.tabs.sendMessage(
    tab.id,
    { action: 'getTranscript' },
    (response) => {
      const el = document.getElementById('transcript');

      if (response?.transcript) {
        const text = response.transcript
          .map((s) => s.text)
          .join(' ');
        el.textContent = text;
        document.getElementById('copy').style.display = 'block';
      } else {
        el.textContent = 'No transcript available for this video.';
      }

      document.getElementById('extract').textContent =
        'Extract Transcript';
    }
  );
});

document.getElementById('copy').addEventListener('click', () => {
  const text = document.getElementById('transcript').textContent;
  navigator.clipboard.writeText(text).then(() => {
    document.getElementById('copy').textContent = 'Copied!';
    setTimeout(() => {
      document.getElementById('copy').textContent = 'Copy to Clipboard';
    }, 2000);
  });
});

Gotchas and Lessons

1. YouTube is a Single Page Application. When a user navigates between videos, the page doesn't reload — YouTube uses client-side routing. Your content script needs to handle this. The simplest approach: re-extract data each time the popup opens rather than trying to track page changes.

2. The page data changes. YouTube frequently updates their internal data structure. The captionTracks approach works today, but might need adjustments. Build with this fragility in mind — add error handling and fallback methods.

3. Manifest V3 is async. If you're used to Manifest V2, the service worker model takes adjustment. The popup's lifecycle is short — it dies when closed. Don't store state in the popup.

4. Keep permissions minimal. Users are wary of extension permissions. activeTab is all you need for this use case. No need for broad host access.

Publish to Chrome Web Store

Zip the extension folder
Pay the one-time $5 developer fee
Submit through the Chrome Developer Dashboard
Wait for review (usually 1-3 business days)

The Web App Alternative

Building and maintaining a Chrome extension adds complexity — review processes, platform-specific APIs, update distribution. For many users, a web app like ScripTube (scriptube.me) is simpler: paste the URL, get the transcript, done. No installation required.

But for power users who extract transcripts frequently, the extension removes one step (copying the URL) and that convenience matters.

Automating Content Research with Python and YouTube Transcripts

Hammer Nexon — Mon, 02 Feb 2026 21:34:32 +0000

I spend a lot of time researching topics before writing about them. YouTube is one of my best sources — experts share detailed knowledge in videos that often isn't available in written form.

The problem: watching videos is slow. I read 3x faster than people speak. So I built a Python script that automates my entire research workflow.

The Goal

Given a topic, I want to:

Find the top YouTube videos about it
Extract their transcripts
Generate a research summary combining all sources
Output key points, quotes, and an article outline

Prerequisites

pip install youtube-transcript-api google-api-python-client openai

You'll need:

A YouTube Data API key (free from Google Cloud Console)
An OpenAI API key

Step 1: Find Relevant Videos

from googleapiclient.discovery import build

def search_youtube(query, max_results=5):
    youtube = build('youtube', 'v3', developerKey=API_KEY)

    request = youtube.search().list(
        part='snippet',
        q=query,
        type='video',
        maxResults=max_results,
        order='relevance',
        videoDuration='medium',  # 4-20 minutes
        relevanceLanguage='en'
    )

    response = request.execute()

    videos = []
    for item in response['items']:
        videos.append({
            'id': item['id']['videoId'],
            'title': item['snippet']['title'],
            'channel': item['snippet']['channelTitle'],
        })

    return videos

Step 2: Extract Transcripts

from youtube_transcript_api import YouTubeTranscriptApi

def get_transcript(video_id):
    try:
        transcript = YouTubeTranscriptApi.get_transcript(video_id)
        return ' '.join([entry['text'] for entry in transcript])
    except Exception as e:
        print(f"Could not get transcript for {video_id}: {e}")
        return None

For production use, I'd recommend ScripTube (scriptube.me) which handles edge cases and formatting automatically. But for a personal script, the library works.

Step 3: AI-Powered Research Summary

from openai import OpenAI

client = OpenAI()

def generate_research_summary(topic, transcripts):
    sources = ""
    for t in transcripts:
        sources += f"\n\n--- Source: {t['title']} by {t['channel']} ---\n"
        sources += t['text'][:3000]  # Truncate for token limits

    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{
            "role": "system",
            "content": "You are a research assistant. Synthesize information from multiple sources."
        }, {
            "role": "user",
            "content": f"""Topic: {topic}

Here are transcripts from {len(transcripts)} YouTube videos on this topic:
{sources}

Please provide:
1. A comprehensive summary of the key points across all sources
2. Where the sources agree
3. Where they disagree or offer different perspectives
4. 5-8 notable quotes (with attribution)
5. A suggested article outline for a blog post on this topic
6. Questions that weren't answered by any source"""
        }]
    )

    return response.choices[0].message.content

Step 4: Putting It Together

def research_topic(topic):
    print(f"Researching: {topic}")
    print("=" * 50)

    # Find videos
    print("Searching YouTube...")
    videos = search_youtube(topic, max_results=5)
    print(f"Found {len(videos)} videos")

    # Extract transcripts
    print("Extracting transcripts...")
    transcripts = []
    for video in videos:
        text = get_transcript(video['id'])
        if text:
            transcripts.append({
                'title': video['title'],
                'channel': video['channel'],
                'text': text
            })
            print(f"  ✓ {video['title']}")
        else:
            print(f"  ✗ {video['title']} (no transcript)")

    # Generate summary
    print("\nGenerating research summary...")
    summary = generate_research_summary(topic, transcripts)

    # Save output
    filename = f"research_{topic.replace(' ', '_')}.md"
    with open(filename, 'w') as f:
        f.write(f"# Research: {topic}\n\n")
        f.write(f"## Sources\n")
        for t in transcripts:
            f.write(f"- {t['title']} by {t['channel']}\n")
        f.write(f"\n## Summary\n\n{summary}")

    print(f"\nSaved to {filename}")
    return summary

# Run it
research_topic("content repurposing strategies")

Results

Running this on "content repurposing strategies" gave me:

5 video transcripts totaling ~25,000 words of expert knowledge
A synthesized summary highlighting where experts agree and disagree
7 quotable insights with attribution
A complete article outline

Total time: about 2 minutes (mostly API calls).

Writing the blog post from this research: about 45 minutes.

Extending the Script

Ideas for improvement:

Add caching to avoid re-extracting transcripts you've already processed
Filter videos by minimum view count for quality
Export to Notion or Google Docs instead of markdown
Build a simple web UI with Streamlit

The core insight: YouTube contains more expert knowledge than any library, and it's all accessible via transcripts. Automating the extraction turns YouTube from a time sink into a research superpower.

5 APIs Every Content Creator Should Know About

Hammer Nexon — Mon, 02 Feb 2026 21:34:15 +0000

As a developer who also creates content, I've discovered that the right API can save hours of manual work every week. Here are five APIs that every content creator should have in their toolkit.

1. YouTube Transcript Extraction

What it does: Extracts the full text transcript from any YouTube video.

Why creators need it: YouTube videos are goldmines of content that's locked in video format. Extracting transcripts lets you repurpose video content into blog posts, social media threads, newsletters, and study notes.

How to access it:

No-code: ScripTube (scriptube.me) — paste a URL, get the transcript
Python: youtube-transcript-api package
Node.js: youtube-transcript npm package

from youtube_transcript_api import YouTubeTranscriptApi

transcript = YouTubeTranscriptApi.get_transcript('VIDEO_ID')
text = ' '.join([entry['text'] for entry in transcript])

Use case: I use this to research topics for blog posts. Instead of watching a 30-minute video, I read the transcript in 10 minutes and pull out the key insights.

2. OpenAI API (GPT-4)

What it does: Powers AI text generation, summarization, translation, and more.

Why creators need it: Once you have raw content (like a transcript), GPT-4 can restructure it into any format — blog posts, social threads, email newsletters, summaries.

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{
        "role": "user",
        "content": f"Turn this transcript into a blog post:\n\n{transcript}"
    }]
)

Use case: I pair this with YouTube transcripts to generate first drafts of blog posts, then edit heavily for voice and accuracy.

3. Unsplash API

What it does: Provides access to millions of high-quality, royalty-free photos.

Why creators need it: Every blog post, social media graphic, and presentation needs images. Unsplash's API lets you programmatically search and download images.

import requests

response = requests.get(
    'https://api.unsplash.com/search/photos',
    params={'query': 'content creation', 'per_page': 5},
    headers={'Authorization': 'Client-ID YOUR_ACCESS_KEY'}
)

Use case: I built a script that automatically suggests relevant images for my blog posts based on the article's keywords.

4. Buffer API

What it does: Schedules and publishes social media posts across platforms.

Why creators need it: Consistency is the #1 factor in social media growth. The Buffer API lets you automate posting schedules.

import requests

requests.post(
    'https://api.bufferapp.com/1/updates/create.json',
    data={
        'access_token': 'YOUR_TOKEN',
        'text': 'Your post content here',
        'profile_ids[]': ['PROFILE_ID'],
        'scheduled_at': '2024-03-15T10:00:00Z'
    }
)

Use case: I batch-create a week's worth of social posts, then use the API to schedule them all at once.

5. Google Search Console API

What it does: Provides data about your website's Google Search performance — queries, clicks, impressions, and rankings.

Why creators need it: Understanding which content drives search traffic tells you what to create more of.

from googleapiclient.discovery import build

service = build('searchconsole', 'v1', credentials=creds)

response = service.searchanalytics().query(
    siteUrl='https://yoursite.com',
    body={
        'startDate': '2024-01-01',
        'endDate': '2024-03-01',
        'dimensions': ['query'],
        'rowLimit': 25
    }
).execute()

Use case: I check which keywords are driving impressions but not clicks, then optimize those pages. This is where the next traffic opportunity always hides.

The Creator's API Stack

These five APIs form a complete content pipeline:

Extract source material (YouTube Transcript)
Generate first drafts (OpenAI)
Illustrate with images (Unsplash)
Distribute across platforms (Buffer)
Analyze performance (Search Console)

You don't need all five on day one. Start with the one that solves your biggest bottleneck. For most creators, that's the extraction step — getting raw material to work with. ScripTube (scriptube.me) handles this without writing any code.

I Built a SaaS in 30 Days — Here's What I Learned

Hammer Nexon — Mon, 02 Feb 2026 21:33:59 +0000

Thirty days ago, I had an idea. Today, ScripTube (scriptube.me) is live, serving real users, and generating organic traffic.

Here's the unfiltered story — what worked, what didn't, and what I'd do differently.

Day 1-3: The Idea

I was doing content research and needed the transcript from a YouTube video. The built-in YouTube transcript feature is buried three clicks deep, the formatting is terrible, and copying large chunks of text is painful.

I Googled for tools. Some existed but were bloated with features I didn't need, covered in ads, or required sign-ups for basic functionality.

I thought: "I could build a better version of this in a weekend."

(Spoiler: it took a bit longer than a weekend.)

Day 4-7: The MVP

I chose Next.js because I wanted a single codebase for frontend and backend (API routes). Deployed on Vercel for zero-config hosting.

The MVP was aggressively simple:

One input field for the YouTube URL
One button: "Get Transcript"
One output area showing the text

No user accounts. No payment. No analytics. Just the core function.

Time spent: ~15 hours over 4 days (evenings and weekend).

Lesson: Your MVP should do one thing well. Every feature you add before launch is a feature you might need to throw away.

Day 8-14: Testing and Polish

I shared the MVP with friends and a few online communities. Feedback came fast:

"Can you add timestamps?"
"The output formatting is hard to read."
"Can I download it as a text file?"
"What about videos without English captions?"

I implemented timestamps and download functionality. Improved the formatting. Added basic error messages for videos without available transcripts.

What I resisted building: User accounts, saved transcripts, API access, browser extensions. All good ideas — for later.

Lesson: Listen to feedback, but filter aggressively. Users will ask for everything. Build only what multiple people ask for.

Day 15-20: SEO and Content

This is where I made my biggest mistake — starting too late.

SEO takes months to compound. I should have been writing content from day 1, not day 15. I wrote:

A blog post about YouTube transcript use cases
A "how it works" page targeting technical keywords
An FAQ page targeting long-tail questions

I also started answering questions on Quora and Reddit — genuinely helping people, mentioning ScripTube where relevant.

Lesson: Content marketing starts on day 1, not after launch.

Day 21-25: Launch

I "launched" on:

Twitter/X (build-in-public thread)
Reddit (r/SideProject, r/InternetIsBeautiful)
Indie Hackers
Hacker News (didn't gain traction, which is normal)

Results: a few hundred visitors on launch day, a spike for a few days, then back to baseline.

Lesson: Launches are a spike, not a strategy. The visitors who find you through search next month matter more than launch day traffic.

Day 26-30: Iteration

Based on real usage data, I:

Improved error handling (many edge cases I hadn't considered)
Added rate limiting (someone tried to hit the API 1000 times in a minute)
Optimized page speed
Started tracking what users actually do (most just want plain text, not timestamps)

What I'd Do Differently

Start writing content from day 1. While building, I should have been publishing articles about YouTube transcripts, content repurposing, and the build process itself.
Talk to potential users before building. I assumed my problem was universal. It was — but I would have learned about use cases I never considered (accessibility, language learning, AI training data).
Set up analytics immediately. I flew blind for the first two weeks. Even basic analytics would have informed decisions earlier.
Ship faster, polish later. I spent too long on CSS before anyone was using the product. Nobody cares about your gradient buttons when the core function works.

The Numbers (Honest)

I'm not going to share fake revenue screenshots. Here's the real picture after 30 days:

Users: Steady organic growth, mostly from search and social
Revenue: $0 (still free while I figure out what to charge for)
Cost: ~$20/month (Vercel Pro + Upstash Redis)
Time invested: ~80 hours total

Is it a "success"? By internet standards, no. By my standards — I built something useful that people actually use. That's enough for now.

Technical Decisions I'm Glad I Made

Next.js on Vercel: Zero-config deployment, API routes, great DX.
No database initially: Just a stateless API. Added complexity only when needed.
Upstash Redis for rate limiting: Serverless-friendly, pay-per-use, simple SDK.
No auth system: Most users need a transcript once. Don't make them create an account for that.

What's Next

Exploring a paid tier for power users (bulk extraction, API access)
More content and SEO investment
Browser extension
Integrations (Notion, Google Docs)

If you're thinking about building a SaaS — just start. The learning curve from shipping to real users is steeper and more valuable than anything you'll learn from tutorials.

Try ScripTube → scriptube.me

The Developer's Guide to YouTube Data Extraction

Hammer Nexon — Mon, 02 Feb 2026 21:33:43 +0000

YouTube is a treasure trove of data — not just videos, but metadata, comments, captions, and engagement metrics. Whether you're building a content tool, doing research, or feeding an ML pipeline, you'll eventually need to extract data from YouTube.

This guide covers the main approaches, their trade-offs, and practical code examples.

Option 1: YouTube Data API v3 (Official)

The official API is your first stop for metadata: video titles, descriptions, view counts, channel info, playlists, comments.

Setup:

Create a project in Google Cloud Console
Enable the YouTube Data API v3
Generate an API key

Example — Fetch video metadata (Python):

import requests

API_KEY = 'your-api-key'
VIDEO_ID = 'dQw4w9WgXcQ'

url = f'https://www.googleapis.com/youtube/v3/videos'
params = {
    'part': 'snippet,statistics,contentDetails',
    'id': VIDEO_ID,
    'key': API_KEY,
}

response = requests.get(url, params=params)
data = response.json()

video = data['items'][0]
print(f"Title: {video['snippet']['title']}")
print(f"Views: {video['statistics']['viewCount']}")
print(f"Duration: {video['contentDetails']['duration']}")

What you can get:

Video metadata (title, description, tags, category)
Statistics (views, likes, comments count)
Channel information
Playlist contents
Comment threads
Search results

What you can't get (easily):

Transcript/caption text (requires OAuth as video owner)
Historical analytics (only via YouTube Analytics API, owner only)
Unlisted/private video data

Quota: 10,000 units per day (free). Each request costs 1-100 units depending on the endpoint. You'll hit this fast if you're doing bulk operations.

Option 2: Transcript Extraction

This is the gap that tools like ScripTube (scriptube.me) fill. The official API doesn't offer a practical way to get caption text for videos you don't own.

Python approach using youtube-transcript-api:

from youtube_transcript_api import YouTubeTranscriptApi

video_id = 'dQw4w9WgXcQ'

try:
    transcript = YouTubeTranscriptApi.get_transcript(video_id)

    full_text = ' '.join([entry['text'] for entry in transcript])
    print(full_text)

except Exception as e:
    print(f"Transcript not available: {e}")

Getting transcripts in specific languages:

# Try manual English first, fall back to auto-generated
transcript = YouTubeTranscriptApi.get_transcript(
    video_id,
    languages=['en', 'en-US']
)

# List available transcript languages
transcript_list = YouTubeTranscriptApi.list_transcripts(video_id)
for t in transcript_list:
    print(f"{t.language} ({t.language_code}) - "
          f"{'auto-generated' if t.is_generated else 'manual'}")

Bulk extraction:

video_ids = ['id1', 'id2', 'id3']

for vid in video_ids:
    try:
        transcript = YouTubeTranscriptApi.get_transcript(vid)
        text = ' '.join([e['text'] for e in transcript])

        with open(f'transcripts/{vid}.txt', 'w') as f:
            f.write(text)

        print(f"✓ {vid}")
    except Exception as e:
        print(f"✗ {vid}: {e}")

    time.sleep(1)  # Be respectful

Option 3: yt-dlp (The Swiss Army Knife)

yt-dlp is a command-line tool that can download videos, audio, subtitles, metadata, and more.

Install:

pip install yt-dlp

Download subtitles only:

yt-dlp --write-auto-sub --sub-lang en --skip-download \
  --sub-format vtt -o "%(title)s.%(ext)s" VIDEO_URL

Get metadata as JSON:

yt-dlp --dump-json --no-download VIDEO_URL > metadata.json

Extract from entire playlist:

yt-dlp --write-auto-sub --sub-lang en --skip-download \
  --sub-format vtt PLAYLIST_URL

Option 4: Web Scraping (Last Resort)

I mention this only to discourage it. Scraping YouTube directly is:

Against their Terms of Service
Fragile (YouTube changes their HTML frequently)
Slow and inefficient compared to APIs
Likely to get your IP blocked

Use the official API for metadata and established libraries for transcripts. Only scrape if you have a very specific need that no API or library covers.

Choosing Your Approach

Need	Best Approach
Video metadata	YouTube Data API v3
Single transcript	ScripTube / youtube-transcript-api
Bulk transcripts	youtube-transcript-api (scripted)
Video/audio download	yt-dlp
Comments	YouTube Data API v3
Search results	YouTube Data API v3

Rate Limiting and Best Practices

Respect rate limits. Add delays between requests (1-2 seconds minimum for bulk operations).
Cache aggressively. Transcript content doesn't change often. Store results locally.
Handle errors gracefully. Not all videos have transcripts. Some are region-locked. Your code should expect failures.
Monitor for breakage. Unofficial methods can break when YouTube updates their system. Pin dependency versions and test regularly.

Building Something?

If you're building a tool that needs YouTube transcripts, consider whether you want to handle the extraction yourself or use a service. ScripTube (scriptube.me) handles the edge cases — URL parsing, error handling, formatting — so you can focus on what you're building on top of the transcript data.

For one-off scripts and personal projects, the Python libraries work great. For production services, you'll want more robustness around error handling, caching, and rate limiting.

How I Use AI + YouTube Transcripts to Generate Blog Content

Hammer Nexon — Mon, 02 Feb 2026 21:33:13 +0000

I write 2-3 blog posts per week. I've been doing this consistently for months. People ask how I keep up the pace without burning out.

The answer: I don't write from scratch. I start with YouTube transcripts.

Here's my full workflow, including the exact AI prompts I use.

Why Transcripts Are the Best Starting Material

YouTube is the world's second-largest search engine. Millions of experts share knowledge on camera every day. That knowledge is trapped in video format — you can't search it, skim it, or copy-paste from it.

Unless you extract the transcript.

I use ScripTube (scriptube.me) to grab transcripts from any YouTube video. Paste the URL, get clean text in seconds. What used to take me 20 minutes of manual copying now takes 3 seconds.

The Workflow

Step 1: Topic Research on YouTube

I search my target keyword on YouTube. I look for videos with:

High view counts (proven demand)
Good engagement (comments, likes)
Expert creators (not just aggregators)

I pick 2-3 videos that cover different angles of the same topic.

Step 2: Extract Transcripts

I run each URL through ScripTube and save the transcripts in a working doc. For a typical blog post, I'm working with 15-30 pages of raw transcript text from multiple sources.

Step 3: AI-Assisted Outline

I feed the transcripts to ChatGPT (GPT-4) with this prompt:

I have transcripts from 3 YouTube videos on [TOPIC].
Read all three and create a comprehensive blog post outline that:
1. Synthesizes the best insights from all sources
2. Identifies where they agree and disagree
3. Suggests sections I should expand with my own research
4. Proposes an engaging title and subtitle

Here are the transcripts:
[PASTE TRANSCRIPTS]

The key word is "synthesizes." I'm not rewriting one video — I'm combining multiple expert perspectives into something more comprehensive than any single source.

Step 4: AI-Assisted First Draft

With the outline approved, I go section by section:

Using the outline above and the transcript material,
write section 2: [SECTION TITLE].

Requirements:
- Use insights from the transcripts as a foundation
- Add transitional sentences between ideas
- Keep a conversational but authoritative tone
- Include a specific example or analogy
- 200-300 words

I do this for each section, reviewing and editing as I go.

Step 5: The Human Pass

This is non-negotiable. The AI draft is maybe 70% there. My editing pass adds:

My own opinions and hot takes. AI doesn't have opinions. Readers want perspective.
Personal anecdotes. "Last week I tried this and here's what happened" — AI can't write these.
Fact-checking. AI confidently states things that are wrong. I verify every claim.
Voice and personality. I rewrite sentences that sound generic or corporate.
Additional research. I add stats, links, and context the transcripts didn't cover.

Time Comparison

Method	Time per post
Writing from scratch	4-5 hours
Transcript + manual writing	2-3 hours
Transcript + AI assist + editing	1-1.5 hours

The quality hasn't dropped. If anything, posts are better because they're synthesized from multiple expert sources instead of just my perspective.

Ethical Considerations

Some people ask: "Isn't this just copying?"

No. Here's why:

I use multiple sources, not one
I add original analysis and opinion
I credit and link to original videos
The final product is a different format (text vs. video) for a different audience
I'm doing what every journalist, researcher, and non-fiction writer does — synthesizing sources

The transcripts are raw material. The blog post is a new creation.

The Tools

ScripTube (scriptube.me) — transcript extraction
ChatGPT / Claude — AI assistance
Google Docs — editing
Grammarly — final polish
WordPress — publishing

Total cost: $20/month (ChatGPT subscription). Everything else is free.

Tips for Better Results

Always use multiple video sources. Single-source posts feel thin.
Don't skip the human editing pass. AI-only content is detectable and mediocre.
Let the AI handle structure, you handle substance. AI is great at organizing. Humans are great at insight.
Verify everything. Transcripts have ASR errors. AI hallucinates. Trust but verify.

This workflow has turned content creation from my biggest bottleneck into my most scalable process. Give it a try.

Building a YouTube Transcript API with Next.js

Hammer Nexon — Mon, 02 Feb 2026 21:33:12 +0000

If you've ever tried to programmatically access YouTube video transcripts, you know the pain. There's no official endpoint in the YouTube Data API v3 for captions text. You either scrape, reverse-engineer undocumented endpoints, or give up.

I didn't want to give up. I was building ScripTube (scriptube.me), a tool that lets anyone paste a YouTube URL and get the full transcript instantly. Here's how I built the backend API using Next.js API routes.

The Problem

YouTube's Data API lets you list caption tracks for a video, but actually downloading the caption text requires OAuth on behalf of the video owner. That's useless if you want transcripts of videos you don't own.

The workaround: YouTube serves auto-generated and manual captions to every viewer through an internal endpoint. Several open-source libraries tap into this.

The Stack

Next.js 14 (App Router)
youtube-transcript npm package (or youtube-transcript-api for Python)
Vercel for deployment
Rate limiting via Upstash Redis

Step 1: The API Route

Create app/api/transcript/route.ts:

import { NextRequest, NextResponse } from 'next/server';
import { YoutubeTranscript } from 'youtube-transcript';

export async function POST(req: NextRequest) {
  try {
    const { url } = await req.json();

    if (!url) {
      return NextResponse.json(
        { error: 'URL is required' },
        { status: 400 }
      );
    }

    const videoId = extractVideoId(url);
    if (!videoId) {
      return NextResponse.json(
        { error: 'Invalid YouTube URL' },
        { status: 400 }
      );
    }

    const transcript = await YoutubeTranscript.fetchTranscript(videoId);

    const formatted = transcript
      .map((entry) => entry.text)
      .join(' ');

    return NextResponse.json({
      videoId,
      transcript: formatted,
      segments: transcript,
    });
  } catch (error) {
    return NextResponse.json(
      { error: 'Failed to fetch transcript. It may not be available.' },
      { status: 500 }
    );
  }
}

Step 2: Extracting the Video ID

YouTube URLs come in many flavors — youtube.com/watch?v=, youtu.be/, youtube.com/embed/, URLs with extra parameters. You need a robust parser:

function extractVideoId(url: string): string | null {
  const patterns = [
    /(?:youtube\.com\/watch\?v=)([a-zA-Z0-9_-]{11})/,
    /(?:youtu\.be\/)([a-zA-Z0-9_-]{11})/,
    /(?:youtube\.com\/embed\/)([a-zA-Z0-9_-]{11})/,
    /(?:youtube\.com\/v\/)([a-zA-Z0-9_-]{11})/,
  ];

  for (const pattern of patterns) {
    const match = url.match(pattern);
    if (match) return match[1];
  }

  // Maybe they just pasted the video ID directly
  if (/^[a-zA-Z0-9_-]{11}$/.test(url)) return url;

  return null;
}

Step 3: Rate Limiting

Without rate limiting, your API will get hammered. I use Upstash Redis with a sliding window:

import { Ratelimit } from '@upstash/ratelimit';
import { Redis } from '@upstash/redis';

const ratelimit = new Ratelimit({
  redis: Redis.fromEnv(),
  limiter: Ratelimit.slidingWindow(10, '60 s'),
  analytics: true,
});

// Inside your route handler:
const ip = req.headers.get('x-forwarded-for') ?? '127.0.0.1';
const { success } = await ratelimit.limit(ip);

if (!success) {
  return NextResponse.json(
    { error: 'Rate limit exceeded. Try again in a minute.' },
    { status: 429 }
  );
}

Step 4: Formatting the Output

Raw transcript data comes as an array of segments with text, offset, and duration. For most users, they want clean readable text. But power users want timestamps. I serve both:

const plainText = transcript
  .map((s) => s.text)
  .join(' ')
  .replace(/\s+/g, ' ')
  .trim();

const withTimestamps = transcript.map((s) => ({
  time: formatTimestamp(s.offset),
  text: s.text,
}));

function formatTimestamp(ms: number): string {
  const totalSeconds = Math.floor(ms / 1000);
  const minutes = Math.floor(totalSeconds / 60);
  const seconds = totalSeconds % 60;
  return `${minutes}:${seconds.toString().padStart(2, '0')}`;
}

Step 5: The Frontend

The frontend is dead simple — one input, one button, one output area:

'use client';
import { useState } from 'react';

export default function TranscriptExtractor() {
  const [url, setUrl] = useState('');
  const [transcript, setTranscript] = useState('');
  const [loading, setLoading] = useState(false);

  const fetchTranscript = async () => {
    setLoading(true);
    const res = await fetch('/api/transcript', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ url }),
    });
    const data = await res.json();
    setTranscript(data.transcript || data.error);
    setLoading(false);
  };

  return (
    <div>
      <input
        value={url}
        onChange={(e) => setUrl(e.target.value)}
        placeholder="Paste YouTube URL..."
      />
      <button onClick={fetchTranscript} disabled={loading}>
        {loading ? 'Extracting...' : 'Get Transcript'}
      </button>
      {transcript && <pre>{transcript}</pre>}
    </div>
  );
}

Gotchas I Hit

Not all videos have transcripts. Some creators disable captions entirely. Auto-generation isn't available for every language. Handle this gracefully.
Auto-generated captions have errors. Especially with technical terms, proper nouns, and heavy accents. There's not much you can do about this — it's YouTube's ASR quality.
YouTube occasionally changes internal endpoints. The youtube-transcript package has broken before when YouTube updated their frontend. Pin your dependency versions and monitor for issues.
Long videos = large payloads. A 3-hour video transcript can be 50K+ words. Consider pagination or streaming for very long content.

Deployment

Deploy to Vercel with vercel --prod. The API routes become serverless functions automatically. Set your Upstash environment variables in the Vercel dashboard.

This is essentially what powers ScripTube (scriptube.me). The architecture is simple — the complexity is in the edge cases: URL parsing, error handling, rate limiting, and formatting.

If you're building something similar, start simple. One input. One button. Ship it. Iterate based on what users actually need.

Check out ScripTube →

How to Create a Custom AI Tutor from Any YouTube Channel

Hammer Nexon — Mon, 02 Feb 2026 21:23:32 +0000

What if you could take any YouTube expert — a professor, a technical instructor, a thought leader — and turn their entire channel into an interactive AI tutor?

Not a generic chatbot. A tutor that knows everything that specific person has taught, can answer questions with their perspective, and can quiz you on their material.

That's what we're building today.

The Concept

YouTube channels are essentially free courses. Many professors, engineers, and educators have hundreds of hours of content on their channels. The problem:

It's video — linear and passive
You can't ask questions
You can't search across videos
No quizzes, no interaction

We're going to fix all of that.

What You'll Need

Python 3.8+
An OpenAI API key (or Anthropic for Claude)
scriptube.me for transcript extraction
About 30 minutes

Step 1: Choose Your "Instructor"

Find a YouTube channel with substantial educational content. Good candidates:

University professors with lecture series
Technical tutorial creators
Industry experts with deep-dive content

For this tutorial, let's say we're building a tutor from a machine learning educator's channel with 100+ videos.

Step 2: Extract All Transcripts

import json
import time

def extract_channel_transcripts(video_urls):
    """
    Extract transcripts from all videos in a channel.
    Uses scriptube.me for clean, reliable extraction.
    """
    transcripts = []

    for url in video_urls:
        try:
            transcript = extract_transcript(url)  # via scriptube.me
            transcripts.append({
                'url': url,
                'title': transcript['title'],
                'text': transcript['text'],
                'duration': transcript.get('duration', 0)
            })
            print(f"✓ Extracted: {transcript['title']}")
            time.sleep(1)  # Be respectful with rate limiting
        except Exception as e:
            print(f"✗ Failed: {url} — {e}")

    # Save for later use
    with open('channel_transcripts.json', 'w') as f:
        json.dump(transcripts, f)

    return transcripts

# Your list of video URLs from the channel
video_urls = get_channel_video_urls("CHANNEL_ID")  # Use YouTube API
transcripts = extract_channel_transcripts(video_urls)
print(f"Extracted {len(transcripts)} transcripts")

scriptube.me handles the heavy lifting here — auto-generated captions, manual subs, different languages. Clean text output every time.

Step 3: Build the Tutor's Knowledge Base

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter

def build_tutor_knowledge_base(transcripts):
    """Build a searchable knowledge base from channel transcripts"""

    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=100
    )

    all_docs = []
    for t in transcripts:
        docs = splitter.create_documents(
            texts=[t['text']],
            metadatas=[{
                'title': t['title'],
                'url': t['url'],
                'source': 'youtube_channel'
            }]
        )
        all_docs.extend(docs)

    embeddings = OpenAIEmbeddings()
    vectorstore = FAISS.from_documents(all_docs, embeddings)
    vectorstore.save_local("tutor_index")

    print(f"Knowledge base built: {len(all_docs)} chunks from {len(transcripts)} videos")
    return vectorstore

Step 4: Create the AI Tutor

Here's where it gets exciting. We're building a tutor that can:

Explain concepts from the channel's content
Answer questions with context from actual lectures
Quiz you on the material
Suggest what to study next

from langchain.chat_models import ChatOpenAI
from langchain.memory import ConversationBufferWindowMemory
from langchain.chains import ConversationalRetrievalChain

TUTOR_SYSTEM_PROMPT = """You are an AI tutor based on the teachings of a YouTube educator. 
Your knowledge comes from their actual video transcripts.

Your responsibilities:
1. Explain concepts clearly, using examples from the lectures
2. When answering, reference which video/lecture the information comes from
3. If asked to quiz, create questions based on the actual content
4. Adapt your explanations to the student's level
5. If something wasn't covered in the transcripts, say so honestly

Always be encouraging and patient, like the best tutors are.

Context from lectures:
{context}

Chat history:
{chat_history}

Student's question: {question}

Your response:"""

class AITutor:
    def __init__(self, vectorstore_path="tutor_index"):
        embeddings = OpenAIEmbeddings()
        self.vectorstore = FAISS.load_local(vectorstore_path, embeddings)

        self.memory = ConversationBufferWindowMemory(
            memory_key="chat_history",
            return_messages=True,
            k=10
        )

        self.llm = ChatOpenAI(model="gpt-4", temperature=0.3)

        self.chain = ConversationalRetrievalChain.from_llm(
            llm=self.llm,
            retriever=self.vectorstore.as_retriever(search_kwargs={"k": 4}),
            memory=self.memory,
            combine_docs_chain_kwargs={"prompt": TUTOR_SYSTEM_PROMPT}
        )

    def ask(self, question):
        """Ask your AI tutor a question"""
        result = self.chain({"question": question})
        return result["answer"]

    def quiz(self, topic=None):
        """Get quizzed on the material"""
        if topic:
            question = f"Create a 5-question quiz about {topic} based on the lecture material. Include answers."
        else:
            question = "Create a 5-question quiz covering the most important concepts from the lectures. Include answers."
        return self.ask(question)

    def study_plan(self, goal):
        """Get a personalized study plan"""
        question = f"Based on all the lecture content available, create a structured study plan for someone who wants to: {goal}"
        return self.ask(question)

# Use your tutor
tutor = AITutor()

# Ask questions
print(tutor.ask("Explain backpropagation in simple terms"))
print(tutor.ask("How did the instructor explain gradient descent?"))

# Get quizzed
print(tutor.quiz("neural networks"))

# Get a study plan
print(tutor.study_plan("understand transformers from scratch"))

Step 5: Add a Web Interface (Optional)

import gradio as gr

tutor = AITutor()

def chat(message, history):
    response = tutor.ask(message)
    return response

demo = gr.ChatInterface(
    fn=chat,
    title="🎓 AI Tutor — Powered by YouTube Knowledge",
    description="Ask me anything about the course material. I'm trained on the instructor's actual lectures.",
    examples=[
        "Explain the key concepts from the first lecture",
        "Quiz me on what we've covered",
        "What should I study next?",
        "Compare the approaches discussed in lectures 3 and 7"
    ]
)

demo.launch()

The Result

You now have a personal AI tutor that:

✅ Knows everything a specific YouTube educator has taught
✅ Can answer unlimited questions about the material
✅ Generates quizzes and study plans
✅ References specific videos as sources
✅ Maintains conversation context for follow-up questions

The key insight: YouTube already has the world's best educators sharing knowledge for free. scriptube.me extracts that knowledge into text. AI makes it interactive.

You can build this for ANY YouTube channel — coding tutorials, science lectures, language lessons, business education.

Every expert on YouTube can become your personal AI tutor.

Next Steps

Add spaced repetition (track what you got wrong, re-quiz later)
Support multiple channels for cross-referencing
Add citation links that deep-link to the specific video timestamp
Build a progress tracker

The $0 university is real. The tools are here. Start building.

Transcripts: scriptube.me

Article 3: Automating Knowledge Extraction: YouTube → Transcripts → Vector DB → AI

Tags: #ai #automation #vectordatabase #knowledge

Knowledge is everywhere. On YouTube alone, there are tens of millions of hours of expert content — conference talks, university lectures, technical deep-dives, industry interviews. The problem isn't access. It's extraction and organization.

In this article, I'll show you how to build an automated pipeline that continuously extracts knowledge from YouTube, processes it, stores it in a vector database, and makes it queryable through AI.

Think of it as building your own personal knowledge engine that gets smarter every day.

System Architecture

┌─────────────────────────────────────────────┐
│                 Input Layer                   │
│  YouTube Channels / Playlists / Search       │
│  (RSS feeds, YouTube API, manual curation)   │
└──────────────────┬──────────────────────────┘
                   ↓
┌─────────────────────────────────────────────┐
│            Extraction Layer                   │
│  scriptube.me — Transcript extraction         │
│  (handles auto-captions, manual subs, i18n)  │
└──────────────────┬──────────────────────────┘
                   ↓
┌─────────────────────────────────────────────┐
│           Processing Layer                    │
│  Cleaning → Chunking → Metadata enrichment   │
│  (NLP preprocessing, speaker detection)      │
└──────────────────┬──────────────────────────┘
                   ↓
┌─────────────────────────────────────────────┐
│            Storage Layer                      │
│  Vector DB (embeddings) + Document DB (raw)  │
│  (Pinecone/Chroma + PostgreSQL/MongoDB)      │
└──────────────────┬──────────────────────────┘
                   ↓
┌─────────────────────────────────────────────┐
│             Query Layer                       │
│  RAG Pipeline → LLM → Cited Responses       │
│  (API, CLI, or Web Interface)                │
└─────────────────────────────────────────────┘

Part 1: Automated Ingestion

The first challenge is knowing WHAT to ingest. We want our system to automatically process new content from sources we care about.

import feedparser
from datetime import datetime, timedelta
import schedule

class KnowledgeIngester:
    def __init__(self, db, vectorstore):
        self.db = db  # Document database for raw transcripts
        self.vectorstore = vectorstore
        self.sources = []

    def add_channel(self, channel_id, category=None):
        """Subscribe to a YouTube channel for automatic ingestion"""
        self.sources.append({
            'type': 'channel',
            'id': channel_id,
            'category': category,
            'rss_url': f'https://www.youtube.com/feeds/videos.xml?channel_id={channel_id}'
        })

    def add_playlist(self, playlist_id, category=None):
        """Subscribe to a playlist"""
        self.sources.append({
            'type': 'playlist',
            'id': playlist_id,
            'category': category
        })

    def check_new_content(self):
        """Check all sources for new videos"""
        new_videos = []

        for source in self.sources:
            if source['type'] == 'channel':
                feed = feedparser.parse(source['rss_url'])
                for entry in feed.entries:
                    video_url = entry.link
                    if not self.db.is_processed(video_url):
                        new_videos.append({
                            'url': video_url,
                            'title': entry.title,
                            'published': entry.published,
                            'category': source.get('category'),
                            'channel_id': source['id']
                        })

        return new_videos

    def ingest_video(self, video_info):
        """Full pipeline for a single video"""
        # Step 1: Extract transcript via scriptube.me
        transcript = extract_transcript(video_info['url'])

        # Step 2: Preprocess
        cleaned = preprocess_transcript(transcript)

        # Step 3: Chunk with metadata
        chunks = smart_chunk(cleaned, {
            **video_info,
            'speaker': transcript.get('speaker', 'Unknown'),
            'duration': transcript.get('duration', 0)
        })

        # Step 4: Store raw in document DB
        self.db.store_transcript(video_info['url'], transcript)

        # Step 5: Embed and store in vector DB
        self.vectorstore.add_documents(chunks)

        print(f"✓ Ingested: {video_info['title']} ({len(chunks)} chunks)")

    def run_ingestion_cycle(self):
        """Check for and process new content"""
        new_videos = self.check_new_content()
        print(f"Found {len(new_videos)} new videos")

        for video in new_videos:
            try:
                self.ingest_video(video)
            except Exception as e:
                print(f"✗ Error processing {video['url']}: {e}")

# Set up the ingester
ingester = KnowledgeIngester(db, vectorstore)

# Subscribe to channels
ingester.add_channel("UC_CHANNEL_1", category="machine_learning")
ingester.add_channel("UC_CHANNEL_2", category="software_engineering")
ingester.add_channel("UC_CHANNEL_3", category="business")

# Run on schedule
schedule.every(6).hours.do(ingester.run_ingestion_cycle)

Part 2: Smart Chunking for Transcripts

Spoken content requires different chunking strategies than written text:

import spacy

nlp = spacy.load("en_core_web_sm")

def smart_chunk(text, metadata, max_chunk_size=1000):
    """
    Intelligent chunking that respects topic boundaries.
    Spoken content shifts topics more fluidly than written text,
    so we use sentence-level analysis.
    """
    doc = nlp(text)
    sentences = list(doc.sents)

    chunks = []
    current_chunk = []
    current_size = 0

    for sent in sentences:
        sent_text = sent.text.strip()
        sent_size = len(sent_text)

        if current_size + sent_size > max_chunk_size and current_chunk:
            chunk_text = ' '.join(current_chunk)
            chunks.append({
                'text': chunk_text,
                'metadata': {
                    **metadata,
                    'chunk_index': len(chunks),
                    'chunk_type': 'topic_segment'
                }
            })
            # Keep last sentence for overlap
            current_chunk = [current_chunk[-1], sent_text]
            current_size = len(current_chunk[0]) + sent_size
        else:
            current_chunk.append(sent_text)
            current_size += sent_size

    # Don't forget the last chunk
    if current_chunk:
        chunks.append({
            'text': ' '.join(current_chunk),
            'metadata': {
                **metadata,
                'chunk_index': len(chunks),
                'chunk_type': 'topic_segment'
            }
        })

    return chunks

Part 3: The Query Interface

from anthropic import Anthropic

class KnowledgeEngine:
    def __init__(self, vectorstore):
        self.vectorstore = vectorstore
        self.client = Anthropic()

    def query(self, question, filters=None, k=5):
        """
        Query the knowledge base with optional filters.
        Returns an AI response grounded in expert transcripts.
        """
        # Retrieve relevant chunks
        search_kwargs = {"k": k}
        if filters:
            search_kwargs["filter"] = filters

        docs = self.vectorstore.similarity_search(question, **search_kwargs)

        # Build context
        context = "\n\n---\n\n".join([
            f"Source: {doc.metadata['title']} (by {doc.metadata.get('speaker', 'Unknown')})\n"
            f"Content: {doc.page_content}"
            for doc in docs
        ])

        # Generate response
        response = self.client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=2000,
            messages=[{
                "role": "user",
                "content": f"""Based on the following expert transcript excerpts, answer the question.
                Always cite which source you're drawing from.

                SOURCES:
                {context}

                QUESTION: {question}

                Provide a thorough, well-cited answer:"""
            }]
        )

        return {
            "answer": response.content[0].text,
            "sources": [
                {"title": d.metadata["title"], "url": d.metadata.get("url", "")}
                for d in docs
            ]
        }

    def compare(self, topic, speakers=None):
        """Compare perspectives on a topic across different experts"""
        question = f"What are the different perspectives on {topic}?"
        if speakers:
            question += f" Focus on views from: {', '.join(speakers)}"
        return self.query(question, k=10)

    def summarize_recent(self, category=None, days=7):
        """Summarize recently ingested content"""
        filters = {}
        if category:
            filters["category"] = category

        return self.query(
            f"Summarize the key themes and insights from content added in the last {days} days",
            filters=filters,
            k=15
        )

# Usage
engine = KnowledgeEngine(vectorstore)

# Ask anything
result = engine.query("What are the latest approaches to prompt engineering?")
print(result["answer"])

# Compare expert views
result = engine.compare("scaling laws", speakers=["Ilya Sutskever", "Andrej Karpathy"])
print(result["answer"])

# Weekly digest
result = engine.summarize_recent(category="machine_learning", days=7)
print(result["answer"])

Part 4: Deployment

For a production-ready system, wrap it in a FastAPI service:

from fastapi import FastAPI, Query
from pydantic import BaseModel

app = FastAPI(title="Knowledge Engine API")

class QueryRequest(BaseModel):
    question: str
    category: str = None
    max_results: int = 5

class QueryResponse(BaseModel):
    answer: str
    sources: list

@app.post("/query", response_model=QueryResponse)
async def query_knowledge(request: QueryRequest):
    filters = {"category": request.category} if request.category else None
    result = engine.query(request.question, filters=filters, k=request.max_results)
    return result

@app.post("/ingest")
async def trigger_ingestion():
    ingester.run_ingestion_cycle()
    return {"status": "ingestion complete"}

@app.get("/stats")
async def get_stats():
    return {
        "total_videos": db.count_videos(),
        "total_chunks": vectorstore.count(),
        "sources": len(ingester.sources)
    }

The Big Picture

What we've built is a knowledge extraction pipeline that:

Monitors YouTube channels you care about
Extracts transcripts automatically (via scriptube.me)
Processes them into searchable chunks
Stores them in a vector database
Answers questions grounded in real expert knowledge

This is how knowledge work changes in the AI era. YouTube has unlimited expert knowledge — millions of hours of it. The bottleneck was never access. It was processing and retrieval.

With this pipeline, you've eliminated that bottleneck.

Tools used:

ScripTube — Transcript extraction
LangChain / Anthropic SDK — LLM integration
Chroma / Pinecone — Vector storage
FastAPI — API layer

The world's experts are sharing their knowledge on YouTube every day. Now you can actually USE all of it.

Start building your knowledge engine.

Building a RAG Pipeline from YouTube Transcripts

Hammer Nexon — Mon, 02 Feb 2026 21:22:54 +0000

YouTube is the world's largest repository of unstructured expert knowledge. Conference talks, technical tutorials, research paper walkthroughs, engineering deep-dives — millions of hours of valuable content locked inside video format.

In this article, I'll walk you through building a RAG (Retrieval-Augmented Generation) pipeline that transforms YouTube transcripts into a queryable, AI-powered knowledge base.

Why YouTube Transcripts?

Before we build, let's talk about why this data source is uniquely valuable:

Expert knowledge density: A 1-hour conference talk might contain insights that took years of experience to develop
Conversational format: Unlike papers, talks include practical examples, Q&A nuance, and real-world context
Volume: Major conferences alone produce hundreds of hours of technical content yearly
Freshness: YouTube content is often more current than published papers or textbooks

The problem? This knowledge is trapped in video. You can't search inside it, query it, or combine insights across talks. That's what we're fixing.

Architecture Overview

YouTube Videos
    ↓
Transcript Extraction (scriptube.me API)
    ↓
Text Preprocessing & Chunking
    ↓
Embedding Generation
    ↓
Vector Database (Pinecone/Chroma/Weaviate)
    ↓
RAG Query Pipeline
    ↓
LLM Response (with source citations)

Step 1: Transcript Extraction

The foundation of our pipeline is clean, reliable transcript extraction. I use ScripTube for this because it:

Handles various transcript sources (auto-generated, manual, multi-language)
Preserves timestamps (critical for citation)
Provides clean text output ready for processing
Works reliably at scale

import requests

def extract_transcript(video_url):
    """Extract transcript using ScripTube"""
    # Use scriptube.me to get the transcript
    # The clean output saves significant preprocessing time
    response = requests.get(f"https://scriptube.me/api/transcript?url={video_url}")
    return response.json()

# Extract from a conference talk
transcript = extract_transcript("https://youtube.com/watch?v=example")

For batch extraction (like processing all talks from a conference), scriptube.me handles multiple videos efficiently — which matters when you're building a knowledge base from hundreds of talks.

Step 2: Preprocessing & Chunking

YouTube transcripts have unique characteristics that affect chunking strategy:

from langchain.text_splitter import RecursiveCharacterTextSplitter
import re

def preprocess_transcript(raw_transcript):
    """Clean and prepare transcript for chunking"""
    # Remove filler words common in spoken content
    text = re.sub(r'\b(um|uh|like|you know)\b', '', raw_transcript['text'])

    # Normalize whitespace
    text = re.sub(r'\s+', ' ', text).strip()

    return text

def chunk_transcript(text, metadata):
    """Chunk with overlap, preserving context"""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=512,
        chunk_overlap=50,
        separators=["\n\n", "\n", ". ", " "]
    )

    chunks = splitter.create_documents(
        [text],
        metadatas=[{
            "source": metadata["video_url"],
            "title": metadata["title"],
            "speaker": metadata["speaker"],
            "channel": metadata["channel"],
            "date": metadata["upload_date"]
        }]
    )

    return chunks

Chunking Strategies That Work for Transcripts

Fixed-size with overlap: 512 tokens, 50-token overlap. Simple and effective for most use cases.
Timestamp-based: If scriptube.me provides timestamps, use natural pauses as chunk boundaries:

def chunk_by_timestamp(timestamped_transcript, max_chunk_seconds=120):
    """Group transcript segments by time windows"""
    chunks = []
    current_chunk = []
    chunk_start = 0

    for segment in timestamped_transcript:
        if segment['start'] - chunk_start > max_chunk_seconds and current_chunk:
            chunks.append({
                'text': ' '.join([s['text'] for s in current_chunk]),
                'start_time': chunk_start,
                'end_time': current_chunk[-1]['start']
            })
            current_chunk = [segment]
            chunk_start = segment['start']
        else:
            current_chunk.append(segment)

    return chunks

Topic-based: Use a lightweight classifier to detect topic shifts — most effective for long talks.

Step 3: Embedding & Storage

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma

def build_vector_store(chunks, collection_name="youtube_knowledge"):
    """Create vector store from transcript chunks"""
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    vectorstore = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        collection_name=collection_name,
        persist_directory="./chroma_db"
    )

    return vectorstore

Pro tip: Embed the metadata as part of the text for better retrieval:

def enrich_chunk_text(chunk):
    """Prepend metadata for better semantic search"""
    prefix = f"Speaker: {chunk.metadata['speaker']}. "
    prefix += f"Topic: {chunk.metadata['title']}. "
    chunk.page_content = prefix + chunk.page_content
    return chunk

Step 4: RAG Query Pipeline

from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

template = """You are a knowledgeable assistant with access to expert talks and lectures.
Use the following transcript excerpts to answer the question. 
Always cite which speaker/talk your information comes from.
If the transcripts don't contain relevant information, say so.

Context from transcripts:
{context}

Question: {question}

Answer (with citations):"""

def build_rag_chain(vectorstore):
    """Build the RAG query chain"""
    llm = ChatOpenAI(model="gpt-4", temperature=0)

    prompt = PromptTemplate(
        template=template,
        input_variables=["context", "question"]
    )

    chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=vectorstore.as_retriever(
            search_type="mmr",  # Maximum Marginal Relevance for diversity
            search_kwargs={"k": 5}
        ),
        chain_type_kwargs={"prompt": prompt},
        return_source_documents=True
    )

    return chain

# Query your knowledge base
chain = build_rag_chain(vectorstore)
result = chain({"query": "What are the best practices for fine-tuning LLMs discussed at recent conferences?"})
print(result["result"])

Step 5: Putting It All Together

Here's the complete pipeline for building a knowledge base from a YouTube playlist or channel:

class YouTubeKnowledgeBase:
    def __init__(self, collection_name="youtube_kb"):
        self.collection_name = collection_name
        self.vectorstore = None
        self.chain = None

    def ingest_videos(self, video_urls):
        """Ingest a list of YouTube videos"""
        all_chunks = []

        for url in video_urls:
            # Extract transcript via scriptube.me
            transcript = extract_transcript(url)

            # Preprocess
            clean_text = preprocess_transcript(transcript)

            # Chunk
            chunks = chunk_transcript(clean_text, transcript['metadata'])

            # Enrich
            chunks = [enrich_chunk_text(c) for c in chunks]
            all_chunks.extend(chunks)

        # Build vector store
        self.vectorstore = build_vector_store(all_chunks, self.collection_name)
        self.chain = build_rag_chain(self.vectorstore)

        print(f"Ingested {len(video_urls)} videos, {len(all_chunks)} chunks")

    def query(self, question):
        """Query the knowledge base"""
        result = self.chain({"query": question})
        return {
            "answer": result["result"],
            "sources": [
                {
                    "title": doc.metadata["title"],
                    "speaker": doc.metadata["speaker"],
                    "excerpt": doc.page_content[:200]
                }
                for doc in result["source_documents"]
            ]
        }

# Usage
kb = YouTubeKnowledgeBase("ml_conferences_2024")

# Ingest all talks from a conference
conference_urls = [
    "https://youtube.com/watch?v=talk1",
    "https://youtube.com/watch?v=talk2",
    # ... hundreds of talks
]

kb.ingest_videos(conference_urls)

# Query with natural language
result = kb.query("What novel approaches to context windows were discussed?")
print(result["answer"])

Optimization Tips

Re-rank before LLM: Use a cross-encoder to re-rank retrieved chunks before feeding to the LLM. This dramatically improves answer quality.
Hybrid search: Combine dense (embedding) and sparse (BM25) retrieval for better recall.
Transcript quality matters: Garbage in, garbage out. This is why I use scriptube.me — clean transcripts mean clean embeddings mean better retrieval.
Metadata filtering: Store rich metadata so you can filter by speaker, date, conference, or topic before semantic search.
Incremental updates: Design your pipeline to add new transcripts without rebuilding the entire index.

What You Can Build With This

Personal research assistant grounded in expert knowledge (not hallucinations)
Conference digest tool — "summarize everything about [topic] from [conference]"
Technical learning system — query across hundreds of tutorials
Competitive intelligence — analyze industry talks systematically

The knowledge is on YouTube. Transcripts (via scriptube.me) make it extractable. RAG makes it queryable.

Start building. The world's experts are waiting to be your knowledge base.

How I Handle Edge Cases in a YouTube Transcript Tool

Hammer Nexon — Mon, 02 Feb 2026 21:19:20 +0000

Building ScripTube (scriptube.me) taught me that the "happy path" is about 20% of the engineering work. The other 80% is edge cases.

Here's a tour of the edge cases I've encountered and how I handle them.

Edge Case 1: URL Variations

Users paste URLs in every imaginable format:

https://www.youtube.com/watch?v=dQw4w9WgXcQ
https://youtu.be/dQw4w9WgXcQ
https://youtube.com/watch?v=dQw4w9WgXcQ
http://www.youtube.com/watch?v=dQw4w9WgXcQ
https://www.youtube.com/watch?v=dQw4w9WgXcQ&t=120
https://www.youtube.com/embed/dQw4w9WgXcQ
https://m.youtube.com/watch?v=dQw4w9WgXcQ
youtube.com/watch?v=dQw4w9WgXcQ
dQw4w9WgXcQ
www.youtube.com/watch?v=dQw4w9WgXcQ&list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf

Some users paste the video ID directly. Some paste URLs with tracking parameters. Some paste playlist URLs where the video ID is buried in query params.

Solution: A robust URL parser that handles all of these, plus validation that the extracted ID is exactly 11 characters matching [a-zA-Z0-9_-].

Edge Case 2: No Transcript Available

Not all videos have transcripts. Reasons:

Creator disabled captions
Video is too new (auto-generation hasn't run yet)
Language not supported by auto-captioning
Music/instrumental content with no speech
Live streams (sometimes)

Solution: Clear, specific error messages. Not just "Error" — tell the user WHY and WHAT to do:

"This video doesn't have captions available. 
The creator may have disabled them, or auto-generation 
isn't available for this video's language."

Edge Case 3: Age-Restricted Videos

YouTube requires sign-in for age-restricted content. The transcript API call fails because it can't authenticate.

Solution: Detect this specific error and tell the user clearly. I considered implementing cookie-based auth to bypass this, but decided against it for privacy reasons.

Edge Case 4: Auto-Generated vs. Manual Captions

Some videos have both. The quality difference is significant — manual captions are accurate, auto-generated ones have errors.

Solution: Prefer manual captions when available. Show the user which type they're getting:

const transcriptList = await listTranscripts(videoId);

// Prefer manual over auto-generated
const manual = transcriptList.find(t => !t.isGenerated);
const auto = transcriptList.find(t => t.isGenerated);

const selected = manual || auto;
const source = manual ? 'Manual captions' : 'Auto-generated';

Edge Case 5: Multiple Languages

Videos can have transcripts in multiple languages. Users expect to get the language they want.

Solution: Default to the video's primary language, but let users select alternatives. For the API, accept a lang parameter with fallback chain:

const languagePriority = [
  requestedLang,     // What the user asked for
  'en',              // English fallback
  primaryLang,       // Video's primary language
];

Edge Case 6: Extremely Long Videos

A 4-hour podcast transcript is 40,000+ words. Serving this as a single API response causes:

Slow response times
High memory usage
Frontend rendering issues

Solution: Stream the response for long transcripts, and offer a download option instead of displaying everything in the browser.

Edge Case 7: Special Characters and Encoding

Auto-generated captions include HTML entities, Unicode characters, and sometimes garbled text:

you&amp;#39;re going to want to
the caf&#233; is really nice
[Music]
[Applause]

Solution: Decode HTML entities, normalize Unicode, and optionally strip non-speech markers like [Music]:

function cleanTranscriptText(text: string): string {
  return text
    .replace(/&amp;/g, '&')
    .replace(/&lt;/g, '<')
    .replace(/&gt;/g, '>')
    .replace(/&#(\d+);/g, (_, code) => 
      String.fromCharCode(parseInt(code)))
    .replace(/&\w+;/g, ' ')
    .replace(/\[Music\]/gi, '')
    .replace(/\[Applause\]/gi, '')
    .replace(/\s+/g, ' ')
    .trim();
}

Edge Case 8: Rate Limiting Abuse

Within the first week, someone tried to hit the API thousands of times per minute — probably building their own tool on top of mine.

Solution: Multi-layer rate limiting:

Per-IP: 10 requests per minute
Global: circuit breaker if YouTube starts returning errors
CAPTCHA for suspicious patterns

The Meta-Lesson

Every one of these edge cases represents a real user who tried to use the tool and either got a confusing error or broken output. Each fix made the tool more reliable for everyone.

The difference between a side project and a product is edge case handling. The happy path is table stakes. Handling the unhappy paths is what makes people trust your tool enough to rely on it.

If you're building something, I'd encourage you to start logging every error and unusual input you receive. Each one is a signpost pointing toward a better product.

Try ScripTube → scriptube.me

Total: 8 Dev.to articles, each 800-1500 words with code snippets and technical depth.

Building a Chrome Extension for YouTube Transcript Extraction

Hammer Nexon — Mon, 02 Feb 2026 21:18:43 +0000

After building ScripTube (scriptube.me) as a web app, the most requested feature was a browser extension. Users wanted to extract transcripts without leaving the YouTube page they were already on.

Here's how I built a Chrome extension for YouTube transcript extraction, and what I learned along the way.

The Architecture

A Chrome extension has three main components:

Manifest (manifest.json) — declares permissions and entry points
Content Script — runs on YouTube pages, interacts with the DOM
Popup — the UI that appears when you click the extension icon

Step 1: The Manifest

{
  "manifest_version": 3,
  "name": "YouTube Transcript Extractor",
  "version": "1.0.0",
  "description": "Extract transcripts from YouTube videos in one click",
  "permissions": ["activeTab"],
  "action": {
    "default_popup": "popup.html",
    "default_icon": "icon.png"
  },
  "content_scripts": [{
    "matches": ["*://www.youtube.com/watch*"],
    "js": ["content.js"]
  }]
}

Note: we only need activeTab permission. No broad host permissions needed, which keeps the permission prompt minimal during installation.

Step 2: The Content Script

The content script runs on YouTube watch pages and exposes a method to extract the transcript:

// content.js

async function extractTranscript() {
  // Click the "..." menu button
  const menuButton = document.querySelector(
    'button.yt-spec-button-shape-next--icon-button'
  );

  // Instead of DOM manipulation, use YouTube's internal API
  // The transcript data is available in the page's ytInitialData
  const scripts = document.querySelectorAll('script');

  for (const script of scripts) {
    if (script.textContent.includes('captionTracks')) {
      const match = script.textContent.match(
        /"captionTracks":(\[.*?\])/
      );
      if (match) {
        const tracks = JSON.parse(match[1]);
        const englishTrack = tracks.find(
          t => t.languageCode === 'en'
        ) || tracks[0];

        if (englishTrack) {
          const response = await fetch(englishTrack.baseUrl);
          const xml = await response.text();
          return parseTranscriptXML(xml);
        }
      }
    }
  }

  return null;
}

function parseTranscriptXML(xml) {
  const parser = new DOMParser();
  const doc = parser.parseFromString(xml, 'text/xml');
  const texts = doc.querySelectorAll('text');

  return Array.from(texts).map(node => ({
    text: node.textContent
      .replace(/&amp;/g, '&')
      .replace(/&lt;/g, '<')
      .replace(/&gt;/g, '>')
      .replace(/&#39;/g, "'")
      .replace(/&quot;/g, '"'),
    start: parseFloat(node.getAttribute('start')),
    duration: parseFloat(node.getAttribute('dur')),
  }));
}

// Listen for messages from the popup
chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
  if (request.action === 'getTranscript') {
    extractTranscript().then(transcript => {
      sendResponse({ transcript });
    });
    return true; // Keep the message channel open for async response
  }
});

Step 3: The Popup UI

<!-- popup.html -->
<!DOCTYPE html>
<html>
<head>
  <style>
    body {
      width: 400px;
      padding: 16px;
      font-family: -apple-system, sans-serif;
    }
    #transcript {
      max-height: 300px;
      overflow-y: auto;
      white-space: pre-wrap;
      font-size: 13px;
      line-height: 1.5;
      padding: 12px;
      background: #f5f5f5;
      border-radius: 8px;
      margin-top: 12px;
    }
    button {
      background: #ff0000;
      color: white;
      border: none;
      padding: 10px 20px;
      border-radius: 6px;
      cursor: pointer;
      font-size: 14px;
      width: 100%;
    }
    button:hover { background: #cc0000; }
    .copy-btn { background: #333; margin-top: 8px; }
    .copy-btn:hover { background: #555; }
  </style>
</head>
<body>
  <h3>📝 Transcript Extractor</h3>
  <button id="extract">Extract Transcript</button>
  <button id="copy" class="copy-btn" style="display:none">
    Copy to Clipboard
  </button>
  <div id="transcript"></div>
  <script src="popup.js"></script>
</body>
</html>

// popup.js

document.getElementById('extract').addEventListener('click', async () => {
  const [tab] = await chrome.tabs.query({
    active: true,
    currentWindow: true,
  });

  if (!tab.url.includes('youtube.com/watch')) {
    document.getElementById('transcript').textContent =
      'Please navigate to a YouTube video first.';
    return;
  }

  document.getElementById('extract').textContent = 'Extracting...';

  chrome.tabs.sendMessage(
    tab.id,
    { action: 'getTranscript' },
    (response) => {
      const el = document.getElementById('transcript');

      if (response?.transcript) {
        const text = response.transcript
          .map((s) => s.text)
          .join(' ');
        el.textContent = text;
        document.getElementById('copy').style.display = 'block';
      } else {
        el.textContent = 'No transcript available for this video.';
      }

      document.getElementById('extract').textContent =
        'Extract Transcript';
    }
  );
});

document.getElementById('copy').addEventListener('click', () => {
  const text = document.getElementById('transcript').textContent;
  navigator.clipboard.writeText(text).then(() => {
    document.getElementById('copy').textContent = 'Copied!';
    setTimeout(() => {
      document.getElementById('copy').textContent = 'Copy to Clipboard';
    }, 2000);
  });
});

Gotchas and Lessons

3. Manifest V3 is async. If you're used to Manifest V2, the service worker model takes adjustment. The popup's lifecycle is short — it dies when closed. Don't store state in the popup.

4. Keep permissions minimal. Users are wary of extension permissions. activeTab is all you need for this use case. No need for broad host access.

Publish to Chrome Web Store

Zip the extension folder
Pay the one-time $5 developer fee
Submit through the Chrome Developer Dashboard
Wait for review (usually 1-3 business days)

The Web App Alternative

But for power users who extract transcripts frequently, the extension removes one step (copying the URL) and that convenience matters.