DEV Community: Jonathan Geiger

SocialKit Make.com Integration: Automate YouTube APIs with Make

Jonathan Geiger — Fri, 01 Aug 2025 14:26:19 +0000

We're excited to announce that SocialKit now integrates with Make.com (formerly Integromat), bringing powerful visual automation to our YouTube APIs. This integration allows you to connect SocialKit's YouTube Video Summarizer, Transcript Extractor, and Video Analytics APIs with over 1,000 other apps using Make's intuitive drag-and-drop interface.

Quick Start

Get SocialKit YouTube API access on Make.com:

SocialKit on Make.com
Alternative Integration Link - Direct access link

🚀 What's New: SocialKit on Make.com

Our Make.com integration enables you to automate YouTube content workflows with powerful visual scenarios. Whether you're a content creator, marketer, researcher, or business owner, you can now build sophisticated automation workflows using Make's advanced features like data transformation, conditional logic, and error handling.

Available SocialKit Actions in Make.com

YouTube Video Summarizer: Generate AI-powered summaries from YouTube videos
YouTube Transcript Extractor: Extract timestamped transcripts from any YouTube video
YouTube Video Analytics: Retrieve comprehensive video statistics and metadata

Example Automation Scenarios

Content Creator Scenario:

New video added to YouTube playlist (YouTube trigger)
Extract transcript using SocialKit (SocialKit action)
Generate summary using SocialKit (SocialKit action)
Save to Google Sheets for analysis (Google Sheets action)

Marketing Research Scenario:

New competitor video found (RSS/Webhook trigger)
Get video analytics using SocialKit (SocialKit action)
Summarize video content using SocialKit (SocialKit action)
Send insights to Slack channel (Slack action)

Educational Content Scenario:

Student submits YouTube video link (Google Forms trigger)
Extract transcript using SocialKit (SocialKit action)
Create study notes in Notion (Notion action)
Email summary to student (Email action)

⚡ Getting Started with SocialKit on Make.com

Step 1: Connect Your SocialKit Account

Visit our Make.com integration page and connect your SocialKit account using your API key.

Step 2: Create Your First Scenario

Choose from our pre-built templates or create custom workflows combining SocialKit with your favorite apps using Make's visual editor.

Step 3: Automate YouTube Content Processing

Set up triggers and actions to automatically process YouTube videos whenever specific events occur in your scenario.

Make.com Integration Links:

SocialKit on Make.com - Main integration page
Alternative Integration Link - Direct access link

🎯 Popular Use Cases

For Content Creators

Competitor Analysis: Automatically summarize competitor videos and track trends
Content Repurposing: Extract transcripts to create blog posts, social media content, and newsletters
Research Automation: Gather insights from educational videos for content planning

For Marketers

Campaign Research: Analyze video marketing campaigns across industries
Trend Monitoring: Track viral video content and extract key themes
Performance Tracking: Monitor video analytics and compile reports automatically

For Educators & Researchers

Lecture Processing: Extract and organize educational content from YouTube
Research Data Collection: Gather video transcripts for academic analysis
Student Resources: Create automated study materials from video content

For Businesses

Training Content: Process training videos and create documentation
Market Intelligence: Monitor industry videos and extract insights
Customer Research: Analyze customer testimonial videos and feedback

🔧 Technical Benefits

Visual Workflow Builder: Design complex automation scenarios with an intuitive drag-and-drop interface
Advanced Data Processing: Transform and manipulate data between steps with built-in functions
Conditional Logic: Create smart workflows that adapt based on data conditions
Error Handling: Robust error handling with custom retry logic and fallback actions
Real-time Execution: Instant processing when triggers fire
Multi-path Scenarios: Create complex branching workflows for sophisticated automation

💡 Integration Examples

Google Sheets + SocialKit

Automatically populate spreadsheets with YouTube video data, summaries, and transcripts. Use Make's data transformation features to format and organize the data perfectly.

Slack + SocialKit

Send video summaries and insights directly to team channels with rich formatting and custom notifications based on video content or analytics.

Notion + SocialKit

Create comprehensive content databases with automated video analysis. Use Make's advanced features to structure and categorize content automatically.

Airtable + SocialKit

Build robust content management systems with automated video processing, tagging, and categorization using Make's powerful data manipulation capabilities.

Discord + SocialKit

Share video insights and summaries in Discord servers, perfect for community-driven content curation and research sharing.

🚀 Ready to Automate Your YouTube Workflows?

The SocialKit Make.com integration brings enterprise-grade automation capabilities to YouTube content processing. With Make's advanced visual workflow builder and powerful data processing features, you can create sophisticated automation scenarios that go beyond simple trigger-action pairs.

Get Started Today:

Transform your YouTube workflow with SocialKit and Make.com's powerful visual automation. Build sophisticated scenarios that process, analyze, and act on YouTube content automatically.

Best YouTube Transcript APIs in 2025

Jonathan Geiger — Wed, 30 Jul 2025 18:35:19 +0000

If you're building applications that need to extract transcripts from YouTube videos, you've probably realized that doing this reliably at scale is harder than it looks. YouTube's interface changes frequently, and handling different video formats (especially YouTube Shorts) adds complexity.

That's where YouTube transcript APIs come in. Instead of managing headless browsers and building your own scraper, you can get reliable, structured transcript data through simple API request.

Let's dive into the top YouTube transcript APIs in 2025 (woohoo)

1. SocialKit YouTube Transcript API

Okay, I'll be honest here, I'm starting with our own API, and yes, I'm biased. But I built SocialKit's YouTube Transcript API because I can bring my scraping expertise to the table.

Key Features:

YouTube & YouTube Shorts support: Handles both regular videos and Shorts seamlessly
Timestamped segments: Get precise timing for each transcript segment
High reliability: Built-in retry logic and error handling
Fast response times: Optimized infrastructure for quick processing
Clean JSON responses: Well-structured data that's easy to work with
Comprehensive ecosystem: Works alongside our YouTube Summarizer API and YouTube Stats API
Future-ready: TikTok and Instagram APIs coming soon

Pricing:

Free tier: 20 requests per month
Starter: $13/month for 2,000 requests ($6.5 per 1,000 requests)
Scale: Higher volume plans available

Best for: Developers building applications that need reliable transcript data, content creators analyzing videos, and businesses doing content research.

2. Supadata.ai

Supadata.ai is primarily a web scraping service, but they've expanded into YouTube transcript extraction. While not exclusively focused on YouTube, they offer a solid API for transcript extraction.

Key Features:

Web scraping focus: Part of a broader web scraping platform
Multiple data sources: YouTube is one of many supported platforms
Decent reliability: Generally stable, though not YouTube-specialized
Standard JSON responses: Clean data format

Pricing:

Free tier: 100 credits per month
Paid plans: Starting at $9 per 1,000 credits

Pros:

Good for projects that need both YouTube transcripts and general web scraping
Established web scraping infrastructure
Reasonable pricing for mixed-use cases

Best for: Projects that need YouTube transcripts as part of broader web scraping needs.

3. ScrapCreators.com

ScrapCreators offers various social media APIs with a focus on stats, details, and metrics. Their YouTube transcript API is part of a larger suite of social media data extraction tools.

Key Features:

Multi-platform approach: YouTube transcripts alongside other social media APIs
Stats and metrics focus: Strong emphasis on data analytics
Various social media platforms: Broader social media data extraction
Pay-per-use model: Only pay for what you use

Pricing:

Free trial: 100 credits (one-time, not monthly)
Pay-per-use: $10 for 5,000 credits (no monthly subscription)

Pros:

No monthly commitments with pay-per-use model
Good for projects needing multiple social media APIs
Focus on comprehensive social media data

Best for: Businesses needing social media analytics with occasional transcript extraction needs.

4. YouTube-transcript.io

YouTube-transcript.io is a dedicated YouTube transcript service with a straightforward approach to transcript extraction.

Key Features:

YouTube-focused: Dedicated specifically to YouTube transcript extraction
Simple API: Straightforward implementation
Monthly subscription model: Predictable pricing

Pricing:

Free tier: 25 credits per month
Subscription: $9.99/month for 1,000 transcripts

Pros:

Focused specifically on YouTube transcripts
Simple, predictable pricing
Dedicated to one use case

Best for: Small to medium projects with consistent, predictable transcript needs.

Pricing Comparison

Service	Free Tier	Cost per 1,000 Requests	Model
SocialKit	20/month	$6.5	Subscription
Supadata.ai	100/month	$9	Subscription
ScrapCreators	100 total	$2*	Pay-per-use
YouTube-transcript.io	25/month	$9.99	Subscription

*Calculated based on $10 for 5,000 credits

Use Case Recommendations

For Developers and No-Code Users

Go with SocialKit if you need:

Reliable, high-volume transcript extraction
YouTube Shorts support
Additional YouTube APIs (Summary, Stats)
No-Code platforms Like: Make.com, Zapier, and N8N coming soon

For Mixed Web Scraping Projects

Consider Supadata.ai if you need:

YouTube transcripts plus general web scraping
Established scraping infrastructure
Multiple data sources in one platform

For Social Media Analytics

ScrapCreators might work if you need:

Multiple social media APIs
Pay-per-use flexibility
Focus on metrics and analytics

For Simple, Predictable Needs

YouTube-transcript.io could work if you have:

Consistent, low-volume needs
Simple integration requirements
Preference for subscription models

Ready to Get Started?

The best way to choose is to test. Most of these APIs offer free tiers, so you can try them with your actual data and see which works best for your project.

If you want to try SocialKit's YouTube Transcript API, you can get started with 20 free requests per month. We also offer a free YouTube transcript extractor tool if you want to test the quality before integrating the API.

If you're interested in more YouTube & YouTube Shorts APIs, these pages might be for you:

YouTube Transcript API: Extract timestamped transcripts
YouTube Summarizer API: Get AI-powered summaries
YouTube Stats API: Access video metrics and analytics

Happy coding, and may your transcript extraction be forever reliable! 🚀

How to Extract YouTube Video Details Using Puppeteer

Jonathan Geiger — Wed, 30 Jul 2025 18:33:06 +0000

YouTube video details contain valuable metadata that developers, content creators, and researchers often need to extract programmatically. While YouTube provides an API, it requires authentication and has rate limits. For simpler use cases or when you need complete control over the extraction process, web scraping with Puppeteer offers a powerful alternative.

In this comprehensive guide, we'll explore how to extract YouTube video details using Puppeteer, including titles, view counts, likes, comments, channel information, and more. We'll cover everything from basic setup to advanced error handling techniques.

Prerequisites

Before diving into the implementation, ensure you have:

Node.js installed (version 14 or higher)
Basic knowledge of JavaScript and async/await
Understanding of DOM manipulation and CSS selectors
Familiarity with browser automation concepts
Experience with handling dynamic content loading

Setting Up the Project

Create a new project and install dependencies:

mkdir youtube-details-scraper
cd youtube-details-scraper
npm init -y
npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

The stealth plugin helps avoid detection by making our automated browser behavior appear more natural.

Basic Implementation

Let's start with a basic implementation that extracts core video details:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

// Use stealth plugin to avoid detection
puppeteer.use(StealthPlugin());

const extractYouTubeDetails = async (url) => {
  const browser = await puppeteer.launch({
    headless: "new",
    ignoreDefaultArgs: ["--enable-automation"]
  });

  try {
    const page = await browser.newPage();
    await page.setViewport({ width: 1280, height: 800 });

    // Navigate to YouTube video
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
    await page.waitForTimeout(1000);

    // Handle cookie banner (EU compliance)
    try {
      await page.evaluate(() => {
        const cookieButton = document.querySelector('button[aria-label*="cookies"]');
        if (cookieButton) {
          cookieButton.click();
          console.log('Closed cookie banner');
        }
      });
      await page.waitForTimeout(1000);
    } catch (e) {
      console.log('No cookie banner found');
    }

    // Scroll down to load below-the-fold content
    await page.evaluate(() => window.scrollBy(0, 300));
    await page.waitForTimeout(600);

    // Extract video details
    const videoDetails = await page.evaluate(() => {
      const extractNumber = (text) => {
        if (!text) return 0;

        const cleanText = text.replace(/[^\d.,KMB\s]/g, '').trim();
        const match = cleanText.match(/(\d{1,3}(?:,\d{3})*(?:\.\d+)?[KMB]?|\d+(?:\.\d+)?[KMB]?)/);
        if (!match) return 0;

        const numStr = match[0];
        const suffix = numStr.slice(-1);

        if (['K', 'M', 'B'].includes(suffix)) {
          const num = parseFloat(numStr.slice(0, -1).replace(/,/g, ''));
          if (isNaN(num)) return 0;

          switch(suffix) {
            case 'K': return Math.floor(num * 1000);
            case 'M': return Math.floor(num * 1000000);
            case 'B': return Math.floor(num * 1000000000);
          }
        } else {
          const num = parseFloat(numStr.replace(/,/g, ''));
          return isNaN(num) ? 0 : Math.floor(num);
        }

        return 0;
      };

      const data = {};

      // Extract title
      const titleElement = document.querySelector('h1.ytd-watch-metadata yt-formatted-string');
      data.title = titleElement ? titleElement.textContent.trim() : '';

      // Extract channel information
      const channelElement = document.querySelector('ytd-channel-name a');
      if (channelElement) {
        data.channelName = channelElement.textContent.trim();
        data.channelLink = channelElement.href || '';
      }

      // Extract views
      const viewsElement = document.querySelector('#info span[class*="view"]');
      data.views = viewsElement ? extractNumber(viewsElement.textContent) : 0;

      // Extract likes
      const likesElement = document.querySelector('.ytd-watch-metadata .yt-spec-button-view-model .yt-spec-button-shape-next__button-text-content');
      data.likes = likesElement ? extractNumber(likesElement.textContent) : 0;

      return data;
    });

    return videoDetails;

  } catch (error) {
    console.error('Error extracting video details:', error);
    throw error;
  } finally {
    await browser.close();
  }
};

// Usage
const videoUrl = 'https://www.youtube.com/watch?v=dQw4w9WgXcQ';
extractYouTubeDetails(videoUrl)
  .then(details => console.log('Video Details:', details))
  .catch(error => console.error('Failed to extract details:', error));

Advanced Implementation with Comprehensive Data Extraction

Here's a more robust version that extracts all available video metadata:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

puppeteer.use(StealthPlugin());

const scrapeYouTubeDetails = async (url) => {
  const browser = await puppeteer.launch({
    headless: "new",
    ignoreDefaultArgs: ["--enable-automation"]
  });

  try {
    const page = await browser.newPage();
    await page.setViewport({
      width: 1280,
      height: 1024,
      deviceScaleFactor: 1,
    });

    // Navigate to YouTube video
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
    await page.waitForTimeout(1000);

    // Handle cookie banner
    try {
      await page.evaluate(() => {
        const cookieButton = document.querySelector('button[aria-label*="cookies"]');
        if (cookieButton) {
          cookieButton.click();
        }
      });
      await page.waitForTimeout(1000);
    } catch (e) {
      console.log('No cookie banner found');
    }

    // Scroll down to load content below the fold
    try {
      await page.evaluate(() => window.scrollBy(0, 300));
      await page.waitForTimeout(600);
    } catch (e) {
      console.log('Could not scroll page');
    }

    // Extract comprehensive metadata
    const metadata = await page.evaluate(() => {
      const extractNumber = (text) => {
        if (!text) return 0;

        const cleanText = text.replace(/[^\d.,KMB\s]/g, '').trim();
        const match = cleanText.match(/(\d{1,3}(?:,\d{3})*(?:\.\d+)?[KMB]?|\d+(?:\.\d+)?[KMB]?)/);
        if (!match) return 0;

        const numStr = match[0];
        const suffix = numStr.slice(-1);

        if (['K', 'M', 'B'].includes(suffix)) {
          const num = parseFloat(numStr.slice(0, -1).replace(/,/g, ''));
          if (isNaN(num) || num < 0 || num > 999999) return 0;

          switch(suffix) {
            case 'K': return Math.floor(num * 1000);
            case 'M': return Math.floor(num * 1000000);
            case 'B': return Math.floor(num * 1000000000);
          }
        } else {
          const num = parseFloat(numStr.replace(/,/g, ''));
          if (isNaN(num) || num < 0 || num > 999999999999) return 0;
          return Math.floor(num);
        }

        return 0;
      };

      const data = {
        title: '',
        channelName: '',
        channelLink: '',
        views: 0,
        likes: 0,
        comments: 0,
        publishDate: '',
        description: '',
        thumbnailUrl: ''
      };

      // Title
      const titleElement = document.querySelector('h1.ytd-watch-metadata yt-formatted-string');
      if (titleElement) {
        data.title = titleElement.textContent.trim();
      }

      // Channel Name and Link
      const channelElement = document.querySelector('ytd-channel-name a');
      if (channelElement) {
        data.channelName = channelElement.textContent.trim();
        data.channelLink = channelElement.href || '';
      }

      // Views - try multiple selectors
      const viewsSelectors = [
        '#info span[class*="view"]',
        '#info .style-scope.yt-formatted-string',
        '#info .view-count',
      ];

      for (const selector of viewsSelectors) {
        const viewsElement = document.querySelector(selector);
        if (viewsElement && viewsElement.textContent.trim()) {
          const text = viewsElement.textContent.trim();
          if (text.includes('views') || text.includes('view') || /[\d,]+[KMB]?\s*(views?|watching)/i.test(text)) {
            data.views = extractNumber(text);
            break;
          }
        }
      }

      // Likes
      const likesElement = document.querySelector(
        '.ytd-watch-metadata .yt-spec-button-view-model .yt-spec-button-shape-next__button-text-content'
      );
      if (likesElement) {
        data.likes = extractNumber(likesElement.textContent);
      }

      // Comments
      const commentsElement = document.querySelector('#title #count span');
      if (commentsElement) {
        data.comments = extractNumber(commentsElement.textContent);
      }

      // Publish date
      const publishElement = document.querySelector('ytd-watch-metadata #info-strings yt-formatted-string:nth-child(2)');
      if (publishElement) {
        data.publishDate = publishElement.textContent.trim();
      }

      // Description
      const descriptionElement = document.querySelector('ytd-watch-metadata #description-text');
      if (descriptionElement) {
        data.description = descriptionElement.textContent.trim().substring(0, 500) + '...';
      }

      // Thumbnail
      const thumbnailElement = document.querySelector('video');
      if (thumbnailElement) {
        data.thumbnailUrl = thumbnailElement.poster || '';
      }

      return data;
    });

    return {
      url,
      extractedAt: new Date().toISOString(),
      ...metadata
    };

  } catch (error) {
    console.error('Error scraping video details:', error);
    throw error;
  } finally {
    await browser.close();
  }
};

// Usage
const videoUrl = 'https://www.youtube.com/watch?v=dQw4w9WgXcQ';
scrapeYouTubeDetails(videoUrl)
  .then(details => {
    console.log('Video Title:', details.title);
    console.log('Channel:', details.channelName);
    console.log('Views:', details.views.toLocaleString());
    console.log('Likes:', details.likes.toLocaleString());
    console.log('Published:', details.publishDate);
  })
  .catch(error => console.error('Failed to scrape details:', error));

Handling Dynamic Content and Common Issues

1. Waiting for Elements to Load

YouTube loads content dynamically, so proper timing is crucial:

// Wait for specific elements before extraction
await page.waitForSelector('h1.ytd-watch-metadata', { timeout: 10000 });
await page.waitForSelector('#info', { timeout: 5000 });

2. Scrolling to Load Below-the-Fold Content

Some elements only load when they come into view:

// Scroll to load engagement metrics
await page.evaluate(() => {
  window.scrollBy(0, 300);
});
await page.waitForTimeout(600);

3. Number Extraction and Formatting

YouTube uses abbreviated numbers (1.2M, 45K). Our extractNumber function handles:

Comma-separated numbers (1,234,567)
Abbreviated suffixes (K, M, B)
Decimal points (1.2M)
Validation and bounds checking

4. Multiple Selector Fallbacks

YouTube's interface changes frequently. Use multiple selectors:

const viewsSelectors = [
  '#info span[class*="view"]',
  '#info .style-scope.yt-formatted-string',
  '#info .view-count',
];

for (const selector of viewsSelectors) {
  const element = document.querySelector(selector);
  if (element && element.textContent.includes('views')) {
    // Extract views
    break;
  }
}

Best Practices

Respect Rate Limits: Add delays between requests to avoid being blocked
Use Stealth Mode: Puppeteer-extra-plugin-stealth makes detection harder
Handle Errors Gracefully: Always wrap extraction code in try-catch blocks
Validate Data: Check extracted numbers for reasonable bounds
Cache Results: Store extracted data to avoid repeated requests
Monitor Selector Changes: YouTube updates its interface regularly

Alternative: Using SocialKit YouTube API

For production applications requiring reliable video details extraction, consider using SocialKit's YouTube Stats API:

curl "https://api.socialkit.dev/youtube/stats?access_key=<your-access-key>&url=https://youtube.com/watch?v=dQw4w9WgXcQ"

Example Response

{
    "success": true,
    "data": {
        "url": "https://youtube.com/watch?v=dQw4w9WgXcQ",
        "title": "Rick Astley - Never Gonna Give You Up (Official Video)",
        "channelName": "Rick Astley",
        "channelLink": "https://youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
        "views": 1428567890,
        "likes": 16234567,
        "comments": 4567890
    }
}

Benefits of using SocialKit:

Reliable extraction: Handles YouTube's interface changes automatically
No browser overhead: Faster and more resource-efficient
Consistent formatting: Standardized data structure across all videos
Built-in retry logic: Automatic error handling and retries
Scale-ready: Handle thousands of videos without infrastructure concerns
Always up-to-date: Adapts to YouTube changes without code updates

Free YouTube Tools

Need quick video insights without coding? Try our free tools:

YouTube Video Summarizer Tool

Get AI-powered summaries and insights with our free YouTube Video Summarizer tool:

Generate detailed summaries of any YouTube video or YouTube Shorts
Extract key insights including main topics, quotes, and takeaways
Analyze video content for tone, target audience, and themes
Get instant results without any setup or registration

Try the Free YouTube Video Summarizer

YouTube Transcript Extractor Tool

Extract accurate transcripts with our free YouTube Transcript Extractor tool:

Extract timestamped transcripts from any YouTube video
Copy segments individually or get the complete transcript
Perfect for content analysis and accessibility purposes
100% free with no API keys required

Try the Free YouTube Transcript Extractor

Both tools complement video details extraction by providing deeper content insights for researchers, content creators, and developers.

Advanced Use Cases

Batch Processing Multiple Videos

const processVideoList = async (urls) => {
  const results = [];

  for (const url of urls) {
    try {
      const details = await scrapeYouTubeDetails(url);
      results.push(details);

      // Add delay between requests
      await new Promise(resolve => setTimeout(resolve, 2000));
    } catch (error) {
      console.error(`Failed to process ${url}:`, error.message);
      results.push({ url, error: error.message });
    }
  }

  return results;
};

Channel Analysis

const analyzeChannel = async (channelUrl) => {
  // Navigate to channel and extract video URLs
  // Then process each video for detailed analytics
  const videoDetails = await processVideoList(videoUrls);

  return {
    channelName: channelDetails.name,
    totalVideos: videoDetails.length,
    totalViews: videoDetails.reduce((sum, video) => sum + video.views, 0),
    averageViews: videoDetails.reduce((sum, video) => sum + video.views, 0) / videoDetails.length,
    videos: videoDetails
  };
};

Conclusion

Extracting YouTube video details with Puppeteer provides complete control over the data extraction process. While the implementation requires careful handling of dynamic content loading, element timing, and YouTube's evolving interface, the results enable powerful video analytics and content research capabilities.

For production applications or when processing large volumes of videos, consider using SocialKit's YouTube API for reliability and scale. For quick insights and analysis, our free YouTube tools provide immediate value without any setup.

Remember to respect YouTube's terms of service, implement appropriate rate limiting, and handle errors gracefully to build robust video data extraction systems.

Whether you're building content analytics dashboards, research tools, or social media management platforms, automated YouTube video details extraction opens up endless possibilities for data-driven insights.

How to Extract YouTube Shorts Video Details Using Puppeteer

Jonathan Geiger — Wed, 30 Jul 2025 18:31:40 +0000

YouTube Shorts have transformed the social media landscape, generating billions of views with their bite-sized vertical video format. However, extracting metadata and analytics data from these short-form videos presents unique technical challenges that differ from regular YouTube videos.

The YouTube Shorts URL Challenge

The fundamental difference between YouTube Shorts and regular videos lies in their URL structure:

YouTube Shorts URL Pattern:

https://www.youtube.com/shorts/VIDEO_ID

Regular YouTube URL Pattern:

https://www.youtube.com/watch?v=VIDEO_ID

To extract video details from YouTube Shorts, we must convert the Shorts URL to the regular format, as the detailed metadata interface is only accessible through the standard video player.

URL Conversion Implementation

const convertShortsToRegularUrl = (url) => {
    // Define regex patterns for both URL types
    const shortsPattern = /youtube\.com\/shorts\/([^&\n?#]+)/;
    const regularPattern = /youtube\.com\/watch\?v=([^&\n?#]+)/;

    if (shortsPattern.test(url)) {
        // Extract video ID from Shorts URL
        const videoId = url.match(shortsPattern)[1];
        return `https://www.youtube.com/watch?v=${videoId}`;
    } else if (regularPattern.test(url)) {
        // Already a regular YouTube URL
        return url;
    } else {
        throw new Error('Invalid YouTube URL format');
    }
};

// Example usage
const shortsUrl = 'https://www.youtube.com/shorts/ABC123xyz';
const regularUrl = convertShortsToRegularUrl(shortsUrl);
// Result: 'https://www.youtube.com/watch?v=ABC123xyz'

Project Setup

Initialize your YouTube Shorts scraper project:

mkdir youtube-shorts-details-scraper
cd youtube-shorts-details-scraper
npm init -y
npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

The stealth plugin helps bypass detection mechanisms that might block automated browser sessions.

Basic YouTube Shorts Details Extractor

Here's a fundamental implementation that converts Shorts URLs and extracts core video metadata:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

// Enable stealth mode to avoid detection
puppeteer.use(StealthPlugin());

const convertShortsToRegularUrl = (url) => {
    const shortsPattern = /youtube\.com\/shorts\/([^&\n?#]+)/;
    const regularPattern = /youtube\.com\/watch\?v=([^&\n?#]+)/;

    if (shortsPattern.test(url)) {
        const videoId = url.match(shortsPattern)[1];
        return `https://www.youtube.com/watch?v=${videoId}`;
    } else if (regularPattern.test(url)) {
        return url;
    } else {
        throw new Error('Invalid YouTube URL format');
    }
};

const extractYouTubeShortsDetails = async (shortsUrl) => {
    // Convert Shorts URL to regular YouTube format
    const url = convertShortsToRegularUrl(shortsUrl);
    console.log(`Converting: ${shortsUrl} -> ${url}`);

    const browser = await puppeteer.launch({
        headless: 'new',
        ignoreDefaultArgs: ['--enable-automation'],
    });

    try {
        const page = await browser.newPage();
        await page.setViewport({ width: 1280, height: 800 });

        // Navigate to converted YouTube URL
        await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
        await page.waitForTimeout(1500);

        // Handle cookie consent banner
        try {
            await page.evaluate(() => {
                const cookieButton = document.querySelector(
                    'button[aria-label*="cookies"]'
                );
                if (cookieButton) {
                    cookieButton.click();
                    console.log('Cookie banner closed');
                }
            });
            await page.waitForTimeout(1000);
        } catch (e) {
            console.log('No cookie banner detected');
        }

        // Scroll to trigger content loading
        await page.evaluate(() => window.scrollBy(0, 300));
        await page.waitForTimeout(800);

        // Extract video details
        const videoDetails = await page.evaluate(() => {
            const extractNumber = (text) => {
                if (!text) return 0;

                const cleanText = text.replace(/[^\d.,KMB\s]/g, '').trim();
                const match = cleanText.match(
                    /(\d{1,3}(?:,\d{3})*(?:\.\d+)?[KMB]?|\d+(?:\.\d+)?[KMB]?)/
                );
                if (!match) return 0;

                const numStr = match[0];
                const suffix = numStr.slice(-1);

                if (['K', 'M', 'B'].includes(suffix)) {
                    const num = parseFloat(numStr.slice(0, -1).replace(/,/g, ''));
                    if (isNaN(num)) return 0;

                    switch (suffix) {
                        case 'K':
                            return Math.floor(num * 1000);
                        case 'M':
                            return Math.floor(num * 1000000);
                        case 'B':
                            return Math.floor(num * 1000000000);
                    }
                } else {
                    const num = parseFloat(numStr.replace(/,/g, ''));
                    return isNaN(num) ? 0 : Math.floor(num);
                }

                return 0;
            };

            const data = {};

            // Extract video title
            const titleElement = document.querySelector(
                'h1.ytd-watch-metadata yt-formatted-string'
            );
            data.title = titleElement ? titleElement.textContent.trim() : '';

            // Extract channel information
            const channelElement = document.querySelector('ytd-channel-name a');
            if (channelElement) {
                data.channelName = channelElement.textContent.trim();
                data.channelLink = channelElement.href || '';
            }

            // Extract view count
            const viewsElement = document.querySelector('#info span[class*="view"]');
            data.views = viewsElement ? extractNumber(viewsElement.textContent) : 0;

            // Extract likes
            const likesElement = document.querySelector(
                '.ytd-watch-metadata .yt-spec-button-view-model .yt-spec-button-shape-next__button-text-content'
            );
            data.likes = likesElement ? extractNumber(likesElement.textContent) : 0;

            return data;
        });

        return {
            originalUrl: shortsUrl,
            convertedUrl: url,
            ...videoDetails,
        };
    } catch (error) {
        console.error('Error extracting Shorts details:', error);
        throw error;
    } finally {
        await browser.close();
    }
};

// Usage example
const shortsUrl = 'https://www.youtube.com/shorts/ABC123xyz';
extractYouTubeShortsDetails(shortsUrl)
    .then((details) => console.log('Shorts Details:', details))
    .catch((error) => console.error('Extraction failed:', error));

Comprehensive YouTube Shorts Metadata Extraction

For production use, here's an advanced implementation with robust error handling and comprehensive data extraction:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

puppeteer.use(StealthPlugin());

const convertShortsToRegularUrl = (url) => {
    const shortsPattern = /youtube\.com\/shorts\/([^&\n?#]+)/;
    const regularPattern = /youtube\.com\/watch\?v=([^&\n?#]+)/;

    if (shortsPattern.test(url)) {
        const videoId = url.match(shortsPattern)[1];
        return `https://www.youtube.com/watch?v=${videoId}`;
    } else if (regularPattern.test(url)) {
        return url;
    } else {
        throw new Error('Invalid YouTube URL format');
    }
};

const scrapeYouTubeShortsDetails = async (shortsUrl) => {
    const url = convertShortsToRegularUrl(shortsUrl);
    console.log(`Processing: ${shortsUrl} -> ${url}`);

    const browser = await puppeteer.launch({
        headless: 'new',
        ignoreDefaultArgs: ['--enable-automation'],
    });

    try {
        const page = await browser.newPage();
        await page.setViewport({
            width: 1280,
            height: 1024,
            deviceScaleFactor: 1,
        });

        // Navigate to YouTube video
        await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
        await page.waitForTimeout(1500);

        // Handle cookie banner
        try {
            await page.evaluate(() => {
                const cookieButton = document.querySelector(
                    'button[aria-label*="cookies"]'
                );
                if (cookieButton) {
                    cookieButton.click();
                }
            });
            await page.waitForTimeout(1000);
        } catch (e) {
            console.log('No cookie banner found');
        }

        // Scroll to load below-the-fold content
        try {
            await page.evaluate(() => window.scrollBy(0, 300));
            await page.waitForTimeout(800);
        } catch (e) {
            console.log('Could not scroll page');
        }

        // Extract comprehensive metadata
        const metadata = await page.evaluate(() => {
            const extractNumber = (text) => {
                if (!text) return 0;

                const cleanText = text.replace(/[^\d.,KMB\s]/g, '').trim();
                const match = cleanText.match(
                    /(\d{1,3}(?:,\d{3})*(?:\.\d+)?[KMB]?|\d+(?:\.\d+)?[KMB]?)/
                );
                if (!match) return 0;

                const numStr = match[0];
                const suffix = numStr.slice(-1);

                if (['K', 'M', 'B'].includes(suffix)) {
                    const num = parseFloat(numStr.slice(0, -1).replace(/,/g, ''));
                    if (isNaN(num) || num < 0 || num > 999999) return 0;

                    switch (suffix) {
                        case 'K':
                            return Math.floor(num * 1000);
                        case 'M':
                            return Math.floor(num * 1000000);
                        case 'B':
                            return Math.floor(num * 1000000000);
                    }
                } else {
                    const num = parseFloat(numStr.replace(/,/g, ''));
                    if (isNaN(num) || num < 0 || num > 999999999999) return 0;
                    return Math.floor(num);
                }

                return 0;
            };

            const data = {
                title: '',
                channelName: '',
                channelLink: '',
                views: 0,
                likes: 0,
                comments: 0,
                publishDate: '',
                description: '',
                thumbnailUrl: '',
            };

            // Extract title
            const titleElement = document.querySelector(
                'h1.ytd-watch-metadata yt-formatted-string'
            );
            if (titleElement) {
                data.title = titleElement.textContent.trim();
            }

            // Extract channel information
            const channelElement = document.querySelector('ytd-channel-name a');
            if (channelElement) {
                data.channelName = channelElement.textContent.trim();
                data.channelLink = channelElement.href || '';
            }

            // Extract views with multiple fallback selectors
            const viewsSelectors = [
                '#info span[class*="view"]',
                '#info .style-scope.yt-formatted-string',
                '#info .view-count',
            ];

            for (const selector of viewsSelectors) {
                const viewsElement = document.querySelector(selector);
                if (viewsElement && viewsElement.textContent.trim()) {
                    const text = viewsElement.textContent.trim();
                    if (
                        text.includes('views') ||
                        text.includes('view') ||
                        /[\d,]+[KMB]?\s*(views?|watching)/i.test(text)
                    ) {
                        data.views = extractNumber(text);
                        break;
                    }
                }
            }

            // Extract likes
            const likesElement = document.querySelector(
                '.ytd-watch-metadata .yt-spec-button-view-model .yt-spec-button-shape-next__button-text-content'
            );
            if (likesElement) {
                data.likes = extractNumber(likesElement.textContent);
            }

            // Extract comments count
            const commentsElement = document.querySelector('#title #count span');
            if (commentsElement) {
                data.comments = extractNumber(commentsElement.textContent);
            }

            // Extract publish date
            const publishElement = document.querySelector(
                'ytd-watch-metadata #info-strings yt-formatted-string:nth-child(2)'
            );
            if (publishElement) {
                data.publishDate = publishElement.textContent.trim();
            }

            // Extract description
            const descriptionElement = document.querySelector(
                'ytd-watch-metadata #description-text'
            );
            if (descriptionElement) {
                data.description =
                    descriptionElement.textContent.trim().substring(0, 300) + '...';
            }

            // Extract thumbnail
            const thumbnailElement = document.querySelector('video');
            if (thumbnailElement) {
                data.thumbnailUrl = thumbnailElement.poster || '';
            }

            return data;
        });

        return {
            originalShortsUrl: shortsUrl,
            convertedUrl: url,
            extractedAt: new Date().toISOString(),
            ...metadata,
        };
    } catch (error) {
        console.error('Error scraping Shorts details:', error);
        throw error;
    } finally {
        await browser.close();
    }
};

// Usage example
const shortsUrl = 'https://www.youtube.com/shorts/ABC123xyz';
scrapeYouTubeShortsDetails(shortsUrl)
    .then((details) => {
        console.log('Shorts Title:', details.title);
        console.log('Channel:', details.channelName);
        console.log('Views:', details.views.toLocaleString());
        console.log('Likes:', details.likes.toLocaleString());
    })
    .catch((error) => console.error('Failed to scrape Shorts details:', error));

Technical Challenges and Solutions

1. URL Structure Conversion

The most critical aspect of YouTube Shorts scraping is proper URL conversion:

// URL validation and conversion
const validateAndConvertUrl = (url) => {
    try {
        const urlObj = new URL(url);
        if (
            urlObj.hostname !== 'www.youtube.com' &&
            urlObj.hostname !== 'youtube.com'
        ) {
            throw new Error('Not a YouTube URL');
        }
        return convertShortsToRegularUrl(url);
    } catch (error) {
        throw new Error(`Invalid URL: ${error.message}`);
    }
};

2. Dynamic Content Loading

YouTube Shorts interface elements load asynchronously:

// Wait for critical elements before extraction
await page.waitForSelector('h1.ytd-watch-metadata', { timeout: 10000 });
await page.waitForSelector('#info', { timeout: 5000 });

// Additional wait for engagement metrics
await page.evaluate(() => window.scrollBy(0, 300));
await page.waitForTimeout(800);

Alternative: SocialKit YouTube Stats API

For production applications requiring reliable YouTube Shorts analytics, consider SocialKit's YouTube Stats API:

curl "https://api.socialkit.dev/youtube/stats?access_key=<your-access-key>&url=https://youtube.com/watch?v=dQw4w9WgXcQ"

Example Response

{
    "success": true,
    "data": {
        "url": "https://youtube.com/watch?v=dQw4w9WgXcQ",
        "title": "Rick Astley - Never Gonna Give You Up (Official Video)",
        "channelName": "Rick Astley",
        "channelLink": "https://youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
        "views": 1428567890,
        "likes": 16234567,
        "comments": 4567890
    }
}

API Benefits:

Automatic URL handling: Processes both Shorts and regular YouTube URLs
No conversion needed: Handles URL transformation internally
Consistent data structure: Standardized response format across all video types
Real-time accuracy: Always up-to-date with current video statistics
Scale-ready: Handle thousands of Shorts without rate limits
Global availability: Works worldwide without geo-restrictions

Free YouTube Tools

Need instant access to YouTube Shorts data? Try our free tools:

YouTube Video Summarizer Tool

Get AI-powered insights with our free YouTube Video Summarizer tool:

Analyze YouTube Shorts content with AI-powered summaries
Extract key themes and trending topics from short-form videos
Identify viral content patterns for your own content strategy
Get instant insights without any setup or registration required

YouTube Transcript Extractor Tool

Extract content from Shorts with our free YouTube Transcript Extractor tool:

Extract transcripts from YouTube Shorts automatically
Get timestamped segments for precise content analysis
Perfect for accessibility and content repurposing
100% free with support for both Shorts and regular videos

Both tools automatically handle YouTube Shorts URLs and provide immediate value for content creators, social media managers, and digital marketers.

Conclusion

Extracting YouTube Shorts video details with Puppeteer requires mastering the critical URL conversion technique that transforms Shorts URLs into standard YouTube video URLs. This conversion unlocks access to YouTube's comprehensive metadata interface, enabling extraction of views, likes, comments, and other valuable analytics data.

Best YouTube Summarizers in 2025

Jonathan Geiger — Sat, 12 Jul 2025 13:43:26 +0000

Whether you're analyzing competitor content, conducting market research, or building applications that process video data at scale, having the right YouTube summarizer can save you countless hours and unlock valuable insights from video content.

In this comprehensive comparison, we'll analyze the top YouTube summarizer tools and APIs available in 2025, helping you choose the perfect solution for your specific needs.

Our Methodology

We evaluated each YouTube summarizer based on these key criteria:

Accuracy: How well does the tool capture the main points and key insights?
Features: What additional capabilities does it offer beyond basic summarization?
Pricing: What are the costs for different usage levels?
API Availability: Does it offer programmatic access for developers?
User Experience: How intuitive and easy to use is the interface?
Content Types: Does it handle both regular videos and YouTube Shorts?

The Best YouTube Summarizers in 2025

1. SocialKit YouTube Summarizer API - Best Overall

Best for: No-code users, developers, marketers, businesses, and anyone needing both free tools and API access

SocialKit's YouTube Video Summarizer stands out as the most comprehensive YouTube summarizer solution in 2025, offering both free tools for individual use and powerful APIs for bulk processing and integration.

Key Features:

Free YouTube Video Summarizer Tool: No registration required
YouTube Summarizer API: Programmatic access for developers
YouTube Transcript API: Extract timestamped transcripts
YouTube Stats API: Get video metrics and analytics
Supports YouTube Shorts: Handles both regular videos and Shorts
AI-powered summaries: Extracts key topics, main points, and important quotes
Multiple output formats: JSON API responses and human-readable summaries
Bulk processing: Handle thousands of videos via API
Upcoming social media APIs: TikTok, Instagram, and more coming soon

Pricing:

Free Tool: 100% free with unlimited use and no registration
API: 20 requests/month free, then $14/month for 2,000 requests, $27/month for 10,000 requests, $79/month for 50,000 requests
Enterprise: Custom pricing for high-volume usage

Pros:

Only solution offering both free tools and APIs
Comprehensive suite of YouTube-related APIs
Excellent documentation and developer support
Fast processing and high accuracy
Handles both regular videos and YouTube Shorts

Cons:

API requires technical integration (though this is also a strength)

Use Cases:

Content creators: Analyzing competitor content and trends
Developers: Building applications that process YouTube data
Marketers: Bulk analysis of video content for campaigns
Researchers: Academic and market research at scale
No-code users: Zapier/Make.com integrations for automation

2. YouTube Summarized - Best Chrome Extension

Best for: Casual users who want quick summaries while browsing YouTube

YouTube Summarized is a popular Chrome extension that adds a summary button directly to YouTube videos, making it convenient for users who want instant summaries without leaving the platform.

Key Features:

Chrome extension integration
One-click summarization
Supports multiple languages
Simple, clean interface
Chapter-based summaries

Pricing:

Free version with limited summaries
Monthly Plan: €7.99/month
Annual Plan: €3.99/month (billed annually)

Pros:

Seamless YouTube integration
Easy to use interface
Good accuracy for most content
Multiple language support

Cons:

No API access for developers
Limited to Chrome browser
No bulk processing capabilities
Requires browser extension installation

3. Eightify - Best for Mobile Users

Best for: Mobile users and those who want AI-powered key insights

Eightify offers both Chrome extension and mobile app, providing AI-powered summaries with a focus on extracting key insights and actionable takeaways from videos.

Key Features:

Chrome extension and mobile app
AI-powered key insights
Timestamped summaries
Share summaries easily
Works with various video lengths

Pricing:

7-day free trial
Monthly Plan: $9.99/month
Annual Plan: $4.99/month (billed annually)

Pros:

Available on multiple platforms
Good mobile experience
Focuses on actionable insights
Attractive user interface

Cons:

No API for developers
Limited free tier
No bulk processing
Premium pricing for regular use

4. NoteGPT - Best Free Alternative

Best for: Students and researchers who need free access to video summaries

NoteGPT offers a free YouTube video summarizer tool that provides comprehensive summaries without requiring registration or payment.

Key Features:

Completely free to use
No registration required
Supports long-form videos
Note-taking integration
Multiple export formats

Pricing:

Free for basic summarization
Pro: $9.99-$29/month or $6.92-$19.92/month (billed annually)

Pros:

Completely free basic version
No registration barriers
Good for educational use
Integrates with note-taking workflow

Cons:

No API access
Limited advanced features in free version
No bulk processing
Interface could be more modern

Feature Comparison

Feature	SocialKit	YouTube Summarized	Eightify	NoteGPT
Free Tool	✅	✅ (Limited)	✅ (Limited)	✅
API Access	✅	❌	❌	❌
Bulk Processing	✅	❌	❌	❌
YouTube Shorts	✅	✅	✅	✅
Timestamped Segments	✅	✅	✅	❌
Chrome Extension	❌	✅	✅	❌
Mobile App	❌	❌	✅	❌
Multiple Languages	✅	✅	✅	✅
Export Options	✅ (JSON/Text)	✅	✅	✅
Developer Documentation	✅	❌	❌	❌

Pricing Comparison

Tool	Free Tier	Paid Plans	Best Value
SocialKit	100% free tool + 20 API requests/month	$14/month (2K requests), $27/month (10K requests), $79/month (50K requests)	Best for API users
YouTube Summarized	Limited summaries	€7.99/month or €3.99/month annually	Good for extension users
Eightify	7-day free trial	$9.99/month or $4.99/month annually	Good for mobile users
NoteGPT	Basic free access	$9.99-$29/month or $6.92-$19.92/month annually	Mixed value

Use Cases and Recommendations

For Individual Users

Casual browsing: YouTube Summarized or Eightify chrome extensions
Students/researchers: NoteGPT for free unlimited access
Mobile users: Eightify mobile app
Quick summaries: SocialKit free tool

For Businesses and Developers

Content analysis at scale: SocialKit API for bulk processing
Application development: SocialKit comprehensive API suite
Marketing campaigns: SocialKit for analyzing competitor content
Research projects: SocialKit for academic and market research

For Automation and Integration

Zapier/Make.com workflows: SocialKit API integration
Custom applications: SocialKit developer-friendly APIs
Bulk content processing: SocialKit enterprise solutions
Multi-platform analysis: SocialKit (with upcoming TikTok/Instagram APIs)

Why SocialKit Stands Out

While all the tools we've reviewed offer good summarization capabilities, SocialKit is the only solution that provides both free tools for individual use AND powerful APIs for developers and businesses. This dual approach makes it suitable for everyone from casual users to enterprise developers.

Key differentiators:

API-first approach: Only YouTube summarizer offering comprehensive APIs
Complete ecosystem: Summary, Transcript, and Stats APIs in one platform
Bulk processing: Handle thousands of videos programmatically
Developer-friendly: Excellent documentation and support
Future-ready: Upcoming social media APIs for TikTok, Instagram, and more
Transparent pricing: Clear, scalable pricing tiers for different usage levels

Getting Started

Free Tools

YouTube Video Summarizer: Try our free tool instantly
YouTube Transcript Extractor: Get timestamped transcripts

For Developers

API Documentation: Complete integration guides
Pricing: Transparent pay-per-use pricing
Sign Up: Start building with our APIs

Conclusion

The YouTube summarizer landscape in 2025 offers several good options, but the choice depends on your specific needs:

For casual users: YouTube Summarized or Eightify chrome extensions provide convenient browsing integration
For students: NoteGPT offers the best free access without limitations
For developers and businesses: SocialKit is the clear winner with its comprehensive API suite and bulk processing capabilities

SocialKit's unique combination of free tools and powerful APIs makes it the most versatile solution, suitable for everyone from individual users to enterprise developers. With upcoming social media APIs and continuous feature updates, it's positioned to remain the leading choice for YouTube content analysis in 2025 and beyond.

Whether you need to analyze a single video or process thousands of videos programmatically, there's a YouTube summarizer solution that fits your needs. For most users, we recommend starting with the free tools to test accuracy and features, then upgrading to paid plans or APIs based on your usage requirements.

Ready to get started? Try our free YouTube Video Summarizer or explore our comprehensive APIs for your next project.

How to Scrape YouTube Shorts Video Transcripts With Puppeteer

Jonathan Geiger — Fri, 11 Jul 2025 08:57:10 +0000

YouTube Shorts have revolutionized short-form video content, but extracting transcripts from these vertical videos presents unique challenges. Unlike regular YouTube videos, Shorts use a different URL structure that requires special handling when automating transcript extraction.

In this comprehensive guide, we'll explore how to scrape YouTube Shorts transcripts using Puppeteer, with a focus on the critical URL conversion technique that makes this process possible. Whether you're analyzing trending short-form content or building accessibility features, this tutorial provides everything you need to extract transcripts from YouTube Shorts automatically.

Why YouTube Shorts Transcripts Matter

YouTube Shorts generate billions of views daily, making them a goldmine for:

Content trend analysis - Understanding viral short-form content patterns
Accessibility compliance - Providing text alternatives for hearing-impaired users
Market research - Analyzing competitor short-form content strategies
SEO optimization - Extracting keywords from popular short videos
Content repurposing - Converting video content to written formats

Prerequisites

Before we dive into scraping YouTube Shorts transcripts, ensure you have:

Node.js installed (version 14 or higher)
Basic understanding of JavaScript and async/await
Familiarity with DOM manipulation and CSS selectors
Knowledge of regular expressions for URL parsing
Understanding of browser automation concepts

Understanding YouTube Shorts URL Structure

The key difference between regular YouTube videos and Shorts lies in their URL structure:

Regular YouTube video: https://www.youtube.com/watch?v=DS4OsxHR9EQ
YouTube Shorts: https://www.youtube.com/shorts/DS4OsxHR9EQ

To extract transcripts from YouTube Shorts, we need to convert the Shorts URL to the regular YouTube format, as the transcript functionality is only available in the standard video player interface.

URL Conversion Pattern

// Regex pattern to extract video ID from Shorts URL
const shortsPattern = /youtube\.com\/shorts\/([^&\n?#]+)/;

// Example conversion
const shortsUrl = 'https://www.youtube.com/shorts/DS4OsxHR9EQ';
const videoId = shortsUrl.match(shortsPattern)[1]; // 'DS4OsxHR9EQ'
const regularUrl = `https://www.youtube.com/watch?v=${videoId}`;

Setting Up the Project

Create a new project directory and install dependencies:

mkdir youtube-shorts-scraper
cd youtube-shorts-scraper
npm init -y

Install the required packages:

npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

Basic YouTube Shorts Transcript Scraper

Here's a complete implementation that handles YouTube Shorts URL conversion and transcript extraction using the same methods as regular YouTube videos:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

// Use stealth plugin to avoid detection
puppeteer.use(StealthPlugin());

const convertShortsToRegularUrl = (url) => {
  // Handle both regular YouTube URLs and Shorts URLs
  const shortsPattern = /youtube\.com\/shorts\/([^&\n?#]+)/;
  const regularPattern = /youtube\.com\/watch\?v=([^&\n?#]+)/;

  if (shortsPattern.test(url)) {
    // Extract video ID from Shorts URL
    const videoId = url.match(shortsPattern)[1];
    return `https://www.youtube.com/watch?v=${videoId}`;
  } else if (regularPattern.test(url)) {
    // Already a regular YouTube URL
    return url;
  } else {
    throw new Error('Invalid YouTube URL format');
  }
};

const scrapeYouTubeShortsTranscript = async (shortsUrl) => {
  // Convert Shorts URL to regular YouTube URL
  const url = convertShortsToRegularUrl(shortsUrl);
  console.log(`Converting: ${shortsUrl} -> ${url}`);

  const browser = await puppeteer.launch({
    headless: "new",
    ignoreDefaultArgs: ["--enable-automation"]
  });

  try {
    const page = await browser.newPage();
    await page.setViewport({ width: 1280, height: 800 });

    // Navigate to YouTube video
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
    await page.waitForTimeout(2000);

    // Handle cookie banner (EU compliance)
    try {
      await page.evaluate(() => {
        const cookieButton = document.querySelector('button[aria-label*="cookies"]');
        if (cookieButton) {
          cookieButton.click();
          console.log('Closed cookie banner');
        }
      });
    } catch (e) {
      console.log('No cookie banner found');
    }

    // Click transcript button
    await page.waitForSelector('ytd-video-description-transcript-section-renderer button', { timeout: 10000 });
    await page.evaluate(() => {
      const transcriptButton = document.querySelector('ytd-video-description-transcript-section-renderer button');
      if (transcriptButton) {
        transcriptButton.click();
        console.log('Clicked transcript button');
      }
    });

    await page.waitForTimeout(2000);

    // Extract transcript text
    const transcriptText = await page.evaluate(() => {
      const segments = Array.from(document.querySelectorAll('#segments-container yt-formatted-string'));
      return segments.map(element => element.textContent?.trim()).filter(text => text && text.length > 0);
    });

    return transcriptText.join(' ');

  } catch (error) {
    console.error('Error scraping transcript:', error);
    throw error;
  } finally {
    await browser.close();
  }
};

// Usage
const shortsUrl = 'https://www.youtube.com/shorts/DS4OsxHR9EQ';
scrapeYouTubeShortsTranscript(shortsUrl)
  .then(transcript => console.log('Transcript:', transcript))
  .catch(error => console.error('Failed to scrape transcript:', error));

Advanced YouTube Shorts Scraper with Error Handling

The basic implementation works for most videos, but YouTube's interface can vary. Here's a more robust version with comprehensive error handling that uses the same methods as regular YouTube transcript scraping:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

puppeteer.use(StealthPlugin());

const convertShortsToRegularUrl = (url) => {
  const shortsPattern = /youtube\.com\/shorts\/([^&\n?#]+)/;
  const regularPattern = /youtube\.com\/watch\?v=([^&\n?#]+)/;

  if (shortsPattern.test(url)) {
    const videoId = url.match(shortsPattern)[1];
    return `https://www.youtube.com/watch?v=${videoId}`;
  } else if (regularPattern.test(url)) {
    return url;
  } else {
    throw new Error('Invalid YouTube URL format');
  }
};

const formatTimestamp = (seconds) => {
  const mins = Math.floor(seconds / 60);
  const secs = seconds % 60;
  return `${mins.toString().padStart(2, '0')}:${secs.toString().padStart(2, '0')}`;
};

const scrapeYouTubeShortsTranscriptAdvanced = async (shortsUrl) => {
  // Convert Shorts URL to regular YouTube URL
  const url = convertShortsToRegularUrl(shortsUrl);
  console.log(`Converting: ${shortsUrl} -> ${url}`);

  const browser = await puppeteer.launch({
    headless: "new",
    ignoreDefaultArgs: ["--enable-automation"]
  });

  try {
    const page = await browser.newPage();
    await page.setViewport({
      width: 1280,
      height: 1024,
      deviceScaleFactor: 1,
    });

    // Navigate to YouTube video
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
    await page.waitForTimeout(2000);

    // Try to close cookie banner if present
    try {
      await page.evaluate(() => {
        const cookieButton = document.querySelector('button[aria-label*="cookies"]');
        if (cookieButton) {
          cookieButton.click();
          console.log('Closed cookie banner');
        }
      });
    } catch (e) {
      console.log('No cookie banner found');
    }

    // Click transcript button with multiple fallback selectors
    try {
      await page.waitForSelector('ytd-video-description-transcript-section-renderer button', { timeout: 10000 });
      await page.evaluate(() => {
        const transcriptButton = document.querySelector('ytd-video-description-transcript-section-renderer button');
        if (transcriptButton) {
          transcriptButton.click();
          console.log('Clicked transcript button');
        }
      });
      await page.waitForTimeout(2000);
    } catch (e) {
      console.log('Transcript button not found, trying alternative selectors');

      // Try alternative selectors
      try {
        await page.evaluate(() => {
          const selectors = [
            'button[aria-label*="transcript"]',
            'button[aria-label*="Transcript"]',
            '[data-target-id="engagement-panel-transcript"] button',
            '#transcript-button',
            'button[aria-label*="Show transcript"]'
          ];

          for (const selector of selectors) {
            const button = document.querySelector(selector);
            if (button) {
              button.click();
              console.log('Clicked transcript button with selector:', selector);
              return;
            }
          }
        });
        await page.waitForTimeout(2000);
      } catch (e2) {
        console.log('Alternative transcript button selectors failed');
      }
    }

    // Extract transcript text with multiple fallback selectors
    let transcriptText = await page.evaluate(() => {
      const segments = Array.from(document.querySelectorAll('#segments-container yt-formatted-string'));
      return segments.map(element => element.textContent?.trim()).filter(text => text && text.length > 0);
    });

    if (transcriptText.length === 0) {
      // Try alternative transcript selectors
      const alternativeTranscript = await page.evaluate(() => {
        const selectors = [
          '#segments-container span',
          '#segments-container div',
          '[data-target-id="engagement-panel-transcript"] span',
          '[data-target-id="engagement-panel-transcript"] div',
          '.ytd-transcript-segment-renderer span',
          '.ytd-transcript-segment-renderer div'
        ];

        for (const selector of selectors) {
          const elements = document.querySelectorAll(selector);
          if (elements.length > 0) {
            const texts = Array.from(elements).map(el => el.textContent?.trim()).filter(text => text && text.length > 0);
            if (texts.length > 0) {
              return texts;
            }
          }
        }
        return [];
      });

      if (alternativeTranscript.length === 0) {
        throw new Error('No transcript available for this video.');
      }

      transcriptText = alternativeTranscript;
    }

    // Format transcript with timestamps
    const transcript = transcriptText.map((text, index) => ({
      text,
      start: index * 5,
      duration: 5,
      timestamp: formatTimestamp(index * 5),
    }));

    const fullText = transcript.map(entry => entry.text).join(' ');

    return {
      url,
      transcript,
      fullText,
      wordCount: fullText.split(' ').length,
      segments: transcript.length,
    };

  } catch (error) {
    console.error('Error scraping transcript:', error);
    throw error;
  } finally {
    await browser.close();
  }
};

// Usage
const shortsUrl = 'https://www.youtube.com/shorts/DS4OsxHR9EQ';
scrapeYouTubeShortsTranscriptAdvanced(shortsUrl)
  .then(result => {
    console.log('Full transcript:', result.fullText);
    console.log('Word count:', result.wordCount);
    console.log('Segments:', result.segments);
    console.log('First few segments:', result.transcript.slice(0, 5));
  })
  .catch(error => console.error('Failed to scrape transcript:', error));

Handling Common Issues

Since YouTube Shorts use the same transcript interface as regular videos after URL conversion, they face the same challenges:

1. Cookie Banners

YouTube shows cookie consent banners in EU regions. Our code handles this by looking for buttons with "cookies" in their aria-label and clicking them automatically.

2. Transcript Button Variations

YouTube's interface changes frequently. We use multiple selectors to find the transcript button, ensuring compatibility across different layouts.

3. Dynamic Content Loading

Transcripts are loaded dynamically. We use waitForTimeout() to ensure content is fully loaded before attempting to extract it.

4. Rate Limiting

To avoid being blocked, consider:

Adding random delays between requests
Using residential proxies
Implementing retry logic with exponential backoff

Best Practices

Respect robots.txt: Always check YouTube's robots.txt file
Implement caching: Store transcripts locally to avoid repeated requests
Error handling: Always wrap your scraping code in try-catch blocks
User agent rotation: Use different user agents to appear more natural
Respect rate limits: Don't overwhelm YouTube's servers
URL validation: Always validate and convert Shorts URLs before processing

Alternative: Using SocialKit YouTube Transcript API

While web scraping works, managing browser instances and handling YouTube's changing interface can be complex. For production applications, consider using SocialKit's YouTube Transcript API, which handles both regular videos and Shorts automatically:

curl "https://api.socialkit.dev/youtube/transcript?access_key=YOUR_ACCESS_KEY&url=https://youtube.com/shorts/DS4OsxHR9EQ"

Example Response for YouTube Shorts

{
  "success": true,
  "data": {
    "url": "https://youtube.com/shorts/DS4OsxHR9EQ",
    "transcript": "Hey everyone! Today I'm showing you this amazing quick tip that will change everything. Watch this transformation happen in just 30 seconds. Isn't that incredible? Make sure to follow for more!",
    "transcriptSegments": [
      {
        "text": "Hey everyone! Today I'm showing you this amazing quick tip",
        "start": 0,
        "duration": 4,
        "timestamp": "00:00"
      },
      {
        "text": "that will change everything. Watch this transformation",
        "start": 4,
        "duration": 3,
        "timestamp": "00:04"
      },
      {
        "text": "happen in just 30 seconds. Isn't that incredible?",
        "start": 7,
        "duration": 4,
        "timestamp": "00:07"
      },
      {
        "text": "Make sure to follow for more!",
        "start": 11,
        "duration": 2,
        "timestamp": "00:11"
      }
    ],
    "wordCount": 32,
    "segments": 4
  }
}

Benefits of Using SocialKit API:

Automatic URL handling: Works with both Shorts and regular YouTube URLs
No browser management: Eliminates Puppeteer complexity and resource usage
Consistent reliability: Built-in handling of YouTube's interface changes
Scale-ready infrastructure: Process thousands of videos without rate limits
Structured data: Returns properly formatted timestamps and segments
Global availability: Works worldwide without geo-restrictions

Free YouTube Tools

Need quick access to YouTube content without building your own scraper? Try our free tools:

YouTube Video Summarizer Tool

Get AI-powered summaries with our free YouTube Video Summarizer tool:

Generate AI-powered summaries of any YouTube video or YouTube Shorts
Extract key insights including main topics, key points, and important quotes
Analyze video tone and identify target audience
Get instant results without any setup or API keys required

Try the Free YouTube Video Summarizer

YouTube Transcript Extractor Tool

Extract accurate transcripts with our free YouTube Transcript Extractor tool:

Extract accurate transcripts from any YouTube video or YouTube Shorts
Get timestamped segments for easy navigation and reference
Copy individual segments or the complete transcript
Perfect for accessibility and content analysis
100% free with no registration required

Try the Free YouTube Transcript Extractor

Both tools automatically handle YouTube Shorts URLs and are perfect for content creators, students, researchers, and anyone who wants to quickly extract valuable information from YouTube content.

Conclusion

Scraping YouTube Shorts transcripts with Puppeteer requires understanding the unique URL structure and conversion process that transforms Shorts URLs into regular YouTube video URLs. While the technical implementation involves careful handling of dynamic content and various edge cases, the ability to extract transcripts from short-form video content opens up powerful possibilities for content analysis, accessibility, and automation.

The key to success lies in the URL conversion technique - transforming youtube.com/shorts/VIDEO_ID to youtube.com/watch?v=VIDEO_ID - which allows access to YouTube's standard transcript interface. Combined with robust error handling and rate limiting, this approach enables reliable extraction from one of the web's fastest-growing content formats.

For production applications or when dealing with large volumes of videos, consider using a dedicated API service like SocialKit's YouTube Transcript API to ensure reliability and save development time. The API automatically handles both Shorts and regular YouTube videos, eliminating the need for manual URL conversion.

If you're looking for quick ways to extract information from video content without coding, check out our free tools: YouTube Video Summarizer for instant AI-powered summaries and YouTube Transcript Extractor for accurate timestamped transcripts.

Remember to always respect YouTube's terms of service, implement appropriate rate limiting, and consider the ethical implications of automated content extraction. Happy scraping!

How to Scrape YouTube Transcripts With Puppeteer

Jonathan Geiger — Thu, 03 Jul 2025 20:17:38 +0000

YouTube transcripts are invaluable for content creators, researchers, and developers who need to extract spoken content from videos. While YouTube provides transcripts through its interface, automating this process can save significant time when dealing with multiple videos or building applications that require transcript data.

In this comprehensive guide, we'll explore how to scrape YouTube transcripts using Puppeteer, a powerful Node.js library that controls headless Chrome browsers. We'll cover everything from basic setup to advanced error handling and provide a complete working solution.

Prerequisites

Before diving into the code, ensure you have the following:

Node.js installed on your machine (version 14 or higher)
Basic knowledge of JavaScript and async/await
Understanding of DOM manipulation and CSS selectors
Familiarity with browser automation concepts

Understanding YouTube's Transcript Structure

YouTube's transcript feature isn't always visible by default. The transcript panel appears after clicking the "Show transcript" button, usually located in the video description area. The transcript content is dynamically loaded and structured as follows:

Transcript Button: Located in ytd-video-description-transcript-section-renderer button
Transcript Container: Found in #segments-container
Individual Segments: Each transcript line is in yt-formatted-string elements

Setting Up the Project

First, create a new project directory and initialize it:

mkdir youtube-transcript-scraper
cd youtube-transcript-scraper
npm init -y

Install the required dependencies:

npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

The stealth plugin helps avoid detection by making our automated browser appear more like a real user.

Basic Implementation

Let's start with a basic implementation that extracts transcripts from a YouTube video:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

// Use stealth plugin to avoid detection
puppeteer.use(StealthPlugin());

const scrapeYouTubeTranscript = async (url) => {
  const browser = await puppeteer.launch({
    headless: "new",
    ignoreDefaultArgs: ["--enable-automation"]
  });

  try {
    const page = await browser.newPage();
    await page.setViewport({ width: 1280, height: 800 });

    // Navigate to YouTube video
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
    await page.waitForTimeout(2000);

    // Handle cookie banner (EU compliance)
    try {
      await page.evaluate(() => {
        const cookieButton = document.querySelector('button[aria-label*="cookies"]');
        if (cookieButton) {
          cookieButton.click();
          console.log('Closed cookie banner');
        }
      });
    } catch (e) {
      console.log('No cookie banner found');
    }

    // Click transcript button
    await page.waitForSelector('ytd-video-description-transcript-section-renderer button', { timeout: 10000 });
    await page.evaluate(() => {
      const transcriptButton = document.querySelector('ytd-video-description-transcript-section-renderer button');
      if (transcriptButton) {
        transcriptButton.click();
        console.log('Clicked transcript button');
      }
    });

    await page.waitForTimeout(2000);

    // Extract transcript text
    const transcriptText = await page.evaluate(() => {
      const segments = Array.from(document.querySelectorAll('#segments-container yt-formatted-string'));
      return segments.map(element => element.textContent?.trim()).filter(text => text && text.length > 0);
    });

    return transcriptText.join(' ');

  } catch (error) {
    console.error('Error scraping transcript:', error);
    throw error;
  } finally {
    await browser.close();
  }
};

// Usage
const videoUrl = 'https://www.youtube.com/watch?v=dQw4w9WgXcQ';
scrapeYouTubeTranscript(videoUrl)
  .then(transcript => console.log('Transcript:', transcript))
  .catch(error => console.error('Failed to scrape transcript:', error));

Advanced Implementation with Error Handling

The basic implementation works for most videos, but YouTube's interface can vary. Here's a more robust version with comprehensive error handling:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

puppeteer.use(StealthPlugin());

const formatTimestamp = (seconds) => {
  const mins = Math.floor(seconds / 60);
  const secs = seconds % 60;
  return `${mins.toString().padStart(2, '0')}:${secs.toString().padStart(2, '0')}`;
};

const scrapeYouTubeTranscriptAdvanced = async (url) => {
  const browser = await puppeteer.launch({
    headless: "new",
    ignoreDefaultArgs: ["--enable-automation"]
  });

  try {
    const page = await browser.newPage();
    await page.setViewport({
      width: 1280,
      height: 1024,
      deviceScaleFactor: 1,
    });

    // Navigate to YouTube video
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
    await page.waitForTimeout(2000);

    // Try to close cookie banner if present
    try {
      await page.evaluate(() => {
        const cookieButton = document.querySelector('button[aria-label*="cookies"]');
        if (cookieButton) {
          cookieButton.click();
          console.log('Closed cookie banner');
        }
      });
    } catch (e) {
      console.log('No cookie banner found');
    }

    // Click transcript button with multiple fallback selectors
    try {
      await page.waitForSelector('ytd-video-description-transcript-section-renderer button', { timeout: 10000 });
      await page.evaluate(() => {
        const transcriptButton = document.querySelector('ytd-video-description-transcript-section-renderer button');
        if (transcriptButton) {
          transcriptButton.click();
          console.log('Clicked transcript button');
        }
      });
      await page.waitForTimeout(2000);
    } catch (e) {
      console.log('Transcript button not found, trying alternative selectors');

      // Try alternative selectors
      try {
        await page.evaluate(() => {
          const selectors = [
            'button[aria-label*="transcript"]',
            'button[aria-label*="Transcript"]',
            '[data-target-id="engagement-panel-transcript"] button',
            '#transcript-button',
            'button[aria-label*="Show transcript"]'
          ];

          for (const selector of selectors) {
            const button = document.querySelector(selector);
            if (button) {
              button.click();
              console.log('Clicked transcript button with selector:', selector);
              return;
            }
          }
        });
        await page.waitForTimeout(2000);
      } catch (e2) {
        console.log('Alternative transcript button selectors failed');
      }
    }

    // Extract transcript text with multiple fallback selectors
    let transcriptText = await page.evaluate(() => {
      const segments = Array.from(document.querySelectorAll('#segments-container yt-formatted-string'));
      return segments.map(element => element.textContent?.trim()).filter(text => text && text.length > 0);
    });

    if (transcriptText.length === 0) {
      // Try alternative transcript selectors
      const alternativeTranscript = await page.evaluate(() => {
        const selectors = [
          '#segments-container span',
          '#segments-container div',
          '[data-target-id="engagement-panel-transcript"] span',
          '[data-target-id="engagement-panel-transcript"] div',
          '.ytd-transcript-segment-renderer span',
          '.ytd-transcript-segment-renderer div'
        ];

        for (const selector of selectors) {
          const elements = document.querySelectorAll(selector);
          if (elements.length > 0) {
            const texts = Array.from(elements).map(el => el.textContent?.trim()).filter(text => text && text.length > 0);
            if (texts.length > 0) {
              return texts;
            }
          }
        }
        return [];
      });

      if (alternativeTranscript.length === 0) {
        throw new Error('No transcript available for this video.');
      }

      transcriptText = alternativeTranscript;
    }

    // Format transcript with timestamps
    const transcript = transcriptText.map((text, index) => ({
      text,
      start: index * 5,
      duration: 5,
      timestamp: formatTimestamp(index * 5),
    }));

    const fullText = transcript.map(entry => entry.text).join(' ');

    return {
      url,
      transcript,
      fullText,
      wordCount: fullText.split(' ').length,
      segments: transcript.length,
    };

  } catch (error) {
    console.error('Error scraping transcript:', error);
    throw error;
  } finally {
    await browser.close();
  }
};

// Usage
const videoUrl = 'https://www.youtube.com/watch?v=dQw4w9WgXcQ';
scrapeYouTubeTranscriptAdvanced(videoUrl)
  .then(result => {
    console.log('Full transcript:', result.fullText);
    console.log('Word count:', result.wordCount);
    console.log('Segments:', result.segments);
    console.log('First few segments:', result.transcript.slice(0, 5));
  })
  .catch(error => console.error('Failed to scrape transcript:', error));

Handling Common Issues

1. Cookie Banners

YouTube shows cookie consent banners in EU regions. Our code handles this by looking for buttons with "cookies" in their aria-label and clicking them automatically.

2. Transcript Button Variations

YouTube's interface changes frequently. We use multiple selectors to find the transcript button, ensuring compatibility across different layouts.

3. Dynamic Content Loading

Transcripts are loaded dynamically. We use waitForTimeout() to ensure content is fully loaded before attempting to extract it.

4. Rate Limiting

To avoid being blocked, consider:

Adding random delays between requests
Using residential proxies
Implementing retry logic with exponential backoff

Best Practices

Respect robots.txt: Always check YouTube's robots.txt file
Implement caching: Store transcripts locally to avoid repeated requests
Error handling: Always wrap your scraping code in try-catch blocks
User agent rotation: Use different user agents to appear more natural
Respect rate limits: Don't overwhelm YouTube's servers

Alternative: Using SocialKit YouTube Transcript API

If you're looking for a more reliable solution without managing browser instances and handling YouTube's constantly changing interface, consider using SocialKit's YouTube Transcript API:

curl "https://api.socialkit.dev/youtube/transcript?access_key=YOUR_ACCESS_KEY&url=https://youtube.com/watch?v=dQw4w9WgXcQ"

Example Response

{
  "success": true,
  "data": {
    "url": "https://youtube.com/watch?v=dQw4w9WgXcQ",
    "transcript": "[♪♪♪] ♪ We're no strangers to love ♪ ♪ You know the rules\nand so do I ♪ ♪ A full commitment's\nwhat I'm thinking of ♪ ♪ You wouldn't get this\nfrom any other guy ♪ ♪ I just wanna tell you\nhow I'm feeling ♪ ♪ Gotta make you understand ♪ ♪ Never gonna give you up ♪",
    "transcriptSegments": [
      {
        "text": "[♪♪♪]",
        "start": 0,
        "duration": 5,
        "timestamp": "00:00"
      },
      {
        "text": "♪ We're no strangers to love ♪",
        "start": 5,
        "duration": 5,
        "timestamp": "00:05"
      }
    ],
    "wordCount": 458,
    "segments": 61
  }
}

Benefits of using SocialKit:

Reliable extraction: Built-in handling of YouTube's interface changes
No browser management: No need to manage Puppeteer instances
Consistent results: Tested across thousands of videos
Automatic retries: Built-in retry logic for failed requests
Structured data: Returns both full text and timestamped segments
Scale-ready: Handle multiple videos without resource constraints

Conclusion

Scraping YouTube transcripts with Puppeteer is a powerful technique for automating content extraction. While the implementation requires careful handling of YouTube's dynamic interface and various edge cases, the results can be extremely valuable for content analysis, accessibility, and automation workflows.

Remember to always respect YouTube's terms of service and implement appropriate rate limiting to avoid being blocked. Happy scraping!

How to Hide Cookie Banners with Puppeteer

Jonathan Geiger — Thu, 15 May 2025 18:00:10 +0000

Let's face it, cookie consent banners are annoying, especially when you're trying to take clean screenshots with Puppeteer. In this guide, I'll show you how to automatically handle these pesky banners and get perfect screenshots every time.

Installing Required Dependencies

Before we dive into the code examples, you'll need to install the necessary packages:

# Install Puppeteer
npm install puppeteer

# Install AdBlocker for Puppeteer
npm install @cliqz/adblocker-puppeteer

# Install AutoConsent (optional, for Method 2)
npm install @duckduckgo/autoconsent

Method 1: Using AdBlocker Filter Lists

One of the most efficient ways to handle cookie banners is by leveraging existing filter lists designed for ad blockers. The @cliqz/adblocker-puppeteer package provides an excellent solution:

const puppeteer = require('puppeteer');
const { PuppeteerBlocker } = require('@cliqz/adblocker-puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    try {
        const page = await browser.newPage();

        // Initialize the blocker with cookie banner filter list
        const blocker = await PuppeteerBlocker.fromLists(fetch, [
            'https://secure.fanboy.co.nz/fanboy-cookiemonster.txt',
        ]);

        // Enable blocking in the page
        await blocker.enableBlockingInPage(page);

        await page.goto('https://example.com');
        await page.screenshot({ path: 'screenshot.png' });
    } catch (error) {
        console.error('Error:', error);
    } finally {
        await browser.close();
    }
})();

This approach is particularly effective because:

It uses community-maintained filter lists
It's regularly updated with new cookie banner patterns
It's lightweight and efficient
It works across multiple websites

Method 2: Using AutoConsent

For more complex scenarios, we can combine the filter list approach with AutoConsent, a tool specifically designed to handle cookie consent dialogs:

const puppeteer = require('puppeteer');
const { PuppeteerBlocker } = require('@cliqz/adblocker-puppeteer');
const autoconsent = require('@duckduckgo/autoconsent');

(async () => {
    const browser = await puppeteer.launch();
    try {
        const page = await browser.newPage();

        // Initialize blocker
        const blocker = await PuppeteerBlocker.fromLists(fetch, [
            'https://secure.fanboy.co.nz/fanboy-cookiemonster.txt',
        ]);
        await blocker.enableBlockingInPage(page);

        // Inject AutoConsent
        await page.evaluateOnNewDocument(autoconsent.script);

        await page.goto('https://example.com');
        await page.screenshot({ path: 'screenshot.png' });
    } catch (error) {
        console.error('Error:', error);
    } finally {
        await browser.close();
    }
})();

Method 3: Custom Cookie Banner Handler

For maximum control, you can implement a custom handler that combines multiple approaches:

const puppeteer = require('puppeteer');

async function handleCookieBanners(page) {
    // Common cookie banner selectors
    const selectors = [
        "[id*='cookie']",
        "[class*='cookie']",
        "[id*='consent']",
        "[class*='consent']",
        "[id*='gdpr']",
        "[class*='gdpr']",
        '.modal__overlay',
        '.cookie-banner',
        '.consent-banner',
    ];

    // Common accept button texts
    const acceptTexts = [
        'Accept',
        'Accept All',
        'Accept Cookies',
        'Allow All',
        'I agree',
        'Got it',
        'Continue',
    ];

    // Try to click accept buttons
    await page.evaluate((texts) => {
        const buttons = Array.from(
            document.querySelectorAll('button, a, [role="button"]')
        );
        const acceptButton = buttons.find((button) =>
            texts.includes(button.innerText.trim())
        );
        if (acceptButton) acceptButton.click();
    }, acceptTexts);

    // Hide any remaining banners
    await page.evaluate((selectors) => {
        document
            .querySelectorAll(selectors.join(', '))
            .forEach((el) => (el.style.display = 'none'));
    }, selectors);
}

(async () => {
    const browser = await puppeteer.launch();
    try {
        const page = await browser.newPage();
        await page.goto('https://example.com');

        // Handle cookie banners
        await handleCookieBanners(page);

        // Take screenshot
        await page.screenshot({ path: 'screenshot.png' });
    } catch (error) {
        console.error('Error:', error);
    } finally {
        await browser.close();
    }
})();

Best Practices

When implementing cookie banner handling, consider these best practices:

Wait for Page Load: Ensure the page is fully loaded before handling banners
Multiple Approaches: Combine different methods for better coverage
Error Handling: Implement proper error handling for failed banner removal
Testing: Test across different websites and banner types
Maintenance: Regularly update selectors and patterns

Alternative: Using CaptureKit Screenshot API

If you're looking for a more reliable solution without managing browser instances, consider using CaptureKit's Screenshot API:

curl "https://api.capturekit.dev/capture?url=https://example.com&remove_cookie_banners=true&access_key=YOUR_ACCESS_KEY"

Benefits of using CaptureKit:

Built-in cookie banner handling
No need to manage browser instances
Consistent results across different websites
Regular updates to handle new banner types

Why Cookie Banners Matter

Now that we've covered the solutions, let's understand why handling cookie banners is crucial for your screenshot automation:

Content Visibility: Banners often cover critical content, making your screenshots incomplete
Consistency: Different banner states can lead to inconsistent results across captures
Layout Issues: Banners can trigger different page layouts or behaviors
Professional Quality: Clean screenshots without banners look more professional
Automation Reliability: Banners can interfere with automated testing and monitoring

Conclusion

Handling cookie banners is crucial for producing clean, professional screenshots. By combining filter lists, AutoConsent, and custom handlers, you can effectively manage most cookie banner scenarios. For production use cases, consider using CaptureKit's API to eliminate the complexity of browser automation.

Happy capturing! 📸

If you found this post useful, you might also enjoy these related Puppeteer tutorials:

How to Block Requests with Puppeteer

Jonathan Geiger — Fri, 02 May 2025 11:43:19 +0000

When working with Puppeteer for web automation and scraping, you'll often encounter situations where you need to block specific requests to improve performance or reduce bandwidth usage. In this guide, we'll explore effective techniques for intercepting and blocking unwanted network requests in Puppeteer.

Why Block Requests in Puppeteer?

Blocking unnecessary requests offers several benefits:

Faster page loading - Skipping images, analytics, and ads can dramatically reduce load times
Reduced bandwidth usage - Especially important for cloud-based scraping or serverless functions
Lower memory consumption - Fewer resources to process means less memory used
Improved stability - Fewer network requests means fewer potential points of failure

Let's dive into the practical implementations.

Basic Request Interception in Puppeteer

The foundation of request blocking in Puppeteer is the request interception mechanism. Here's a simple example:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();

    // Enable request interception
    await page.setRequestInterception(true);

    // Add event listener to intercept requests
    page.on('request', (request) => {
      // Your logic to determine which requests to block
      // ...

      // Either abort or continue the request
      request.continue();
    });

    await page.goto('https://example.com');

    // Continue with your automation
    await page.screenshot({ path: 'example.png' });

  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

This is the basic structure, but the real power comes in how you decide which requests to block.

Blocking Requests by Resource Type

One of the most common approaches is to block requests based on resource type. Puppeteer allows you to identify the type of each request:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();

    // Enable request interception
    await page.setRequestInterception(true);

    // Block requests by resource type
    page.on('request', (request) => {
      const resourceType = request.resourceType();

      // Define which resources to block
      const blockResources = ['image', 'stylesheet', 'font', 'media'];

      if (blockResources.includes(resourceType)) {
        request.abort();
      } else {
        request.continue();
      }
    });

    await page.goto('https://example.com');

    // Take a screenshot to see the effect
    await page.screenshot({ path: 'no-images-css.png' });

    console.log('Page loaded without images, CSS, fonts, and media!');

  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

Puppeteer supports the following resource types:

document
stylesheet
image
media
font
script
texttrack
xhr
fetch
eventsource
websocket
manifest
other

Choose which types to block based on your specific use case. For pure data extraction, blocking images, stylesheets, fonts, and media is often a good choice.

Blocking Requests by URL Pattern

Another approach is to block requests that match specific URL patterns:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();

    await page.setRequestInterception(true);

    // Block requests by URL pattern
    page.on('request', (request) => {
      const url = request.url();

      // Define patterns to block
      const blockedPatterns = [
        'google-analytics.com',
        'googletagmanager.com',
        'doubleclick.net',
        'facebook.net',
        'ads',
        'tracking',
        '.png',
        '.jpg',
        '.jpeg',
        '.gif'
      ];

      // Check if URL contains any blocked pattern
      if (blockedPatterns.some(pattern => url.includes(pattern))) {
        request.abort();
      } else {
        request.continue();
      }
    });

    await page.goto('https://example.com');

    await page.screenshot({ path: 'no-trackers-images.png' });

    console.log('Page loaded without tracking scripts and images!');

  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

This technique is particularly useful for:

Blocking analytics and tracking scripts
Preventing ad networks from loading
Filtering out certain file types
Blocking third-party resources

Combining Multiple Blocking Strategies

For more comprehensive request blocking, combine both resource type and URL pattern approaches:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();

    await page.setRequestInterception(true);

    // Combined blocking strategy
    page.on('request', (request) => {
      const resourceType = request.resourceType();
      const url = request.url();

      // Block by resource type
      const blockedResourceTypes = ['image', 'media', 'font'];

      // Block by URL pattern
      const blockedUrlPatterns = [
        'analytics',
        'tracking',
        'advertisement',
        'doubleclick',
      ];

      // Block if either condition is met
      if (
        blockedResourceTypes.includes(resourceType) ||
        blockedUrlPatterns.some(pattern => url.includes(pattern))
      ) {
        request.abort();
      } else {
        request.continue();
      }
    });

    await page.goto('https://example.com');

    // Continue with your automation

  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

This combined approach gives you the most control over which requests are allowed.

Using Wildcards for URL Pattern Matching

For more flexible URL matching, consider using a wildcard matching library:

const puppeteer = require('puppeteer');
const wildcardMatch = require('wildcard-match');

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();

    await page.setRequestInterception(true);

    // Create matchers for different patterns
    const isBlockedDomain = wildcardMatch(['*.analytics.com', '*.ads.*', 'tracker.*']);
    const isBlockedFile = wildcardMatch(['*.png', '*.jpg', '*.gif']);

    page.on('request', (request) => {
      const url = request.url();

      if (isBlockedDomain(url) || isBlockedFile(url)) {
        request.abort();
      } else {
        request.continue();
      }
    });

    await page.goto('https://example.com');

  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

You'll need to install the wildcard-match package:

npm install wildcard-match

Measuring Performance Improvements

To understand the impact of request blocking, you can measure the performance difference:

const puppeteer = require('puppeteer');

async function measureLoadTime(blockRequests) {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();

    // Set up request interception if enabled
    if (blockRequests) {
      await page.setRequestInterception(true);
      page.on('request', (request) => {
        const resourceType = request.resourceType();
        if (['image', 'stylesheet', 'font'].includes(resourceType)) {
          request.abort();
        } else {
          request.continue();
        }
      });
    }

    // Measure performance
    const startTime = Date.now();

    await page.goto('https://example.com', {
      waitUntil: 'networkidle2',
    });

    const loadTime = Date.now() - startTime;
    return loadTime;
  } finally {
    await browser.close();
  }
}

(async () => {
  console.log('Testing performance...');

  // Without blocking
  const regularLoadTime = await measureLoadTime(false);
  console.log(`Regular load time: ${regularLoadTime}ms`);

  // With blocking
  const optimizedLoadTime = await measureLoadTime(true);
  console.log(`Optimized load time: ${optimizedLoadTime}ms`);

  // Calculate improvement
  const improvement = ((regularLoadTime - optimizedLoadTime) / regularLoadTime * 100).toFixed(2);
  console.log(`Performance improvement: ${improvement}%`);
})();

This script helps you quantify the benefits of request blocking for your specific use case.

Potential Issues with Request Interception

While request interception is powerful, be aware of these potential issues:

Performance impact - In older versions of Puppeteer, request interception disables the native browser cache, which can slow down navigation
Breaking functionality - Blocking certain resources might break website functionality if you need to interact with the page
Race conditions - Complex request handling can sometimes lead to race conditions

For simple use cases, these issues are rarely a problem, but keep them in mind for more complex scenarios.

Simplified Solution: Using a Screenshot API

If you're using Puppeteer primarily for taking screenshots, CaptureKit Screenshot API offers a simpler approach with built-in request blocking:

curl "https://api.capturekit.dev/capture?url=https://example.com&block_resources=image,media,font&access_key=YOUR_ACCESS_KEY"

With CaptureKit, you can specify which resource types to block without managing browser instances or request interception:

Parameter	Type	Description
`block_resources`	string	Comma-separated list of resource types to block (e.g., "image,stylesheet,font")
`block_urls`	string	Comma-separated list of URL patterns to block (e.g., "analytics,tracking,advertisement")

This approach eliminates the complexity of managing browser instances and request interception directly.

Conclusion

Blocking unnecessary requests in Puppeteer can significantly improve performance, reduce bandwidth usage, and make your automation scripts more efficient. By selectively filtering requests based on resource type, URL patterns, or a combination of both, you can focus on the content that matters for your specific use case.

Whether you implement request blocking directly with Puppeteer or use a service like CaptureKit API, this technique should be part of your web automation toolkit.

Happy optimizing! 🚀

If you found this post useful, you might also enjoy these related Puppeteer tutorials:

How to Take Full Page Screenshots with Puppeteer

Jonathan Geiger — Sat, 26 Apr 2025 18:16:14 +0000

While Puppeteer offers a simple fullPage: true option for taking full page screenshots, the reality is often more complex. In this guide, we'll explore how to reliably capture full-page screenshots with Puppeteer, addressing common challenges and offering practical solutions.

Basic Full Page Screenshot with Puppeteer

Let's start with the simplest approach – using Puppeteer's built-in fullPage option:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();
    await page.setViewport({ width: 1280, height: 800 });

    await page.goto('https://example.com', { 
      waitUntil: ['load', 'domcontentloaded'] 
    });

    await page.screenshot({
      path: 'full-page-screenshot.png',
      fullPage: true
    });

    console.log('Screenshot captured successfully!');
  } catch (error) {
    console.error('Error capturing screenshot:', error);
  } finally {
    await browser.close();
  }
})();

This code should work for simple websites, but modern sites often present challenges that this basic approach doesn't handle.

Common Challenges with Full Page Screenshots

When taking full-page screenshots, you might encounter several issues:

Lazy-loaded images and content - Elements that load only when they enter the viewport
Sticky headers and footers - Elements that may appear multiple times in the screenshot
Animations triggered by scrolling - Visual effects that only appear during user interaction
Very long pages - Websites where Puppeteer's built-in fullPage option might fail
Dynamic height content - Pages that change height as they load or as the user interacts

Let's address these challenges one by one.

Handling Lazy-Loaded Content

The most common issue with full-page screenshots is lazy-loaded images and content that only load when scrolled into view. Here's a technique to trigger lazy loading by scrolling through the entire page before capturing:

const puppeteer = require('puppeteer');

async function scrollPageToBottom(page) {
  await page.evaluate(async () => {
    await new Promise((resolve) => {
      let totalHeight = 0;
      const distance = 100;
      const timer = setInterval(() => {
        const scrollHeight = document.body.scrollHeight;
        window.scrollBy(0, distance);
        totalHeight += distance;

        if (totalHeight >= scrollHeight) {
          clearInterval(timer);
          window.scrollTo(0, 0); // Scroll back to top
          resolve();
        }
      }, 100);
    });
  });
}

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();
    await page.setViewport({ width: 1280, height: 800 });

    await page.goto('https://example.com', { 
      waitUntil: 'networkidle2'
    });

    // Scroll to trigger lazy loading
    await scrollPageToBottom(page);

    // Wait a moment for images to finish loading
    await page.waitForTimeout(1000);

    // Now take the full page screenshot
    await page.screenshot({
      path: 'full-page-with-lazy-content.png',
      fullPage: true
    });

    console.log('Full page screenshot with lazy-loaded content captured!');
  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

This approach scrolls the page gradually, triggering lazy-loaded content to appear, and then captures the full page.

For more details on ensuring pages fully load before capturing, check out our guide on how to wait for page to load in Puppeteer.

Advanced Technique: Section-by-Section Capture

For more complex cases, especially with animations that play during scrolling or very long pages, a more reliable approach is to capture the page in sections and merge them together:

const puppeteer = require('puppeteer');
const merge = require('merge-img');
const Jimp = require('jimp');

(async () => {
  const browser = await puppeteer.launch();
  try {
    const page = await browser.newPage();
    await page.goto('https://example.com');

    const path = 'sectioned-full-page.png';

    // Calculate sections needed based on page height
    const { pages, extraHeight, viewport } = await page.evaluate(() => {
      window.scrollTo(0, 0);
      const pageHeight = Math.max(
        document.body.scrollHeight,
        document.documentElement.scrollHeight,
        document.body.offsetHeight,
        document.documentElement.offsetHeight,
        document.body.clientHeight,
        document.documentElement.clientHeight
      );

      return {
        pages: Math.ceil(pageHeight / window.innerHeight),
        extraHeight: (pageHeight % window.innerHeight) * window.devicePixelRatio,
        viewport: {
          height: window.innerHeight * window.devicePixelRatio,
          width: window.innerWidth * window.devicePixelRatio,
        },
      };
    });

    console.log(`Taking screenshot in ${pages} sections`);

    // Take screenshots of each section
    const sectionScreenshots = [];
    for (let i = 0; i < pages; i++) {
      // Scroll to position
      await page.evaluate((i, innerHeight) => {
        window.scrollTo(0, i * innerHeight);
      }, i, await page.evaluate(() => window.innerHeight));

      // Wait for any animations or content to settle
      await page.waitForTimeout(400);

      // Capture this section
      const screenshot = await page.screenshot({
        type: 'png',
        captureBeyondViewport: false,
      });

      sectionScreenshots.push(screenshot);
    }

    // Handle single-section case
    if (pages === 1) {
      const screenshot = await Jimp.read(sectionScreenshots[0]);
      screenshot.write(path);
      console.log('Single section screenshot saved');
      return;
    }

    // Handle final section if it's partial
    if (extraHeight > 0) {
      const cropped = await Jimp.read(sectionScreenshots.pop())
        .then((image) =>
          image.crop(
            0,
            viewport.height - extraHeight,
            viewport.width,
            extraHeight
          )
        )
        .then((image) => image.getBufferAsync(Jimp.AUTO));

      sectionScreenshots.push(cropped);
    }

    // Merge all screenshots
    const result = await merge(sectionScreenshots, { direction: true });

    // Save the final image
    await new Promise((resolve) => {
      result.write(path, () => {
        console.log('Full sectioned screenshot saved successfully!');
        resolve();
      });
    });
  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

This approach:

Divides the page into viewport-sized sections
Captures each section separately
Processes the final section if it's not a complete viewport
Merges all sections into a single image

You'll need to install the following packages:

npm install puppeteer merge-img jimp

Ultimate Technique: Smart Scrolling with Viewport Measurement

For the most reliable results across a wide range of websites, we can implement a comprehensive solution that:

Precisely measures the total page height
Implements gradual scrolling to trigger all lazy-loaded content
Waits for animations to complete
Returns to the top before capturing the full screenshot

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  try {
    const page = await browser.newPage();

    // Set a reasonable viewport
    await page.setViewport({ width: 1280, height: 800 });

    // Navigate to the target page with appropriate wait conditions
    await page.goto('https://example.com', {
      waitUntil: ['load', 'domcontentloaded', 'networkidle2'],
    });

    // Calculate the true height of the page
    const pageHeight = await page.evaluate(() => {
      return Math.max(
        document.body.scrollHeight,
        document.documentElement.scrollHeight,
        document.body.offsetHeight,
        document.documentElement.offsetHeight,
        document.body.clientHeight,
        document.documentElement.clientHeight
      );
    });

    // Get the viewport height
    const viewportHeight = await page.evaluate(() => window.innerHeight);

    console.log(`Page height: ${pageHeight}px, Viewport height: ${viewportHeight}px`);

    // Implement smart scrolling to trigger lazy loading
    console.log('Scrolling page to trigger lazy loading...');
    await page.evaluate(
      async (pageHeight, scrollDuration) => {
        // Start at the top
        window.scrollTo(0, 0);
        await new Promise((resolve) => setTimeout(resolve, 100));

        // Scroll to bottom gradually to trigger lazy loading
        const steps = 20;
        const stepSize = pageHeight / steps;
        const stepDelay = scrollDuration / steps;

        for (let i = 1; i <= steps; i++) {
          window.scrollTo(0, i * stepSize);
          await new Promise((resolve) => setTimeout(resolve, stepDelay));
        }

        // Ensure we reach the bottom
        window.scrollTo(0, pageHeight);
        await new Promise((resolve) => setTimeout(resolve, 300));

        // Return to top
        window.scrollTo(0, 0);
        await new Promise((resolve) => setTimeout(resolve, 300));
      },
      pageHeight,
      2000 // Scroll duration in ms
    );

    // Wait for everything to settle after scrolling
    console.log('Waiting for page to settle...');
    await page.waitForTimeout(1000);

    // Finally, capture the full page screenshot
    console.log('Taking full page screenshot...');
    const screenshot = await page.screenshot({
      path: 'advanced-full-page.png',
      fullPage: true,
    });

    console.log('Full page screenshot captured successfully!');
  } catch (error) {
    console.error('Error capturing screenshot:', error);
  } finally {
    await browser.close();
  }
})();

This approach provides the most reliable results for most websites, but it may still struggle with extremely long pages or complex layouts.

For a simpler approach to taking screenshots with Puppeteer, check out our basic guide on how to take screenshots with Puppeteer.

Simplified Solution: Using CaptureKit API

Setting up Puppeteer for reliable full-page screenshots requires handling numerous edge cases. If you need a faster, more reliable solution without the complexity, CaptureKit API offers a simple alternative:

curl "https://api.capturekit.dev/capture?url=https://example.com&full_page=true&access_key=YOUR_ACCESS_KEY"

With CaptureKit, you can control the lazy-loading behavior and scroll duration:

curl "https://api.capturekit.dev/capture?url=https://example.com&full_page=true&full_page_scroll=true&full_page_scroll_duration=800&access_key=YOUR_ACCESS_KEY"

Benefits of Using CaptureKit for Full-Page Screenshots

No browser management - Forget about setting up and maintaining Puppeteer
Smart lazy-loading - Automatically handles lazy-loaded content and animations
Reliable capture - Works with even the most complex websites and layouts
Fine-tuned control - Configure scroll behavior and timing as needed

Available Options for Full-Page Screenshots

Parameter	Type	Description
`full_page`	boolean	Capture the entire page instead of just the visible viewport (default: false)
`full_page_scroll`	boolean	Scroll the page to fully load lazy-loaded elements before capturing (default: true)
`full_page_scroll_duration`	number	Time in milliseconds for scrolling before capturing (default: 400)

Conclusion

Taking reliable full-page screenshots with Puppeteer requires understanding and addressing several challenges, especially with modern websites that use lazy loading and dynamic content. The techniques in this guide will help you capture complete, accurate screenshots in most scenarios.

For production use cases where reliability and simplicity are priorities, consider using CaptureKit API to eliminate the complexities of browser automation and focus on your core tasks instead.

Happy capturing! 📸

If you encountered this post and found it useful, I write a lot about scraping and Puppeteer. Maybe you will find these scraping tutorials with Puppeteer useful as well:

Markdown Extraction Support

Jonathan Geiger — Tue, 15 Apr 2025 10:30:45 +0000

We've just released a new feature in CaptureKit based on user feedback: Markdown extraction. This update enhances CaptureKit's content extraction capabilities, allowing you to retrieve clean, structured text from web pages.

📝 Markdown Extraction (in /content API)

You can now extract the Markdown representation of web pages using the /content endpoint. This makes it easier to work with the textual content of web pages in a format that's both human-readable and machine-processable.

Example Request

GET https://api.capturekit.dev/content?access_key=<your-access-key>&url=https://capturekit.dev&include_markdown=true

Example Response

{
  "success": true,
  "data": {
    "metadata": { ... },
    "links": { ... },
    "html": "<html><body><h1>Hello, world!</h1></body></html>",
    "markdown": "# Hello, world!"
  }
}

Parameters

url (string, required): URL of the webpage
access_key (string, required): Your API key
include_markdown (boolean, optional): Set to true to include Markdown data (defaults to false)

Why Markdown?

Markdown provides several advantages over raw HTML:

Readability: Markdown is cleaner and easier to read than HTML
Simplicity: It removes unnecessary styling and formatting
Portability: Easy to use in various applications and platforms
Text Processing: Ideal for content analysis, summarization, and AI processing

Use Cases

Content Management: Import web content directly into your CMS
AI Processing: Feed web content to LLMs and other AI systems in a clean format
Documentation: Extract documentation from websites for offline use
Knowledge Bases: Build internal knowledge repositories from web content

Final Notes

This feature was developed in direct response to user feedback. We're committed to building CaptureKit to meet your real-world needs.

Have ideas for more features? Let us know! We're actively developing CaptureKit based on user input.

Thanks for being part of the journey!

How to Convert HTML to Markdown

Jonathan Geiger — Tue, 15 Apr 2025 10:26:55 +0000

HTML is the standard markup language for web pages, but Markdown provides a more readable and writeable format for content creation. Converting HTML to Markdown can be useful for content extraction, documentation, and migrating web content to platforms that use Markdown. In this guide, we'll show you how to convert HTML to Markdown using JavaScript, and then demonstrate how CaptureKit API offers a simpler alternative.

Method 1: Converting HTML to Markdown with JavaScript

To convert HTML to Markdown, we'll use the popular turndown library, which handles most of the heavy lifting. We'll need to:

Set up the Turndown service with appropriate options
Remove unwanted HTML elements
Convert relative URLs to absolute URLs
Process the HTML content

Here's a complete solution using JavaScript with turndown:

import TurndownService from 'turndown';

function convertHtmlToMarkdown(htmlString) {
  try {
    // Initialize Turndown with custom options
    const turndownService = new TurndownService({
      headingStyle: 'atx',
      codeBlockStyle: 'fenced',
      emDelimiter: '_',
      hr: '---',
      bulletListMarker: '-',
      strongDelimiter: '**',
    });

    // Remove scripts, styles, and other unwanted elements
    turndownService.remove(['script', 'style', 'noscript', 'iframe']);

    // Convert the HTML to Markdown
    const markdown = turndownService.turndown(htmlString);

    return markdown;
  } catch (error) {
    console.error('Error converting HTML to Markdown:', error);
    return '';
  }
}

// Usage example
function main() {
  const htmlString = `
    <html>
      <body>
        <h1>Hello World</h1>
        <p>This is a <strong>bold</strong> statement.</p>
        <ul>
          <li>Item 1</li>
          <li>Item 2</li>
        </ul>
        <a href="https://example.com">Example Link</a>
      </body>
    </html>
  `;

  const markdown = convertHtmlToMarkdown(htmlString);
  console.log('Markdown content:', markdown);
}

main();

Handling Relative URLs

One common issue when converting HTML to Markdown is dealing with relative URLs in links and images. To ensure all links and images work correctly, we need to convert relative URLs to absolute URLs:

import TurndownService from 'turndown';

function convertHtmlToMarkdownWithAbsoluteUrls(htmlString, baseUrl) {
  try {
    // Initialize Turndown with custom options
    const turndownService = new TurndownService({
      headingStyle: 'atx',
      codeBlockStyle: 'fenced',
      emDelimiter: '_',
      hr: '---',
      bulletListMarker: '-',
      strongDelimiter: '**',
    });

    // Remove scripts, styles, and other unwanted elements
    turndownService.remove(['script', 'style', 'noscript', 'iframe']);

    // Parse the domain from the URL
    const domain = new URL(baseUrl).origin;

    // Convert relative URLs to absolute URLs in links
    turndownService.addRule('links', {
      filter: 'a',
      replacement: function (content, node) {
        const href = node.getAttribute('href');
        if (!href) return content;

        // Convert relative URLs to absolute
        let absoluteUrl = href;
        if (href.startsWith('/')) {
          absoluteUrl = domain + href;
        } else if (
          !href.startsWith('http://') &&
          !href.startsWith('https://') &&
          !href.startsWith('#')
        ) {
          absoluteUrl = domain + '/' + href;
        }

        return '[' + content + '](' + absoluteUrl + ')';
      },
    });

    // Convert relative URLs to absolute URLs in images
    turndownService.addRule('images', {
      filter: 'img',
      replacement: function (content, node) {
        const src = node.getAttribute('src');
        const alt = node.getAttribute('alt') || '';
        if (!src) return '';

        // Convert relative URLs to absolute
        let absoluteUrl = src;
        if (src.startsWith('/')) {
          absoluteUrl = domain + src;
        } else if (
          !src.startsWith('http://') &&
          !src.startsWith('https://') &&
          !src.startsWith('data:')
        ) {
          absoluteUrl = domain + '/' + src;
        }

        return '![' + alt + '](' + absoluteUrl + ')';
      },
    });

    // Convert the HTML to Markdown
    const markdown = turndownService.turndown(htmlString);

    return markdown;
  } catch (error) {
    console.error('Error converting HTML to Markdown:', error);
    return '';
  }
}

// Usage example
function main() {
  const htmlString = `
    <html>
      <body>
        <h1>Example Website</h1>
        <p>Check out our <a href="/about">About page</a>.</p>
        <img src="/images/logo.png" alt="Logo">
        <a href="products/new">New Products</a>
      </body>
    </html>
  `;

  const baseUrl = 'https://example.com';
  const markdown = convertHtmlToMarkdownWithAbsoluteUrls(htmlString, baseUrl);
  console.log('Markdown content:', markdown);
}

main();

How It Works

This code handles several important aspects of HTML to Markdown conversion:

Custom Formatting: Configures the output Markdown style (headings, code blocks, etc.)
Element Filtering: Removes unwanted elements like scripts and styles
URL Handling: Converts relative URLs to absolute URLs in both links and images
Content Processing: Converts the full HTML document to clean Markdown

Method 2: Using CaptureKit API (Recommended)

While the JavaScript approach is flexible, it requires handling HTTP requests, HTML parsing, and URL transformations. CaptureKit API offers a simpler solution that handles all these complexities for you.

Here's how to use CaptureKit API to convert HTML to Markdown:

curl "https://api.capturekit.dev/content?url=https://example.com&access_key=YOUR_ACCESS_KEY&include_markdown=true"

The API response includes the Markdown content along with other useful website information:

{
  "success": true,
  "data": {
    "metadata": {
      "title": "CaptureKit - Turn any website into a screenshot with our powerful Screenshot API",
      "description": "CaptureKit is a powerful API for capturing screenshots, extracting HTML, gathering links, and summarizing content—all with a simple request.",
      "favicon": "https://capturekit.dev/favicon.ico",
      "ogImage": "https://capturekit-assets.s3.amazonaws.com/capturekit-og+(1).png"
    },
    "links": {
      "internal": [
        "https://capturekit.dev/",
        "https://capturekit.dev/dashboard",
        "https://capturekit.dev/pricing",
        "https://capturekit.dev/blog"
      ],
      "external": [
        "https://docs.capturekit.dev",
        "https://zapier.com/apps/capturekit-website-screenshots-p/integrations",
        "https://www.nextupkit.com"
      ],
      "social": [
        "https://github.com/CaptureKit-Web-Scraping-API",
        "https://x.com/capturekit"
      ]
    },
    "markdown": "# CaptureKit - Turn any website into a screenshot with our powerful Screenshot API\n\nCaptureKit is a powerful API for capturing screenshots, extracting HTML, gathering links, and summarizing content—all with a simple request.\n\n..."
  }
}

API Parameters

url string (Required): The URL of the webpage to capture.
access_key string (Required): Your API access key. Can be provided via the access_key query parameter, x-access-key header, or request body.
include_html boolean (Optional, Default: false): Include the raw HTML of the webpage in the response.
include_markdown boolean (Optional, Default: false): Include the Markdown of the webpage in the response.
include_sitemap boolean (Optional, Default: false): Include sitemap data of the webpage in the response.

Benefits of Using CaptureKit API

Simplicity: One API call instead of dozens of lines of code
Reliability: Handles all edge cases and complex HTML structures
Performance: Optimized for speed and efficiency
Additional Data: Get website metadata alongside the Markdown content
No Maintenance: No need to update your code when HTML standards change

Conclusion

Converting HTML to Markdown is essential for content extraction, documentation, and migration tasks. While our JavaScript solution provides a comprehensive approach with full control, CaptureKit API offers a more convenient alternative that handles the complexities for you.

Choose the method that best fits your needs:

Use the JavaScript solution if you need full control over the conversion process
Use CaptureKit API if you want a quick, reliable solution with minimal code

By converting HTML to Markdown, you can transform complex web content into a clean, readable format that's easy to work with and integrate into various platforms.