DEV Community

Coco
Coco

Posted on

How to scrape YouTube in 2026?

#python #webscraping #tutorial #api

Three Python functions that scrape YouTube video metadata, a channel's uploads and a full timed transcript, using the Chocodata API. Every snippet below was executed on 27 July 2026 and the output blocks are copied from that run.

Here is what the finished script prints:

NASA  15,000,000 subscribers  6,100 videos  more=True
pulled 30 rows from page 1
newest: Moon Base: June 2026 Update  (199,184 views)
  199,192 views | 8,156 likes | 65s | Science & Technology
  transcript: 13 segments, 94 words, 1 languages
  [Dramatic music] [Jared Isaacman] People are looking up again, believi
wrote youtube_channel_videos.csv
Enter fullscreen mode Exit fullscreen mode

TL;DR

  • The video endpoint returns 26 top-level fields. There is no data wrapper, so r.json()["title"] is correct and r.json()["data"] raises KeyError.
  • A channel request returns 30 videos per page plus has_more and a 2,372-character continuation token.
  • Transcripts come back as timestamped segments. The Rick Astley video gives 61 segments, 487 words, 2,089 characters.
  • comment_count is None and keywords is [] on plenty of videos. Both crash naive code.

Why is it hard to scrape YouTube?

YouTube ships the watch page as a shell and builds the visible values in the browser from a JSON blob, so a plain requests.get plus an HTML parser returns markup with no title, no view count and no captions. Channel uploads compound it by paginating with continuation tokens instead of page numbers, and transcripts are not in the page at all. The part that eats the most time is the transcript, because captions are served from a separate call whose parameters are signed and short lived.

Prerequisites

To scrape YouTube with this code you need three things.

1. A free Chocodata API key. Sign up and copy it from the dashboard. Free tier, no card.

The Chocodata dashboard showing where to copy a free API key

2. Python 3.9+ and requests. Tested on Python 3.13.7 with requests 2.34.2 in July 2026.

pip install requests
Enter fullscreen mode Exit fullscreen mode

3. A target URL. The watch URL for videos and transcripts, the channel URL for uploads.

Scrape one video's metadata

To scrape a YouTube video, send a single GET with the watch URL and read the fields off the top level of the response.

1. Send the request and check the status 

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"

r = requests.get(f"{BASE}/youtube/video",
                 params={"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
                         "api_key": API_KEY}, timeout=30)
r.raise_for_status()
v = r.json()
print(r.status_code, len(v), "fields")
print(v["video_id"], v["title"])
Enter fullscreen mode Exit fullscreen mode

Output:

200 26 fields
dQw4w9WgXcQ Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)
Enter fullscreen mode Exit fullscreen mode

Terminal showing HTTP 200, 26 fields and the parsed video title

2. Map the fields you actually need

Pick the fields explicitly rather than storing the whole payload. The counts are real integers, not display strings.

row = {k: v[k] for k in ("video_id", "title", "channel_name", "view_count",
                         "like_count", "duration_seconds", "category",
                         "publish_date")}
print(row["view_count"], row["like_count"], row["duration_seconds"])
print(row["channel_name"], "|", row["category"], "|", row["publish_date"][:10])
Enter fullscreen mode Exit fullscreen mode

Output:

1797072750 19279452 213
Rick Astley | Music | 2009-10-24
Enter fullscreen mode Exit fullscreen mode

Terminal printing the mapped video fields with view and like counts

3. Handle the null and empty fields

comment_count came back None on this video and keywords is an empty list on many others, so coalesce before doing arithmetic or indexing.

print("comment_count:", repr(v["comment_count"]))
print("keywords:", len(v["keywords"]), "| thumbnails:", len(v["thumbnails"]),
      "| related:", v["related_count"])
engagement = (v["like_count"] or 0) + (v["comment_count"] or 0)
print("engagement:", engagement)
Enter fullscreen mode Exit fullscreen mode

Output:

comment_count: None
keywords: 27 | thumbnails: 5 | related: 12
engagement: 19279452
Enter fullscreen mode Exit fullscreen mode

Terminal showing the null comment count handled with a coalesce

Pull a channel's video list

To scrape a YouTube channel's uploads, use the channel parameter, which accepts the channel URL as its value.

1. Request the channel 

r = requests.get(f"{BASE}/youtube/channel",
                 params={"channel": "https://www.youtube.com/@NASA",
                         "api_key": API_KEY}, timeout=60)
r.raise_for_status()
ch = r.json()
print(ch["channel_name"], ch["channel_id"],
      ch["subscriber_count"], ch["video_count"])
Enter fullscreen mode Exit fullscreen mode

Output:

NASA UCLA_DiR1FfKNvjuUpBHmylQ 15000000 6100
Enter fullscreen mode Exit fullscreen mode

Terminal resolving the channel URL to a channel ID and counts

2. Flatten the videos list into rows

ch["videos"] is a list of dicts with position, id, title, url, thumbnail, channel, views and published. Note that views is a display string and published is relative, so keep id as your key.

rows = [{"position": x["position"], "video_id": x["id"], "title": x["title"],
         "views": x["views"], "published": x["published"], "url": x["url"]}
        for x in ch["videos"]]

print(len(rows), "rows")
for x in rows[:3]:
    print(x["position"], x["video_id"], x["views"], "|", x["title"][:44])
Enter fullscreen mode Exit fullscreen mode

Output:

30 rows
1 aRSBZN2UF0Q 199,184 views | Moon Base: June 2026 Update
2 75-H-i9ctkE 39,288 views | Independence Day Wishes from the Space Stati
3 84id2dzpwU0 45,248 views | Astronauts on the Space Station Celebrate th
Enter fullscreen mode Exit fullscreen mode

Terminal listing 30 flattened channel rows with IDs and view strings

3. Follow has_more for the next page

Pagination is a token, not an offset, so check has_more and carry next_page_token forward instead of incrementing a number.

print("videos_count:", ch["videos_count"], "| has_more:", ch["has_more"],
      "| token chars:", len(ch["next_page_token"]))

if ch["has_more"]:
    r = requests.get(f"{BASE}/youtube/channel",
                     params={"page_token": ch["next_page_token"],
                             "api_key": API_KEY}, timeout=60)
    r.raise_for_status()
    nxt = r.json()
    print("page", nxt["page"], "->", nxt["videos_count"], "more videos")
Enter fullscreen mode Exit fullscreen mode

Output:

videos_count: 30 | has_more: True | token chars: 2372
page 2 -> 30 more videos
Enter fullscreen mode Exit fullscreen mode

Positions restart at 1 on each page, so build your key from video_id and not from position.

Terminal showing has_more true and the continuation token length

Extract a video transcript

To scrape a YouTube transcript, send the watch URL to the transcript endpoint and read the segments list.

1. Fetch the transcript for a video URL 

r = requests.get(f"{BASE}/youtube/transcript",
                 params={"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
                         "api_key": API_KEY}, timeout=60)
r.raise_for_status()
tr = r.json()

if not tr["transcript_available"]:
    raise SystemExit("no captions for this video")

print(tr["source"], tr["language"], tr["segment_count"],
      tr["word_count"], tr["char_count"])
Enter fullscreen mode Exit fullscreen mode

Output:

native en 61 487 2089
Enter fullscreen mode Exit fullscreen mode

source is native when the captions were authored rather than machine generated, which is_generated also reports as False.

Terminal showing transcript source, language and segment totals

2. Filter segments by timestamp

Every segment carries start and duration in seconds, so slicing a time window is a list comprehension.

window = [s for s in tr["segments"] if 40 <= s["start"] < 60]
print(len(window), "segments between 40s and 60s")
for s in window[:4]:
    print(f"{s['start']:>6.2f} +{s['duration']:.2f}  {s['text']}")
Enter fullscreen mode Exit fullscreen mode

Output:

7 segments between 40s and 60s
 40.52 +2.40  ♪ Gotta make you understand ♪
 43.00 +2.12  ♪ Never gonna give you up ♪
 45.20 +1.88  ♪ Never gonna let you down ♪
 47.32 +3.80  ♪ Never gonna run around and desert you ♪
Enter fullscreen mode Exit fullscreen mode

Terminal printing transcript segments filtered to a time window

3. Join the segments into one text block

For search or summarisation you want the flat text, which is a join away.

text = " ".join(s["text"] for s in tr["segments"])
print(len(text), "chars |", len(text.split()), "words")
print(text[:70])
Enter fullscreen mode Exit fullscreen mode

Output:

2089 chars | 487 words
[♪♪♪] ♪ We're no strangers to love ♪ ♪ You know the rules and so
Enter fullscreen mode Exit fullscreen mode

Terminal showing the joined transcript text and its length

The part that breaks

Three failures, all hit while writing this, none of them in the network layer.

Assuming a data wrapper. The fields are at the top level. Reaching for ["data"] is the single most common mistake porting from other APIs:

Traceback (most recent call last):
  File "scrape_youtube.py", line 12, in <module>
    title = r.json()["data"]["title"]
            ~~~~~~~~^^^^^^^^
KeyError: 'data'
Enter fullscreen mode Exit fullscreen mode

comment_count is None, not 0. Summing it straight is a TypeError that only fires on the videos where it is missing:

TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
Enter fullscreen mode Exit fullscreen mode

keywords is empty on many videos. The Rick Astley video has 27, the NASA clip has zero, and indexing the empty one raises:

IndexError: list index out of range
Enter fullscreen mode Exit fullscreen mode

One more that is easy to miss: available_languages can repeat a code. The TEDx talk lists en twice inside 28 entries, so wrap it in sorted(set(...)) before you count languages.

Full script 

"""YouTube scraper: video metadata, channel uploads, transcripts.
Tested: Python 3.13.7, requests 2.34.2, 27 July 2026."""
import csv
import sys
import time

import requests

sys.stdout.reconfigure(encoding="utf-8")

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def call(job, timeout=60, tries=3, **params):
    """One GET against the API. Fields come back at the top level, no wrapper.

    Retried, because a long-running batch should not die on one slow response.
    """
    params["api_key"] = API_KEY
    for attempt in range(tries):
        try:
            r = requests.get(f"{BASE}/{job}", params=params, timeout=timeout)
            if r.ok:
                return r.json()
        except requests.exceptions.RequestException:
            if attempt == tries - 1:
                raise
        time.sleep(2)
    r.raise_for_status()


def get_video(url):
    v = call("youtube/video", url=url)
    return {
        "video_id": v["video_id"],
        "title": v["title"],
        "channel": v["channel_name"],
        "views": v["view_count"],
        "likes": v["like_count"],
        "comments": v["comment_count"] or 0,      # null on many videos
        "seconds": v["duration_seconds"],
        "category": v["category"],
        "published": v["publish_date"][:10],
        "keywords": len(v["keywords"]),           # empty list on many videos
    }


def get_channel_videos(channel_url, limit=30):
    c = call("youtube/channel", channel=channel_url)
    rows = [{"position": v["position"], "video_id": v["id"], "title": v["title"],
             "views": v["views"], "published": v["published"], "url": v["url"]}
            for v in c["videos"][:limit]]
    return {"name": c["channel_name"], "channel_id": c["channel_id"],
            "subscribers": c["subscriber_count"], "total_videos": c["video_count"],
            "has_more": c["has_more"]}, rows


def get_transcript(url):
    t = call("youtube/transcript", url=url)
    if not t["transcript_available"]:
        return None
    return {
        "video_id": t["video_id"],
        "language": t["language"],
        "generated": t["is_generated"],
        "segments": t["segment_count"],
        "words": t["word_count"],
        "languages": sorted(set(t["available_languages"])),   # duplicates appear
        "text": " ".join(s["text"] for s in t["segments"]),
    }


def to_csv(rows, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader()
        w.writerows(rows)
    return path


if __name__ == "__main__":
    meta, rows = get_channel_videos("https://www.youtube.com/@NASA")
    print(f"{meta['name']}  {meta['subscribers']:,} subscribers  "
          f"{meta['total_videos']:,} videos  more={meta['has_more']}")
    print(f"pulled {len(rows)} rows from page 1")

    newest = rows[0]
    print(f"newest: {newest['title']}  ({newest['views']})")

    video = get_video(newest["url"])
    print(f"  {video['views']:,} views | {video['likes']:,} likes | "
          f"{video['seconds']}s | {video['category']}")

    tr = get_transcript(newest["url"])
    if tr:
        print(f"  transcript: {tr['segments']} segments, {tr['words']} words, "
              f"{len(tr['languages'])} languages")
        print(f"  {tr['text'][:70]}")

    to_csv(rows, "youtube_channel_videos.csv")
    print("wrote youtube_channel_videos.csv")
Enter fullscreen mode Exit fullscreen mode

The chain is the useful part: one channel call produces 30 URLs, and each URL feeds both the video and transcript endpoints without any ID extraction in between. The CSV it wrote has a header plus 30 rows.

Summary

Three endpoints cover the YouTube jobs most pipelines need, and all three take the URL you already have: the video endpoint returns 26 top-level fields with integer view and like counts, the channel endpoint returns 30 uploads per page with a continuation token for the rest, and the transcript endpoint returns timestamped segments plus a language list. The thing to carry away is that the failures are in the data shape rather than the request, so coalesce comment_count, guard keywords, and never reach for a data wrapper that does not exist. Python is used throughout here, though any language that can send a GET works the same way.

FAQ

Is it legal to scrape YouTube?

Public video metadata and captions sit in a different position from private or logged-in data, so stay on publicly visible pages and check YouTube's terms for your specific use.

How many videos does one channel request return?

Thirty per page, with has_more and a next_page_token telling you whether to request another page.

Why is my view count a string instead of a number?

You are reading view_count_text or the channel list's views field, both of which are display strings, while view_count on the video endpoint is a real integer.

Top comments (0)