Reading a 40,000-comment YouTube thread with the Data API

Most tutorials that show you how to read YouTube comments stop at the
first page. Here is what actually happens when you try to read a thread
that has 40,000 of them.

The endpoint, and what it actually costs

Comments live behind commentThreads.list on the YouTube Data API v3:

GET https://www.googleapis.com/youtube/v3/commentThreads
    ?part=snippet,replies
    &videoId=VIDEO_ID
    &maxResults=100
    &pageToken=NEXT_PAGE
    &key=YOUR_API_KEY

Two things bite people here.

It is quota-metered, not rate-limited. commentThreads.list costs 1 unit
per call against a default 10,000-unit daily quota. At maxResults=100 that
is 100 calls — and 100 calls is 1% of your day. A busy video with 40,000
comments needs 400 calls. You will hit the ceiling faster than you expect,
and the error is a plain 403 quotaExceeded, not a retryable 429.

maxResults is capped at 100 and the cap is silent. Ask for 500 and you
get 100 with no warning. Loop until a page returns fewer items than you asked
for, or you will spin forever on a video whose comment count is a multiple
of 100.

Pagination that actually terminates

The naive loop breaks in three places: the last page, a video with comments
disabled, and a video that is age-restricted.

import requests

def read_thread(video_id: str, api_key: str):
    url = "https://www.googleapis.com/youtube/v3/commentThreads"
    token, seen = None, 0

    while True:
        params = {
            "part": "snippet,replies",
            "videoId": video_id,
            "maxResults": 100,
            "key": api_key,
        }
        if token:
            params["pageToken"] = token

        resp = requests.get(url, params=params, timeout=15)

        # 403 covers both "comments disabled" and "quota exhausted".
        # Read the reason, do not just retry.
        if resp.status_code == 403:
            reason = resp.json()["error"]["errors"][0]["reason"]
            if reason == "commentsDisabled":
                return seen
            raise RuntimeError(f"quota or permission problem: {reason}")

        payload = resp.json()
        items = payload.get("items", [])
        seen += len(items)

        token = payload.get("nextPageToken")
        if not token:
            return seen

The nextPageToken is absent on the final page — that is your exit
condition. Do not compare against pageInfo.totalResults; it is an estimate
and it drifts on large threads.

Replies are a second request

commentThreads.list returns the top-level comment plus, by default, a
single reply. Getting the rest means calling comments.list with
parentId. On a thread with heavy back-and-forth this doubles or triples
your quota spend, and there is no way to know in advance how much.

If you only need the top-level conversation, do not request replies at all —
you get roughly 3x the coverage for the same quota.

When you do not want to write any of this

I built Apriocity because I kept writing this same
pagination loop for one-off research tasks. It does the paging, the quota
accounting and the reply expansion for you, and hands back CSV or JSON:

• paste a video link, read the whole thread, search it by keyword
• export the slice as CSV or JSON
• batch mode: up to ten videos, one ZIP back

It uses the same public API described above, so what you see is exactly
what is publicly visible — nothing is stored on the server side.

The short version

• commentThreads.list = 1 quota unit per call, maxResults capped at 100
• nextPageToken missing = done; do not trust totalResults
• 403 is overloaded: check the reason field before retrying
• Replies cost extra calls — skip them unless you need them

If you are doing this once, the API is fine. If you are doing it weekly,
that loop is worth automating away.