Build a YouTube comment dataset for training or evaluating a text classifier, without leaking test data

YouTube comments are a good source of short, informal, real-world text for spam, toxicity, intent or sentiment classifiers. They are also an easy way to build a dataset that looks great in evaluation and fails in production: one viral video dominates it, the same copy-pasted comment ends up in train and test, and a model learns the video instead of the label. This post builds the dataset carefully: sampling, pseudonymised authors, de-duplication, a split by video, and a check that your labels mean something.

Decide what the dataset is for

Write down the task and the population before collecting anything:

• The label set, with one example and one counter-example for each label. "Spam: promotes a link or channel unrelated to the video" is a definition; "spam: bad comments" is not.
• Where the classifier will run. Comments on your own channel, comments in one language, comments on gaming videos? The dataset should look like that, not like YouTube in general.
• The test set's job. If the model has to work on channels it has never seen, the test set must come from channels the training set does not include.

Sampling: many videos, a small cap each

Comment counts are extremely skewed. One video can have more comments than a hundred others together, so "collect everything" gives you a dataset about that video. Instead:

• Many videos, a fixed cap per video. For example 100-200 comments from each of a few hundred videos across many channels.
• Mix the ordering. sortBy: "relevance" (YouTube's "Top comments") over-represents popular, well-written comments. sortBy: "time" gives you the recent, unfiltered stream, including the spam and low-effort comments a spam classifier needs to see. Collect both and record which one each row came from.
• Decide about replies. Replies are conversational and often meaningless without their parent. Leave includeReplies off unless the task is about threads, and remember that replies count toward the per-video cap.

Best Damn YouTube Comments Scraper reads comments through the official YouTube Data API v3. One run per ordering:

{
    "startUrls": ["https://www.youtube.com/watch?v=VIDEO_ID_1", "https://www.youtube.com/@SomeChannel"],
    "maxVideosPerChannel": 10,
    "sortBy": "time",
    "maxComments": 150,
    "authorData": "hash"
}

To get a few hundred videos across channels in your domain, use Best Damn YouTube Scraper in search mode, or list the channels you care about.

Pseudonymise authors from the start

authorData: "hash" replaces each commenter's channel ID with a 16-character hash and drops the name, channel URL and avatar. The hash is stable across your runs, so you can still keep one person's comments on one side of the split and cap how many comments one prolific person contributes. It is not reversible on its own, but anyone who knows a candidate channel ID and the salt could test for a match, so treat it as pseudonymous, not anonymous. With "omit" all author fields are removed.

Commenter names are useless as features and risky as data. A model trained on names learns who someone is, not what they wrote. Don't collect what you don't need.

Clean, de-duplicate and split

The script below combines both runs, drops failures and the creator's own comments, removes exact and normalised duplicates, caps comments per author, and assigns each video to train, validation or test with a hash of its videoId. Every comment of a video lands in the same split, so near-identical comments in one thread can't leak across.

import hashlib, json, re
from collections import Counter
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
VIDEOS = ["https://www.youtube.com/watch?v=VIDEO_ID_1", "https://www.youtube.com/@SomeChannel"]
MAX_PER_AUTHOR = 5

rows, seen, per_author = [], set(), Counter()
for order in ("relevance", "time"):
    run = client.actor("josh99smith/youtube-comments-scraper").call(run_input={
        "startUrls": VIDEOS, "maxVideosPerChannel": 10, "sortBy": order,
        "maxComments": 150, "authorData": "hash"})
    for c in client.dataset(run["defaultDatasetId"]).iterate_items():
        if not c["success"] or c["isByVideoOwner"]:
            continue
        norm = re.sub(r"\s+", " ", re.sub(r"https?://\S+", "<url>", c["text"].lower())).strip()
        if len(norm) < 3 or c["commentId"] in seen or norm in seen:
            continue
        if per_author[c["authorChannelId"]] >= MAX_PER_AUTHOR:
            continue
        seen.update([c["commentId"], norm])
        per_author[c["authorChannelId"]] += 1
        rows.append({
            "id": c["commentId"], "video_id": c["videoId"], "channel_id": c["videoChannelId"],
            "author": c["authorChannelId"], "text": c["text"], "likes": c["likeCount"],
            "is_reply": c["isReply"], "published_at": c["publishedAt"], "order": order, "label": None,
        })

def split_for(key, val=0.1, test=0.1):
    bucket = int(hashlib.sha256(key.encode()).hexdigest(), 16) % 1000 / 1000
    return "test" if bucket < test else "val" if bucket < test + val else "train"

for r in rows:
    r["split"] = split_for(r["video_id"])  # use r["channel_id"] to test on unseen channels

with open("comments.jsonl", "w", encoding="utf-8") as f:
    for r in rows:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")
print(len(rows), Counter(r["split"] for r in rows), Counter(r["order"] for r in rows))

Choices worth explaining:

• Duplicates are matched on normalised text, with links replaced by <url>, so the same spam posted with different links counts once. Duplicates you remove are useful in their own right: a text that appears under many videos is almost always spam, which makes it a cheap, high-precision label.
• The author cap stops one person's 200 comments from teaching the model their style.
• Splitting by hash is deterministic: rerun the script with more videos and the old videos stay in the split they were in, so your test set does not quietly change.
• Splitting by channel is the stricter option. Comments on one channel share vocabulary, in-jokes and spam campaigns, so a by-video split can still be optimistic.
• Emoji, casing and typos are kept in text. They carry signal for tone and spam, and a model in production will see them.

Label, then check the labels

Label a few hundred rows before labelling thousands. Have two people label the same 200 rows independently, then measure agreement. Cohen's kappa corrects for agreement by chance:

def cohen_kappa(a, b):
    labels = set(a) | set(b)
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    expected = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)
    return (observed - expected) / (1 - expected) if expected < 1 else 1.0

Low agreement means the label definitions are ambiguous, not that the labellers are careless. Read the disagreements, rewrite the definitions, and label again. If two people can't agree on whether a comment is sarcastic, a model trained on one person's labels will learn that person, and its test accuracy measures agreement with them.

For an LLM-labelled dataset the same check applies: compare the model's labels with a human-labelled sample before trusting them.

Cost and limits

• Price. $0.0004 per delivered comment ($0.40 per 1,000). 300 videos at 150 comments in two orderings is at most 90,000 comments, so at most $36, and usually less: many videos have fewer comments than the cap, and the two orderings overlap. Failures, comments-off videos and quota errors are free, and a run stops at the maximum cost you set.
• Use your own API key for collection at this size. The Actor's built-in key has a 10,000-unit daily quota shared by every user, and 90,000 comments in pages of up to 100 per video takes well over 1,000 units. Create a free key in the Google Cloud Console, pass it as apiKey, and you get your own 10,000 units a day. When a quota runs out, the run stops cleanly and the remaining videos come back as free rate-limited records.
• Public comments only. Held, removed and members-only comments are not returned, so the dataset under-represents the worst content. A moderation classifier trained on it will see more extreme spam in production than in training.

Rules that apply to the data

Comments are personal data under GDPR and similar laws, even pseudonymised, so you need a legitimate purpose and a retention plan. The comments' authors own their text. YouTube's API Services Developer Policies also set rules on storing and using API data, including how long it may be kept. Read them, and get advice if the dataset is going to be published or used commercially. Keep authorData on hash or omit, don't publish raw comment dumps, and delete what you no longer need.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/youtube-comments-scraper. YouTube is a trademark of Google; the Actor is not affiliated with Google.