Sooner or later, most teams that work with video data hit the same wall. You get access to a large video dataset — millions of rows of video IDs, titles, channel names, languages, upload dates, durations, and engagement counters across platforms like YouTube and TikTok — and the plan is to train something on it. A clickbait-title classifier. A product-spam filter. A content-category tagger. Something that turns rows into a model and justifies the data budget.
Then the annotation quote arrives, and the project dies in a spreadsheet. Even at the cheap end, a single human label costs real money, and multiplying that by a few million rows is not a budget conversation anyone wins. The usual workaround — label "enough" rows quickly with whatever tooling is lying around — tends to mean accepting label quality you cannot defend three months later.
The right response is neither "label everything" nor "give up". It is to decide, deliberately, which few thousand rows deserve a human label, and to design that decision so the resulting sample can still answer the question you actually have. This is sampling design, and in my experience it gets far less attention than the model architecture that comes after it, even though a bad sample quietly poisons everything downstream: training, evaluation, and every number you eventually put in front of stakeholders.
This post is the method I use, with runnable DuckDB-based code, for cutting a large video dataset down to a labeled sample that fits a real budget.
Why the obvious approach fails
The instinctive move is df.sample(n=5000) or ORDER BY random() LIMIT 5000. It feels safe, because random sampling is unbiased. And it is — unbiased for the population. The problem is that engagement in video data follows a power law so extreme that "unbiased" and "useless for your question" can be true at the same time.
Consider view counts. In a large video corpus the median video has modest viewership while the top fraction of a percent has tens of millions of views. If viral videos make up 0.2% of rows, a random 5,000-row sample contains about ten of them. You cannot evaluate how a classifier behaves on viral content from ten examples. Worse, virality usually correlates with the thing you are labeling — viral titles are stylistically different from the rest, spam concentrates in specific engagement bands — so the naive sample gives your model almost no supervision exactly where mistakes are most expensive, and your eval set will report near-zero head behavior with a perfectly straight face.
The same starvation logic applies along every skewed axis in the table. Language distribution is heavy-tailed: a few big languages dominate and dozens of others survive only as residue. Platform mix is skewed. Upload recency is skewed by whenever the collection window happened to run. None of these are exotic properties; they are the normal shape of video metadata. And every one of them is a dimension along which a naive random draw can silently starve you.
The deeper framing: random sampling answers "what does the population look like, on average". But a classifier does not experience the population on average. It experiences specific slices, and it gets judged on its worst slice.
Four designs, one decision
There are four standard ways to allocate a labeling budget, and they answer different questions.
Naive random is the baseline we just dismissed. It preserves the population distribution but gives no control over rare-slice coverage. Fine for demographic-style estimation, wrong for supervision.
Proportional stratification keeps the population shape but samples within predefined strata — platform × engagement decile × language group, say. It is basically random sampling with guardrails: strata you care about cannot go to zero just because they are rare. This is my default for training samples.
Equal allocation gives every stratum the same quota. Coverage is guaranteed, but the sample no longer represents the population — a stratum with 40,000 rows and a stratum with 4 million rows contribute equally. That is exactly what you want for an evaluation set ("how does the model do on each slice?") and exactly what you do not want for a training set, unless you reweight afterwards.
Neyman allocation is the principled middle ground: allocate per stratum in proportion to the stratum's size times the standard deviation of whatever you are estimating inside it. It minimizes total variance for a fixed budget. The catch is that you need a variance estimate per stratum before you have labels — usually a proxy from weak labels, heuristics, or a pilot round. When I have a pilot, I use Neyman. When I don't, proportional-with-floors is the honest default.
The rule of thumb I've settled on: training samples want proportional-with-floors; evaluation samples want the slices you will actually be judged on, allocated roughly equally. Know which artifact you are building before you set a single quota. Teams conflate the two constantly, and the result is a training set that over-represents the head or an eval set that can't see it.
Choosing strata without exploding the grid
Strata are only useful if you can afford them. The natural dimensions for video metadata are platform, engagement decile, language, upload recency, and duration. Cross all of them and the grid detonates: 3 platforms × 10 deciles × 8 language groups × 4 recency buckets × 3 duration buckets is 2,880 cells. With an 8,000-row budget that is under three labels per cell, which is not supervision, it is noise.
My working recipe: pick one backbone dimension that directly tracks the thing you care about — for engagement-related tasks that is the view-count decile, computed within each platform, never globally (a top-decile TikTok video and a top-decile YouTube video live in different view ranges, and a global decile quietly reorders one platform's scale). Use language as a floor dimension: every language group gets a minimum number of rows regardless of what proportional allocation says. Leave recency and duration out of the cross-product; sample them as secondary floors if the task needs them. Any cell that lands below a minimum size gets merged into its parent.
Two more structural guards go in before any sampling happens. First, deduplicate on the platform-video identity, keeping the freshest row if the dataset was collected incrementally. Second, a near-duplicate guard: the same clip reposted by three channels is one piece of content wearing three row-IDs. If two near-duplicates land in your sample, you burn two labels on one video and — worse — you create a leakage path between train and eval splits later. A cheap approximation is an md5 of the normalized title within platform; it will not catch re-edited reposts (embedding similarity would), but it catches the lazy copy-paste majority, which is where most of the damage is.
The code
The script below is self-contained: if it doesn't find a video_metadata.jsonl, it generates a synthetic corpus with a heavy-tailed view distribution so you can run it end-to-end before pointing it at real data. Swap in your own file with the same columns and it works the same. You need pip install duckdb pandas.
One production note before the code: the per-stratum draw interpolates stratum keys into SQL with f-strings, which is fine when the keys come from your own data pipeline and catastrophic when they come from anywhere else. Parameterize the WHERE clauses if the data source is not fully under your control.
"""Stratified sampling for a video metadata dataset, before you spend labeling budget.
Reads video_metadata.jsonl (or generates a synthetic corpus so the script runs
end-to-end anywhere), deduplicates, builds per-platform engagement strata,
allocates quotas (proportional with floors), draws a reproducible sample with
population weights, and verifies coverage.
Requires: pip install duckdb (tested with duckdb 1.x, Python 3.10+)
"""
import json
import os
import random
import datetime as dt
import duckdb
import pandas as pd
DATA_FILE = "video_metadata.jsonl"
SAMPLE_FILE = "sample.csv"
MANIFEST_FILE = "sample_manifest.json"
BUDGET = 8000 # total rows you can afford to label
MIN_CELL = 30 # floor per stratum so rare slices still get supervision
SEED = "42" # deterministic draw key
# ----------------------------------------------------------------------------
# 0. Corpus: real file if you have one, synthetic otherwise
# ----------------------------------------------------------------------------
SYNTH_N = 250_000
def make_synthetic(path: str, n: int = SYNTH_N) -> None:
"""Power-law views, skewed platforms/languages, 24-month upload window."""
rng = random.Random(7)
platforms = ["youtube"] * 6 + ["tiktok"] * 3 + ["shorts"]
langs = ["en"] * 5 + ["es"] * 2 + ["pt", "id", "other"]
base = dt.date(2026, 9, 1)
with open(path, "w", encoding="utf-8") as f:
for i in range(n):
views = int(500 * (rng.paretovariate(1.16) ** 2)) # heavy tail
upload = base - dt.timedelta(days=rng.randint(0, 730))
f.write(json.dumps({
"platform": rng.choice(platforms),
"video_id": f"v{i:07d}",
"title": f"sample title {i}",
"language": rng.choice(langs),
"upload_date": upload.isoformat(),
"duration_s": rng.randint(15, 3600),
"views": views,
"likes": int(views * rng.uniform(0.01, 0.08)),
"comments": int(views * rng.uniform(0.001, 0.01)),
"collection_ts": "2026-09-20T00:00:00Z",
}) + "\n")
if not os.path.exists(DATA_FILE):
make_synthetic(DATA_FILE)
print(f"[setup] no {DATA_FILE} found -> generated synthetic corpus ({SYNTH_N} rows)")
# ----------------------------------------------------------------------------
# 1. Load, clean, and guard against duplicates / near-duplicates
# ----------------------------------------------------------------------------
con = duckdb.connect()
con.execute(f"""
CREATE TABLE raw AS
SELECT * FROM read_json_auto('{DATA_FILE}')
""")
con.execute("""
CREATE TABLE clean AS
SELECT * FROM (
SELECT
*,
-- keep the freshest row per (platform, video_id)
row_number() OVER (
PARTITION BY platform, video_id
ORDER BY collection_ts DESC
) AS dedup_rn
FROM raw
WHERE video_id IS NOT NULL
) WHERE dedup_rn = 1
""")
con.execute("""
CREATE TABLE guarded AS
SELECT * FROM (
SELECT
*,
-- cheap near-duplicate guard: same normalized title on the same
-- platform is treated as one content unit; keep one representative
md5(platform || '|' ||
regexp_replace(lower(trim(title)), '[^a-z0-9 ]', '', 'g')) AS dup_group,
row_number() OVER (
PARTITION BY md5(platform || '|' ||
regexp_replace(lower(trim(title)), '[^a-z0-9 ]', '', 'g'))
ORDER BY views DESC
) AS dup_rn
FROM clean
) WHERE dup_rn = 1
""")
# ----------------------------------------------------------------------------
# 2. Strata: engagement decile computed WITHIN each platform
# ----------------------------------------------------------------------------
con.execute("""
CREATE TABLE strata AS
SELECT
*,
NTILE(10) OVER (PARTITION BY platform ORDER BY views) AS view_decile
FROM guarded
""")
con.execute("""
CREATE TABLE stratum_counts AS
SELECT platform, view_decile, count(*) AS n
FROM strata
GROUP BY platform, view_decile
""")
strata = con.execute("""
SELECT platform, view_decile, n FROM stratum_counts ORDER BY n DESC
""").fetchall()
total = sum(s[2] for s in strata)
# ----------------------------------------------------------------------------
# 3. Quotas: proportional, then floored, then rescaled back to budget
# ----------------------------------------------------------------------------
quotas = {s[:2]: max(MIN_CELL, round(BUDGET * s[2] / total)) for s in strata}
over = sum(quotas.values()) - BUDGET
if over > 0: # shrink only cells that are above the floor
shrinkable = {k: v - MIN_CELL for k, v in quotas.items() if v > MIN_CELL}
shrink_total = sum(shrinkable.values())
for k, excess in shrinkable.items():
quotas[k] -= round(over * excess / shrink_total)
# ----------------------------------------------------------------------------
# 4. Draw: deterministic hash-order sampling, one pass per stratum
# ----------------------------------------------------------------------------
pops = {(p, d): n for p, d, n in strata}
sample_parts = []
for (platform, decile), quota in quotas.items():
pop = pops[(platform, decile)]
rows = con.execute(f"""
SELECT * EXCLUDE (dedup_rn, dup_rn, dup_group),
{quota} AS quota,
{pop} AS stratum_pop
FROM (
SELECT * FROM strata
WHERE platform = '{platform}' AND view_decile = {decile}
ORDER BY md5(video_id || '{SEED}')
LIMIT {quota}
)
""").fetchdf()
sample_parts.append(rows)
sample = pd.concat(sample_parts, ignore_index=True)
sample["weight"] = sample["stratum_pop"] / sample["quota"] # N_s / n_s
sample.to_csv(SAMPLE_FILE, index=False)
# ----------------------------------------------------------------------------
# 5. Verify: coverage + what a naive random draw would have looked like
# ----------------------------------------------------------------------------
cov = con.execute(f"""
SELECT s.platform, s.view_decile, count(*) AS sampled, min({MIN_CELL}) AS floor
FROM read_csv_auto('{SAMPLE_FILE}') s
GROUP BY 1, 2
ORDER BY sampled ASC LIMIT 5
""").fetchall()
head10_share = con.execute(
"SELECT sum(CASE WHEN view_decile = 10 THEN 1 ELSE 0 END)::DOUBLE / count(*) FROM strata"
).fetchone()[0]
print(f"population rows (post-guard): {total:,}")
print(f"strata: {len(strata)} budget: {BUDGET:,} sampled: {len(sample):,}")
print(f"head-decile share of population: {head10_share:.3%}")
print(f"head-decile rows in sample: {int((sample['view_decile'] == 10).sum())}")
print(f"head-decile rows a naive random {BUDGET}-draw would give: "
f"~{round(BUDGET * head10_share)}")
print("tightest strata (sampled / floor):", cov)
print(f"weight range: {sample['weight'].min():.1f} .. {sample['weight'].max():.1f}")
manifest = {
"generated_at": dt.datetime.now(dt.timezone.utc).isoformat(),
"corpus_file": os.path.abspath(DATA_FILE),
"corpus_sha_prefix": None, # fill with your checksum in production
"budget": BUDGET, "min_cell": MIN_CELL, "seed": SEED,
"strata_definition": "platform x view_decile(10, within platform)",
"allocation": "proportional with floor {floor}, rescaled to budget".format(floor=MIN_CELL),
"sampled": len(sample),
}
with open(MANIFEST_FILE, "w", encoding="utf-8") as f:
json.dump(manifest, f, indent=2)
print(f"wrote {SAMPLE_FILE} + {MANIFEST_FILE}")
The draw itself uses ORDER BY md5(video_id || seed) instead of random(). That is deliberate: a hash-ordered draw is deterministic (same corpus, same seed, same sample), which makes the whole pipeline reproducible and lets a second labeling round exclude exactly what was already labeled by re-running with an exclusion list. random() gives you none of that for free.
The weight column — stratum population divided by sampled count — is the piece teams most often throw away and later need most. More on that below.
What the verification output is telling you
The final block prints three things worth actually reading. The head-decile share versus the naive-random expectation quantifies the entire point of the exercise: if the top decile is a small share of the population, a random draw of your budget gives you a handful of head rows, while the stratified sample gives you the full quota of the backbone strata. That one printed comparison is what I show people who ask why the sampling step "is so complicated".
The tightest-strata listing shows which cells ended up pinned at the floor. Cells at the floor are where your sample is least proportional — the weights there are largest, and any population-level estimate derived from the sample leans on those weights hardest. If half your strata are pinned at the floor, the grid is too fine for the budget: collapse cells (fewer deciles, coarser language grouping) rather than pretending the floors aren't distorting things.
The weight range is the honest summary of how far your sample has wandered from the population. A narrow range means floors barely fired and the sample is nearly proportional. A wide range means you built a deliberately unrepresentative sample — which is legitimate for training, but it obligates you to use the weights, because your raw sample mean is now the mean of your sampling design, not of the market.
The pits, from the bottom up
Forgetting the weights. Train a model on a stratified sample without sample weights (or without matching the deployment distribution at inference time) and the model inherits your quota sheet as a prior. The fix is boring: pass weight into your loss, or resample proportionally to weights before training. Either way the weight column has to survive into the training pipeline, not die in the sampling script.
Treating engagement counters as current facts. View and like columns in a purchased or scraped dataset are "as of collection date" values. They drift — sometimes violently for videos in their first weeks — between when the snapshot was taken and when your labels arrive. Treat them as snapshot features, version your dataset, and don't let anyone downstream describe them as live numbers.
Sampling the head because the head is exciting. Viral rows are fun to look at and dangerous to over-collect. The strata should encode where the model will actually be deployed, not which rows make good demo screenshots in the readout meeting. If your product only ever sees mid-tail content, a head-heavy sample is a very organized way to build the wrong model.
Paying for the same row twice. Second labeling rounds are common and the failure is silent: without a durable list of previously labeled video IDs and an exclusion filter in the draw, you re-purchase labels you already own and your "new round" is smaller than the invoice says.
Sampling comments the same way you sample videos. Comment datasets are a different discipline — a large share of raw comments on popular videos is spam, bots, or copy-paste, and sampling before filtering means your sentiment annotation budget is spent mostly on noise. Filter first, sample second; the strata logic then applies, but on a population that deserves the labels.
None of this is glamorous. The whole deliverable is a CSV and a manifest. But every labeled row you don't waste is budget you keep, and the difference between a designed sample and a random one is usually the difference between an eval report that survives contact with stakeholders and one that gets quietly rerun.
If you're starting from zero data rather than a purchased corpus, the same design logic applies — you'd first assemble the raw video metadata at scale (Thordata's multi-platform video datasets are one starting point, and they offer a free sample for evaluation before payment, which I'd suggest exercising exactly the way this post suggests: audit the skew before you commit). More details on that line of data products here: https://www.thordata.com/products/multi-platform-video-datasets?ls=dev&lk=DEV
Disclosure: I work with Thordata on developer content, so treat the link above accordingly; the sampling method in this post is vendor-agnostic and works on any video metadata table with the columns shown.
Top comments (0)