A view count is not a fact. That sentence took me embarrassingly long to actually believe, because for the first year I treated scraped video metrics like table rows: pull them once, store them once, join them once, done. It worked until the week I had to explain to a stakeholder why the "views" column in our dashboard disagreed with the numbers they could see on their phone by 40 percent. The scraper had not broken. The data had simply aged. Every engagement metric on a video is a measurement of a moving target, and the moment you capture it, it starts going stale.
This article is about what I learned rebuilding that pipeline around deltas instead of snapshots. It is written for anyone building or buying video datasets - YouTube, TikTok, Shorts, anything with counters that move - because the schema conversations (which fields exist, how platforms name engagement) get all the attention, while the harder question, "what does one row of this data actually mean in time?", usually gets skipped until production punishes you for it.
The problem nobody puts in the requirements
Here is the shape of the failure. You collect a batch of ten thousand videos with their view counts on Monday. On Thursday, someone asks how the batch is trending. You collect the same ten thousand again. Now you have two snapshots, and the naive move is to join them on video ID and subtract. Then reality shows up in three flavors at once.
First, moving counters. A video that got pushed by the platform's recommendation system might multiply its views fivefold in two days. A video that got suppressed might barely move. Between your two snapshots, both happened, and a single before/after difference hides them completely.
Second, metric resets and recounts. Platforms recount views, and the counts can go down. A video's like count can drop when a bot purge removes fake engagement, or when the creator deletes comments. If your delta logic assumes monotonic growth, a negative difference either crashes your pipeline or - worse - silently poisons whatever trend model eats it downstream.
Third, disappearing rows. Videos get deleted, set to private, made region-restricted, or removed for policy reasons. If your second snapshot is missing two hundred video IDs that existed on Monday, is that a collection bug, a CAPTCHA page you accidentally captured, or real deletion? If you cannot answer that with confidence, you cannot compute a churn rate either, and churn is often the most interesting signal in the whole dataset.
I have seen all three of these get mislabeled as "scraper quality issues". They are not. A perfect scraper that hits the same URL twice will still observe all three, because the target itself changed. The fix is not better scraping. The fix is designing the data model around the fact that each collection run is a timestamped observation, not a statement about the world.
Snapshots, deltas, and the join that lies
The mental model that finally worked for me: a video metrics dataset is not a table of facts, it is a time series sampled at irregular intervals. Each row is an observation event: "at time T, collector C observed video V with metrics M". Once you accept that, several design decisions fall out naturally.
The primary key of your fact table is not video ID. It is the pair (video ID, capture run). If you store metrics keyed by video ID alone and overwrite in place, you have thrown away the only thing that makes the data valuable - the trajectory - and kept the least valuable thing, a stale number.
Deltas are computed between consecutive observations of the same video, and every delta carries metadata: which two runs produced it, how much wall-clock time elapsed, and whether the video was observed in both runs. A "delta" for a video missing from the second run is not zero and not NULL-as-in-error; it is a churn observation, and it belongs in your dataset as its own record with a reason code.
Velocity beats magnitude. A video with 50,000 new views over three days is unremarkable. A video with 50,000 new views in the first six hours is an outlier worth flagging. Since your capture runs are irregular, express growth as views per hour between the two observations, not raw differences. Every trend dashboard I have built from this kind of data eventually converged on velocity windows.
Ratios need history too. Like-to-view ratios are a useful health check - they are stable within a band for most organic content, and a sudden shift often means deleted engagement, a changed audience, or a data join that silently mixed two different videos. But you cannot evaluate today's ratio without yesterday's distribution, which is one more argument for never overwriting snapshots.
A runnable delta harness
Here is the harness I wish I had written on day one. It is deliberately boring: standard library plus sqlite3, no dependencies, so you can run it against any source of snapshot data - your own collection runs, exports from a dataset vendor, even two hand-pulled CSVs - and get a defensible trend layer.
# snapshot_delta.py
# Metric snapshot store + delta computation for video engagement data.
# stdlib only. Python 3.9+.
import sqlite3, csv, sys
from datetime import datetime, timezone
SCHEMA = """
CREATE TABLE IF NOT EXISTS observations (
obs_id INTEGER PRIMARY KEY AUTOINCREMENT,
video_id TEXT NOT NULL,
platform TEXT NOT NULL,
run_id TEXT NOT NULL, -- one collection batch
captured_at TEXT NOT NULL, -- ISO8601 UTC
view_count INTEGER,
like_count INTEGER,
comment_count INTEGER,
UNIQUE (video_id, run_id)
);
CREATE INDEX IF NOT EXISTS idx_obs_video_time
ON observations (video_id, captured_at);
"""
def ingest(conn, rows):
"""rows: iterable of dicts with the SCHEMA column names."""
conn.executemany(
"""INSERT OR IGNORE INTO observations
(video_id, platform, run_id, captured_at,
view_count, like_count, comment_count)
VALUES (:video_id, :platform, :run_id, :captured_at,
:view_count, :like_count, :comment_count)""",
rows)
conn.commit()
def _parse(ts):
return datetime.fromisoformat(ts.replace("Z", "+00:00"))
def deltas(conn, run_new, run_prev=None):
"""
Compute per-video deltas between run_new and the most recent
earlier run (or an explicit run_prev). Returns one row per video:
velocity, negative-delta flags, and churn (missing) observations.
"""
cur = conn.cursor()
if run_prev is None:
cur.execute("""SELECT DISTINCT run_id FROM observations
WHERE run_id <> ?
ORDER BY (SELECT MIN(captured_at)
FROM observations o2
WHERE o2.run_id = observations.run_id)
DESC LIMIT 1""", (run_new,))
row = cur.fetchone()
if not row:
return []
run_prev = row[0]
cur.execute("""SELECT video_id, platform, captured_at,
view_count, like_count, comment_count
FROM observations WHERE run_id = ?""", (run_prev,))
prev = {r[0]: r for r in cur.fetchall()}
cur.execute("""SELECT video_id, platform, captured_at,
view_count, like_count, comment_count
FROM observations WHERE run_id = ?""", (run_new,))
new = {r[0]: r for r in cur.fetchall()}
out = []
for vid, n in new.items():
p = prev.get(vid)
if p is None:
out.append(dict(video_id=vid, run_prev=run_prev, run_new=run_new,
status="new_in_run", d_views=None,
views_per_hour=None, flag=""))
continue
hours = max((_parse(n[2]) - _parse(p[2])).total_seconds() / 3600.0, 1e-9)
d_views = (n[3] or 0) - (p[3] or 0)
d_likes = (n[4] or 0) - (p[4] or 0)
flags = []
if d_views < 0:
flags.append("view_recount_down") # platform recount / purge
if d_likes < 0:
flags.append("like_recount_down") # deleted engagement
vph = d_views / hours
if vph > 0 and (n[3] or 0) > 0 and vph > 0.05 * (n[3] or 0) / max(hours, 1):
flags.append("viral_velocity") # heuristic, tune per niche
out.append(dict(video_id=vid, run_prev=run_prev, run_new=run_new,
status="ok", d_views=d_views,
views_per_hour=round(vph, 2),
flag=";".join(flags)))
for vid, p in prev.items():
if vid not in new:
out.append(dict(video_id=vid, run_prev=run_prev, run_new=run_new,
status="missing_in_new_run", d_views=None,
views_per_hour=None, flag="possible_churn"))
return out
if __name__ == "__main__":
conn = sqlite3.connect(":memory:")
conn.executescript(SCHEMA)
# demo: three videos observed in two runs, four hours apart
run_a = [
dict(video_id="v1", platform="yt", run_id="r1",
captured_at="2026-09-22T09:00:00Z", view_count=1000,
like_count=80, comment_count=10),
dict(video_id="v2", platform="yt", run_id="r1",
captured_at="2026-09-22T09:00:05Z", view_count=50000,
like_count=3000, comment_count=200),
dict(video_id="v3", platform="yt", run_id="r1",
captured_at="2026-09-22T09:00:10Z", view_count=200,
like_count=25, comment_count=2),
]
run_b = [
dict(video_id="v1", platform="yt", run_id="r2",
captured_at="2026-09-22T13:00:00Z", view_count=1100,
like_count=83, comment_count=11),
dict(video_id="v2", platform="yt", run_id="r2",
captured_at="2026-09-22T13:00:05Z", view_count=48000,
like_count=2900, comment_count=199), # recount down
# v3 missing: churn candidate
]
ingest(conn, run_a); ingest(conn, run_b)
w = csv.DictWriter(sys.stdout, fieldnames=[
"video_id", "run_prev", "run_new", "status",
"d_views", "views_per_hour", "flag"])
w.writeheader()
for row in deltas(conn, "r2"):
w.writerow(row)
Run it and you get exactly the report shape I ended up building dashboards on: v1 grew 100 views over four hours and looks organic; v2 lost 2,000 views and gets tagged view_recount_down so nobody downstream mistakes it for collection noise; v3 is absent from the second run and surfaces as possible_churn instead of silently vanishing in an inner join. That last line is the one that saved my team the most arguments - a LEFT-join-and-coalesce dashboard would have shown v3's "views dropping to zero", which is a completely wrong story about a video that was simply removed.
Traps worth more than the code
A few things I got wrong before this settled down, in the hope you skip them.
Sampling clock matters as much as sample size. Two snapshots taken at 9:00 and 9:15 tell you almost nothing except collector jitter; engagement counters update on the platform's schedule, not yours. For trend work I now want capture runs spaced in hours, not minutes, unless I am explicitly studying near-real-time velocity - and then I accept the noise floor that comes with it.
Timestamp the observation, not the pipeline. If you stamp rows with ingestion time but the collection run itself took forty minutes, every cross-video comparison is skewed by up to forty minutes. Stamp at capture time per item, and also store the run ID so you can always reconstruct batch boundaries.
Negative deltas are data, not defects. The instinct is to filter them out as errors. Do not. Recount drops and deleted-engagement events are some of the most informative rows in the dataset, especially if you are studying platform moderation or audience quality. Route them to a flag, keep the row.
Churn needs a disposition, not an absence. A missing video in the newer run is only "deleted" if your collector is healthy - if that run was half-blocked or hit rate limits, your churn rate is a measurement of your infrastructure. This is why I never compute churn from a single run pair without checking run health first (coverage against the expected ID set, error class distribution, count of obviously-degraded pages). If a run looks sick, mark the whole run's deltas as low-confidence instead of shipping them.
Buying versus collecting, honestly. When I need a broad baseline - say, weeks of history across a niche - a pre-collected dataset is often the sane starting point; Thordata's video datasets line, for instance, advertises 6B original videos from 700M unique channels with 100+ domain-specific datasets, and dataset pricing starts from $0.25 per 1K records as shown on their pricing page (both checked live on 2026-09-22 - prices move, so re-verify before you budget). But a static dataset is, by construction, a set of old snapshots. The moment your product needs trends, you are back to collecting your own deltas, on a schedule - which is exactly the capability the scheduled-batch side of a video data scraper exists for. My honest recommendation is to treat purchased datasets as your historical backfill and your own scheduled collection as the live edge, and design the schema so both produce the same observation rows. That way the delta harness above does not care where a row came from, and neither does your dashboard.
One disclosure
I work with Thordata on content, so read the vendor paragraph above with that in mind - the delta-design advice stands on its own, and the collection method matters far less than capturing time honestly. The harness above is the whole idea in a hundred lines; everything else is discipline about what a snapshot is allowed to claim.
Top comments (0)