My YouTube analytics script ran for three months and reported two tables every day: Top 5 videos by views-per-day and Bottom 5. I used those tables to decide which video archetypes to produce the next day. It seemed reasonable — high performers tell you what's working, low performers tell you what isn't.
The problem, which I finally wrote down explicitly during a directive audit this week, is that Top5/Bottom5 is exactly the wrong sample for comparing archetypes. You're always selecting on the outcome. That means you can't compute a real median for any archetype, can't tell whether a typical video of type A outperforms a typical video of type B, and can't run any form of hook A/B experiment. All you can say is which videos happen to be at the distribution's edges on a given day.
The original report and why it felt informative
The daily cron fetched all video stats from the YouTube Data API v3, computed views-per-day for each video older than 24 hours, sorted by that rate, and wrote the top 5 and bottom 5 into a markdown report in docs/yt-analytics/. The script then read those tables to build a "prefer X, avoid Y" bias hint for the next day's video generation directive.
This caught obvious failures. The build_in_public archetype really did crater — I could see it collapsing from a median around 34 views-per-day to around 8 over about three weeks. But obvious failures are easy. The harder question is: among archetypes that aren't catastrophically bad, which one has a better median? That's exactly what Top5/Bottom5 cannot answer.
Here's why. If product_findindiegame is the dominant archetype — which it is, the directive has favored it for weeks — then it will appear more often in both tails just because there are more of them. A Top5 table with three findindiegame entries and two unknowns tells me nothing about whether findindiegame is better than, say, ossfind. I need the middle of the distribution, not the edges.
The tables are also silent about new archetypes with only 2-3 videos. Those videos rarely hit Top5 or Bottom5 because neither cluster is large enough to dominate. I could run a new archetype for a month and have no data on it at all from this report.
What a valid comparison requires
To compare archetypes fairly, I need three things:
| Approach | What you see | What you miss | Good for |
|---|---|---|---|
| Top5/Bottom5 daily | Extreme performers | Median distribution | Catching catastrophic failures |
| views/day, all videos | Current rate, full fleet | Age confound — a 2-day Short looks worse than a 20-day one | Quick daily snapshot |
| Full-fleet JSONL + day-7 | All videos at age ~7 days | Nothing — full population | Archetype A/B, hook arm tests |
The age problem matters: YouTube Shorts plateau fast. A fresh video has a high views-per-day because its 24-hour velocity is being divided by 1 or 2 days. A 30-day-old video has a low rate even if it accumulated more total views. Comparing views-per-day across videos of different ages is structurally biased toward new content.
Age-controlled: observe every video at the same point in its lifecycle. Day 7 is a good choice because Shorts on this channel reach roughly 99% of their lifetime views by that point — the day-7 number is a close proxy for total views, without the age confound.
To get day-7 observations, I need a snapshot of every video's view count taken seven days after it was published — which means logging all videos daily so I can find the snapshot closest to day 7 in post.
The implementation
The new append_history() function in scripts/yt-analytics/run.py does this logging:
def append_history(stats_videos: list[dict], history_path: Path = HISTORY_PATH) -> int:
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
seen_today: set[str] = set()
if history_path.exists():
for line in history_path.read_text().splitlines():
try:
d = json.loads(line)
except ValueError:
continue
if d.get("date") == today:
seen_today.add(d.get("video_id"))
rows = []
for v in stats_videos:
vid = v.get("id")
if not vid or vid in seen_today:
continue
s = v.get("statistics") or {}
snippet = v.get("snippet") or {}
rows.append(json.dumps({
"date": today,
"video_id": vid,
"title": snippet.get("title", "")[:120],
"published_at": snippet.get("publishedAt", ""),
"archetype": v.get("_archetype", "unknown"),
"hook_arm": v.get("_hook_arm", ""),
"views": _stat_int(s, "viewCount"),
"likes": _stat_int(s, "likeCount"),
"comments": _stat_int(s, "commentCount"),
}, ensure_ascii=False))
if rows:
history_path.parent.mkdir(parents=True, exist_ok=True)
with history_path.open("a") as f:
f.write("\n".join(rows) + "\n")
return len(rows)
Idempotent per (date, video_id) — if the cron runs twice the same day, the second run writes nothing. One pass over the file to build seen_today, then append. No extra API quota: the full channel fetch was already happening; I just stopped discarding the non-tail rows.
The day7_by_archetype() function reads the history and finds each video's closest snapshot in the 6-to-8-day window:
def day7_by_archetype(history_path: Path = HISTORY_PATH):
best: dict[str, tuple[float, int, str]] = {} # vid -> (dist_from_7, views, archetype)
for line in history_path.read_text().splitlines():
d = json.loads(line)
pub = (d.get("published_at") or "")[:10]
date = d.get("date") or ""
age = (datetime.fromisoformat(date) - datetime.fromisoformat(pub)).days
if not 6 <= age <= 8:
continue
dist = abs(age - 7)
cur = best.get(d["video_id"])
if cur is None or dist < cur[0]:
best[d["video_id"]] = (dist, d.get("views", 0), d.get("archetype", "unknown"))
by_arch: dict[str, list[int]] = defaultdict(list)
for _dist, views, arch in best.values():
by_arch[arch].append(views)
return sorted(
((a, statistics.median(vs), len(vs)) for a, vs in by_arch.items()),
key=lambda x: x[1], reverse=True,
), len(best)
The 6-8 day window matters because the cron doesn't run at a precise offset from video publish time. A video might get its first post-launch observation at age 6.2 days or 7.8 days depending on publish time vs cron schedule. Taking the closest observation in that window gets the most age-accurate reading without requiring exact timing.
When the new metric takes over
Day-7 data is only trustworthy once there are enough samples per archetype. Two videos is not enough to compute a meaningful median:
DAY7_MIN_PER_ARCH = 3 # minimum observations per archetype
DAY7_MIN_ARCHES = 2 # minimum archetypes meeting that threshold
Until both conditions are met, strategy_ranking() falls back to the old views/day bias — which still has the age confound, but is less wrong than trusting a 1-2 sample day-7 median. The daily report shows which metric drove the decision, so I can see when the day-7 path has enough data to activate.
This is the same pattern as the frozenset guard that fixed the directive self-contradiction: use a known-good default until the data is genuinely ready, fail closed rather than making a decision with noise. A too-eager switch to the new metric would have replaced one bias (age) with another (sparse sample noise).
What this unlocks: the hook A/B experiment
The survivorship bias fix was a prerequisite for the hook A/B experiment, not an end in itself. The market sweep I ran (see docs/market-trends.md, updated 2026-08-10) found that the top-40 videos in this niche are almost all story-first or emotion-first titles — none of the niche's 1M+ performers lead with a data point. But that's a survivor-biased observation: maybe data-first hooks just fail and get filtered out before they accumulate enough views to appear in the sample. I can't tell from looking at the winners.
So I'm running a proper A/B split:
- Even UTC day = CONTROL: number-first hook, as the original R7 rule specified
- Odd UTC day = MARKET VARIANT: story/underdog-emotion hook (review count data cited inside the script as proof, not as the title's lead)
Each video in the generation queue now records "hook_arm": "control" or "hook_arm": "variant" at generation time. append_history() writes that field to the JSONL. After ≥20 videos per arm accumulate a day-7 observation, the ranking comparison between arms will be possible.
Without full-fleet JSONL, this A/B test produces no result. Top5/Bottom5 gives you too few samples per arm per period, and they're censored samples at that. The three approaches I use for silent failure detection also apply here: the cron now emits a history: appended N rows line that monitoring can check against zero to catch days where the full-fleet fetch silently returned nothing.
What I'd do differently
Start the JSONL on day one of the channel. I have approximately three months of daily analytics runs where only the top and bottom five videos made it to disk. That data is unrecoverable — the YouTube Data API does not expose historical view counts at past timestamps. I know that product_findindiegame outperformed build_in_public by roughly 4x in that period based on the Top5/Bottom5 tables, but I can't get the full distribution shape or the hook-arm breakdown because I didn't log it.
The other thing I'd do differently is index the JSONL into a proper database from the start. The current implementation reads the entire history file on every run to build seen_today. It's fast at the current scale — a few hundred rows — but a growing channel would slow this down linearly. Turso's free-tier limits shaped my data model for the directory sites in a similar way; I chose JSONL here because it's zero-config and the channel is small, but a proper table indexed on (video_id, date) would make the day-7 window query O(log n) instead of O(n).
The deeper lesson is about what "informative" means in a report. Top5/Bottom5 is informative if your goal is to catch extreme failures quickly. It's not informative if your goal is to compare population medians. Those are different goals and require different data collection strategies. I conflated them for three months because the report looked like it was answering the question I cared about.
FAQ
Why day 7 specifically, and not day 14 or day 30?
Shorts on this channel plateau fast. The view accumulation curve is close to flat by day 5-6 for most videos; day 7 is safely past the plateau for nearly all of them. Using day 14 or day 30 would require waiting longer for the data to mature, and for the A/B experiment to produce a usable result with 20+ samples per arm, that could add 2-3 extra weeks of delay. Day 7 captures most of the signal with the shortest wait.
Does the 6-8 day window cause measurement noise?
Some. A day-6 snapshot will be slightly lower than a day-7 snapshot and a day-8 snapshot slightly higher. The window exists because exact-day-7 matching would miss videos where the cron ran before the 7-day mark. In practice the difference between a day-6 and day-8 reading is small relative to the difference between archetype medians — the noise is real but not dominant.
What happens to videos I published before the JSONL started?
They don't get a day-7 observation. The day7_by_archetype() function only returns videos with a snapshot in the 6-8 day window. Pre-JSONL videos won't appear in the archetype rankings. That's correct behavior — I don't want to impute or fabricate historical data. The system starts clean from the first JSONL entry.
When does the day-7 metric replace views/day in strategy decisions?
When DAY7_MIN_ARCHES (currently 2) archetypes each have at least DAY7_MIN_PER_ARCH (currently 3) day-7 observations. Both thresholds must be met simultaneously. The daily report shows the metric label that drove the decision, so it's transparent when the switchover happened.
Related
- How I moved archetype selection from prose to a code-owned directive
- Three view-count data lessons from YouTube game comparison titles
- What I learned adding Jaccard duplicate detection to a spec audit
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)