I've been running a hook A/B test on my YouTube Shorts for 14 days. The variant arm leads on views-per-day by a meaningful-looking margin. The data is also telling me that the margin is almost entirely explained by one outlier video on each side — not by the hook text.
Here's what the numbers look like and why I'm not changing anything yet.
The test setup
The YouTube short-generation pipeline assigns hook arm by the publish day's UTC parity. Even UTC day = CONTROL: number-first hook, leading with a concrete data point. Odd UTC day = VARIANT: story-first or underdog-emotion hook, burying the data inside the script rather than leading with it.
Every video spec now records "hook_arm": "control" or "hook_arm": "variant" in its JSON. The analytics history appends this to the per-video JSONL on every daily fetch. The A/B logic is described in the survivorship bias fix write-up — without full-fleet JSONL, there would be no arm-level data to analyze.
The 14-day window (2026-07-29 to 2026-08-12) covers 11 videos: 3 in the control arm, 8 in the variant arm. The imbalance is calendar arithmetic — videos only upload when specs are ready, not on a fixed cadence, so the even/odd split doesn't guarantee equal arm sizes.
The 14-day numbers
Parity-inferred medians (these are computed from current views-per-day, not age-controlled day-7 data — the JSONL doesn't have enough history for day-7 on all 11 videos yet):
| Arm | n | Videos | Median vpd |
|---|---|---|---|
| Control (even day) | 3 | Brotato/AC Mirage, Undertale/Spider-Man, Project Zomboid | 5.3 vpd |
| Variant (odd day) | 8 | Geometry Dash, Binding of Isaac, RimWorld/FF7R, Undertale/Jul29, Don't Starve, Celeste, Valheim, Among Us | 7.4 vpd |
The two-day threshold for flagging a directional signal — variant leading for 2 consecutive days — was crossed today.
The confound
The control arm contains one video with 0.7 vpd: Brotato/AC Mirage. This is a "flop anchor" pair. AC Mirage had poor anchor quality relative to the live-table rankings at the time of publication, and the video's performance reflects that.
The variant arm contains Geometry Dash (18.7 vpd, age 3 days). This video uses a current live-table anchor with EA FC 26 — a different quality tier from anything in the control arm. Geometry Dash's high early vpd is consistent with a strong anchor pairing, not with the hook text being particularly effective.
When I exclude one outlier from each arm — Geometry Dash from variant, Brotato from control — the medians converge:
- Variant (n=7, ex-Geometry Dash): 6.0 vpd
- Control (n=2, ex-Brotato): 5.7 vpd
A 0.3 vpd gap on n=2 vs n=7 is not interpretable. The original 2.1 vpd gap was driven by which anchor each arm happened to get, not by the hook text.
Why the outliers ended up in these arms
The arm assignment is mechanical: even UTC publish day = control, odd = variant. Geometry Dash was published on an odd day because the generation routine fired on that day. Brotato was published on an even day for the same reason. Neither assignment was intentional from a design standpoint — the calendar parity doesn't track anchor quality.
This is the core design problem: the arm assignment correlates with publish date, and anchor quality partially correlates with publish date (because the live-table ranking changes over time, and the best anchors get used when specs happen to be ready). Getting a clean measurement requires either randomizing anchor quality across arms or stratifying — making sure each arm has a comparable distribution of anchor tiers.
Currently the live-table anchor gate ensures minimum quality (R6b compliance) but doesn't control for distribution within arms.
What I'm waiting for before drawing a conclusion
Three conditions need to be met before the hook text comparison is valid:
1. Tagged data instead of inferred data. The hook_arm field was added partway through the 14-day window. Several of the videos in the parity table are "parity-inferred" — I know their publish day and computed their arm from that, but they don't have hook_arm in their spec JSON. Inferred arms are correct, but they make the history brittle to any change in how the pipeline runs.
2. At least 5 tagged observations per arm. The current control arm has n=3, with one outlier consuming most of the signal. At n=5 (minimum), excluding one outlier still leaves 4 data points — enough to compute a median that isn't dominated by the sample distribution.
3. Anchor parity between arms. Before publishing the next batch of videos under the hold, I'm tracking which anchor tier lands in which arm. If the next three control-arm videos all get weak anchors and the next three variant-arm videos get strong anchors again, the arm comparison stays confounded regardless of sample size.
The anchor parity check is the one I can't automate easily with the current pipeline. It requires comparing the live-table ranking of the anchor at the time of publish across the two arms — which means storing the table state at generation time, not just the anchor name. That's a data collection change, not a filtering change.
What the 2-day threshold is actually telling me
The "variant leads for 2 consecutive days" signal is real and worth noting. It means there's a genuine directional trend in the current data, not one-day noise. But the threshold was designed to trigger review, not to trigger a directive change. The next step is the anchor-parity audit, not switching all future videos to the variant hook.
If after 5 tagged observations per arm with comparable anchors the variant still leads, that's a different situation. At that point the hook text itself is the more plausible explanation for the gap.
Until then: record hook_arm in every new spec JSON, and wait.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)