MORINAGA

Posted on Sep 8

How I found deletion bias in my own YouTube shorts analytics

#webdev #showdev #javascript #indiehackers

I delete YouTube shorts that sit under 50 views after their first week. The reasoning seemed sound — keep the performing content, sharpen the sample, get a cleaner signal on what works.

On 2026-08-11 I ran a routine check and found the headline median was 3.3 views/day. A month earlier it had been 2.9. I started writing a note about the improvement.

Then I recomputed it correctly. The corrected figure was 2.91.

The entire apparent gain was the deletion.

What I was measuring, and why it lied

The pipeline tracks a rolling views/day figure for every video. When a video is deleted, it disappears from the sample. That's where the bias enters.

The videos I delete are the worst performers — the ones with the lowest views/day. When I remove them, the sample average rises not because the remaining videos improved, but because the distribution got trimmed from the bottom. The sample is missing-not-at-random: the measurement outcome (views rate) is exactly the selection criterion for removal. Deleting the 0.3 views/day videos and then reporting that my average is up to 3.3 is the same shape of error as measuring customer satisfaction only among people who didn't churn.

Statisticians call this outcome-dependent censoring. I'd seen the term in econometrics contexts but had never applied the concept to my own feed.

The naive number wasn't wrong as a description of surviving videos. It was wrong as an answer to "how is the channel doing?" — which is the question the metric was supposed to answer.

The LOCF fix

The standard patch for this kind of censoring is Last Observation Carried Forward (LOCF). When a video is deleted, you record its last known views/day value and carry it forward in all subsequent aggregate computations. You are not pretending the video still exists; you're preserving the measurement you had before it was removed.

Two implementation choices mattered:

1. Track deletions explicitly before they happen. Before any deletion, the pipeline now records: video id, deletion date, and the 7-day rolling views/day at that moment. The raw file has deleted_at and last_known_vpd columns. Without this step, LOCF is impossible — you'd have to reconstruct the value from API history, and that data has retention limits.

2. Refuse to render when imputed share exceeds 20%. If more than 20% of the current sample's views/day figure is coming from carried-forward values, the report prints n/a (imputed share: X%) rather than a number. LOCF values degrade in accuracy as time passes — a video deleted at 0.4 views/day six months ago shouldn't be treated as 0.4 today. The 20% cap is a circuit breaker, not a long-term solution.

With LOCF applied to the 2026-08-11 snapshot, the median became 2.91 — statistically flat versus the 2.9 baseline. No improvement; no regression. The channel was where it had been.

I renamed the metrics in the report, and added a note to the pipeline health monitor that flags when the imputed share crosses 20%:

surviving_cohort_median — the naive figure, useful as a description of what's currently live
channel_median — the LOCF figure, useful for trend measurement; shows n/a when imputed share is too high

Neither is hidden. They answer different questions and the report now says so explicitly.

What it did to a running A/B test

During the same period I'd been running a format test: faster narration versus the original pacing, split across 20 videos. The test ran for six weeks.

The deletions hit both arms, but not evenly. Arm A lost 36% of its videos to deletion. Arm B lost 44%.

When I computed effect size treating the remaining videos as the full sample — the naive approach — the result was 0.99x. Essentially neutral. When I recomputed using LOCF, carrying deleted videos at their last known values, the result was 1.89x in favor of the faster-narration arm.

Nearly a two-fold difference from the same raw data, determined entirely by how deletions were handled.

I marked the test SUSPENDED rather than claiming either result. The sample started at 20 videos; differential attrition left each arm at a different size; the effect direction was sensitive to the imputation method. There was no valid winner to declare. The right next step is a cleaner run: no in-flight deletions while the test is active.

The table I should have built earlier

Method	Median views/day	Notes
Naive (surviving videos only)	3.3	What the original script produced
LOCF carry-forward	2.91	Deleted videos carried at last known value
Difference	+0.39	100% attributable to deletion policy
A/B effect size (naive)	0.99×	Neutral result
A/B effect size (LOCF)	1.89×	Favor faster-narration arm

The 0.4 views/day gap looked meaningful until it evaporated under LOCF. The A/B direction is actually more interesting — but the test design doesn't support claiming it. I shouldn't have been running deletions during an active arm comparison.

A note from a different project

For the shelf-empty detector I run on a Raspberry Pi, I have 19 scans in the dataset. I've never deleted one.

This isn't principled rigor — the scan files are small and there's no reason to prune them. But the effect on signal quality is real. The temporal majority vote that determines whether a detection is "confirmed" looks at the last three scans. If I'd been deleting scans where the model fired incorrectly, the historical false-positive rate would look artificially clean — the same shape as the deleted-video problem.

The mAP50 for that model on the held-out test set is 0.844 (277 images, 1,255 instances — model trained on 11,667 images from Roboflow Universe under CC BY 4.0). That number was measured on data that wasn't filtered by correctness after the fact. The inference runs at a median 8.5 seconds per scan on the Pi 3's CPU — not fast, but the hourly schedule makes it acceptable. What matters is that neither the accuracy number nor the inference number has had its failures removed from the sample.

The data hygiene principle is identical to the YouTube case: measuring only the observations you liked gives you a number that describes your curation policy, not the underlying thing you were trying to measure. My three post-processing layers for the shelf detector were all about handling reality correctly rather than filtering it away.

Where deletion bias hides elsewhere

Once I found it in views/day, I looked for the same pattern elsewhere.

The 97-upload analysis — where I measured why videos underperformed — had a lower-bound bias in the "before" cohort because history collection started after the earliest videos were already live. Not deletion per se, but the same missing-not-at-random shape: the least-observed observations are the early ones, and early ones may be systematically different from later ones.

The Steam VoC engine has a version of the same problem: reviews from players who stopped playing early are less likely to be in the corpus. The engine handles this explicitly by joining topic assignments to playtime, making the selection condition visible. The approach for silent-failure detection in cron pipelines also had to account for the fact that green runs don't prove correct output — a job can complete without producing anything useful, and the absence of a failure log doesn't mean success.

In all of these cases, the same question applies: are the observations I'm using to measure something a random sample, or did a selection criterion that correlates with the outcome trim the sample before I looked at it?

FAQ

Why not just stop deleting videos?

Deletion is a distribution decision — remove content I don't want live — not an analytics decision. The fix is to preserve the measurement even when removing the video. Deletion policy and accounting policy are separate. LOCF lets me keep both.

Doesn't LOCF get worse over time as carried-forward values diverge from reality?

Yes. A video deleted at 0.3 views/day in month 1 shouldn't be assumed to still be at 0.3 in month 12. The 20% imputed-share cap manages this: once imputed values dominate the sample, the channel median is printed as n/a rather than a misleading number. For cumulative historical analysis over months, I don't use LOCF at all — I use the naive count with a note on the deleted-video share.

Was the A/B test outcome real at 1.89x?

I don't know. The differential attrition (36% vs 44%) means even the LOCF number can't be trusted cleanly — the deleted videos in arm B weren't random draws from that arm, and the imputed values for 44% of an arm pull the estimate toward the observed remainder. The honest answer is that I need a new test with a no-deletion constraint while it's active.

How do I know the 20% threshold is right?

I don't. It's a judgment call: at 20% imputed share, I consider the metric noisy enough to suppress. A more principled approach would compute a confidence interval explicitly, which would require stronger assumptions about what the deleted videos would have done. I chose the simpler circuit breaker.

What does this affect besides the headline median?

Anything computed over the full video set: median views, mean views, engagement rate, retention estimates, cohort comparisons. The audit of fabricated claims found similar-shaped errors in articles: numbers that were computed correctly once but then quoted in contexts where the original selection conditions no longer held. The root is the same — a number is only meaningful relative to the set it was computed over, and that set needs to be explicit.

DEV Community