Before the spec audit, my CI pipeline could upload a perfectly formatted video—correct word count, all JSON fields present, proper hook variants—that was structurally identical to something uploaded three days ago. After adding Jaccard similarity checks against the last 30 uploaded specs, those near-duplicates fail in under 50ms before any TTS synthesis or ffmpeg step runs. The threshold calibration matters more than the algorithm: 0.76 for titles and 0.82 for opening 28-word windows, derived from manual inspection of uploaded specs rather than theory.
The CI pipeline that generates YouTube Shorts ran for several weeks before I noticed a pattern in the upload history: the titles varied, the game matchups differed, but the opening lines were converging. The AI generating scripts had found a template that worked—open with a hard number, frame an underdog—and was reusing the same sentence skeleton almost verbatim. Not the same video, but close enough that a viewer who'd watched the last five would notice.
Format validation doesn't catch this. You can verify that a spec has a title, a script between 55 and 200 words, at least three hook_variants, and data_panels with HTTPS source URLs—and still upload something that feels like a rerun. The content quality gate I wrote for articles has the same limitation: it catches broken frontmatter and missing required fields, but it can't tell whether you've written essentially the same piece twice with different nouns.
Jaccard similarity is the simplest thing that addresses this. It's used in plagiarism detection and recommendation systems; it's three lines of code with no dependencies; and it's directly interpretable when it fires. The formal definition is intersection over union of two sets. Here's how I applied it to video spec validation.
What Jaccard measures, and what it won't catch
Jaccard similarity between two texts: tokens in both / tokens in either. Tokens are lowercased words longer than two characters, punctuation stripped. The algorithm is order-blind—it treats "Risk of Rain 2 has more reviews than Hollow Knight" and "Hollow Knight has fewer reviews than Risk of Rain 2" as identical. That's a weakness if you're trying to catch intentional paraphrase. It's an acceptable tradeoff here because I'm not trying to catch paraphrase. I'm trying to catch the case where the generation routine reused the same sentence structure and vocabulary because a particular template happened to score well in the analytics classifier.
function tokenSet(text) {
return new Set(
String(text).toLowerCase()
.replace(/[^a-z0-9 ]+/g, " ")
.split(/\s+/)
.filter((t) => t.length > 2)
);
}
function jaccard(a, b) {
const aa = tokenSet(a), bb = tokenSet(b);
const overlap = [...aa].filter((token) => bb.has(token)).length;
return overlap / Math.max(1, new Set([...aa, ...bb]).size);
}
I compute this twice per new spec—once on the full title, once on the first 28 words of the script—and compare against every spec in the last 30 uploads. Title check fires at 0.76; opening check fires at 0.82.
Why separate thresholds for titles vs openings
The right threshold depends on how much vocabulary overlap is natural given the content. Titles in this pipeline almost always name two specific games or software products. Two titles covering different matchups share almost no vocabulary. A threshold of 0.76 on the title means "three-quarters of the title tokens appear in both texts"—that only happens when two specs are targeting the same product comparison with nearly identical phrasing.
Opening lines are harder. The pipeline's best-performing hook pattern leads with a hard number in the first three words: "Risk of Rain 2 has 693,000 Steam reviews while Hollow Knight sits at 400,000." A different spec might open with "Balatro has 380,000 Steam reviews against Final Fantasy XVI's 12,000." These share nearly nothing despite following the same template, because the proper nouns dominate the vocabulary.
The problem arises when the generation routine gets into a rut after several consecutive product_findindiegame specs. I've seen it output "X has N reviews, making it the clear winner" across three consecutive specs with only the game names swapped. The Jaccard score for that pattern reached 0.84 on the opening window—above the 0.82 threshold. At 0.70, genuine numeric hooks on different matchups would have occasionally collided. At 0.90, rut cases would slip through.
Choosing a 28-word window for the opening comparison came from looking at where structurally similar openings diverged from each other. At 20 words, the window was too short to catch real convergence; two specs that opened with different games but the same template still scored below 0.70. At 40 words, unrelated specs that both transitioned into a "here's why this matters" framing started scoring too high due to shared transition vocabulary. Twenty-eight words—roughly the first two sentences of a 55-word Short script—was the inflection point for this corpus.
The claims provenance map: tying every assertion to a source
Jaccard catches structural similarity. The claims[] field catches something different: assertions that are factually grounded but not traceable from the spec itself.
Every item in claims must link a specific factual assertion to the HTTPS URL where I confirmed it, and that URL must also appear in data_panels. The cross-reference constraint prevents a pattern I saw in early generated specs: a claims[] entry citing a Steam page, but data_panels containing a different URL that was the actual source the generation routine used. If those diverge, either the claim was generated from a source not recorded in the panels, or the panels list sources that don't back the specific claims.
"claims": [
{
"claim": "Risk of Rain 2 has 693,000 Steam reviews as of 2026-07-10",
"source": "https://store.steampowered.com/app/632360/Risk_of_Rain_2/"
}
],
"data_panels": [
{
"label": "Steam review counts",
"url": "https://store.steampowered.com/app/632360/Risk_of_Rain_2/"
}
]
The audit enforces this: every claim.source must appear in the flat list of URLs extracted from data_panels. A spec that cites source A in claims but lists source B in panels fails the audit regardless of whether either URL is actually correct.
Combined with verified_at (a YYYY-MM-DD date field required on every spec), this creates a paper trail: each claim has a source, each source has a date when I confirmed it, and the audit verifies the linkage.
Quarantine instead of halting the queue
The pipeline health monitor watches the queue at the workflow level—it fires if nothing has shipped in 36 hours. But a spec that fails quality validation is a narrower problem: one bad spec shouldn't stall the day's queue.
The audit runs as the first step in the CI job, before any compute-intensive work. If it fails, the spec moves to content/yt-queue/rejected/ with the error list appended to its filename, the CI job exits non-zero, and the next spec in the queue is unaffected. The quarantine directory also serves as a record: I've retrieved and manually fixed rejected specs twice, and I've occasionally discovered that a threshold miscalibration was producing false positives before any specs were lost.
This matters more for the two-host longform pipeline than for Shorts. A longform spec takes 8-15 minutes to synthesize and render. A bad spec that passes format validation but fails midway—at the thumbnail generation step, for example—wastes the entire CI run. The three-tier thumbnail fallback pipeline was partly built in response to a spec that failed at thumbnail generation after TTS had already completed; the quarantine model would have caught it before any of that compute ran.
With the audit as a fast first gate, failure is cheap: exit non-zero in under 100ms, commit the rejected spec with its error list, continue with the next item. The health monitor's 36-hour threshold is large enough to absorb several consecutive audit failures without alerting.
How the archetype directive and the spec audit interact
The analytics-driven archetype directive is the upstream control. If the directive correctly identifies which archetype to produce and the generation routine follows it with sufficient variation in game matchups, the Jaccard audit should rarely fire. Different matchups produce different vocabulary; Jaccard scores for unrelated specs typically stay below 0.40.
The audit becomes necessary when the directive guidance is followed but the generation routine still converges. That can happen when the same archetype runs for several days and the language model finds a phrase pattern that consistently passes the hook quality checks—not because it's copying verbatim, but because it's exploiting the same structural template.
The thumbnail rules derived from 19 videos of analytics have an analogous property: they enforce visual differentiation at the image layer while the spec audit enforces differentiation at the text layer. Neither catches the other's failure mode—a visually distinct thumbnail can accompany a near-duplicate script, and vice versa.
The quality gates article covers why I think fail-closed gates matter at every layer; this post is about one specific gate and how it's calibrated.
Limits I haven't solved
Semantic drift. If the generation routine rotates vocabulary deliberately—"critically acclaimed" instead of "award-winning", "dominating" instead of "winning"—Jaccard scores will be low even if the underlying framing is identical. I don't have a good solution here at the scale of 30 videos. TF-IDF cosine similarity would be more sensitive to rare vocabulary, but the corpus is too small for meaningful IDF weights.
The lookback window ages out. If I publish 30 videos on the same archetype category—roughly seven weeks at current cadence—the oldest specs age out of the window, and a rephrase of the earliest ones could theoretically slip through. I'd need either a longer window or a topic-slug–based deduplication layer that persists beyond the rolling 30.
Threshold drift. Both thresholds are calibrated to the current script style. If I significantly change the hook pattern or switch to a different generation model, the natural base similarity of both titles and openings will shift and the thresholds will need recalibration. I haven't built any automated way to flag threshold drift; it requires noticing the audit's false positive rate climbing.
FAQ
Why Jaccard and not cosine similarity or BLEU?
Jaccard is three lines of code with zero dependencies. Cosine similarity over TF-IDF vectors would be more sensitive to vocabulary frequency, but TF-IDF requires a meaningful corpus for IDF weighting—30 specs is too small. BLEU requires per-test reference translations, which don't apply here. For this scale, Jaccard is accurate enough and trivially debuggable: when the audit fires, I can compute the overlap manually in my head.
Does the lookback include longform specs?
Yes, but the similarity checks only compare a new spec against previous specs of the same type. A Short spec is checked against the 30 most recent Short specs; a longform is checked against longform. The isLongform detection is based on whether the spec has a segments array rather than a single script field.
What's actually in the lookback directory?
content/yt-queue/uploaded/ contains one JSON file per uploaded spec, named by upload date. The audit reads all .json files in that directory, sorts by filename, and takes the last 30. It does not read from the YouTube API—just from the committed spec files.
How long does the audit take?
On the GitHub Actions runner, about 40ms for a queue with 19 uploaded specs. The hot path is the nested loop over uploaded specs (O(n) Jaccard computations per new spec), and both n and the token set sizes are small enough that this doesn't register in the workflow's timing.
What happens to a rejected spec?
It moves to content/yt-queue/rejected/ with the full error list written to a companion .txt file. The rejected directory is committed and pushed. The generation routine doesn't automatically re-queue rejected specs; I handle those manually or let the next daily run produce a fresh spec for the same topic.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)