What I learned about detecting template residue in AI-generated directory content

The conclusion first: once you have more than 200 AI-generated directory entries, reading them to check for sameness is useless. The script I wrote after hitting that wall — scripts/lint-humanization.mjs — catches two distinct problems that visual review misses entirely. The first is named phrase contamination: specific stock phrases that Claude reaches for repeatedly, even when it's not using the fallback template. The second is structural sentence duplication: sentences that are semantically identical after stripping numbers, repo names, and punctuation. Both show up in real content. Neither is visible at a glance.

This article explains how both detection approaches work, where they fail, and what the data looked like across 1,664 AI model entries, 120 indie game entries, and 80 SaaS entries in this project.

The two kinds of sameness

AI-generated directory content has a failure mode that doesn't show up in standard quality checks. Cliché detection, word count gates, and tag pool validation — the original quality gate I built in June — all catch structural problems. None of them detect when Claude keeps writing the same sentence 40 times across different entries, each time with different nouns substituted in.

The sameness problem is specific to high-volume generation. When you're generating 20 entries, variation is natural — the model has different context each call and produces different structure. At 1,664 entries (the current count in the AI Tools directory), the model starts converging on stock phrases. Not because of the fallback template — those entries are marked fallback-template in model_used and easy to count. The harder problem is entries where model_used = "claude-haiku-4-5" but the output is still near-identical across tens of rows.

There are two reasons this happens:

System prompt convergence: when your system prompt describes a consistent structure ("produce a 2-3 sentence summary, then 3-5 use cases, 3-5 pros, 3-5 cons"), the model learns to fill each slot with similar-sounding content. The slots vary — the model isn't copying — but the sentence starters, transition phrases, and hedging language repeat.

Fallback-adjacent phrasing: even without triggering the actual parseOrFallback function, Claude reaches for phrasing that sounds like it could have been generated by the fallback template. Phrases like "typically requires self-hosting" or "OSS alternatives provide" appear in both fallback output and real Claude output. The fallback template was trained to sound like Claude, so the boundary is blurry.

RESIDUE_RULES: named phrases with thresholds

The first detection layer in lint-humanization.mjs is explicit: an array of specific phrases, each with a maximum allowed count and a severity level:

const RESIDUE_RULES = [
  { phrase: "The main gap", max: 0, severity: "error" },
  { phrase: "plays like", max: 12, severity: "warn" },
  { phrase: "because it combines", max: 6, severity: "warn" },
  { phrase: "handles instruction prompts, multi-turn dialogue, and open-ended text generation", max: 0, severity: "warn" },
  { phrase: "It follows chat template conventions and supports system-level role instructions", max: 0, severity: "warn" },
  { phrase: "OSS alternatives provide", max: 12, severity: "warn" },
  { phrase: "typically absent", max: 12, severity: "warn" },
  { phrase: "vendor lock-in", max: 24, severity: "warn" },
];

The max: 0 entries are phrases that should never appear at all. Those two long strings — "handles instruction prompts, multi-turn dialogue, and open-ended text generation" and "It follows chat template conventions" — are lifted verbatim from a previous version of the system prompt. They somehow ended up in early model entries that slipped through before I noticed. Setting max: 0 with severity error means the script fails with exit code 1 if they appear anywhere in the dataset. That's a CI-breaking error, not a warning.

The max: 12 and max: 24 entries allow some tolerance. "Vendor lock-in" appearing in 24 of 80 SaaS entries is acceptable — it's a real concept in that domain. The same phrase appearing in 60 of 80 entries means the model is using it as a crutch rather than as a precise descriptor. The thresholds are manually calibrated based on what I saw in the actual data.

What I'd do differently: the thresholds need to be per-dataset rather than global. The indie game entries get different stock phrases than the AI model entries ("plays like" is legitimate at low counts for games, meaningless for AI models). The current implementation applies the full RESIDUE_RULES array to all three datasets, which means some rules don't fire where they should and some warn incorrectly.

Sentence normalization: the technique that catches everything else

Named phrase detection only works when you know which phrases to target. The sentence normalization approach works without any prior knowledge of what phrases to look for.

The normalizeSentence function strips everything that makes sentences look different:

function normalizeSentence(s) {
  return s
    .toLowerCase()
    .replace(/\b\d+(?:[,.]\d+)*\b/g, "<num>")
    .replace(/\b[a-z0-9_.-]+\/[a-z0-9_.-]+\b/g, "<repo>")
    .replace(/[^a-z0-9<> ]+/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}

After normalization, "Llama-3.2-8B supports long-context reasoning up to 128,000 tokens" and "Qwen2.5-72B supports extended context windows up to 32,000 tokens" become the same string: "llama num supports long context reasoning up to num tokens" vs "qwen num supports extended context windows up to num tokens". Those aren't identical. But "The model supports long-context reasoning tasks and multi-turn conversation" and "The model supports multi-turn dialogue and long-context reasoning workflows" normalize to something very close — different word order, but the same <num>-stripped core.

Then repeatedSentences runs across all rows in a dataset, collecting normalized versions of every sentence longer than 48 characters and counting how many distinct entries contain each. Anything appearing in 6 or more entries gets flagged:

.filter((item) => item.ids.length >= 6)
.sort((a, b) => b.ids.length - a.ids.length);

The 48-character floor is important. Short sentences ("This is a fast model", "No GPU required") appear everywhere legitimately — they're just short, factual, and obvious. The normalization approach becomes noise if it flags those. At 48 characters after normalization, you're catching sentences with enough specificity that duplication is actually meaningful.

I set the threshold at 6 entries (not 10 or 20) because anything appearing in 6 of 80 SaaS entries is already at 7.5% of the dataset. That's the level where a reader browsing the directory would start noticing the pattern.

Where the --strict flag belongs

The script has two modes. Without --strict, it exits 0 unless there are errors (the max: 0 violations). Warnings are printed but don't block. With --strict, any warning becomes an exit code 1.

In CI, I run without --strict. The content refresh cron runs nightly and adds entries as new models appear on HuggingFace. Blocking the whole ETL because 8 entries out of 1,664 share a sentence pattern is too aggressive — the pipeline would stall, the new entries wouldn't be added, and I'd need to manually investigate a false positive at 3am.

The strict mode is for the three-tier quality ladder I described earlier: when I manually run a quality audit pass and deliberately want it to fail on warnings so I have to address them. That's a human-in-the-loop operation, not an automated one. The --strict flag makes the gate adjustable to context rather than binary.

The same pattern applies to the four content QC scripts I wrote — each one has a severity model that distinguishes CI-blocking errors from advisory warnings. The decision about which level to use has to account for who is running the script and why.

What I'd do differently

Three things I'd change knowing what I know now:

Per-dataset thresholds from the start. The residue rules evolved based on what I found in the OSS alternatives dataset first, then got copy-pasted to cover all three datasets. The indie games dataset has different stock phrases than the AI model dataset. I should have designed the rule structure to be dataset-specific from the beginning. The current workaround is accepting warnings for phrases that don't really matter for some datasets.

Semantic similarity, not just string equality. The sentence normalization approach catches structural similarity after stripping numbers. It doesn't catch sentences that express the same concept with different words. "Requires a GPU with at least 16GB VRAM" and "Needs dedicated GPU with substantial memory" normalize to different strings. Both are stock phrases. This is the limit of the normalization approach — it catches the easy cases, not the subtle ones. Getting to semantic deduplication would require embedding every sentence and clustering by cosine distance, which is a different order of complexity.

Track residue trend over time. Right now the script is stateless — it checks the current dataset against current rules. I have no way to see whether the warning count is going up or down over time. Adding a simple JSON log of warning counts per run would let me see whether the nightly content refresh is making things better or worse without reading raw output.

FAQ

Does this catch the fallback-template entries specifically?

Yes, separately. The script checks row.model_used === "fallback-template" and reports a count as a warning. That's distinct from the sentence residue checks — it's just counting rows where the ETL explicitly fell back to the template rather than calling Claude. All three datasets currently show 0 fallback-template rows, which means the Anthropic API has been available during recent ETL runs. The three-tier quality ladder uses this count as a health signal.

What's the difference between this and the audit-articles.mjs quality gate?

audit-articles.mjs checks individual article files in content/articles/ — it enforces the quality_contract v2 requirements, cliché detection, word count, and tag pool validity. lint-humanization.mjs checks the JSON data files for the three directory sites — it operates at dataset scale, looking for cross-entry patterns rather than per-article structural issues. They're solving different problems.

At what entry count does sentence normalization start finding real signal?

I started finding meaningful patterns around 150 entries in the AI tools dataset. Below that, 6 occurrences in 80 entries is statistically noisy. At 1,664 entries, the repeated sentence list has 12 entries flagging across 20+ rows each. The technique scales with dataset size — more entries means more statistical signal, not more noise.

How much of the noindex-gate approach does this affect?

Pages with the highest residue counts are candidates for noindex — if 40% of a model's description is shared with other entries, the page probably doesn't add unique value. I haven't automated this linkage yet, but the residue score per entry would be a useful feature for the noindex decision threshold.

Does the lint rules approach in boilerplate overlap with this?

three-lint-rules covers article-level linting at generation time — clichés, tag pool, word count. This covers dataset-level quality at scale across the JSON files. Different scope and different timing.

---

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.