Three lint rules that caught persistent boilerplate in 1500 AI-generated directory entries

When you generate content for 1500 directory entries using Claude Haiku, the output is not uniform boilerplate — but it's not fully unique either. The model has patterns. Certain phrases recur across entries that happen to share a HuggingFace pipeline tag or category. You don't notice until you're reviewing ten entries side by side and realize they all start with "It follows chat template conventions and supports system-level role instructions."

That phrase showed up in 87 entries. Verbatim.

How template text gets in

There are two entry points for boilerplate in this setup.

First, the prompt template. My system prompt for AI model entries asks for "a 2-3 sentence technical summary." The model knows what a text-generation model does, and it tends to express that knowledge the same way. "Handles instruction prompts, multi-turn dialogue, and open-ended text generation" is a factually accurate description of every chat model. Which means it appears in the description of every chat model.

Second, the fallback content. Before the Claude API is available (in CI with no API key, or for new entries before the first enrichment run), I seed entries with template content. The three-tier quality ladder covers the tiering. The problem: even after Claude runs on an entry, the generated content sometimes closely echoes the fallback phrasing. The model saw similar entries with similar prompts and converged on similar output.

I could fix this by making the prompts more specific per entry or by adding explicit "do not use these phrases" instructions. I did that too, as the seeded-variant-generation approach shows. But I also wanted a gate that catches regressions without relying on the prompt being perfect.

What the lint pass does

scripts/lint-humanization.mjs reads the three dataset JSON files (models, saas, games), extracts text fields from every row, and checks for two things:

1. Phrase frequency: specific strings that should appear rarely or never
2. Sentence clustering: normalized sentence deduplication across all entries

For sentence clustering, normalization matters. You want "loads 7 billion parameters" and "loads 3 billion parameters" to count as the same sentence shape, not as different sentences. I replace numbers with <num>, repository paths with <repo>, and strip punctuation before comparing:

function normalizeSentence(s) {
  return s
    .toLowerCase()
    .replace(/\b\d+(?:[,.]\d+)*\b/g, "<num>")
    .replace(/\b[a-z0-9_.-]+\/[a-z0-9_.-]+\b/g, "<repo>")
    .replace(/[^a-z0-9<> ]+/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}

Any normalized sentence that appears in more than N entries is a candidate for review.

Three rules that surfaced real problems

Rule 1: "The main gap" — max 0, severity error

This phrase appeared in my initial OSS alternatives entries. The prompt asked for "comparison notes," and the model consistently opened with "The main gap between SaaS X and its open-source alternatives is..." It's not wrong — it is often the main gap — but having it verbatim in 40 entries reads as templated.

max: 0 means the lint pass errors (not warns) if it appears at all. After adding this rule, I rewrote the prompt to ask for comparison notes in a different structure. The phrase disappeared.

Rule 2: "vendor lock-in" — max 24, severity warn

I can't ban "vendor lock-in" entirely — it's genuinely the right term in many contexts. But appearing in 24+ out of 500 SaaS entries means the model is reaching for it as a default framing rather than choosing it specifically. The max: 24 threshold is approximately 5% of the dataset; above that, it signals a prompt pattern, not genuine use.

When this triggered, I found 31 occurrences. I didn't rewrite those entries manually. Instead I adjusted the prompt to ask for "data portability and self-hosting tradeoffs" rather than "reasons to seek OSS alternatives," which shifted the vocabulary away from vendor lock-in framing.

Rule 3: "handles instruction prompts, multi-turn dialogue, and open-ended text generation" — max 0, severity warn

This one is specific enough that I flagged it at max 0 even though I kept the severity at warn (not error). It's an almost verbatim sentence from my system prompt example output — I had shown the model what a good entry looked like, and it had memorized the phrasing from my example.

The fix was removing the example from the system prompt and replacing it with structural guidance only. Entry quality went down slightly on the first run (the model was less clear on format without an example), then recovered once I found the right balance of instruction vs. example.

The two-tier severity: error vs warn

All eight rules have severity: "error" or severity: "warn". The lint script exits with a non-zero status only on errors (or with --strict, on warnings too).

The distinction:

• error: the phrase indicates a template that got through and shouldn't exist in production. Block deploys.
• warn: the phrase is legitimate but overcrowded. Track it, but don't block deploys. Fix the prompt at the next review.

Running with --strict is useful for auditing. Running without is useful for CI — you want to catch actual regressions, not get blocked every time the model slightly overuses "open-source ecosystem."

What this didn't catch

The lint pass is phrase-based and sentence-level. It doesn't catch:

• Structural repetition (every entry has four pros and three cons with the same surface structure)
• Factual errors (a model described as supporting Japanese when its only language tag is English)
• Missing specificity (valid sentences that are technically true but say nothing)

For those, I rely on the content quality gate scripts and manual spot-checks. The lint pass is one layer, not the whole quality story.

The --dry-run flag reports violations without modifying anything. I run it weekly against the current datasets. If violations spike — which they do after a large batch ETL run — I investigate before the next content refresh.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.