How three automated writers confused our content-publish gate with one shared label

scripts/polish.py picks summaries from a deterministic pool of template sentences, seeded by the md5 hash of each model slug. It makes no LLM call. The label it stamps on every row it writes is now "polish-py-template".

Until September 2026, it stamped "claude-routine-polish". So did scripts/humanize-aiappdex.mjs. So did the Claude Code Routine that generates curated model entries via API.

Three generators, one label. The curation gate in apps/ai-tools/src/lib/curation.ts couldn't tell any of them apart.

What the gate checks

The publishable check is a single set membership test:

export const PUBLISHABLE_MODEL_USED_LIST = ["claude-routine", "human-edited"];
export const PUBLISHABLE_MODEL_USED: ReadonlySet<string> = new Set(PUBLISHABLE_MODEL_USED_LIST);

if (!m.model_used || !PUBLISHABLE_MODEL_USED.has(m.model_used)) {
  return "unpublished:model_used";
}

"claude-routine-polish" is not in that set. When all three generators stamped it, the gate was blocking all of them — including the genuinely LLM-generated rows. Template content and Claude-drafted content were both unpublished, both indistinguishable, both labelled the same way.

The three generators

Each produces meaningfully different output but all arrived at the same label:

scripts/polish.py: Pure template. An md5 seed selects from a pool of summary sentences with {n} (download count) and {a} (author) placeholders filled in. Consistent structure, no variation beyond the placeholders. Makes no API call.

scripts/humanize-aiappdex.mjs: Builds sentences from HuggingFace model metadata: pipeline tag, language list, license. Also deterministic, also no LLM call. The header of scripts/relabel-aiappdex-provenance.py describes its shape as "one of its 6 openers AND one of its 8 caveats" — a fixed opener-caveat sentence structure.

The Claude Code Routine: Calls Claude Haiku 4.5 via the Anthropic API. Produces entries with genuine prose variation, specific technical context, and actual model-level detail that the template pools don't have access to.

These three are not equivalent. A curation gate designed to ensure pages are built only from the third class was working from a label that didn't encode which class any row belonged to.

What the relabeling script found

scripts/relabel-aiappdex-provenance.py ran a template-matching pass over apps/ai-tools/src/data/models.json to assign the correct label per row after the fact. It produced four classes:

• claude-routine — summary didn't match any known template shape. Residual class: likely LLM-generated but not positively identified as such.
• polish-py-template — verbatim match against the scripts/polish.py sentence pools (with {n}, {a}, {ln} wildcarded).
• metadata-derived — matched the humanize-aiappdex.mjs opener-plus-caveat structure.
• fallback-template — the ETL's own fallback sentence, written by the ETL when no generator ran.

The script's own header is explicit about one caveat: "claude-routine" is the residual class, not a positive identification. A row lands there because the script couldn't attribute it to any known template pattern. An older revision of one of the template generators whose shape wasn't encoded in the classifier would also land there. The agreement between the relabeling script's output and an earlier generation audit isn't independent corroboration — it's the same template-matching method applied to the same corpus, so reproducibility was guaranteed.

What this looked like from the output side

The publish gate gates page generation, not data ingestion. All rows were stored in models.json regardless of model_used. But only rows labeled "claude-routine" or "human-edited" became pages.

During the period when all three generators stamped "claude-routine-polish", the rows that should have been published as pages weren't. Astro's build gate (apps/ai-tools/src/lib/curation.ts is called at build time) filtered them out silently. There was no build error — the build succeeded, it just produced fewer pages than it should have.

Silent misses of this kind are harder to notice than failures. A build error stops everything. A correctly behaving gate returning the wrong answer from a wrong label produces a running site with missing pages.

The fix is obvious in hindsight

Unique label per generator, set at write time.

scripts/polish.py now stamps "polish-py-template". scripts/humanize-aiappdex.mjs stamps "metadata-derived". The Claude Routine stamps "claude-routine". The gate's publishable set matches what the generators actually write.

The template residue detection work I wrote about earlier ran the opposite direction: infer provenance from content text after the fact. That article found the same tradeoff — text-based classification at scale is expensive and fragile compared to asserting the class at write time. Labels are cheap. Classifiers built to recover missing labels are not.

The broader lesson: any field that drives automated decisions downstream should carry the correct value at the moment of write. A label that needs a one-off relabeling script to become meaningful is a label that was wrong from the start.

---

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.