I run three directory sites and a content pipeline that generates and cross-posts articles to Dev.to, Hashnode, and Bluesky automatically. The pipeline has been running for about six weeks. Early on I found a category of failure that no amount of CI infrastructure was catching: content quality problems. Wrong tags. Cliché phrases that slipped past self-review. Articles that implied specific traffic metrics I couldn't back up. Fabricated specificity disguised as honest reporting.
The solution was scripts/audit-articles.mjs — a lint-style quality gate that runs on every new article before the publish step. It works the way eslint works for code: structured checks, clear error/warning distinction, strict mode for pre-publish, lenient mode for a historical baseline.
Why a lint gate and not a manual review
The specific failure mode I was trying to prevent was this: automated generation leaves an article that reads fine on first pass but fails on systematic inspection. A cliché phrase at the start of a section. The tag "seo" slipping in when the pool explicitly forbids it. A word count of 580 when the spec requires 600-900 for a lightweight article. These aren't hard to catch — they're tedious to catch every single time, and tedium is where manual review degrades.
The pre-post Bluesky QC gate I built earlier applies the same principle to social posts: systematic gates catch what self-review misses reliably. For articles, the gate runs before the publish workflow can touch the file.
The JSON-LD audit script runs a similar check post-deploy against live pages. audit-articles.mjs runs pre-commit, against local markdown. Catching it before it ships is always cheaper than chasing it down after.
What the gate checks
The script runs about 12 distinct checks per file, split into errors (fail hard in strict mode) and warnings (report but continue by default).
Frontmatter structure. Four required keys must be present: title, description, tags, publish_to. Missing any of them is an error. publish_to must contain only known targets. tags must be an array of exactly 4 items, and each one is checked against an explicit pool of 22 allowed values:
const TAG_POOL = new Set([
"ai", "astro", "webdev", "showdev", "typescript", "javascript",
"indiehackers", "productivity", "opensource", "programming", "tutorial",
"machinelearning", "claude", "anthropic", "vercel", "turso", "tailwindcss",
"githubactions", "react", "nextjs", "node", "python",
]);
Only one tag is a hard error: "seo", which is explicitly prohibited in the spec. Any other tag outside the pool produces a warning (tag "x" not in canonical tag pool — verify intent) rather than a failure, so an off-pool tag still forces me to look at it without blocking a publish outright. The pool exists to prevent organic drift toward one-off tags that don't build topic coherence. The editorial layer over programmatic content has a similar constraint: structure imposed at ingestion time is easier to maintain than structure enforced by convention.
Title and description length. Titles over 90 characters are an error — Dev.to and Hashnode truncate feed titles beyond that. Descriptions over 200 characters are an error — the meta description budget for Google and Hashnode display.
Word count. The length check itself has only two thresholds, and they're deliberately far apart from the per-archetype targets in the spec: under 500 words is a hard error, over 3,500 words is a warning. Anything in between produces no length finding, because the right length varies by archetype and the script doesn't know which archetype a file is. (Word count is still read by the link checks further down, which use 600- and 1,200-word cutoffs — but those flag missing links, not length.) A lightweight at 580 words gets no length flag at all — that judgment stays with me. The word count always appears in the summary line regardless, so there's no hiding from it.
Cliché detection. 13 literal phrases are checked case-insensitively against the full body. The list covers diving metaphors, fast-paced-world openers, hyperbolic tech superlatives, and in-this-article throat-clearing — the standard set of AI-assisted writing tics. I won't reproduce all 13 here for an ironic reason I'll cover in "What I'd do differently."
The check is simple String.prototype.includes, not regex, because the phrases are specific enough that false positives aren't a real concern. An article using the typical "world of AI" framing is probably using clichéd structure; the check forcing me to see it is the point.
Fabricated metric detection. Three regex patterns catch the most common forms of fabricated social proof:
const FABRICATED_METRIC_PATTERNS = [
/\b(\d{2,3},?\d{3,}|\d{4,})\s+(visit|view|readers?|users?|subscribers?)\b/gi,
/\branked #1\b/gi,
/\bmillion(?:s)?\s+of\s+(?:users?|readers?|developers?)/gi,
];
The first pattern catches round numbers followed by social-proof nouns like "visitors", "users", or "subscribers" — numbers presented as fact when the sites are young enough that no credible claim to that kind of traffic exists. The pattern requires a four-digit-or-higher number followed directly by a social-proof noun. It catches claimed traffic figures but not numeric references in code or data discussion.
This check produces an error, not a warning, because fabricated metrics are the one category I genuinely can't allow. The whole premise of the three sites experiment depends on readers trusting that I'm reporting honestly. One fabricated stat poisons that trust retroactively.
Internal and external link presence. For pillar-sized articles (over 1200 words), the check warns if internal links number fewer than 5 — the spec requires 5-10 for pillar archetype. For any article over 600 words, it warns if there are zero external links. The check uses a simple regex:
const internalLinks = (body.match(/\]\(\/[^)]+\)/g) || []).length;
if (wc > 1200 && internalLinks < 5) {
warnings.push(
`pillar-sized (${wc} words) has ${internalLinks} internal links — spec requires 5-10`
);
}
The noindex gate for programmatic pages uses a similar "catch before ship" logic at the infrastructure level. The article gate applies it at the content level.
Sentence repetition. Any sentence appearing three or more times in the same article generates a warning. This catches a specific AI-assisted writing failure: a paragraph occasionally gets reformulated twice, and the reformulation ends up identical to the original a few hundred words later. The check normalizes to lowercase and trims whitespace before comparing.
Strict versus lenient mode
The gate has two distinct behaviors depending on how it's called. The Node.js process.exitCode documentation is the relevant primitive — returning 1 from main() is enough to block any downstream step that checks exit codes.
When run against a single file (node scripts/audit-articles.mjs path/to/article.md) or with --strict, errors cause a non-zero exit code and the publish step stops.
When run against all articles without explicit paths (node scripts/audit-articles.mjs), errors are reported but don't fail the process. This baseline scan mode exists because older articles pre-date some current rules, and a retroactive constraint would block everything. The three-tier content quality ladder applies here: different content tiers have different quality expectations.
The article generation routine runs strict mode on each newly staged file before committing — that's the only place the gate is currently wired in. publish-articles.yml doesn't call it at all; it hands the whole content/articles/*.md array to pnpm post:article and relies on the per-platform skip logic. And there's no scheduled job running the all-articles baseline scan either — that mode exists, but today I only invoke it by hand via pnpm audit:articles.
Title duplicate detection across the repo
When scanning all articles (not single-file mode), the gate runs a cross-article deduplication check. Titles are normalized to lowercase alphanumeric, whitespace collapsed, then compared:
function detectTitleDuplicates(reports) {
const titles = new Map();
for (const [path, r] of reports) {
if (!r.meta?.title) continue;
const key = r.meta.title.toLowerCase().replace(/[^a-z0-9]+/g, " ").trim();
const existing = titles.get(key) ?? [];
existing.push(path);
titles.set(key, existing);
}
return [...titles.entries()]
.filter(([, paths]) => paths.length > 1)
.map(([key, paths]) => ({ key, paths }));
}
The comparison is exact Map-key equality on the normalized string, so what this actually catches is duplicates that differ only in punctuation, casing, or spacing — not rephrases. "Why I use Turso for my Astro monorepo" and "Using Turso libSQL in an Astro monorepo" normalize to two different keys and would both sail through. The check does no similarity scoring and no semantic comparison, which is the obvious next thing to add, given how the pipeline-aware content variants approach works.
What actually got caught
Since adding the gate, here are the real categories it's flagged that I fixed before publishing:
Tag outside pool: twice. Both times were "seo" appearing in generated frontmatter. Reliable pattern: when generating frontmatter from a prompt, the model includes obviously relevant tags that are explicitly off-limits in the spec.
Word count: nothing, actually. Three lightweight articles came in at 540-580 words, under the 600-900 the spec calls for but comfortably above the script's 500-word floor, so the gate stayed quiet and I caught them reading the summary line. Two got an additional section and reached spec. One I accepted shorter — the content was complete and adding words would have been padding.
Fabricated metric match: once. The generated draft said "early testers report" something in a way that parsed as a quantified claim. Fix was removing the number.
Missing external link: regularly. Generated articles sometimes make claims about tools or APIs without citing the primary source. About half the time I add a link on review; the other half I rephrase to not make a claim that requires one.
What I'd do differently
The cliché list is static. A smarter approach: add phrases from real failures after the fact. When a specific phrase slips through and I notice it post-publish, add it. So far that hasn't happened once — the list is still the same 13 entries it shipped with, and there's no automated feedback loop to grow it. One possible improvement: run the gate on published articles periodically and flag new patterns.
The biggest gap: the checker doesn't strip code blocks before running cliché or metric checks. This article itself demonstrates the problem — I had to describe the cliché list in prose rather than including it as a JavaScript literal, because reproducing the exact phrases inside a code block triggers the gate. The fix is straightforward: split the body on triple-backtick fences, check only the prose regions. A two-pass preprocessing step before any pattern check would resolve this entirely. I haven't shipped it yet because the current behavior just means I write around it, which isn't painful enough to force the fix.
CI integration is partial. The gate runs as part of the article generation routine and as a verify step, but it doesn't run as a blocking step in the publish workflow itself. That means a file committed outside the routine could bypass the gate entirely. The fix is one line in publish-articles.yml — a step I haven't prioritized because the routine path is currently the only real publishing path. (Status as of June 2026. In July I did add it: publish-articles.yml now runs node scripts/audit-articles.mjs --strict as a fail-closed step before images and publishing.)
The EEAT transparency pages I built approach content credibility from the site structure side. The lint gate approaches it from the individual article side. Both serve the same goal: content that doesn't embarrass the project in hindsight.
FAQ
Why not use an existing prose linter like alex or write-good?
alex focuses on inclusive language; write-good checks passive voice and weak qualifiers. Neither checks frontmatter structure, tag pools, or fabricated metrics — the pipeline-specific failures I actually need to catch. A domain-specific gate catches domain-specific failures better than a general-purpose tool.
How does the fabricated metric check handle code blocks?
Currently it doesn't exclude them, which is the known gap. Numeric references in code examples occasionally trigger false positives. I review the flagged line location before treating it as a real failure. Pre-processing the body to strip triple-backtick blocks would fix this.
Why warn on internal link count instead of hard-erroring?
Because article type isn't always deterministic from content alone. A 1200-word article might be a pillar or a borderline lightweight. Treating it as a warning preserves the ability to make a judgment call; treating it as a hard error would block edge cases that are genuinely fine.
What happens to historical articles that fail current checks?
Baseline scan mode reports them without failing. Historical content is in a different tier. New articles must pass strict mode; historical articles are known issues reported for visibility, not blocked.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)